Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

dev to release (#476)

Thomas Hebrarddf5d20cb8b18
diff --git a/.blackboxrules b/.blackboxrules
index 6e236d028..211d85f6e 100644
--- a/.blackboxrules
+++ b/.blackboxrules
@@ -1,1175 +1,329 @@
<!-- BEGIN_PIPELEX_RULES -->
-## Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-### Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-### Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-#### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-#### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-#### Pipe Definitions
-
-### Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-#### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-## Single item (default)
-inputs = { document = "Text" }
-
-## Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-## Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
+# Pipelex Coding Rules
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
+## Commands
-### Structuring Models
+### Linting
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-#### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
-
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
-
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-#### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-#### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-### Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-### PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-#### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-#### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-#### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-### PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-#### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-#### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
+ After making code changes, you must always lint using `make check`.
-### PipeLLM operator
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-#### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-#### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-#### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-#### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-### PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-#### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
+ Always fix any issues reported by these tools before proceeding.
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
+### Running Tests
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
-### PipeCompose operator
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
+### Running Tests with Prints
-#### Basic Usage
+ If anything went wrong, you can run the tests with prints to see the error:
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-## Report Summary
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
-Based on the analysis:
-$data
+### Running specific Tests
-Generated on: {{ current_date }}
-"""
-```
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
+### Running Last Failed Tests
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
+ To rerun only the tests that failed in the previous run, use:
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
-Hi $customer.first_name,
+---
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
+### Prerequisites for running command lines: activate virtual environment
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
+ Do this:
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
+ or do that:
-Looking forward to moving this forward together!
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
+ (adapt the above command to the OS and available virtual environment name)
-#### Key Parameters
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-#### Template Variables
+## Coding Standards & Best Practices for Python Code
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
-### PipeImgGen operator
+### Variables, loops and indexes
-The PipeImgGen operator is used to generate images using AI image generation models.
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
-#### Basic Usage
+### Enums and tests
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
+### Imports
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
+#### **Imports at the top of the file**
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
+ - Import all necessary libraries at the top of the file
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.
-#### Key Parameters
+- **Logging and Pretty Printing**:
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
-#### Input Requirements
+- **StrEnum and Self type**:
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
-The input can be named anything but must contain the prompt text for image generation.
+### Typing
-### PipeFunc operator
+#### **Always Use Type Hints**
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
-#### Basic Usage
+#### **BaseModel / Pydantic Standards**
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
+### Factory Pattern
-#### Key Parameters
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
+### Error Handling
-#### Function Requirements
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
-The Python function must:
+### Documentation
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
+1. **Docstring Format**
```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
pass
```
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-#### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-#### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-### Rules to choose LLM models used in PipeLLMs.
-
-#### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-#### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-#### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-#### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
-
-## Guide to execute a pipeline and write example code
-
-### Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-## Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-## or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-### Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-## start Pipelex
-Pipelex.make()
-## run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-### Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
-from my_project.gantt.gantt_struct import GanttChart
+## Writing tests
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
+### Unit test generalities
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
+#### Test file structure
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete workflows
+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
-## start Pipelex
-Pipelex.make()
+#### Markers
-## run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
+Apply the appropriate markers:
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- "gha_disabled: will not be able to run properly on GitHub Actions"
-### Setting up the input memory
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
-#### Explanation of input memory
+#### Important rules
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
+- Never use the unittest.mock. Use pytest-mock.
-#### Different ways to set up the input memory
+#### Test Class Structure
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
+- Always group the tests of a module into a test class:
```python
-## Here we have a single input and it's a Text.
-## If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-## Here we have a single input and it's a PDF.
-## Because PDFContent is a native concept, we can use it directly as a value,
-## the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-## Here we have a single input and it's an Image.
-## Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-## Here we have a single input, it's an image but
-## its actually a more specific concept gantt.GanttImage which refines Image,
-## so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
)
-
-## Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
- )
-```
-
-### Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
-
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
-
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
-
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
-
- @property
- def main_stuff_as_str(self) -> str:
- ...
-
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
-
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
-
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
-
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
-
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
```
-As you can see, you can extract any variable from the output working memory.
-
-#### Getting the main stuff as a specific type
-
-Simple text as a string:
-
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
+#### Test Data Organization
-If it's a list, you can get a `ListContent` of the specific type.
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
+### Best Practices for Testing
-or if you want, you can get the actual items as a regular python list:
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
+## Writing Docs
----
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
-## Rules to choose LLM models used in PipeLLMs.
+### MkDocs Markdown Requirements
-### LLM Configuration System
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
+## Test-Driven Development Guide
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
+This document outlines our test-driven development (TDD) process and the tools available for testing.
-### LLM Handles
+### TDD Cycle
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
+1. **Write a Test First**
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+3. **Run Linting and Type Checking**
+4. **Validate tests**
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
<!-- END_PIPELEX_RULES -->
diff --git a/.cursor/rules/commands.mdc b/.cursor/rules/commands.mdc
index 6c017d195..279b9b820 100644
--- a/.cursor/rules/commands.mdc
+++ b/.cursor/rules/commands.mdc
@@ -4,5 +4,92 @@ description: Guidelines for running commands
---
# Commands
- - When you want to run commands such as `python`, `pytest` or any of our CLI such as `pipelex` or `cocode`, ALWAYS use the current obvious virtual env. If the installaton is standard, the venv is named `.venv` so always check that first.
-
\ No newline at end of file
+## Linting
+
+ After making code changes, you must always lint using `make check`.
+
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
+
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
+
+ Always fix any issues reported by these tools before proceeding.
+
+## Running Tests
+
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
+
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
+
+## Running Tests with Prints
+
+ If anything went wrong, you can run the tests with prints to see the error:
+
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
+
+## Running specific Tests
+
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
+
+## Running Last Failed Tests
+
+ To rerun only the tests that failed in the previous run, use:
+
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
+
+---
+
+## Prerequisites for running command lines: activate virtual environment
+
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
+
+ Do this:
+
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
+
+ or do that:
+
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
+
+ (adapt the above command to the OS and available virtual environment name)
+
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
+
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
+
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
diff --git a/.cursor/rules/docs.mdc b/.cursor/rules/docs.mdc
index 1a166508b..f81bc9c42 100644
--- a/.cursor/rules/docs.mdc
+++ b/.cursor/rules/docs.mdc
@@ -6,8 +6,6 @@ globs:
---
# Writing Docs
-Write docs and answer questions about writing docs.
-
We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
## MkDocs Markdown Requirements
diff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdc
deleted file mode 100644
index 7622b337d..000000000
--- a/.cursor/rules/llms.mdc
+++ /dev/null
@@ -1,85 +0,0 @@
----
-alwaysApply: false
-description: LLM configuration and usage guidelines
-globs:
-- '*.plx'
-- '*.toml'
----
-# Rules to choose LLM models used in PipeLLMs.
-
-## LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-## LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-## Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-## LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
diff --git a/.cursor/rules/pytest_standards.mdc b/.cursor/rules/pytest_standards.mdc
index b1c2f11e2..2b1a92002 100644
--- a/.cursor/rules/pytest_standards.mdc
+++ b/.cursor/rules/pytest_standards.mdc
@@ -4,11 +4,11 @@ description: Guidelines for writing unit tests
globs:
- tests/**/*.py
---
-# Writing unit tests
+# Writing tests
## Unit test generalities
-NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
### Test file structure
@@ -19,9 +19,8 @@ NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.
- `tests/e2e/` - for end-to-end tests that test complete workflows
- `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
-- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
-- Always put test inside Test classes.
-- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
### Markers
@@ -63,45 +62,23 @@ class TestFooBar:
# Test implementation
```
-- Never more than 1 class per test file.
+- Never more than 1 class per test module.
- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
### Test Data Organization
- If it's not already there, create a `test_data.py` file in the proper test directory
-- Define test cases using `StuffBlueprint`:
-
-```python
-class TestCases:
- CASE_BLUEPRINT_1 = StuffBlueprint(
- name="test_case_1",
- concept_code="domain.ConceptName1",
- value="test_value"
- )
- CASE_BLUEPRINT_2 = StuffBlueprint(
- name="test_case_2",
- concept_code="domain.ConceptName2",
- value="test_value"
- )
-
- CASE_BLUEPRINTS: ClassVar[list[tuple[str, str]]] = [ # topic, blueprint"
- ("topic1", CASE_BLUEPRINT_1),
- ("topic2", CASE_BLUEPRINT_2),
- ]
-```
-
-Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
-Also note that we provide a topic for the test case, which is purely for convenience.
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
## Best Practices for Testing
-- Whenever possible, use strong asserts to test value, not just type and presence.
+- Use strong asserts: test value, not just type and presence.
- Use parametrize for multiple test cases
- Test both success and failure cases
- Verify working memory state
- Check output structure and content
- Use meaningful test case names
-- Include docstrings explaining test purpose but not on top of the file
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
- Log outputs for debugging
-- Generate reports for cost tracking
diff --git a/.cursor/rules/python_standards.mdc b/.cursor/rules/python_standards.mdc
index 2e2e0a9b2..980b3c7d6 100644
--- a/.cursor/rules/python_standards.mdc
+++ b/.cursor/rules/python_standards.mdc
@@ -20,6 +20,10 @@ This document outlines the core coding standards, best practices, and quality co
- When defining enums related to string values, always inherit from `StrEnum`
- Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
- As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
## Imports
@@ -82,17 +86,17 @@ This document outlines the core coding standards, best practices, and quality co
## Factory Pattern
- - Use Factory Pattern for object creation when dealing with multiple implementations
+ - Use factory pattern for object creation when dealing with multiple implementations
- Our factory methods are named `make_from_...` and such
## Error Handling
- Always catch exceptions at the place where you can add useful context to it.
- Use try/except blocks with specific exceptions
- - Convert third-party exceptions to our custom ones
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
- NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
- - NEVER raise generic exceptions like ValueError or TypeError, create new error classes and raise them instead
- - Always add `from exc` to the exception
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
```python
try:
@@ -102,8 +106,6 @@ This document outlines the core coding standards, best practices, and quality co
raise PipelexSetupError(msg) from exc
```
- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.
-
## Documentation
1. **Docstring Format**
@@ -129,48 +131,3 @@ This document outlines the core coding standards, best practices, and quality co
Provides methods for resizing, converting, and optimizing images.
"""
```
-
-## Code Quality Checks
-
-### Linting and Type Checking
-
-Before finalizing a task, run:
-```bash
-make fix-unused-imports
-make check
-```
-
-This runs multiple code quality tools:
-- Pyright: Static type checking
-- Ruff: Fast Python linter
-- Mypy: Static type checker
-
-Always fix any issues reported by these tools before proceeding.
-
-### Running Tests
-
-1. **Quick Test Run** (no LLM/image generation):
- ```bash
- make tp
- ```
- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)`
-
-2. **Specific Tests**:
- ```bash
- make tp TEST=TestClassName
- # or
- make tp TEST=test_function_name
- ```
- Note: Matches names starting with the provided string.
-
-**Important**: Never run `make ti`, `make test-inference`, `make te`, `make test-extract`, `make tg`, or `make test-img-gen` - these use costly inference.
-
-## Pipelines
-
-- Always validate pipelines after creation/edit with `make validate`.
- Iterate if there are errors.
-
-## Project Structure
-
-- **Tests**: `tests/` directory
-- **Documentation**: `docs/` directory
diff --git a/.cursor/rules/run_pipelex.mdc b/.cursor/rules/run_pipelex.mdc
deleted file mode 100644
index 78aed7449..000000000
--- a/.cursor/rules/run_pipelex.mdc
+++ /dev/null
@@ -1,246 +0,0 @@
----
-alwaysApply: false
-description: Guidelines for running Pipelex pipelines
-globs:
-- examples/**/*.py
----
-# Guide to execute a pipeline and write example code
-
-## Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-# Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-# or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-## Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-# start Pipelex
-Pipelex.make()
-# run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-## Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
-
-from my_project.gantt.gantt_struct import GanttChart
-
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
-
-
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
-
-
-# start Pipelex
-Pipelex.make()
-
-# run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
-
-## Setting up the input memory
-
-### Explanation of input memory
-
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
-
-### Different ways to set up the input memory
-
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
-
-```python
-# Here we have a single input and it's a Text.
-# If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-# Here we have a single input and it's a PDF.
-# Because PDFContent is a native concept, we can use it directly as a value,
-# the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-# Here we have a single input and it's an Image.
-# Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-# Here we have a single input, it's an image but
-# its actually a more specific concept gantt.GanttImage which refines Image,
-# so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
-
-# Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
- )
-```
-
-## Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
-
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
-
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
-
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
-
- @property
- def main_stuff_as_str(self) -> str:
- ...
-
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
-
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
-
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
-
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
-
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
-```
-
-As you can see, you can extract any variable from the output working memory.
-
-### Getting the main stuff as a specific type
-
-Simple text as a string:
-
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
-
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
-
-If it's a list, you can get a `ListContent` of the specific type.
-
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
-
-or if you want, you can get the actual items as a regular python list:
-
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
-
----
-
diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdc
index 4b4f058b5..0342df336 100644
--- a/.cursor/rules/tdd.mdc
+++ b/.cursor/rules/tdd.mdc
@@ -9,7 +9,6 @@ This document outlines our test-driven development (TDD) process and the tools a
## TDD Cycle
1. **Write a Test First**
-[pytest.mdc](pytest.mdc)
2. **Write the Code**
- Implement the minimum amount of code needed to pass the test
@@ -17,12 +16,8 @@ This document outlines our test-driven development (TDD) process and the tools a
- Keep it simple - don't write more than needed
3. **Run Linting and Type Checking**
-[coding_standards.mdc](coding_standards.mdc)
-4. **Refactor if needed**
-If the code needs refactoring, with the best practices [coding_standards.mdc](coding_standards.mdc)
+4. **Validate tests**
-5. **Validate tests**
-
-Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
diff --git a/.cursor/rules/write_pipelex.mdc b/.cursor/rules/write_pipelex.mdc
deleted file mode 100644
index b46e4f23d..000000000
--- a/.cursor/rules/write_pipelex.mdc
+++ /dev/null
@@ -1,862 +0,0 @@
----
-alwaysApply: false
-description: Guidelines for writing Pipelex pipelines
-globs:
-- '**/*.plx'
-- '**/pipelines/**/*.py'
----
-# Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-## Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-## Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-### Pipe Definitions
-
-## Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-# Single item (default)
-inputs = { document = "Text" }
-
-# Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-# Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
-
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
-
-## Structuring Models
-
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
-
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
-
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-## Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-## PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-## PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
-
-## PipeLLM operator
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-## PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
-
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
-
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
-
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
-
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
-
-## PipeCompose operator
-
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
-
-### Basic Usage
-
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-# Report Summary
-
-Based on the analysis:
-$data
-
-Generated on: {{ current_date }}
-"""
-```
-
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
-
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
-
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
-
-Hi $customer.first_name,
-
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
-
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
-
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
-
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
-
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
-
-Looking forward to moving this forward together!
-
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
-
-### Key Parameters
-
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
-
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
-
-### Template Variables
-
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
-
-## PipeImgGen operator
-
-The PipeImgGen operator is used to generate images using AI image generation models.
-
-### Basic Usage
-
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
-
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
-
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
-
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
-
-### Key Parameters
-
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
-
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
-
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
-
-### Input Requirements
-
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
-
-The input can be named anything but must contain the prompt text for image generation.
-
-## PipeFunc operator
-
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
-
-### Basic Usage
-
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
-
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
-
-### Key Parameters
-
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
-
-### Function Requirements
-
-The Python function must:
-
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
- ```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
- pass
- ```
-
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-## Rules to choose LLM models used in PipeLLMs.
-
-### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
index 6e236d028..211d85f6e 100644
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -1,1175 +1,329 @@
<!-- BEGIN_PIPELEX_RULES -->
-## Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-### Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-### Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-#### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-#### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-#### Pipe Definitions
-
-### Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-#### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-## Single item (default)
-inputs = { document = "Text" }
-
-## Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-## Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
+# Pipelex Coding Rules
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
+## Commands
-### Structuring Models
+### Linting
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-#### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
-
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
-
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-#### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-#### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-### Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-### PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-#### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-#### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-#### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-### PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-#### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-#### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
+ After making code changes, you must always lint using `make check`.
-### PipeLLM operator
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-#### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-#### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-#### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-#### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-### PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-#### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
+ Always fix any issues reported by these tools before proceeding.
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
+### Running Tests
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
-### PipeCompose operator
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
+### Running Tests with Prints
-#### Basic Usage
+ If anything went wrong, you can run the tests with prints to see the error:
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-## Report Summary
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
-Based on the analysis:
-$data
+### Running specific Tests
-Generated on: {{ current_date }}
-"""
-```
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
+### Running Last Failed Tests
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
+ To rerun only the tests that failed in the previous run, use:
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
-Hi $customer.first_name,
+---
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
+### Prerequisites for running command lines: activate virtual environment
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
+ Do this:
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
+ or do that:
-Looking forward to moving this forward together!
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
+ (adapt the above command to the OS and available virtual environment name)
-#### Key Parameters
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-#### Template Variables
+## Coding Standards & Best Practices for Python Code
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
-### PipeImgGen operator
+### Variables, loops and indexes
-The PipeImgGen operator is used to generate images using AI image generation models.
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
-#### Basic Usage
+### Enums and tests
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
+### Imports
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
+#### **Imports at the top of the file**
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
+ - Import all necessary libraries at the top of the file
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.
-#### Key Parameters
+- **Logging and Pretty Printing**:
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
-#### Input Requirements
+- **StrEnum and Self type**:
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
-The input can be named anything but must contain the prompt text for image generation.
+### Typing
-### PipeFunc operator
+#### **Always Use Type Hints**
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
-#### Basic Usage
+#### **BaseModel / Pydantic Standards**
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
+### Factory Pattern
-#### Key Parameters
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
+### Error Handling
-#### Function Requirements
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
-The Python function must:
+### Documentation
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
+1. **Docstring Format**
```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
pass
```
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-#### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-#### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-### Rules to choose LLM models used in PipeLLMs.
-
-#### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-#### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-#### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-#### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
-
-## Guide to execute a pipeline and write example code
-
-### Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-## Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-## or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-### Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-## start Pipelex
-Pipelex.make()
-## run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-### Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
-from my_project.gantt.gantt_struct import GanttChart
+## Writing tests
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
+### Unit test generalities
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
+#### Test file structure
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete workflows
+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
-## start Pipelex
-Pipelex.make()
+#### Markers
-## run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
+Apply the appropriate markers:
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- "gha_disabled: will not be able to run properly on GitHub Actions"
-### Setting up the input memory
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
-#### Explanation of input memory
+#### Important rules
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
+- Never use the unittest.mock. Use pytest-mock.
-#### Different ways to set up the input memory
+#### Test Class Structure
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
+- Always group the tests of a module into a test class:
```python
-## Here we have a single input and it's a Text.
-## If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-## Here we have a single input and it's a PDF.
-## Because PDFContent is a native concept, we can use it directly as a value,
-## the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-## Here we have a single input and it's an Image.
-## Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-## Here we have a single input, it's an image but
-## its actually a more specific concept gantt.GanttImage which refines Image,
-## so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
)
-
-## Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
- )
-```
-
-### Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
-
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
-
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
-
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
-
- @property
- def main_stuff_as_str(self) -> str:
- ...
-
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
-
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
-
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
-
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
-
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
```
-As you can see, you can extract any variable from the output working memory.
-
-#### Getting the main stuff as a specific type
-
-Simple text as a string:
-
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
+#### Test Data Organization
-If it's a list, you can get a `ListContent` of the specific type.
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
+### Best Practices for Testing
-or if you want, you can get the actual items as a regular python list:
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
+## Writing Docs
----
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
-## Rules to choose LLM models used in PipeLLMs.
+### MkDocs Markdown Requirements
-### LLM Configuration System
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
+## Test-Driven Development Guide
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
+This document outlines our test-driven development (TDD) process and the tools available for testing.
-### LLM Handles
+### TDD Cycle
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
+1. **Write a Test First**
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+3. **Run Linting and Type Checking**
+4. **Validate tests**
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
<!-- END_PIPELEX_RULES -->
diff --git a/.github/workflows/package-check.yml b/.github/workflows/package-check.yml
new file mode 100644
index 000000000..b01217fda
--- /dev/null
+++ b/.github/workflows/package-check.yml
@@ -0,0 +1,27 @@
+name: package-check
+
+on:
+ pull_request:
+
+jobs:
+ uv-lock-check:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v4
+
+ - name: Install uv
+ uses: astral-sh/setup-uv@v3
+ with:
+ enable-cache: true
+
+ - name: Check if uv.lock is up to date
+ run: |
+ uv lock --locked
+ if ! git diff --exit-code uv.lock; then
+ echo "❌ uv.lock is out of date!"
+ echo "Please run 'uv lock' to update the lock file."
+ exit 1
+ else
+ echo "✅ uv.lock is up to date!"
+ fi
diff --git a/.pipelex/inference/backends.toml b/.pipelex/inference/backends.toml
index 52fe3cbfc..2f002afc8 100644
--- a/.pipelex/inference/backends.toml
+++ b/.pipelex/inference/backends.toml
@@ -71,7 +71,7 @@ api_key = "${OPENAI_API_KEY}"
[vertexai]
display_name = "Google Vertex AI"
-enabled = true
+enabled = false # This is the only one we disable beacuse setting it up requires internet access just to get credentials so it fails in CI sandboxes
gcp_project_id = "${GCP_PROJECT_ID}"
gcp_location = "${GCP_LOCATION}"
gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
diff --git a/.pipelex/inference/backends/anthropic.toml b/.pipelex/inference/backends/anthropic.toml
index 6b9c9128c..a3a485379 100644
--- a/.pipelex/inference/backends/anthropic.toml
+++ b/.pipelex/inference/backends/anthropic.toml
@@ -98,3 +98,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 100
costs = { input = 1.0, output = 5.0 }
+
+["claude-4.5-opus"]
+model_id = "claude-opus-4-5-20251101"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 5.0, output = 25.0 }
diff --git a/.pipelex/inference/backends/azure_openai.toml b/.pipelex/inference/backends/azure_openai.toml
index 6054b124d..ee04f7ad6 100644
--- a/.pipelex/inference/backends/azure_openai.toml
+++ b/.pipelex/inference/backends/azure_openai.toml
@@ -97,24 +97,43 @@ model_id = "gpt-5-mini-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.25, output = 2.0 }
+constraints = ["temperature_must_be_1"]
[gpt-5-nano]
model_id = "gpt-5-nano-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.05, output = 0.4 }
+constraints = ["temperature_must_be_1"]
[gpt-5-chat]
model_id = "gpt-5-chat-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
[gpt-5]
model_id = "gpt-5-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+# --- GPT-5.1 Series -------------------------------------------------------------
+["gpt-5.1"]
+model_id = "gpt-5.1-2025-11-13"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+["gpt-5.1-chat"]
+model_id = "gpt-5.1-chat-2025-11-13"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
################################################################################
# IMAGE GENERATION MODELS
@@ -122,6 +141,7 @@ costs = { input = 1.25, output = 10.0 }
# --- OpenAI Image Generation --------------------------------------------------
[gpt-image-1]
+sdk = "azure_rest"
model_type = "img_gen"
model_id = "gpt-image-1-2025-04-15"
inputs = ["text"]
diff --git a/.pipelex/inference/backends/bedrock.toml b/.pipelex/inference/backends/bedrock.toml
index cea91344d..c4ab17657 100644
--- a/.pipelex/inference/backends/bedrock.toml
+++ b/.pipelex/inference/backends/bedrock.toml
@@ -109,3 +109,12 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 100
costs = { input = 1.0, output = 5.0 }
+
+["claude-4.5-opus"]
+sdk = "bedrock_anthropic"
+model_id = "global.anthropic.claude-opus-4-5-20251101-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 5.0, output = 25.0 }
diff --git a/.pipelex/inference/backends/google.toml b/.pipelex/inference/backends/google.toml
index ff0040a91..0d39dacb1 100644
--- a/.pipelex/inference/backends/google.toml
+++ b/.pipelex/inference/backends/google.toml
@@ -59,3 +59,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 3000
costs = { input = 0.10, output = 0.40 }
+
+# --- Gemini 3.0 Series ----------------------------------------
+["gemini-3.0-pro"]
+model_id = "gemini-3-pro-preview"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 2, output = 12.0 }
diff --git a/.pipelex/inference/backends/openai.toml b/.pipelex/inference/backends/openai.toml
index 485333254..7fcf0afed 100644
--- a/.pipelex/inference/backends/openai.toml
+++ b/.pipelex/inference/backends/openai.toml
@@ -157,6 +157,13 @@ outputs = ["text"]
costs = { input = 1.25, output = 10.0 }
constraints = ["temperature_must_be_1"]
+# --- GPT-5.1 Series -------------------------------------------------------------
+["gpt-5.1"]
+model_id = "gpt-5.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+
################################################################################
# IMAGE GENERATION MODELS
################################################################################
diff --git a/.pipelex/inference/backends/pipelex_inference.toml b/.pipelex/inference/backends/pipelex_inference.toml
index 655105a47..a7dbdfbc3 100644
--- a/.pipelex/inference/backends/pipelex_inference.toml
+++ b/.pipelex/inference/backends/pipelex_inference.toml
@@ -85,6 +85,18 @@ inputs = ["text", "images"]
outputs = ["text"]
costs = { input = 1.25, output = 10.00 }
+["gpt-5.1"]
+model_id = "pipelex/gpt-5.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
+["gpt-5.1-chat"]
+model_id = "pipelex/gpt-5.1-chat"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
# --- Claude LLMs --------------------------------------------------------------
["claude-4-sonnet"]
model_id = "pipelex/claude-4-sonnet"
@@ -110,32 +122,45 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1, output = 5 }
+["claude-4.5-opus"]
+model_id = "pipelex/claude-4.5-opus"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 5, output = 25 }
+
# --- Gemini LLMs --------------------------------------------------------------
["gemini-2.0-flash"]
-model_id = "gemini/gemini-2.0-flash"
+model_id = "pipelex/gemini-2.0-flash"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.10, output = 0.40 }
["gemini-2.5-pro"]
-model_id = "gemini/gemini-2.5-pro"
+model_id = "pipelex/gemini-2.5-pro"
inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 3000
costs = { input = 1.25, output = 10.0 }
["gemini-2.5-flash"]
-model_id = "gemini/gemini-2.5-flash"
+model_id = "pipelex/gemini-2.5-flash"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.30, output = 2.50 }
["gemini-2.5-flash-lite"]
-model_id = "gemini/gemini-2.5-flash-lite"
+model_id = "pipelex/gemini-2.5-flash-lite"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.10, output = 0.40 }
+["gemini-3.0-pro"]
+model_id = "pipelex/gemini-3.0-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 2, output = 12.0 }
+
# --- XAI LLMs --------------------------------------------------------------
[grok-3]
diff --git a/.pipelex/inference/routing_profiles.toml b/.pipelex/inference/routing_profiles.toml
index fa3f1d42a..0910ff285 100644
--- a/.pipelex/inference/routing_profiles.toml
+++ b/.pipelex/inference/routing_profiles.toml
@@ -94,10 +94,6 @@ default = "openai"
description = "Use Pipelex Inference backend for all its supported models"
default = "pipelex_inference"
-[profiles.all_vertexai]
-description = "Use Vertex AI backend for all its supported models"
-default = "vertexai"
-
[profiles.all_xai]
description = "Use xAI backend for all its supported models"
default = "xai"
diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.toml
index 4a164cd80..9e4f9cbb6 100644
--- a/.pipelex/pipelex.toml
+++ b/.pipelex/pipelex.toml
@@ -44,6 +44,7 @@ excluded_dirs = [
]
[pipelex.builder_config]
+fix_loop_max_attempts = 10
# Uncomment to change where the generated pipelines are saved
# default_output_dir = "."
# default_bundle_file_name = "bundle"
diff --git a/.windsurfrules.md b/.windsurfrules.md
index 6e236d028..211d85f6e 100644
--- a/.windsurfrules.md
+++ b/.windsurfrules.md
@@ -1,1175 +1,329 @@
<!-- BEGIN_PIPELEX_RULES -->
-## Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-### Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-### Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-#### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-#### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-#### Pipe Definitions
-
-### Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-#### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-## Single item (default)
-inputs = { document = "Text" }
-
-## Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-## Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
+# Pipelex Coding Rules
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
+## Commands
-### Structuring Models
+### Linting
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-#### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
-
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
-
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-#### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-#### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-### Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-### PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-#### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-#### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-#### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-### PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-#### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-#### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
+ After making code changes, you must always lint using `make check`.
-### PipeLLM operator
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-#### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-#### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-#### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-#### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-### PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-#### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
+ Always fix any issues reported by these tools before proceeding.
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
+### Running Tests
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
-### PipeCompose operator
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
+### Running Tests with Prints
-#### Basic Usage
+ If anything went wrong, you can run the tests with prints to see the error:
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-## Report Summary
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
-Based on the analysis:
-$data
+### Running specific Tests
-Generated on: {{ current_date }}
-"""
-```
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
+### Running Last Failed Tests
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
+ To rerun only the tests that failed in the previous run, use:
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
-Hi $customer.first_name,
+---
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
+### Prerequisites for running command lines: activate virtual environment
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
+ Do this:
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
+ or do that:
-Looking forward to moving this forward together!
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
+ (adapt the above command to the OS and available virtual environment name)
-#### Key Parameters
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-#### Template Variables
+## Coding Standards & Best Practices for Python Code
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
-### PipeImgGen operator
+### Variables, loops and indexes
-The PipeImgGen operator is used to generate images using AI image generation models.
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
-#### Basic Usage
+### Enums and tests
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
+### Imports
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
+#### **Imports at the top of the file**
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
+ - Import all necessary libraries at the top of the file
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.
-#### Key Parameters
+- **Logging and Pretty Printing**:
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
-#### Input Requirements
+- **StrEnum and Self type**:
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
-The input can be named anything but must contain the prompt text for image generation.
+### Typing
-### PipeFunc operator
+#### **Always Use Type Hints**
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
-#### Basic Usage
+#### **BaseModel / Pydantic Standards**
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
+### Factory Pattern
-#### Key Parameters
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
+### Error Handling
-#### Function Requirements
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
-The Python function must:
+### Documentation
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
+1. **Docstring Format**
```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
pass
```
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-#### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-#### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-### Rules to choose LLM models used in PipeLLMs.
-
-#### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-#### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-#### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-#### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
-
-## Guide to execute a pipeline and write example code
-
-### Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-## Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-## or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-### Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-## start Pipelex
-Pipelex.make()
-## run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-### Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
-from my_project.gantt.gantt_struct import GanttChart
+## Writing tests
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
+### Unit test generalities
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
+#### Test file structure
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete workflows
+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
-## start Pipelex
-Pipelex.make()
+#### Markers
-## run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
+Apply the appropriate markers:
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- "gha_disabled: will not be able to run properly on GitHub Actions"
-### Setting up the input memory
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
-#### Explanation of input memory
+#### Important rules
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
+- Never use the unittest.mock. Use pytest-mock.
-#### Different ways to set up the input memory
+#### Test Class Structure
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
+- Always group the tests of a module into a test class:
```python
-## Here we have a single input and it's a Text.
-## If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-## Here we have a single input and it's a PDF.
-## Because PDFContent is a native concept, we can use it directly as a value,
-## the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-## Here we have a single input and it's an Image.
-## Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-## Here we have a single input, it's an image but
-## its actually a more specific concept gantt.GanttImage which refines Image,
-## so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
)
-
-## Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
- )
-```
-
-### Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
-
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
-
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
-
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
-
- @property
- def main_stuff_as_str(self) -> str:
- ...
-
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
-
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
-
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
-
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
-
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
```
-As you can see, you can extract any variable from the output working memory.
-
-#### Getting the main stuff as a specific type
-
-Simple text as a string:
-
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
+#### Test Data Organization
-If it's a list, you can get a `ListContent` of the specific type.
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
+### Best Practices for Testing
-or if you want, you can get the actual items as a regular python list:
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
+## Writing Docs
----
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
-## Rules to choose LLM models used in PipeLLMs.
+### MkDocs Markdown Requirements
-### LLM Configuration System
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
+## Test-Driven Development Guide
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
+This document outlines our test-driven development (TDD) process and the tools available for testing.
-### LLM Handles
+### TDD Cycle
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
+1. **Write a Test First**
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+3. **Run Linting and Type Checking**
+4. **Validate tests**
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
<!-- END_PIPELEX_RULES -->
diff --git a/AGENTS.md b/AGENTS.md
index 6e236d028..1456cbe11 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,1175 +1,299 @@
<!-- BEGIN_PIPELEX_RULES -->
-## Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-### Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-### Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-#### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-#### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-#### Pipe Definitions
-
-### Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-#### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-## Single item (default)
-inputs = { document = "Text" }
-
-## Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-## Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
-
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
-
-### Structuring Models
-
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-#### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
+# Pipelex Coding Rules
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
+## Codex Cloud Commands
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-#### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-#### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-### Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-### PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-#### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-#### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-#### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-### PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-#### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-#### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
+### Linting
-### PipeLLM operator
+ After making code changes, you must always lint using `make check`.
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-#### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-#### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-#### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-#### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-### PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-#### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
-
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
-
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
-
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
-
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
-
-### PipeCompose operator
-
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
-
-#### Basic Usage
-
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-## Report Summary
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
-Based on the analysis:
-$data
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
-Generated on: {{ current_date }}
-"""
-```
+ Always fix any issues reported by these tools before proceeding.
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
+### Running Tests in Codex Cloud
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
+ To test everything that can be tested from within the Codex Cloud sandbox, run this:
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
+ ```bash
+ make codex-tests
+ # It's equivalent to running pytest with `-m "(dry_runnable or not inference) and not (pipelex_api or codex_disabled)"`
+ # If some test fails, re-run it with `-s -vv` to see more details
+ ```
-Hi $customer.first_name,
+---
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
+### Prerequisites for running command lines: activate virtual environment
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
+ Do this:
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
+ or do that:
-Looking forward to moving this forward together!
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
+ (adapt the above command to the OS and available virtual environment name)
-#### Key Parameters
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-#### Template Variables
+## Coding Standards & Best Practices for Python Code
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
-### PipeImgGen operator
+### Variables, loops and indexes
-The PipeImgGen operator is used to generate images using AI image generation models.
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
-#### Basic Usage
+### Enums and tests
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
+### Imports
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
+#### **Imports at the top of the file**
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
+ - Import all necessary libraries at the top of the file
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.
-#### Key Parameters
+- **Logging and Pretty Printing**:
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
-#### Input Requirements
+- **StrEnum and Self type**:
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
-The input can be named anything but must contain the prompt text for image generation.
+### Typing
-### PipeFunc operator
+#### **Always Use Type Hints**
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
-#### Basic Usage
+#### **BaseModel / Pydantic Standards**
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
+### Factory Pattern
-#### Key Parameters
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
+### Error Handling
-#### Function Requirements
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
-The Python function must:
+### Documentation
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
+1. **Docstring Format**
```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
pass
```
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-#### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-#### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-### Rules to choose LLM models used in PipeLLMs.
-
-#### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-#### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-#### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-#### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
-
-## Guide to execute a pipeline and write example code
-
-### Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-## Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-## or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-### Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-## start Pipelex
-Pipelex.make()
-## run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-### Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
-from my_project.gantt.gantt_struct import GanttChart
+## Writing tests
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
+### Unit test generalities
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
+#### Test file structure
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete workflows
+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
-## start Pipelex
-Pipelex.make()
+#### Markers
-## run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
+Apply the appropriate markers:
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- "gha_disabled: will not be able to run properly on GitHub Actions"
-### Setting up the input memory
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
-#### Explanation of input memory
+#### Important rules
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
+- Never use the unittest.mock. Use pytest-mock.
-#### Different ways to set up the input memory
+#### Test Class Structure
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
+- Always group the tests of a module into a test class:
```python
-## Here we have a single input and it's a Text.
-## If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-## Here we have a single input and it's a PDF.
-## Because PDFContent is a native concept, we can use it directly as a value,
-## the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-## Here we have a single input and it's an Image.
-## Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-## Here we have a single input, it's an image but
-## its actually a more specific concept gantt.GanttImage which refines Image,
-## so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
-
-## Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
)
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
```
-### Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
+#### Test Data Organization
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
+### Best Practices for Testing
- @property
- def main_stuff_as_str(self) -> str:
- ...
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
+## Writing Docs
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
+### MkDocs Markdown Requirements
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
-```
-
-As you can see, you can extract any variable from the output working memory.
-
-#### Getting the main stuff as a specific type
-
-Simple text as a string:
+## Test-Driven Development Guide
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
+This document outlines our test-driven development (TDD) process and the tools available for testing.
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
+### TDD Cycle
-If it's a list, you can get a `ListContent` of the specific type.
+1. **Write a Test First**
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
-
-or if you want, you can get the actual items as a regular python list:
-
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
-
----
-
-## Rules to choose LLM models used in PipeLLMs.
-
-### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+3. **Run Linting and Type Checking**
+4. **Validate tests**
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
<!-- END_PIPELEX_RULES -->
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9464d199c..f6d8113fa 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,32 @@
# Changelog
+## Unreleased
+
+### Fixed
+
+- Fixed some issues with inputs of pipes: The validation methods was not detecting misconceptions with implicit concepts.
+- Fixed validation of pipe inputs: The mulitplicity was not checked properly.
+
+### Changed
+
+- Improved pipe builder by auto-fixing errors, forcing consistency in the inputs and outputs of the pipes.
+- PipeCondition:
+ - Moved the expression/expression_template choosing to the factory.
+
+### Feature
+
+- Library manager now supports multiple libraries. You can now have multiple libraries in your project, each with its own set of concepts, pipes, and stuffs.
+
+### Refactor
+
+- Moved a lot of validation to blueprints instead of pipe instances.
+- Refactored the Blueprint validation errors, and validation functions.
+- Refactored the PipelexInterpreter validation errors.
+- Refactored the pipe builder validation loop.
+- Reorganized the unit tests, and added new ones.
+- Reorganized the config files.
+- Moved `dev_cli` to `cli.dev_cli`.
+
## [v0.15.7] - 2025-11-18
### Fixed
@@ -347,7 +374,7 @@ This is all in the spirit of making Pipelex a declarative language, where you ex
- Changed `PipeLLMPromptTemplate.make_llm_prompt()` to private method `_make_llm_prompt()`
- Updated pipe-builder prompts to include concept specs for better context
- Updated `PipelexBundleSpec.to_blueprint()` to sort pipes by dependencies before creating bundle
- - Changed exception base class from `PipelexError` to `PipelexException` throughout codebase
+ - Changed exception base class from `PipelexError` to `PipelexError` throughout codebase
- Updated Makefile pyright target to use `--pythonpath` flag correctly
- Enhanced `LibraryManager` to use `importlib.resources` for reliable PLX file discovery across all installation modes (wheel, source, relative path)
- Simplified `FuncRegistryUtils` to exclusively register functions with `@pipe_func` decorator (removed `decorator_names` and `require_decorator` parameters)
diff --git a/CLAUDE.md b/CLAUDE.md
index 6e236d028..211d85f6e 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,1175 +1,329 @@
<!-- BEGIN_PIPELEX_RULES -->
-## Guide to write or edit pipelines using the Pipelex language in .plx files
-
-- Always first write your "plan" in natural language, then transcribe it in pipelex.
-- You should ALWAYS RUN validation when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- - For a specific file: `pipelex validate path_to_file.plx`
- - For all pipelines: `pipelex validate all`
- - **IMPORTANT**: Ensure the Python virtual environment is activated before running `pipelex` commands. For standard installations, the venv is named `.venv` - always check that first. The commands will not work without proper venv activation.
-- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)
-
-### Pipeline File Naming
-- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)
-- Files must be `.py` for code defining the data structures
-- Use descriptive names in `snake_case`
-
-### Pipeline File Outline
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-#### Domain Statement
-```plx
-domain = "domain_name"
-description = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-#### Concept Definitions
-
-Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.
-
-```plx
-[concept]
-ConceptName = "Description of the concept"
-```
-
-**Naming Rules:**
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex
-- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-**Native Concepts:**
-Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.
-
-**Refining Native Concepts:**
-To create a concept that specializes a native concept without adding fields:
-
-```plx
-[concept.Landscape]
-description = "A scenic outdoor photograph"
-refines = "Image"
-```
-
-For details on how to structure concepts with fields, see the "Structuring Models" section below.
-
-#### Pipe Definitions
-
-### Pipe Base Definition
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base definition.
-- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
-So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
-variable='['invoice']'``
-That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
-
-NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:
-<NEVER DO THIS>
-```plx
-inputs = {
- input_1 = "ConceptName1",
- input_2 = "ConceptName2"
-}
-```
-</NEVER DO THIS>
-
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
-
-#### Input Multiplicity
-
-By default, inputs expect a single item. Use bracket notation to specify multiple items:
-
-```plx
-## Single item (default)
-inputs = { document = "Text" }
-
-## Variable list - indeterminate number of items
-inputs = { documents = "Text[]" }
-
-## Fixed count - exactly N items
-inputs = { comparison_items = "Image[2]" }
-```
+# Pipelex Coding Rules
-**Key points:**
-- No brackets = single item (default behavior)
-- Use `[]` for lists of unknown length
-- Use `[N]` (where N is an integer) when operation requires exact count (e.g., comparing 2 items)
+## Commands
-### Structuring Models
+### Linting
-Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
-
-#### Three Ways to Structure Concepts
-
-**1. No Structure Needed**
-
-If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.
-
-**2. Inline Structure Definition (RECOMMENDED for most cases)**
-
-For concepts with structured fields, define them inline using TOML syntax:
-
-```plx
-[concept.Invoice]
-description = "A commercial document issued by a seller to a buyer"
-
-[concept.Invoice.structure]
-invoice_number = "The unique invoice identifier" # This will be optional by default
-issue_date = { type = "date", description = "The date the invoice was issued", required = true }
-total_amount = { type = "number", description = "The total invoice amount", required = true }
-vendor_name = "The name of the vendor" # This will be optional by default
-line_items = { type = "list", item_type = "text", description = "List of items" }
-```
-
-**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`
-
-**Field properties:** `type`, `description`, `required` (default: false), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)
-
-**Simple syntax** (creates required text field):
-```plx
-field_name = "Field description"
-```
-
-**Detailed syntax** (with explicit properties):
-```plx
-field_name = { type = "text", description = "Field description", default_value = "default" }
-```
-
-**3. Python StructuredContent Class (For Advanced Features)**
-
-Create a Python class when you need:
-- Custom validation logic (@field_validator, @model_validator)
-- Computed properties (@property methods)
-- Custom methods or class methods
-- Complex cross-field validation
-- Reusable structures across multiple domains
-
-```python
-from pipelex.core.stuffs.structured_content import StructuredContent
-from pydantic import Field, field_validator
-
-class Invoice(StructuredContent):
- """A commercial invoice with validation."""
-
- invoice_number: str = Field(description="The unique invoice identifier")
- total_amount: float = Field(ge=0, description="The total invoice amount")
- tax_amount: float = Field(ge=0, description="Tax amount")
-
- @field_validator('tax_amount')
- @classmethod
- def validate_tax(cls, v, info):
- """Ensure tax doesn't exceed total."""
- total = info.data.get('total_amount', 0)
- if v > total:
- raise ValueError('Tax amount cannot exceed total amount')
- return v
-```
-
-**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.
-
-#### Decision Rules for Agents
-
-**If concept already exists:**
-- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class
-- If it's already a Python class → KEEP IT as Python class
-
-**If creating new concept:**
-1. Does it only refine a native concept without adding fields? → Use concept-only declaration
-2. Does it need custom validation, computed properties, or methods? → Use Python class
-3. Otherwise → Use inline structure (fastest and simplest)
-
-**When to suggest conversion to Python class:**
-- User needs validation logic beyond type checking
-- User needs computed properties or custom methods
-- Structure needs to be reused across multiple domains
-- Complex type relationships or inheritance required
-
-#### Inline Structure Limitations
-
-Inline structures:
-- ✅ Support all common field types (text, number, date, list, dict, etc.)
-- ✅ Support required/optional fields, defaults, choices
-- ✅ Generate full Pydantic models with validation
-- ❌ Cannot have custom validators or complex validation logic
-- ❌ Cannot have computed properties or custom methods
-- ❌ Cannot refine custom (non-native) concepts
-- ❌ Limited IDE autocomplete compared to explicit Python classes
-
-
-### Pipe Controllers and Pipe Operators
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - For creating a sequence of multiple steps
- - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - For parallelizing pipes
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - Generate Text and Objects (include Vision LLM)
- - `PipeExtract` - Extract text and images from an image or a PDF
- - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.
- - `PipeImgGen` - Generate Images
- - `PipeFunc` - For running classic python scripts
-
-### PipeSequence controller
-
-Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.
-
-#### Basic Definition
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-#### Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-#### Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-### PipeCondition controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-#### Basic usage
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditional pipe to decide whether..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-default_outcome = "process_medium"
-
-[pipe.conditional_operation.outcomes]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-#### Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `outcomes`: Dictionary mapping expression results to pipe codes:
- 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`
- 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger
-- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found
-
-Example with fail as default:
-```plx
-[pipe.strict_validation]
-type = "PipeCondition"
-description = "Validate with strict matching"
-inputs = { status = "Status" }
-output = "Text"
-expression = "status.value"
-default_outcome = "fail"
-
-[pipe.strict_validation.outcomes]
-approved = "process_approved"
-rejected = "process_rejected"
-```
+ After making code changes, you must always lint using `make check`.
-### PipeLLM operator
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-#### Basic Usage
-
-Simple Text Generation:
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt = """
-Write a short story about a programmer.
-"""
-```
-
-Structured Data Extraction:
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt = """
-Extract person information from this text:
-@text
-"""
-```
-
-Supports system instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt = "Analyze this data"
-```
-
-#### Multiple Outputs
-
-Generate multiple outputs (fixed number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[3]" # Generate exactly 3 ideas
-prompt = "Generate 3 ideas"
-```
-
-Generate multiple outputs (variable number) - use bracket notation:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea[]" # Let the LLM decide how many to generate
-prompt = "Generate ideas"
-```
-
-#### Vision
-
-Process images with VLMs (image inputs must be tagged in the prompt):
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" }
-output = "ImageAnalysis"
-prompt = """
-Describe what you see in this image:
-
-$image
-"""
-```
-
-You can also reference images inline in meaningful sentences to guide the Visual LLM:
-```plx
-[pipe.compare_images]
-type = "PipeLLM"
-description = "Compare two images"
-inputs = { photo = "Image", painting = "Image" }
-output = "Analysis"
-prompt = "Analyze the colors in $photo and the shapes in $painting."
-```
-
-#### Writing prompts for PipeLLM
-
-**Insert stuff inside a tagged block**
-
-If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.
-
-DO NOT write things like "Here is the expense: @expense".
-DO write simply "@expense" alone in an isolated line.
-
-**Insert stuff inline**
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-DO NOT write "$topic" alone in an isolated line.
-DO write things like "Write an essay about $topic" to include text into an actual sentence.
-
-
-### PipeExtract operator
-
-The PipeExtract operator is used to extract text and images from an image or a PDF
-
-#### Simple Text Extraction
-```plx
-[pipe.extract_info]
-type = "PipeExtract"
-description = "extract the information"
-inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.
-output = "Page"
-```
-
-Using Extract Model Settings:
-```plx
-[pipe.extract_with_model]
-type = "PipeExtract"
-description = "Extract with specific model"
-inputs = { document = "PDF" }
-output = "Page"
-model = "base_extract_mistral" # Use predefined extract preset or model alias
-```
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
-Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`
+ Always fix any issues reported by these tools before proceeding.
-```python
-class TextAndImagesContent(StuffContent):
- text: TextContent | None
- images: list[ImageContent] | None
+### Running Tests
-class PageContent(StructuredContent): # CONCEPT IS "Page"
- text_and_images: TextAndImagesContent
- page_view: ImageContent | None = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
-### PipeCompose operator
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
-The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.
+### Running Tests with Prints
-#### Basic Usage
+ If anything went wrong, you can run the tests with prints to see the error:
-Simple Template Composition:
-```plx
-[pipe.compose_report]
-type = "PipeCompose"
-description = "Compose a report using template"
-inputs = { data = "ReportData" }
-output = "Text"
-template = """
-## Report Summary
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
-Based on the analysis:
-$data
+### Running specific Tests
-Generated on: {{ current_date }}
-"""
-```
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
-Using Named Templates:
-```plx
-[pipe.use_template]
-type = "PipeCompose"
-description = "Use a predefined template"
-inputs = { content = "Text" }
-output = "Text"
-template_name = "standard_report_template"
-```
+### Running Last Failed Tests
-Using Nested Template Section (for more control):
-```plx
-[pipe.advanced_template]
-type = "PipeCompose"
-description = "Use advanced template settings"
-inputs = { data = "ReportData" }
-output = "Text"
-
-[pipe.advanced_template.template]
-template = "Report: $data"
-category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-```
+ To rerun only the tests that failed in the previous run, use:
-CRM Email Template:
-```plx
-[pipe.compose_follow_up_email]
-type = "PipeCompose"
-description = "Compose a personalized follow-up email for CRM"
-inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }
-output = "Text"
-template_category = "html"
-templating_style = { tag_style = "square_brackets", text_format = "html" }
-template = """
-Subject: Following up on our $deal.product_name discussion
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
-Hi $customer.first_name,
+---
-I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.
+### Prerequisites for running command lines: activate virtual environment
-Based on our discussion, I understand that your key requirements are: $deal.customer_requirements
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
-I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:
+ Do this:
-**Next Steps:**
-- Schedule a demo tailored to your specific needs
-- Provide you with a customized quote based on your requirements
-- Connect you with our implementation team
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
-Would you be available for a 30-minute call this week? I have openings on:
-{% for slot in available_slots %}
-- {{ slot }}
-{% endfor %}
+ or do that:
-Looking forward to moving this forward together!
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
-Best regards,
-$sales_rep.name
-$sales_rep.title
-$sales_rep.phone | $sales_rep.email
-"""
-```
+ (adapt the above command to the OS and available virtual environment name)
-#### Key Parameters
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
-- `template`: Inline template string (mutually exclusive with template_name)
-- `template_name`: Name of a predefined template (mutually exclusive with template)
-- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)
-- `templating_style`: Styling options for template rendering
-- `extra_context`: Additional context variables for template
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
-For more control, you can use a nested `template` section instead of the `template` field:
-- `template.template`: The template string
-- `template.category`: Template type
-- `template.templating_style`: Styling options
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-#### Template Variables
+## Coding Standards & Best Practices for Python Code
-Use the same variable insertion rules as PipeLLM:
-- `@variable` for block insertion (multi-line content)
-- `$variable` for inline insertion (short text)
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
-### PipeImgGen operator
+### Variables, loops and indexes
-The PipeImgGen operator is used to generate images using AI image generation models.
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
-#### Basic Usage
+### Enums and tests
-Simple Image Generation:
-```plx
-[pipe.generate_image]
-type = "PipeImgGen"
-description = "Generate an image from prompt"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-```
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
-Using Image Generation Settings:
-```plx
-[pipe.generate_photo]
-type = "PipeImgGen"
-description = "Generate a high-quality photo"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Photo"
-model = { model = "fast-img-gen" }
-aspect_ratio = "16:9"
-quality = "hd"
-```
+### Imports
-Multiple Image Generation:
-```plx
-[pipe.generate_variations]
-type = "PipeImgGen"
-description = "Generate multiple image variations"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image[3]"
-seed = "auto"
-```
+#### **Imports at the top of the file**
-Advanced Configuration:
-```plx
-[pipe.generate_custom]
-type = "PipeImgGen"
-description = "Generate image with custom settings"
-inputs = { prompt = "ImgGenPrompt" }
-output = "Image"
-model = "img_gen_preset_name" # Use predefined preset
-aspect_ratio = "1:1"
-quality = "hd"
-background = "transparent"
-output_format = "png"
-is_raw = false
-safety_tolerance = 3
-```
+ - Import all necessary libraries at the top of the file
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.
-#### Key Parameters
+- **Logging and Pretty Printing**:
-**Image Generation Settings:**
-- `model`: Model choice (preset name or inline settings with model name)
-- `quality`: Image quality ("standard", "hd")
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
-**Output Configuration:**
-- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)
-- `output_format`: File format ("png", "jpeg", "webp")
-- `background`: Background type ("default", "transparent")
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
-**Generation Control:**
-- `seed`: Random seed (integer or "auto")
-- `is_raw`: Whether to apply post-processing
-- `is_moderated`: Enable content moderation
-- `safety_tolerance`: Content safety level (1-6)
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
-#### Input Requirements
+- **StrEnum and Self type**:
-PipeImgGen requires exactly one input that must be either:
-- An `ImgGenPrompt` concept
-- A concept that refines `ImgGenPrompt`
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
-The input can be named anything but must contain the prompt text for image generation.
+### Typing
-### PipeFunc operator
+#### **Always Use Type Hints**
-The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
-#### Basic Usage
+#### **BaseModel / Pydantic Standards**
-Simple Function Call:
-```plx
-[pipe.process_data]
-type = "PipeFunc"
-description = "Process data using custom function"
-inputs = { input_data = "DataType" }
-output = "ProcessedData"
-function_name = "process_data_function"
-```
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
-File Processing Example:
-```plx
-[pipe.read_file]
-type = "PipeFunc"
-description = "Read file content"
-inputs = { file_path = "FilePath" }
-output = "FileContent"
-function_name = "read_file_content"
-```
+### Factory Pattern
-#### Key Parameters
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
-- `function_name`: Name of the Python function to call (must be registered in func_registry)
+### Error Handling
-#### Function Requirements
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
-The Python function must:
+### Documentation
-1. **Be registered** in the `func_registry`
-2. **Accept `working_memory`** as a parameter:
+1. **Docstring Format**
```python
- async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:
- # Function implementation
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
pass
```
-3. **Return appropriate types**:
- - `StuffContent`: Single content object
- - `list[StuffContent]`: Multiple content objects (becomes ListContent)
- - `str`: Simple string (becomes TextContent)
-
-#### Function Registration
-
-Functions must be registered in the function registry before use:
-
-```python
-from pipelex.system.registries.func_registry import func_registry
-
-@func_registry.register("my_function_name")
-async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:
- # Access inputs from working memory
- input_data = working_memory.get_stuff("input_name")
-
- # Process data
- result = process_logic(input_data.content)
-
- # Return result
- return MyResultContent(data=result)
-```
-
-#### Working Memory Access
-
-Inside the function, access pipeline inputs through working memory:
-
-```python
-async def process_function(working_memory: WorkingMemory) -> TextContent:
- # Get input stuff by name
- input_stuff = working_memory.get_stuff("input_name")
-
- # Access the content
- input_content = input_stuff.content
-
- # Process and return
- processed_text = f"Processed: {input_content.text}"
- return TextContent(text=processed_text)
-```
-
----
-
-### Rules to choose LLM models used in PipeLLMs.
-
-#### LLM Configuration System
-
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
-
-#### LLM Handles
-
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-#### Using an LLM Handle in a PipeLLM
-
-Here is an example of using a model to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-#### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
-
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
-
----
-
-ALWAYS RUN validation when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
-- For a specific bundle/file: `pipelex validate path_to_file.plx`
-- For all pipelines: `pipelex validate all`
-- Remember: Ensure your Python virtual environment is activated (typically `.venv` for standard installations) before running `pipelex` commands.
-
-Then, create an example file to run the pipeline in the `examples` folder.
-But don't write documentation unless asked explicitly to.
-
-## Guide to execute a pipeline and write example code
-
-### Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-## Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-## or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
-### Example to execute a pipeline with text output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-
-
-async def hello_world() -> str:
- """
- This function demonstrates the use of a super simple Pipelex pipeline to generate text.
- """
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="hello_world",
- )
-
- return pipe_output.main_stuff_as_str
-
-
-## start Pipelex
-Pipelex.make()
-## run sample using asyncio
-output_text = asyncio.run(hello_world())
-pretty_print(output_text, title="Your first Pipelex output")
-```
-
-### Example to execute a pipeline with structured output
-
-```python
-import asyncio
-
-from pipelex import pretty_print
-from pipelex.pipelex import Pipelex
-from pipelex.pipeline.execute import execute_pipeline
-from pipelex.core.stuffs.image_content import ImageContent
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
-from my_project.gantt.gantt_struct import GanttChart
+## Writing tests
-SAMPLE_NAME = "extract_gantt"
-IMAGE_URL = "assets/gantt/gantt_tree_house.png"
+### Unit test generalities
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
-async def extract_gantt(image_url: str) -> GanttChart:
- # Run the pipe
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
- )
- # Output the result
- return pipe_output.main_stuff_as(content_type=GanttChart)
+#### Test file structure
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete workflows
+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
-## start Pipelex
-Pipelex.make()
+#### Markers
-## run sample using asyncio
-gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))
-pretty_print(gantt_chart, title="Gantt Chart")
-```
+Apply the appropriate markers:
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- "gha_disabled: will not be able to run properly on GitHub Actions"
-### Setting up the input memory
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
-#### Explanation of input memory
+#### Important rules
-The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
-```python
-StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
-PipelineInputs = dict[str, StuffContentOrData]
-```
-As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
+- Never use the unittest.mock. Use pytest-mock.
-#### Different ways to set up the input memory
+#### Test Class Structure
-So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
+- Always group the tests of a module into a test class:
```python
-## Here we have a single input and it's a Text.
-## If you assign a string, by default it will be considered as a TextContent.
- pipe_output = await execute_pipeline(
- pipe_code="master_advisory_orchestrator",
- inputs={
- "user_input": problem_description,
- },
- )
-
-## Here we have a single input and it's a PDF.
-## Because PDFContent is a native concept, we can use it directly as a value,
-## the system knows what content it corresponds to:
- pipe_output = await execute_pipeline(
- pipe_code="power_extractor_dpe",
- inputs={
- "document": PDFContent(url=pdf_url),
- },
- )
-
-## Here we have a single input and it's an Image.
-## Because ImageContent is a native concept, we can use it directly as a value:
- pipe_output = await execute_pipeline(
- pipe_code="fashion_variation_pipeline",
- inputs={
- "fashion_photo": ImageContent(url=image_url),
- },
- )
-
-## Here we have a single input, it's an image but
-## its actually a more specific concept gantt.GanttImage which refines Image,
-## so we must provide it using a dict with the concept and the content:
- pipe_output = await execute_pipeline(
- pipe_code="extract_gantt_by_steps",
- inputs={
- "gantt_chart_image": {
- "concept": "gantt.GanttImage",
- "content": ImageContent(url=image_url),
- }
- },
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
)
-
-## Here is a more complex example with multiple inputs assigned using different ways:
- pipe_output = await execute_pipeline(
- pipe_code="retrieve_then_answer",
- dynamic_output_concept_code="contracts.Fees",
- inputs={
- "text": load_text_from_path(path=text_path),
- "question": {
- "concept": "answer.Question",
- "content": question,
- },
- "client_instructions": client_instructions,
- },
- )
-```
-
-### Using the outputs of a pipeline
-
-All pipe executions return a `PipeOutput` object.
-It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.
-It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:
-
-```python
-
-class PipeOutput(BaseModel):
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
- pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
-
- @property
- def main_stuff(self) -> Stuff:
- ...
-
- def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:
- ...
-
- def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:
- ...
-
- def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:
- ...
-
- @property
- def main_stuff_as_text(self) -> TextContent:
- ...
-
- @property
- def main_stuff_as_str(self) -> str:
- ...
-
- @property
- def main_stuff_as_image(self) -> ImageContent:
- ...
-
- @property
- def main_stuff_as_text_and_image(self) -> TextAndImagesContent:
- ...
-
- @property
- def main_stuff_as_number(self) -> NumberContent:
- ...
-
- @property
- def main_stuff_as_html(self) -> HtmlContent:
- ...
-
- @property
- def main_stuff_as_mermaid(self) -> MermaidContent:
- ...
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
```
-As you can see, you can extract any variable from the output working memory.
-
-#### Getting the main stuff as a specific type
-
-Simple text as a string:
-
-```python
-result = pipe_output.main_stuff_as_str
-```
-Structured object (BaseModel):
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
-```python
-result = pipe_output.main_stuff_as(content_type=GanttChart)
-```
+#### Test Data Organization
-If it's a list, you can get a `ListContent` of the specific type.
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
-```python
-result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)
-```
+### Best Practices for Testing
-or if you want, you can get the actual items as a regular python list:
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
-```python
-result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)
-```
+## Writing Docs
----
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
-## Rules to choose LLM models used in PipeLLMs.
+### MkDocs Markdown Requirements
-### LLM Configuration System
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
-In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
-LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
+## Test-Driven Development Guide
-- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
-- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
-- **Routing**: `.pipelex/inference/routing_profiles.toml`
+This document outlines our test-driven development (TDD) process and the tools available for testing.
-### LLM Handles
+### TDD Cycle
-An llm_handle can be either:
-1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
-2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
+1. **Write a Test First**
-```toml
-[aliases]
-base-claude = "claude-4.5-sonnet"
-base-gpt = "gpt-5"
-base-gemini = "gemini-2.5-flash"
-base-mistral = "mistral-medium"
-```
-
-The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
-
-### Using an LLM Handle in a PipeLLM
-
-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:
-
-```plx
-[pipe.hello_world]
-type = "PipeLLM"
-description = "Write text about Hello World."
-output = "Text"
-model = { model = "gpt-5", temperature = 0.9 }
-prompt = """
-Write a haiku about Hello World.
-"""
-```
-
-As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").
-
-### LLM Presets
-
-Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.
-
-Examples:
-```toml
-llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
-```
-
-The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
-
-```plx
-[pipe.extract_invoice]
-type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_text = "InvoiceText" }
-output = "Invoice"
-model = "llm_to_extract_invoice"
-prompt = """
-Extract invoice information from this invoice:
-
-The category of this invoice is: $invoice_details.category.
-
-@invoice_text
-"""
-```
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
-The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.
-You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
+3. **Run Linting and Type Checking**
+4. **Validate tests**
-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
<!-- END_PIPELEX_RULES -->
diff --git a/MANIFESTO.md b/MANIFESTO.md
deleted file mode 100644
index a6abe44db..000000000
--- a/MANIFESTO.md
+++ /dev/null
@@ -1,61 +0,0 @@
-### This is our manifesto, this is why we built [Pipelex](https://pipelex.com).
-
-First published on our [blog](https://www.pipelex.com/post/repeatable-ai-workflows-knowledge-pipelines) on June 3, 2025.
-
----
-# The Knowledge Pipeline Manifesto
-
-## Agents are brilliant but they're hopeless at repeatability
-
-AI agents are getting impressive at novel tasks: research, planning, coding from scratch. But here's what we're missing: lots of knowledge work isn't novel. It's the same patterns, repeated thousands of times.
-
-Ask an agent to review 10,000 expense reports. Watch it improvise a new approach for each one. Different reasoning paths, different output formats, different edge case handling. Run it twice on the same report? Different results. The intelligence is there, but the consistency isn't. And **someone’s paying for those improv tokens.**
-
-Agents are explorers. But if you know the path, you don't need an expedition. You need a bullet train.
-
-## Enter the knowledge pipeline
-
-The solution should be obvious: delegate the repetitive tasks to deterministic AI workflows. But here's the deeper problem: AI workflows today are handcrafted, not engineered. We prompt, tweak, test, and pray.
-
-Without proper tools for capturing repeatable workflows, every company recreates the same solutions from scratch. The expense categorization that took your team weeks to perfect? The next company starts at zero. We're in the pre-SQL era of AI: brilliant capabilities trapped in proprietary silos, no way to reuse what actually works.
-
-We need a way to capture proven AI workflows and make them as reliable and reusable as any other software component. We call this a **knowledge pipeline.**
-
-A knowledge pipeline is built from modular components called pipes. Each pipe is a knowledge transformer with a simple contract: **knowledge in, knowledge out**. Unlike data pipelines that move bytes or ML pipelines that train models, knowledge pipes transform *meaning*.
-
-This isn't about dumbing down AI. Each pipe leverages full AI intelligence to handle variation in content while guaranteeing its output structure. You get **deterministic structure with adaptive intelligence:** consistency without rigidity.
-
-## The architecture of understanding
-
-### **Knowledge pipes compose like LEGO blocks for repeatable AI workflows:**
-
-- Connect them sequentially for step-by-step transformations
-- Run them in parallel to process multiple perspectives
-- Feed multiple outputs into a single pipe for synthesis
-- Call sub-pipes conditionally based on categorization
-
-Why does this matter? Because real knowledge work isn't linear.
-
-A pipe analyzing contracts might need context from step 2, results from steps 5-7, and external compliance data. The architecture mirrors how knowledge actually flows: not as a conveyor belt, but as a network of understanding.
-
-## The method becomes the artifact
-
-When every pipe guarantees its output structure, you can compose them fearlessly. When each pipe uses AI intelligently, the pipeline adapts without breaking. **When you capture your method as actual code, you can version it, test it, and scale it like software.**
-
-With knowledge pipelines, your process becomes something you can hold, examine, and refine. **Swap out language models to find the sweet spot between cost and quality.** Adjust prompts until they sing. Reconfigure the flow when you discover a shortcut. Each iteration teaches you something new because you're working with stable, measurable outcomes.
-
-Remember when ChatGPT launched and suddenly everyone was sharing prompts? That same collaborative instinct is waiting to be unleashed on knowledge pipelines. A pipeline for extracting customer insights gets forked for user research. A financial audit workflow becomes the skeleton for compliance reviews. The building blocks translate across industries.
-
-These pipeline artifacts are meant to be shared, forked, and composed. Period.
-
-## The path forward
-
-We've seen this movie before. Docker transformed deployment by giving us a simple way to package and share what used to live in scattered shell scripts and tribal knowledge. Suddenly, complex configurations became artifacts you could version, test, and run anywhere. Better yet, Docker made it possible to build on each other's work rather than starting from scratch each time.
-
-Today's AI workflows are ready for the same breakthrough.
-
-This is why we built [**Pipelex**](https://github.com/Pipelex/pipelex) as an open-source standard for repeatable AI workflows. Not just another tool, but a foundation for the community to build on.
-
-The manifesto is simple: **When you discover a method that works, capture it as a knowledge pipeline. When you build a pipeline that roars, share it with the world!**
-
-Join us in building the knowledge infrastructure of tomorrow. One pipe at a time.
diff --git a/Makefile b/Makefile
index a2bbddad3..f47291b81 100644
--- a/Makefile
+++ b/Makefile
@@ -20,7 +20,7 @@ VENV_PIPELEX_DEV := "$(VIRTUAL_ENV)/bin/pipelex-dev"
UV_MIN_VERSION = $(shell grep -m1 'required-version' pyproject.toml | sed -E 's/.*= *"([^<>=, ]+).*/\1/')
-USUAL_PYTEST_MARKERS := "(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)"
+USUAL_PYTEST_MARKERS := "(dry_runnable or not (inference or llm or img_gen or extract)) and not pipelex_api"
define PRINT_TITLE
$(eval PROJECT_PART := [$(PROJECT_NAME)])
@@ -54,9 +54,11 @@ make lint - lint with ruff check
make pyright - Check types with pyright
make mypy - Check types with mypy
+make rules - Install agent rules for contributing to Pipelex
make up-kit-configs - Update kit configs from .pipelex/
make ukc - Shorthand -> up-kit-configs
make check-config-sync - Verify .pipelex and pipelex/kit/configs are in sync
+make check-rules - Verify installed agent rules match kit templates
make ccs - Shorthand -> check-config-sync
make cleanenv - Remove virtual env and lock files
@@ -112,6 +114,7 @@ export HELP
.PHONY: \
all help env lock install update build \
format lint pyright mypy pylint \
+ rules up-kit-configs ukc check-config-sync check-rules ccs \
cleanderived cleanenv cleanall \
test test-xdist t test-quiet tq test-with-prints tp test-inference ti \
test-llm tl test-img-gen tg test-extract te codex-tests gha-tests \
@@ -119,7 +122,6 @@ export HELP
validate v check c cc \
merge-check-ruff-lint merge-check-ruff-format merge-check-mypy merge-check-pyright \
li check-unused-imports fix-unused-imports check-uv check-TODOs docs docs-check docs-deploy \
- up-kit-configs ukc check-config-sync ccs \
test-count check-test-badge
all help:
@@ -174,6 +176,14 @@ build: env
$(call PRINT_TITLE,"Building the wheels")
@uv build
+rules: env
+ $(call PRINT_TITLE,"Installing agent rules for contributing to Pipelex")
+ $(VENV_PIPELEX) kit rules --set coding_standards
+
+check-rules: env
+ $(call PRINT_TITLE,"Checking installed agent rules against templates")
+ $(VENV_PIPELEX_DEV) check-rules --quiet
+
up-kit-configs:
$(call PRINT_TITLE,"Updating kit configs from .pipelex/")
@rsync -av --delete .pipelex/ pipelex/kit/configs/
@@ -183,7 +193,7 @@ ukc: up-kit-configs
check-config-sync: env
$(call PRINT_TITLE,"Checking config sync between .pipelex and pipelex/kit/configs")
- $(VENV_PIPELEX_DEV) check-config-sync
+ $(VENV_PIPELEX_DEV) check-config-sync --quiet
ccs: check-config-sync
@echo "> done: ccs = check-config-sync"
@@ -227,7 +237,7 @@ cleanall: cleanderived cleanenv cleanconfig
codex-tests: env
$(call PRINT_TITLE,"Unit testing for Codex")
@echo "• Running unit tests for Codex (excluding inference and codex_disabled)"
- $(VENV_PYTEST) --exitfirst -m "(dry_runnable or not inference) and not (needs_output or pipelex_api or codex_disabled)" || [ $$? = 5 ]
+ $(VENV_PYTEST) -n auto --exitfirst -m "(dry_runnable or not inference) and not (pipelex_api or codex_disabled)" || [ $$? = 5 ]
gha-tests: env
$(call PRINT_TITLE,"Unit testing for github actions")
@@ -253,7 +263,11 @@ test: env
$(call PRINT_TITLE,"Unit testing without prints but displaying logs via pytest for WARNING level and above")
@echo "• Running unit tests"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -262,7 +276,11 @@ test-xdist: env
$(call PRINT_TITLE,"Unit testing without prints but displaying logs via pytest for WARNING level and above")
@echo "• Running unit tests"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) -n auto -m $(USUAL_PYTEST_MARKERS) -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) -n auto -m $(USUAL_PYTEST_MARKERS) -o log_level=WARNING --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) -n auto -m $(USUAL_PYTEST_MARKERS) -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) -n auto -m $(USUAL_PYTEST_MARKERS) -o log_level=WARNING $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -274,7 +292,11 @@ test-quiet: env
$(call PRINT_TITLE,"Unit testing without prints but displaying logs via pytest for WARNING level and above")
@echo "• Running unit tests"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) -m $(USUAL_PYTEST_MARKERS) -o log_cli=true -o log_level=WARNING $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -286,7 +308,11 @@ test-with-prints: env
$(call PRINT_TITLE,"Unit testing with prints and our rich logs")
@echo "• Running unit tests"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) -s -m $(USUAL_PYTEST_MARKERS) $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -302,7 +328,11 @@ tb: env
test-inference-with-prints: env
$(call PRINT_TITLE,"Unit testing")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --pipe-run-mode live -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --pipe-run-mode live -m "inference" -s -rfE --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --pipe-run-mode live -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --pipe-run-mode live -m "inference" -s -rfE $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -310,7 +340,11 @@ test-inference-with-prints: env
test-inference-fast: env
$(call PRINT_TITLE,"Unit testing")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) -n auto --pipe-run-mode live -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) -n auto --pipe-run-mode live -m "inference" -s -rfE --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) -n auto --pipe-run-mode live -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) -n auto --pipe-run-mode live -m "inference" -s -rfE $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -324,7 +358,11 @@ ti: test-inference-fast
ti-dry: env
$(call PRINT_TITLE,"Unit testing")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --pipe-run-mode dry --exitfirst -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --pipe-run-mode dry --exitfirst -m "inference" -s -rfE --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --pipe-run-mode dry --exitfirst -m "inference" -s -rfE -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --pipe-run-mode dry --exitfirst -m "inference" -s -rfE $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -332,7 +370,11 @@ ti-dry: env
test-llm: env
$(call PRINT_TITLE,"Unit testing LLM")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "llm" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "llm" -s --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "llm" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "llm" -s $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -343,7 +385,11 @@ tl: test-llm
test-extract: env
$(call PRINT_TITLE,"Unit testing Extract")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "extract" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "extract" -s --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "extract" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "extract" -s $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -354,7 +400,11 @@ te: test-extract
test-img-gen: env
$(call PRINT_TITLE,"Unit testing Image Generation")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "img_gen" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "img_gen" -s --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "img_gen" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --pipe-run-mode live --exitfirst -m "img_gen" -s $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -365,7 +415,11 @@ tg: test-img-gen
test-pipelex-api: env
$(call PRINT_TITLE,"Unit testing")
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --exitfirst -m "pipelex_api" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --exitfirst -m "pipelex_api" -s --lf $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ else \
+ $(VENV_PYTEST) --exitfirst -m "pipelex_api" -s -k "$(TEST)" $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
+ fi; \
else \
$(VENV_PYTEST) --exitfirst -m "pipelex_api" -s $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))); \
fi
@@ -377,7 +431,11 @@ cov: env
$(call PRINT_TITLE,"Unit testing with coverage")
@echo "• Running unit tests with coverage"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --lf $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ else \
+ $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ fi; \
else \
$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
fi
@@ -386,7 +444,11 @@ cov-missing: env
$(call PRINT_TITLE,"Unit testing with coverage and missing lines")
@echo "• Running unit tests with coverage and missing lines"
@if [ -n "$(TEST)" ]; then \
- $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing --lf $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ else \
+ $(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing -k "$(TEST)" $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
+ fi; \
else \
$(VENV_PYTEST) --cov=$(if $(PKG),$(PKG),pipelex) --cov-report=term-missing $(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,-v)); \
fi
@@ -476,7 +538,7 @@ docs-check: env
docs-deploy: env
$(call PRINT_TITLE,"Deploying documentation with mkdocs")
- $(VENV_MKDOCS) gh-deploy --force --clean
+ $(VENV_MKDOCS) gh-deploy --force --clean -s
##########################################################################################
### SHORTHANDS
@@ -486,9 +548,9 @@ c: format lint pyright mypy
@echo "> done: c = check"
cc: cleanderived c
- @echo "> done: cc = cleanderived format lint pyright mypy"
+ @echo "> done: cc = cleanderived format lint pyright pylint mypy"
-check: cc check-unused-imports pylint ccs
+check: cc check-unused-imports ccs check-rules pylint
@echo "> done: check"
v: validate
diff --git a/docs/home/3-understand-pipelex/language-spec-v0-1-0.md b/docs/home/3-understand-pipelex/language-spec-v0-1-0.md
new file mode 100644
index 000000000..eb841795d
--- /dev/null
+++ b/docs/home/3-understand-pipelex/language-spec-v0-1-0.md
@@ -0,0 +1,184 @@
+# Pipelex (PLX) – Declarative AI Workflow Spec (v0.1.0)
+
+**Build deterministic, repeatable AI workflows using declarative TOML syntax.**
+
+The Pipelex Language (PLX) uses a TOML-based syntax to define deterministic, repeatable AI workflows. This specification documents version 0.1.0 of the language and establishes the canonical way to declare domains, concepts, and pipes inside `.plx` bundles.
+
+---
+
+## Core Idea
+
+Pipelex is a workflow declaration language that gets interpreted at runtime, we already have a Python runtime (see [github.com/pipelex/pipelex](https://github.com/pipelex/pipelex)).
+
+Pipelex lets you declare **what** your AI workflow should accomplish and **how** to execute it step by step. Each `.plx` file represents a bundle where you define:
+
+- **Concepts** (PascalCase): the structured or unstructured data flowing through your system
+- **Pipes** (snake_case): operations or orchestrators that define your workflow
+- **Domain** (named in snake_case): the topic or field of work this bundle is about
+
+Write once in `.plx` files. Run anywhere. Get the same results every time.
+
+---
+
+## Semantics
+
+Pipelex workflows are **declarative and deterministic**:
+
+- Pipes are evaluated based on their dependencies, not declaration order
+- Controllers explicitly define execution flow (sequential, parallel, or conditional)
+
+All concepts are strongly typed. All pipes declare their inputs and outputs. The runtime validates that data flowing between pipes matches the declared types before execution. Concepts are also conceptually defined, adding meaning on top of the technical typing.
+
+---
+
+## Guarantees & Limits
+
+**Guarantees:**
+
+- Deterministic workflow execution and outputs
+- Strong typing with validation before runtime
+
+**Not supported in v0.1.0:**
+
+- Pipe namespaces per domain
+
+**More:**
+
+- GitHub: [github.com/pipelex/pipelex](https://github.com/pipelex/pipelex)
+
+---
+
+## Complete Example: CV Job Matching Workflow
+
+This workflow analyses candidate CVs against job offer requirements to determine match quality.
+
+```toml
+domain = "cv_job_matching"
+description = "Analyzing candidate CVs against job offer requirements to determine match quality."
+main_pipe = "analyze_cv_job_matches"
+
+[concept.JobRequirements]
+description = "Structured representation of a job position's requirements and details."
+
+[concept.JobRequirements.structure]
+role_title = { type = "text", description = "The job title or position name", required = true }
+key_responsibilities = { type = "text", description = "Main duties and responsibilities of the role", required = true }
+required_qualifications = { type = "text", description = "Mandatory qualifications, skills, or experience needed", required = true }
+preferred_qualifications = { type = "text", description = "Nice-to-have qualifications, skills, or experience" }
+experience_level = { type = "text", description = "Required years or level of experience" }
+technical_skills = { type = "text", description = "Specific technical skills or tools required" }
+soft_skills = { type = "text", description = "Interpersonal or behavioral skills desired" }
+
+[concept.CvMatchAnalysis]
+description = "Assessment of how well a candidate's CV matches job requirements."
+
+[concept.CvMatchAnalysis.structure]
+overall_match_score = { type = "number", description = "Numerical score representing overall fit (0-100)", required = true }
+strengths = { type = "text", description = "Areas where the candidate excels or meets requirements well", required = true }
+weaknesses = { type = "text", description = "Areas where the candidate falls short or lacks requirements", required = true }
+recommendation = { type = "text", description = "Final recommendation on whether to proceed with the candidate", required = true }
+relevant_experience = { type = "text", description = "Summary of candidate's relevant work experience" }
+skills_match = { type = "text", description = "Analysis of how candidate's skills align with job requirements" }
+
+[pipe.analyze_cv_job_matches]
+type = "PipeSequence"
+description = """
+Main pipeline that extracts job requirements from the job offer PDF and analyzes all candidate CVs against those requirements to determine match quality.
+"""
+inputs = { job_offer_pdf = "PDF", cv_pdfs = "PDF[]" }
+output = "CvMatchAnalysis[]"
+steps = [
+ { pipe = "extract_job_offer", result = "job_offer_pages" },
+ { pipe = "structure_job_requirements", result = "job_requirements" },
+ { pipe = "process_all_cvs", result = "cv_match_analyses" },
+]
+
+[pipe.extract_job_offer]
+type = "PipeExtract"
+description = "Extracts text content from the job offer PDF document into pages."
+inputs = { job_offer_pdf = "PDF" }
+output = "Page[]"
+model = "extract_text_from_pdf"
+
+[pipe.structure_job_requirements]
+type = "PipeLLM"
+description = """
+Consolidates all pages from the job offer into a single structured job description with key requirements, qualifications, and responsibilities.
+"""
+inputs = { job_offer_pages = "Page[]" }
+output = "JobRequirements"
+model = "llm_to_retrieve"
+system_prompt = """
+You are an expert HR analyst specializing in extracting and structuring job requirements from job postings. Your task is to produce a structured JobRequirements object based on the provided job offer document.
+"""
+prompt = """
+Analyze the following job offer pages and extract the key information into a structured format.
+
+@job_offer_pages
+
+Extract and organize the job requirements, including the role title, key responsibilities, required and preferred qualifications, experience level, technical skills, and soft skills.
+"""
+
+[pipe.process_all_cvs]
+type = "PipeBatch"
+description = "Processes each CV PDF concurrently to extract and analyze against job requirements."
+inputs = { cv_pdfs = "PDF[]", job_requirements = "JobRequirements" }
+output = "CvMatchAnalysis[]"
+branch_pipe_code = "process_single_cv"
+input_list_name = "cv_pdfs"
+input_item_name = "cv_pdf"
+
+[pipe.process_single_cv]
+type = "PipeSequence"
+description = "Processes a single CV through extraction and analysis against job requirements."
+inputs = { cv_pdf = "PDF", job_requirements = "JobRequirements" }
+output = "CvMatchAnalysis"
+steps = [
+ { pipe = "extract_cv_content", result = "cv_pages" },
+ { pipe = "analyze_cv_match", result = "cv_match_analysis" },
+]
+
+[pipe.extract_cv_content]
+type = "PipeExtract"
+description = "Extracts text content from a single CV PDF into pages."
+inputs = { cv_pdf = "PDF" }
+output = "Page[]"
+model = "extract_text_from_pdf"
+
+[pipe.analyze_cv_match]
+type = "PipeLLM"
+description = """
+Analyzes the CV content against the job requirements and produces a match assessment with scores, strengths, weaknesses, and overall recommendation.
+"""
+inputs = { cv_pages = "Page[]", job_requirements = "JobRequirements" }
+output = "CvMatchAnalysis"
+model = "llm_to_answer_questions"
+system_prompt = """
+You are an expert HR analyst and recruiter. Your task is to analyze candidate CVs against job requirements and produce structured assessments. Be thorough, objective, and provide actionable insights.
+"""
+prompt = """
+Analyze the following CV against the job requirements and provide a comprehensive match assessment.
+
+@cv_pages
+
+Job Requirements:
+- Role: $job_requirements.role_title
+- Key Responsibilities: $job_requirements.key_responsibilities
+- Required Qualifications: $job_requirements.required_qualifications
+- Preferred Qualifications: $job_requirements.preferred_qualifications
+- Experience Level: $job_requirements.experience_level
+- Technical Skills: $job_requirements.technical_skills
+- Soft Skills: $job_requirements.soft_skills
+
+Evaluate how well this candidate matches the job requirements.
+"""
+```
+
+**What happens:**
+
+- Extracts and structures job requirements from a job offer PDF using document extraction + LLM
+- Processes all candidate CVs in parallel (batch processing)
+- Each CV is extracted and analyzed against the structured job requirements using an LLM
+- Produces a scored match analysis for each candidate with strengths, weaknesses, and hiring recommendations
+- Demonstrates sequential orchestration, parallel processing, nested workflows, and strong typing
+
diff --git a/docs/home/3-understand-pipelex/manifesto.md b/docs/home/3-understand-pipelex/manifesto.md
deleted file mode 100644
index a5cc3de90..000000000
--- a/docs/home/3-understand-pipelex/manifesto.md
+++ /dev/null
@@ -1,5 +0,0 @@
----
-title: The Knowledge Pipeline Manifesto
----
-
---8<-- "MANIFESTO.md"
diff --git a/docs/home/3-understand-pipelex/viewpoint.md b/docs/home/3-understand-pipelex/viewpoint.md
new file mode 100644
index 000000000..90d4117a2
--- /dev/null
+++ b/docs/home/3-understand-pipelex/viewpoint.md
@@ -0,0 +1,147 @@
+### This is our viewpoint, this is why we built [Pipelex](https://pipelex.com).
+
+Web version: https://knowhowgraph.com/
+
+---
+# Viewpoint: The Know-How Graph
+
+Declarative, Repeatable AI Workflows as Shared Infrastructure
+
+**TL;DR**
+
+Agents are great at solving new problems, terrible at doing the same thing twice.
+
+We argue that repeatable AI workflows should complement agents: written in a declarative language that both humans and agents can understand, reuse, and compose. These workflows become tools that agents can build, invoke, and share to turn repeatable cognitive work into reliable infrastructure.
+
+At scale, this forms a **Know-How Graph:** a network of reusable methods that become shared infrastructure.
+
+## **Agents Are Brilliant but Hopeless at Repeatability**
+
+When an AI agent encounters the same task for the nth time, it behaves as if it were the first: different reasoning paths, different output structures, different edge-case handling. **This is fine for novel problems, but wasteful for proven processes.**
+
+### **Agents remember knowledge, but not know-how**
+
+This is **the repeatability paradox**. Agents excel at understanding requirements and designing solutions. They can reason, analyze a task, break it down into logical steps, model data structures. They can remember facts and past conversations. But they can't remember the method they used to solve that same task last time. **They end up reinventing the approach every time, and someone's paying for those improv tokens.**
+
+### We Need a Standard for Reusable Methods
+
+The solution is to capture these methods as AI workflows so agents can reuse them.
+
+By "AI workflows" we mean the actual intellectual work that wasn't automatable before LLMs: extracting structured data from unstructured documents, applying complex analyses and business rules, generating reports with reasoning. **This isn’t about API plumbing or app connectors, it’s about the actual intellectual work.**
+
+Yet look at what's happening today: teams everywhere are hand-crafting the same workflows from scratch. To extract data points from contracts and RFPs, to process expense reports, to classify documents, to screen resumes: identical problems solved in isolation, burning engineering hours.
+
+## AI workflows must be formalized
+
+OpenAPI and MCP enable interoperability for software and agents. The remaining problem is formalizing the **methods that assemble the cognitive steps themselves:** extraction, analysis, synthesis, creativity, and decision-making, the part where understanding matters. These formalized methods must be:
+
+- **Consistent:** same input, same output, every time.
+- **Efficient:** use the right AI model for each step, large or small.
+- **Transparent:** no black boxes. Domain experts can audit the logic, spot issues, suggest improvements.
+
+The workflow becomes a shared artifact that humans and AI collaborate on, optimize together, and trust to run at scale.
+
+### Current solutions are inadequate
+
+Engineers building AI workflows today are stuck with bad options.
+
+Code frameworks like LangChain require **maintaining custom software for every workflow,** with business logic buried in implementation details and technical debt accumulating with each new use case.
+
+Visual builders like Zapier, Make, or n8n excel at what they're designed for: connecting APIs and automating data flow between services. **But automation platforms are not cognitive workflow systems.** AI was bolted on as a feature after the fact. They weren't built for intellectual work. When you need actual understanding and multi-step reasoning, these tools quickly become unwieldy.
+
+None of these solutions speak the language of the domain expert. None of them were built for agents to understand, modify, or generate workflows from requirements. They express technical plumbing, not business logic.
+
+At the opposite, agent SDKs and multi-agent frameworks give you flexibility but sacrifice the repeatability you need for production. **You want agents for exploration and problem-solving, but when you've found a solution that works, you need to lock it down.**
+
+> We need a universal workflow language that expresses business logic, not technical plumbing.
+This workflow language must run across platforms, models, and agent frameworks, where the method outlives any vendor or model version.
+>
+
+## We Need a Declarative Language
+
+AI workflows should be first-class citizens of our technical infrastructure: not buried in code or trapped in platforms, but expressed in a language built for the job. The method should be an artifact you can version, diff, test, and optimize.
+
+**We need a declarative language that states what you want, not how to compute it.** As SQL separated intent from implementation for data, we need the same for AI workflows — so we can build a Know-How Graph: a reusable graph of methods that agents and humans both understand.
+
+### The language shouldn’t need documentation: it is the documentation
+
+Traditional programs are instructions a machine blindly executes. The machine doesn’t see your variable names or comments. With LLMs, we can write instructions the machine actually understands, enabling a new kind of human–computer collaboration.
+
+**The abstraction level must be high enough to speak the language of each use case.** It should explicitly define concepts, include necessary explanations, and remove ambiguity. In short: self-documenting.
+
+### Language fosters collaboration: users and agents building together
+
+The language must be readable by everyone who matters: domain experts who know the business logic, engineers who optimize and deploy it, and crucially, AI agents that can build and refine workflows autonomously.
+
+Imagine agents that transform natural language requirements into working workflows. They design each transformation step (or reuse existing ones), test against real or synthetic data, incorporate expert feedback, and iterate to improve quality while reducing costs. Once a workflow is built, agents can invoke it as a reliable tool whenever they need structured, predictable outputs.
+
+> This is how agents finally remember know-how: by encoding methods into reusable workflows they can build, share, and execute on demand.
+>
+
+## The Know-How Graph: a Network of Composable Methods
+
+**Breaking complex work into smaller tasks is a recursive, core pattern.** Each workflow should stand on the shoulders of others, composing like LEGO bricks to build increasingly sophisticated cognitive systems.
+
+What emerges is a **Know-How Graph**: not just static knowledge, but executable methods that connect and build upon one another. **Unlike a knowledge graph mapping facts, this maps procedures: the actual know-how of getting cognitive work done.**
+
+**Example:**
+
+A recruitment workflow doesn't start from scratch. It composes existing workflows:
+
+- ExtractCandidateProfile (experience, education, skills…)
+- ExtractJobOffer (skills, years of experience…).
+
+These feed into your custom ScoreCard logic to produce a MatchAnalysis, which triggers either GenerateRejectionEmail or PrepareInterviewQuestions.
+
+Each component can be assigned to different team members and validated independently by the relevant stakeholders.
+
+> Think of a workflow as a proven route through the work, and the Know-How Graph as the network of all such routes.
+>
+
+### Know-how is as shareable as knowledge
+
+Think about the explosion of prompt sharing since 2023. All those people trading their best ChatGPT prompts on Twitter, GitHub, Reddit, LinkedIn. Now imagine that same viral knowledge sharing, but with complete, tested, composable workflows instead of fragile prompts.
+
+We’ve seen this movie: software package managers, SQL views, Docker, dbt packages. Composable standards create ecosystems where everyone’s work makes everyone else more productive. Generic workflows for common tasks will spread rapidly, while companies keep their differentiating workflows as competitive advantage. That's how we stop reinventing the wheel while preserving secret sauce.
+
+The same principle applies to AI workflows through the Know-How Graph: durable infrastructure that compounds value over time.
+
+> The Know-How Graph will thrive on the open web because workflows are just files: easy to publish, fork, improve, and compose.
+>
+
+### What this unlocks
+
+- Faster time to production (reuse existing workflows + AI writes them for you)
+- Lower run costs (optimize price / performance for each task)
+- Better collaboration between tech and business
+- Better auditability / compliance
+- No vendor lock-in
+
+## Our Solution: Pipelex
+
+[**Pipelex**](https://github.com/Pipelex/pipelex) is our take on this language: open-source (MIT), designed for the Know-How Graph.
+
+Each workflow is built from pipes: modular transformations that guarantee their output structure while applying intelligence to the content. A pipe is a knowledge transformer with a simple contract: knowledge in → knowledge out., each defined conceptually and with explicit structure and validation. The method is readable and editable by humans and agents.
+
+Our Pipelex workflow builder is itself a Pipelex workflow. The tooling builds itself.
+
+## Why This Can Become a Standard
+
+Pipelex is MIT-licensed and designed for portability. Workflows are files, based on TOML syntax (itself well standardized), and the outputs are validated JSON.
+
+Early adopters are contributing to the [cookbook repo](https://github.com/Pipelex/pipelex-cookbook), building integrations, and running workflows in production. The pieces for ecosystem growth are in place: declarative spec, reference implementation, composable architecture.
+
+Building a standard is hard. We're at v0.1.0, with versioning and backward compatibility coming next. The spec will evolve with your feedback.
+
+## Join Us
+
+The most valuable standards are boring infrastructure everyone relies on: SQL, HTTP, JSON. Pipelex aims to be that for AI workflows.
+
+Start with one workflow: extract invoice data, process applications, analyze reports… Share what works. Build on what others share.
+
+**The future of AI needs both:** smarter agents that explore and adapt, AND reliable workflows that execute proven methods at scale. One workflow at a time, let's build the cognitive infrastructure every organization needs.
+
+---
+
+© 2025 Evotis S.A.S.
+Pipelex is a trademark of Evotis S.A.S.
\ No newline at end of file
diff --git a/docs/home/7-configuration/config-pipeline-validation/static-validation-config.md b/docs/home/7-configuration/config-pipeline-validation/static-validation-config.md
index a5023edb4..e789228ea 100644
--- a/docs/home/7-configuration/config-pipeline-validation/static-validation-config.md
+++ b/docs/home/7-configuration/config-pipeline-validation/static-validation-config.md
@@ -1,18 +1,18 @@
-# Static Validation Configuration
+# Validation Error Configuration
-The `StaticValidationConfig` class controls how Pipelex handles validation errors during static analysis.
+The `ValidationErrorConfig` class controls how Pipelex handles validation errors during static analysis.
## Configuration Options
```python
-class StaticValidationReaction(StrEnum):
+class ValidationErrorReaction(StrEnum):
RAISE = "raise" # Raise an exception
LOG = "log" # Log the error but continue
IGNORE = "ignore" # Silently ignore the error
-class StaticValidationConfig(ConfigModel):
- default_reaction: StaticValidationReaction
- reactions: Dict[StaticValidationErrorType, StaticValidationReaction]
+class ValidationErrorConfig(ConfigModel):
+ default_reaction: ValidationErrorReaction
+ reactions: dict[PipeValidationErrorType, ValidationErrorReaction]
```
### Fields
@@ -31,10 +31,10 @@ Each validation error type can be configured to have one of three reactions:
## Example Configuration
```toml
-[pipelex.static_validation_config]
+[pipelex.validation_error_config]
default_reaction = "raise"
-[pipelex.static_validation_config.reactions]
+[pipelex.validation_error_config.reactions]
# Enable one of these to tolerate some static validation errors, like you would for pyright or some linters
# missing_input_variable = "log"
# extraneous_input_variable = "log"
diff --git a/docs/index.md b/docs/index.md
index a83d9b5bf..095fe73cd 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -56,7 +56,7 @@ Pipelex solves these problems by:
Ready to get started?
- [:material-rocket-launch: Get Started](./home/2-get-started/pipe-builder.md){ .md-button .md-button--primary }
-- [:material-book-open: Read the Manifesto](./home/3-understand-pipelex/manifesto.md){ .md-button }
+- [:material-book-open: Read the Know-How Graph](./home/3-understand-pipelex/viewpoint.md){ .md-button }
- [:material-lightbulb: Explore the Paradigm](./home/3-understand-pipelex/pipelex-paradigm/index.md){ .md-button }
**Explore Examples:**
diff --git a/mkdocs.yml b/mkdocs.yml
index bef311f35..75b90f250 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -93,8 +93,9 @@ nav:
- The Pipe Builder: home/2-get-started/pipe-builder.md
- Write Workflows Manually: home/2-get-started/write-workflows-manually.md
- Understand Pipelex:
- - The Knowledge Pipeline Manifesto: home/3-understand-pipelex/manifesto.md
+ - The Know-How Graph: home/3-understand-pipelex/viewpoint.md
- The Pipelex Paradigm: home/3-understand-pipelex/pipelex-paradigm/index.md
+ - Pipelex Language Spec v0.1.0: home/3-understand-pipelex/language-spec-v0-1-0.md
- Cookbook Examples:
- Overview: home/4-cookbook-examples/index.md
- Hello World: home/4-cookbook-examples/hello-world.md
diff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.py
index 17992f133..61ab5212f 100644
--- a/pipelex/base_exceptions.py
+++ b/pipelex/base_exceptions.py
@@ -1,20 +1,16 @@
-class RootException(Exception):
+class PipelexError(Exception):
def __init__(self, message: str):
super().__init__(message)
self.message = message
-class PipelexException(RootException):
+class PipelexUnexpectedError(PipelexError):
pass
-class PipelexUnexpectedError(PipelexException):
+class PipelexConfigError(PipelexError):
pass
-class PipelexConfigError(PipelexException):
- pass
-
-
-class PipelexSetupError(PipelexException):
+class PipelexSetupError(PipelexError):
pass
diff --git a/pipelex/builder/builder.plx b/pipelex/builder/builder.plx
index beebf0fbc..458bff8e7 100644
--- a/pipelex/builder/builder.plx
+++ b/pipelex/builder/builder.plx
@@ -202,7 +202,9 @@ Shape of the contract for PipeOperator is:
**PipeCondition:**
- jinja2_expression_template: Jinja2 expression to evaluate.
- outcomes: Mapping dict[str, str] of expression results to pipe codes.
-- default_outcome: Fallback pipe_code or "fail"/"continue" if no match.
+- default_outcome: Add a default outcome to the outcomes map.
+Becareful, the outputs "fail" and "continue" are special outcomes, they are not pipe codes.
+-> "continue" is a special outcome that allows to continue the sequence, "fail" is a special outcome that fails the whole pipeline. To use with caution.
## More rules:
- For each pipe: give a unique snake_case pipe_code, based on a verb, and craft description of what the pipe does.
diff --git a/pipelex/builder/builder.py b/pipelex/builder/builder.py
index c27013430..95707e099 100644
--- a/pipelex/builder/builder.py
+++ b/pipelex/builder/builder.py
@@ -2,16 +2,13 @@ from typing import TYPE_CHECKING, cast
from pydantic import ValidationError
-from pipelex import pretty_print
from pipelex.builder.builder_errors import (
PipeBuilderError,
PipelexBundleUnexpectedError,
)
-from pipelex.builder.builder_validation import validate_dry_run_bundle_blueprint
from pipelex.builder.bundle_header_spec import BundleHeaderSpec
from pipelex.builder.bundle_spec import PipelexBundleSpec
from pipelex.builder.concept.concept_spec import ConceptSpec
-from pipelex.builder.exceptions import PipelexBundleError
from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec
from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec
from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec
@@ -24,8 +21,6 @@ from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec
from pipelex.builder.pipe.pipe_spec import PipeSpec
from pipelex.builder.pipe.pipe_spec_map import pipe_type_to_spec_class
from pipelex.builder.pipe.pipe_spec_union import PipeSpecUnion
-from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.interpreter import PipelexInterpreter
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.exceptions import StuffContentTypeError
from pipelex.core.stuffs.list_content import ListContent
@@ -61,7 +56,7 @@ async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> Pipelex
PipelexBundleSpec: The constructed pipeline spec.
"""
- # The working memory actually contains ConceptSpec objects (not ConceptSpecDraft)
+ # The working memory actually contains ConceptSpec objects
# but they may have been deserialized incorrectly
try:
concept_specs = working_memory.get_stuff_as_list(
@@ -133,27 +128,3 @@ def reconstruct_bundle_with_pipe_fixes(pipelex_bundle_spec: PipelexBundleSpec, f
pipelex_bundle_spec.pipe[pipe_code] = fixed_pipe_blueprint
return pipelex_bundle_spec
-
-
-async def load_and_validate_bundle(bundle_path: str) -> PipelexBundleBlueprint:
- """Load a bundle file and extract its main_pipe.
-
- Args:
- bundle_path: Path to the .plx bundle file.
-
- Returns:
- PipelexBundleBlueprint: The blueprint of the bundle.
-
- Raises:
- PipelexBundleError: If any pipe failed during dry run.
- """
- bundle_blueprint = PipelexInterpreter.load_bundle_blueprint(bundle_path=bundle_path)
-
- try:
- await validate_dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
- except PipelexBundleError as exc:
- msg = f"Bundle at '{bundle_path}' failed to validate"
- pretty_print(exc, title="Validation Error Details")
- raise PipelexBundleError(message=msg, pipe_failures=exc.pipe_failures) from exc
-
- return bundle_blueprint
diff --git a/pipelex/builder/builder_errors.py b/pipelex/builder/builder_errors.py
index 186538b9a..c862ed7a8 100644
--- a/pipelex/builder/builder_errors.py
+++ b/pipelex/builder/builder_errors.py
@@ -1,41 +1,17 @@
-from pipelex.base_exceptions import PipelexException
-from pipelex.builder.validation_error_data import (
- ConceptFailure,
- PipeFailure,
-)
+from pipelex.base_exceptions import PipelexError
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.types import Self
-class PipeBuilderError(Exception):
+class PipeBuilderError(PipelexError):
def __init__(self: Self, message: str, working_memory: WorkingMemory | None = None) -> None:
self.working_memory = working_memory
super().__init__(message)
-class ConceptSpecError(PipelexException):
+class ConceptSpecError(PipelexError):
"""Details of a single concept failure during dry run."""
- def __init__(self: Self, message: str, concept_failure: ConceptFailure) -> None:
- self.concept_failure = concept_failure
- super().__init__(message)
-
-
-class PipeSpecError(PipelexException):
- """Details of a single pipe failure during dry run."""
-
- def __init__(self: Self, message: str, pipe_failure: PipeFailure) -> None:
- self.pipe_failure = pipe_failure
- super().__init__(message)
-
-
-class ValidateDryRunError(Exception):
- """Raised when validating the dry run of a pipe."""
-
-
-class PipelexBundleNoFixForError(PipelexException):
- """Raised when no fix is found for a static validation error."""
-
-class PipelexBundleUnexpectedError(PipelexException):
+class PipelexBundleUnexpectedError(PipelexError):
"""Raised when an unexpected error occurs during validation."""
diff --git a/pipelex/builder/builder_loop.py b/pipelex/builder/builder_loop.py
index 070fbcb80..9b9644f04 100644
--- a/pipelex/builder/builder_loop.py
+++ b/pipelex/builder/builder_loop.py
@@ -1,24 +1,21 @@
-from pipelex import log, pretty_print
+from pathlib import Path
+
+from pipelex import builder, log
from pipelex.builder.builder import (
PipelexBundleSpec,
PipeSpecUnion,
reconstruct_bundle_with_pipe_fixes,
)
-from pipelex.builder.builder_errors import (
- PipeBuilderError,
- PipelexBundleNoFixForError,
- PipelexBundleUnexpectedError,
-)
-from pipelex.builder.builder_validation import fix_inputs_consistency, validate_bundle_spec
-from pipelex.builder.exceptions import PipelexBundleError
+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec
from pipelex.client.protocol import PipelineInputs
-from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
-from pipelex.core.pipes.exceptions import StaticValidationErrorType
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories
-from pipelex.hub import get_console, get_required_pipe
+from pipelex.config import get_config
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
+from pipelex.core.pipes.pipe_blueprint import PipeCategory
+from pipelex.core.pipes.variable_multiplicity import format_concept_with_multiplicity
+from pipelex.hub import get_required_pipe
from pipelex.language.plx_factory import PlxFactory
-from pipelex.pipeline.exceptions import PipelineExecutionError
from pipelex.pipeline.execute import execute_pipeline
+from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle
from pipelex.tools.misc.file_utils import get_incremental_file_path, save_text_to_path
from pipelex.tools.misc.json_utils import save_as_json_to_path
@@ -26,23 +23,17 @@ from pipelex.tools.misc.json_utils import save_as_json_to_path
class BuilderLoop:
async def build_and_fix(
self,
- pipe_code: str,
+ builder_pipe: str,
inputs: PipelineInputs | None = None,
is_save_first_iteration_enabled: bool = True,
is_save_second_iteration_enabled: bool = True,
is_save_working_memory_enabled: bool = True,
) -> PipelexBundleSpec:
- pretty_print(f"Building and fixing with {pipe_code}")
- try:
- pipe_output = await execute_pipeline(
- pipe_code=pipe_code,
- inputs=inputs,
- )
- except PipelineExecutionError as exc:
- msg = f"Builder loop: Failed to execute pipeline: {exc}."
- console = get_console()
- console.print_exception()
- raise PipeBuilderError(message=msg) from exc
+ pipe_output = await execute_pipeline(
+ pipe_code=builder_pipe,
+ library_path=str(Path(builder.__file__).parent),
+ inputs=inputs,
+ )
if is_save_working_memory_enabled:
working_memory_path = get_incremental_file_path(
@@ -52,11 +43,7 @@ class BuilderLoop:
)
save_as_json_to_path(object_to_save=pipe_output.working_memory.smart_dump(), path=working_memory_path)
- try:
- pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
- except WorkingMemoryStuffNotFoundError as exc:
- msg = f"Builder loop: Failed to get pipelex bundle spec: {exc}."
- raise PipeBuilderError(message=msg, working_memory=pipe_output.working_memory) from exc
+ pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
if is_save_first_iteration_enabled:
@@ -67,96 +54,157 @@ class BuilderLoop:
)
save_text_to_path(text=plx_content, path=first_iteration_path)
- # Fix input consistency for all PipeController pipes before validation
- log.dev("🔄 Calling fix_inputs_consistency() to ensure PipeController inputs are consistent")
- try:
- pipelex_bundle_spec = fix_inputs_consistency(bundle_spec=pipelex_bundle_spec)
- except PipelexBundleError as bundle_error:
- # Let the error fall through to the existing error handling
- pipelex_bundle_spec = self._fix_bundle_error(
- bundle_error=bundle_error,
- pipelex_bundle_spec=pipelex_bundle_spec,
- is_save_second_iteration_enabled=is_save_second_iteration_enabled,
- )
+ bundle_blueprint = pipelex_bundle_spec.to_blueprint()
+ max_attempts = get_config().pipelex.builder_config.fix_loop_max_attempts
+ for attempt in range(1, max_attempts + 1):
+ try:
+ if attempt == 1:
+ await validate_bundle(blueprints=[bundle_blueprint])
+ else:
+ log.info(f"Validating bundle after fixes (attempt {attempt}/{max_attempts})...")
+ fixed_bundle_blueprint = pipelex_bundle_spec.to_blueprint()
+ await validate_bundle(blueprints=[fixed_bundle_blueprint])
- # Save the fixed bundle for debugging if enabled
- if is_save_first_iteration_enabled:
- plx_content_after_fix = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
- first_iteration_after_fix_path = get_incremental_file_path(
- base_path="results",
- base_name="generated_pipeline_1st_iteration_after_fix",
- extension="plx",
- )
- save_text_to_path(text=plx_content_after_fix, path=first_iteration_after_fix_path)
-
- try:
- await validate_bundle_spec(bundle_spec=pipelex_bundle_spec)
- except PipelexBundleError as bundle_error:
- pipelex_bundle_spec = self._fix_bundle_error(
- bundle_error=bundle_error,
- pipelex_bundle_spec=pipelex_bundle_spec,
- is_save_second_iteration_enabled=is_save_second_iteration_enabled,
- )
+ if attempt > 1:
+ log.info(f"✅ Bundle validation passed after fixes (attempt {attempt}/{max_attempts})")
+ break # Validation passed, exit the loop
+
+ except ValidateBundleError as exc:
+ if attempt < max_attempts:
+ log.info(f"⚠️ Validation failed on attempt {attempt}/{max_attempts}, attempting fixes...")
+ pipelex_bundle_spec = self._fix_bundle_validation_error(
+ bundle_error=exc, pipelex_bundle_spec=pipelex_bundle_spec, is_save_second_iteration_enabled=is_save_second_iteration_enabled
+ )
+ else:
+ # Final attempt failed, re-raise the error
+ log.error(f"❌ Validation failed after {max_attempts} attempts, raising error")
+ raise
return pipelex_bundle_spec
- def _fix_bundle_error(
+ def _fix_bundle_validation_error(
self,
- bundle_error: PipelexBundleError,
+ bundle_error: ValidateBundleError,
pipelex_bundle_spec: PipelexBundleSpec,
is_save_second_iteration_enabled: bool,
) -> PipelexBundleSpec:
fixed_pipes: list[PipeSpecUnion] = []
- # Fix static validation errors for PipeController inputs
- if bundle_error.static_validation_error:
- static_error = bundle_error.static_validation_error
- if not static_error.pipe_code:
- msg = "Static validation error had no pipe code"
- raise PipelexBundleUnexpectedError(message=msg) from bundle_error
- if not pipelex_bundle_spec.pipe:
- msg = "Static validation error pipelex_bundle_spec had no pipe section"
- raise PipelexBundleUnexpectedError(message=msg) from bundle_error
- pipe_spec = pipelex_bundle_spec.pipe.get(static_error.pipe_code)
+ for val_error in bundle_error.pipe_validation_error_data:
+ if not val_error.pipe_code or not pipelex_bundle_spec.pipe:
+ continue
+
+ pipe_spec = pipelex_bundle_spec.pipe.get(val_error.pipe_code)
if not pipe_spec:
- msg = f"Static validation error pipelex_bundle_spec had no pipe spec for considered pipe code: '{static_error.pipe_code}'"
- raise PipelexBundleUnexpectedError(message=msg) from bundle_error
- match static_error.error_type:
- case StaticValidationErrorType.MISSING_INPUT_VARIABLE | StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE:
- if not AllowedPipeCategories.is_controller_by_str(category_str=pipe_spec.pipe_category):
- msg = (
- f"Static validation error: pipelex_bundle_spec had an input requirement error for a pipe spec of type "
- f"{pipe_spec.type} for considered pipe code: '{static_error.pipe_code}' but it was not a PipeController. "
- "We don't support fixing this error yet. Dump of the pipelex_bundle_spec:\n\n"
- f"{pipelex_bundle_spec.model_dump_json(serialize_as_any=True, indent=2)}"
- )
- raise PipelexBundleNoFixForError(message=msg) from bundle_error
+ continue
- pipe = get_required_pipe(pipe_code=static_error.pipe_code)
+ match val_error.error_type:
+ case PipeValidationErrorType.INPUT_REQUIREMENT_MISMATCH:
+ # Fix input requirement mismatch by updating the specific mismatched input(s)
+ # This applies to ALL pipe categories (operators and controllers)
+ if not PipeCategory.is_controller_by_str(category_str=pipe_spec.pipe_category):
+ continue
+
+ pipe = get_required_pipe(pipe_code=val_error.pipe_code)
needed_inputs = pipe.needed_inputs()
- # Build the new inputs dict from needed_inputs
- new_inputs: dict[str, str] = {}
- for named_requirement in needed_inputs.named_input_requirements:
- new_inputs[named_requirement.variable_name] = named_requirement.concept.code
- # Update the pipe spec inputs
+
+ # Start with existing inputs, we'll only override the mismatched ones
+ new_inputs: dict[str, str] = dict(pipe_spec.inputs) if pipe_spec.inputs else {}
+
+ # Get the variable names that have mismatches
+ mismatched_variables = val_error.variable_names or []
+
+ # Update only the mismatched inputs with the correct concept from needed_inputs
+ for variable_name in mismatched_variables:
+ for named_requirement in needed_inputs.named_input_requirements:
+ if named_requirement.variable_name == variable_name:
+ old_value = new_inputs.get(variable_name, "NOT SET")
+ concept_code_with_multiplicity = format_concept_with_multiplicity(
+ concept_code_or_string=named_requirement.concept.code,
+ multiplicity=named_requirement.multiplicity,
+ )
+ new_inputs[variable_name] = concept_code_with_multiplicity
+ # TODO: return a structured report of what was done, let the caller decide if they want to print it or act on it
+ log.info(
+ f"🔧 Fixed input requirement mismatch for pipe '{val_error.pipe_code}': input '{variable_name}' \
+ changed from '{old_value}' → '{concept_code_with_multiplicity}'"
+ )
+ break
+
pipe_spec.inputs = new_inputs
fixed_pipes.append(pipe_spec)
- case StaticValidationErrorType.INADEQUATE_INPUT_CONCEPT:
- msg = "Static validation error had inadequate input concept. We don't support fixing this error yet."
- raise PipelexBundleNoFixForError(message=msg) from bundle_error
- case StaticValidationErrorType.TOO_MANY_CANDIDATE_INPUTS:
- msg = "Static validation error had too many candidate inputs. We don't support fixing this error yet."
- raise PipelexBundleNoFixForError(message=msg) from bundle_error
-
- if fixed_pipes and is_save_second_iteration_enabled:
+
+ case PipeValidationErrorType.MISSING_INPUT_VARIABLE | PipeValidationErrorType.EXTRANEOUS_INPUT_VARIABLE:
+ # Fix input variables for PipeController ONLY by copying all requirements from needed_inputs
+ if not PipeCategory.is_controller_by_str(category_str=pipe_spec.pipe_category):
+ continue
+
+ pipe = get_required_pipe(pipe_code=val_error.pipe_code)
+ needed_inputs = pipe.needed_inputs()
+ old_inputs = dict(pipe_spec.inputs) if pipe_spec.inputs else {}
+ fixed_inputs: dict[str, str] = {}
+ for named_requirement in needed_inputs.named_input_requirements:
+ concept_code_with_multiplicity = format_concept_with_multiplicity(
+ concept_code_or_string=named_requirement.concept.code,
+ multiplicity=named_requirement.multiplicity,
+ )
+ fixed_inputs[named_requirement.variable_name] = concept_code_with_multiplicity
+
+ # Only apply fix if it actually changes something (avoid infinite loops)
+ if fixed_inputs != old_inputs:
+ pipe_spec.inputs = fixed_inputs
+ fixed_pipes.append(pipe_spec)
+ log.info(f"🔧 Fixed input variables for pipe '{val_error.pipe_code}': BEFORE={old_inputs} → AFTER={fixed_inputs}")
+ else:
+ log.warning(
+ f"⚠️ Cannot auto-fix MISSING_INPUT_VARIABLE for pipe '{val_error.pipe_code}': needed_inputs() \
+ doesn't include the missing variable '{val_error.variable_names}'. \
+ This might be an intermediate variable that shouldn't be in inputs."
+ )
+
+ case PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT:
+ # Fix output concept mismatch for PipeSequence by updating to match last step's output
+ if not isinstance(pipe_spec, PipeSequenceSpec):
+ continue
+
+ last_step = pipe_spec.steps[-1]
+ last_step_pipe_code = last_step.pipe_code
+
+ # Get the last step's pipe spec to retrieve its output
+ last_step_pipe_spec = pipelex_bundle_spec.pipe.get(last_step_pipe_code)
+ if not last_step_pipe_spec:
+ continue
+
+ old_output = pipe_spec.output
+ new_output = last_step_pipe_spec.output
+
+ # Set the sequence output to match the last step's output
+ pipe_spec.output = new_output
+ fixed_pipes.append(pipe_spec)
+ # TODO: return a structured report of what was done, let the caller decide if they want to print it or act on it
+ log.info(
+ f"🔧 Fixed output concept for pipe '{val_error.pipe_code}': output changed from '{old_output}' → \
+ '{new_output}' (matching last step '{last_step_pipe_code}')"
+ )
+
+ case (
+ PipeValidationErrorType.LLM_OUTPUT_CANNOT_BE_IMAGE
+ | PipeValidationErrorType.IMG_GEN_INPUT_NOT_TEXT_COMPATIBLE
+ | PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
+ | PipeValidationErrorType.CIRCULAR_DEPENDENCY_ERROR
+ ):
+ continue
+
+ # Reconstruct bundle if we made changes
+ if fixed_pipes:
pipelex_bundle_spec = reconstruct_bundle_with_pipe_fixes(pipelex_bundle_spec=pipelex_bundle_spec, fixed_pipes=fixed_pipes)
- pretty_print(pipelex_bundle_spec, title="Pipelex Bundle Spec • 2nd iteration")
- plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
- second_iteration_path = get_incremental_file_path(
- base_path="results",
- base_name="generated_pipeline_2nd_iteration",
- extension="plx",
- )
- save_text_to_path(text=plx_content, path=second_iteration_path)
+ if is_save_second_iteration_enabled:
+ plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
+ second_iteration_path = get_incremental_file_path(
+ base_path="results",
+ base_name="generated_pipeline_2nd_iteration",
+ extension="plx",
+ )
+ save_text_to_path(text=plx_content, path=second_iteration_path)
return pipelex_bundle_spec
diff --git a/pipelex/builder/builder_validation.py b/pipelex/builder/builder_validation.py
deleted file mode 100644
index fdf383416..000000000
--- a/pipelex/builder/builder_validation.py
+++ /dev/null
@@ -1,301 +0,0 @@
-from pipelex import log
-from pipelex.builder.builder_errors import (
- ConceptSpecError,
- PipeFailure,
- PipelexBundleUnexpectedError,
- PipeSpecError,
- ValidateDryRunError,
-)
-from pipelex.builder.bundle_spec import PipelexBundleSpec
-from pipelex.builder.exceptions import PipelexBundleError
-from pipelex.builder.pipe.pipe_spec_map import pipe_type_to_spec_class
-from pipelex.builder.validation_error_data import DomainFailure, PipeInputErrorData, StaticValidationErrorData
-from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.exceptions import (
- ConceptDefinitionErrorData,
-)
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.core.pipes.exceptions import PipeInputError
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories
-from pipelex.hub import get_library_manager, get_required_pipe
-from pipelex.libraries.exceptions import (
- ConceptLoadingError,
- DomainLoadingError,
- PipeDefinitionErrorData,
- PipeLoadingError,
-)
-from pipelex.pipe_run.dry_run import DryRunOutput, dry_run_pipes
-
-
-def fix_inputs_consistency(bundle_spec: PipelexBundleSpec) -> PipelexBundleSpec:
- """Proactively fix input declarations for all PipeController pipes.
-
- This function rebuilds the inputs dict for all PipeController pipes (PipeSequence,
- PipeParallel, PipeCondition, PipeBatch) based on their actual requirements computed
- by the needed_inputs() method. This ensures consistency before validation.
-
- Args:
- bundle_spec: The bundle spec to fix.
-
- Returns:
- The modified bundle spec with fixed inputs.
-
- Raises:
- PipelexBundleError: If loading the bundle fails or other errors occur.
- """
- log.dev(f"🔧 Starting input consistency fix for domain '{bundle_spec.domain}'")
-
- if not bundle_spec.pipe:
- log.dev("No pipes found in bundle spec, skipping input consistency fix")
- return bundle_spec
-
- # Convert to blueprint and load into library
- try:
- bundle_blueprint = bundle_spec.to_blueprint()
- except ConceptSpecError as concept_spec_error:
- concept_failures = [concept_spec_error.concept_failure]
- raise PipelexBundleError(message=concept_spec_error.message, concept_failures=concept_failures) from concept_spec_error
- except PipeSpecError as pipe_spec_error:
- pipe_failures = [pipe_spec_error.pipe_failure]
- raise PipelexBundleError(message=pipe_spec_error.message, pipe_failures=pipe_failures) from pipe_spec_error
-
- log.dev(f"Loading bundle blueprint for domain '{bundle_spec.domain}' into library manager")
- library_manager = get_library_manager()
- try:
- library_manager.load_from_blueprint(blueprint=bundle_blueprint)
- log.dev(f"Successfully loaded bundle with {len(bundle_spec.pipe)} pipes")
- except StaticValidationError as static_validation_error:
- static_validation_error_data = StaticValidationErrorData(
- error_type=static_validation_error.error_type,
- domain=static_validation_error.domain,
- pipe_code=static_validation_error.pipe_code,
- variable_names=static_validation_error.variable_names,
- required_concept_codes=static_validation_error.required_concept_codes,
- provided_concept_code=static_validation_error.provided_concept_code,
- file_path=static_validation_error.file_path,
- explanation=static_validation_error.explanation,
- )
- raise PipelexBundleError(
- message=static_validation_error.desc(), static_validation_error=static_validation_error_data
- ) from static_validation_error
- except DomainLoadingError as domain_loading_error:
- domain_failures = [DomainFailure(domain_code=domain_loading_error.domain_code, error_message=str(domain_loading_error))]
- raise PipelexBundleError(message=domain_loading_error.message, domain_failures=domain_failures) from domain_loading_error
- except ConceptLoadingError as concept_loading_error:
- concept_def_error = concept_loading_error.concept_definition_error
- concept_definition_error_data = ConceptDefinitionErrorData(
- message=str(concept_def_error),
- domain_code=concept_def_error.domain_code,
- concept_code=concept_def_error.concept_code,
- description=concept_def_error.description,
- structure_class_python_code=concept_def_error.structure_class_python_code,
- structure_class_syntax_error_data=concept_def_error.structure_class_syntax_error_data,
- source=concept_def_error.source,
- )
- raise PipelexBundleError(
- message=concept_loading_error.message, concept_definition_errors=[concept_definition_error_data]
- ) from concept_loading_error
- except PipeLoadingError as pipe_loading_error:
- pipe_def_error = pipe_loading_error.pipe_definition_error
- pipe_definition_error_data = PipeDefinitionErrorData(
- message=str(pipe_def_error),
- domain_code=pipe_def_error.domain_code,
- pipe_code=pipe_def_error.pipe_code,
- description=pipe_def_error.description,
- source=pipe_def_error.source,
- )
- raise PipelexBundleError(message=pipe_loading_error.message, pipe_definition_errors=[pipe_definition_error_data]) from pipe_loading_error
- except PipeInputError as pipe_input_error:
- pipe_input_error_data = PipeInputErrorData(
- message=str(pipe_input_error),
- pipe_code=pipe_input_error.pipe_code,
- variable_name=pipe_input_error.variable_name,
- concept_code=pipe_input_error.concept_code,
- )
- raise PipelexBundleError(message=pipe_input_error.message, pipe_input_errors=[pipe_input_error_data]) from pipe_input_error
-
- # Fix inputs for all PipeController pipes
- log.dev("Starting to fix inputs for PipeController pipes")
- fixed_count = 0
- try:
- for pipe_code, pipe_spec in bundle_spec.pipe.items():
- # Check if this is a PipeController
- if AllowedPipeCategories.is_controller_by_str(category_str=pipe_spec.pipe_category):
- log.dev(f" Checking inputs for {pipe_spec.type} pipe '{pipe_code}'")
-
- # Get the loaded pipe instance
- pipe = get_required_pipe(pipe_code=pipe_code)
-
- # Get the actual needed inputs
- needed_inputs = pipe.needed_inputs()
-
- # Store old inputs for logging
- old_inputs = pipe_spec.inputs.copy()
-
- # Rebuild the inputs dict from needed_inputs, preserving multiplicity
- new_inputs: dict[str, str] = {}
- for named_requirement in needed_inputs.named_input_requirements:
- concept_code = named_requirement.concept.code
- # Preserve multiplicity brackets
- if named_requirement.multiplicity is not None:
- if named_requirement.multiplicity is True:
- # Variable-length list []
- concept_code = f"{concept_code}[]"
- else:
- # Fixed-length list [N] where N is an int
- concept_code = f"{concept_code}[{named_requirement.multiplicity}]"
- new_inputs[named_requirement.variable_name] = concept_code
-
- # Update the pipe spec inputs
- pipe_spec.inputs = new_inputs
-
- # Log the changes
- if old_inputs != new_inputs:
- log.dev(f" Old inputs: {old_inputs}")
- log.dev(f" New inputs: {new_inputs}")
- fixed_count += 1
- else:
- log.dev(" ✅")
- finally:
- # Clean up by removing the bundle from library manager
- log.dev("Cleaning up: removing bundle from library manager")
- library_manager.remove_from_blueprint(blueprint=bundle_blueprint)
-
- log.dev(f"✅ Input consistency fix completed: fixed {fixed_count} PipeController pipe(s)")
- return bundle_spec
-
-
-async def validate_bundle_spec(bundle_spec: PipelexBundleSpec):
- try:
- bundle_blueprint = bundle_spec.to_blueprint()
- except ConceptSpecError as concept_spec_error:
- concept_failures = [concept_spec_error.concept_failure]
- raise PipelexBundleError(message=concept_spec_error.message, concept_failures=concept_failures) from concept_spec_error
- except PipeSpecError as pipe_spec_error:
- pipe_failures = [pipe_spec_error.pipe_failure]
- raise PipelexBundleError(message=pipe_spec_error.message, pipe_failures=pipe_failures) from pipe_spec_error
-
- library_manager = get_library_manager()
- dry_run_result = await dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
- library_manager.remove_from_blueprint(blueprint=bundle_blueprint)
-
- dry_run_pipe_failures = extract_pipe_failures_from_dry_run_result(bundle_spec=bundle_spec, dry_run_result=dry_run_result)
- if dry_run_pipe_failures:
- raise PipelexBundleError(message="Pipes failed during dry run", pipe_failures=dry_run_pipe_failures)
-
-
-def extract_pipe_failures_from_dry_run_result(bundle_spec: PipelexBundleSpec, dry_run_result: dict[str, DryRunOutput]) -> list[PipeFailure]:
- dry_run_pipe_failures: list[PipeFailure] = []
- for pipe_code, dry_run_output in dry_run_result.items():
- if dry_run_output.status.is_failure:
- if not bundle_spec.pipe:
- msg = f"No pipes section found in bundle spec but we recorded a dry run failure for pipe '{pipe_code}'"
- raise PipelexBundleUnexpectedError(message="No pipes section found in bundle spec")
- if pipe_code not in bundle_spec.pipe:
- msg = f"Pipe '{pipe_code}' not found in bundle spec but we recorded a dry run failure for it"
- raise PipelexBundleUnexpectedError(message=msg)
-
- pipe_spec = bundle_spec.pipe[pipe_code]
- spec_class = pipe_type_to_spec_class.get(pipe_spec.type)
- if not spec_class:
- msg = f"Unknown pipe type: {pipe_spec.type}"
- raise ValidateDryRunError(msg)
- pipe_spec = spec_class(**pipe_spec.model_dump(serialize_as_any=True))
- pipe_failure = PipeFailure(
- pipe_spec=pipe_spec,
- error_message=dry_run_output.error_message or "",
- )
- dry_run_pipe_failures.append(pipe_failure)
- return dry_run_pipe_failures
-
-
-def document_pipe_failures_from_dry_run_blueprint(
- bundle_blueprint: PipelexBundleBlueprint, dry_run_result: dict[str, DryRunOutput]
-) -> list[PipeFailure]:
- dry_run_pipe_failures: list[PipeFailure] = []
- for pipe_code, dry_run_output in dry_run_result.items():
- if dry_run_output.status.is_failure:
- if not bundle_blueprint.pipe:
- msg = f"No pipes section found in bundle spec but we recorded a dry run failure for pipe '{pipe_code}'"
- raise PipelexBundleUnexpectedError(message="No pipes section found in bundle spec")
- if pipe_code not in bundle_blueprint.pipe:
- msg = f"Pipe '{pipe_code}' not found in bundle spec but we recorded a dry run failure for it"
- raise PipelexBundleUnexpectedError(message=msg)
-
- pipe_spec = bundle_blueprint.pipe[pipe_code]
- spec_class = pipe_type_to_spec_class.get(pipe_spec.type)
- if not spec_class:
- msg = f"Unknown pipe type: {pipe_spec.type}"
- raise ValidateDryRunError(msg)
- pipe_spec = spec_class(**pipe_spec.model_dump(serialize_as_any=True))
- pipe_failure = PipeFailure(
- pipe_spec=pipe_spec,
- error_message=dry_run_output.error_message or "",
- )
- dry_run_pipe_failures.append(pipe_failure)
- return dry_run_pipe_failures
-
-
-async def dry_run_bundle_blueprint(bundle_blueprint: PipelexBundleBlueprint) -> dict[str, DryRunOutput]:
- library_manager = get_library_manager()
- try:
- pipes = library_manager.load_from_blueprint(blueprint=bundle_blueprint)
- dry_run_result = await dry_run_pipes(pipes=pipes, raise_on_failure=True)
- except StaticValidationError as static_validation_error:
- static_validation_error_data = StaticValidationErrorData(
- error_type=static_validation_error.error_type,
- domain=static_validation_error.domain,
- pipe_code=static_validation_error.pipe_code,
- variable_names=static_validation_error.variable_names,
- required_concept_codes=static_validation_error.required_concept_codes,
- provided_concept_code=static_validation_error.provided_concept_code,
- file_path=static_validation_error.file_path,
- explanation=static_validation_error.explanation,
- )
- raise PipelexBundleError(
- message=static_validation_error.desc(), static_validation_error=static_validation_error_data
- ) from static_validation_error
- except DomainLoadingError as domain_loading_error:
- domain_failures = [DomainFailure(domain_code=domain_loading_error.domain_code, error_message=str(domain_loading_error))]
- raise PipelexBundleError(message=domain_loading_error.message, domain_failures=domain_failures) from domain_loading_error
- except ConceptLoadingError as concept_loading_error:
- concept_def_error = concept_loading_error.concept_definition_error
- concept_definition_error_data = ConceptDefinitionErrorData(
- message=str(concept_def_error),
- domain_code=concept_def_error.domain_code,
- concept_code=concept_def_error.concept_code,
- description=concept_def_error.description,
- structure_class_python_code=concept_def_error.structure_class_python_code,
- structure_class_syntax_error_data=concept_def_error.structure_class_syntax_error_data,
- source=concept_def_error.source,
- )
- raise PipelexBundleError(
- message=concept_loading_error.message, concept_definition_errors=[concept_definition_error_data]
- ) from concept_loading_error
- except PipeLoadingError as pipe_loading_error:
- pipe_def_error = pipe_loading_error.pipe_definition_error
- pipe_definition_error_data = PipeDefinitionErrorData(
- message=str(pipe_def_error),
- domain_code=pipe_def_error.domain_code,
- pipe_code=pipe_def_error.pipe_code,
- description=pipe_def_error.description,
- source=pipe_def_error.source,
- )
- raise PipelexBundleError(message=pipe_loading_error.message, pipe_definition_errors=[pipe_definition_error_data]) from pipe_loading_error
- except PipeInputError as pipe_input_error:
- pipe_input_error_data = PipeInputErrorData(
- message=str(pipe_input_error),
- pipe_code=pipe_input_error.pipe_code,
- variable_name=pipe_input_error.variable_name,
- concept_code=pipe_input_error.concept_code,
- )
- raise PipelexBundleError(message=pipe_input_error.message, pipe_input_errors=[pipe_input_error_data]) from pipe_input_error
- return dry_run_result
-
-
-async def validate_dry_run_bundle_blueprint(bundle_blueprint: PipelexBundleBlueprint):
- dry_run_result = await dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
- pipe_failures = document_pipe_failures_from_dry_run_blueprint(bundle_blueprint=bundle_blueprint, dry_run_result=dry_run_result)
- if pipe_failures:
- msg = "Dry run failed for bundle"
- raise PipelexBundleError(message=msg, pipe_failures=pipe_failures)
diff --git a/pipelex/builder/bundle_spec.py b/pipelex/builder/bundle_spec.py
index e23d4a6f9..5c983f2e2 100644
--- a/pipelex/builder/bundle_spec.py
+++ b/pipelex/builder/bundle_spec.py
@@ -4,15 +4,8 @@ from rich.table import Table
from rich.text import Text
from typing_extensions import override
-from pipelex.builder.builder_errors import (
- ConceptFailure,
- ConceptSpecError,
- PipeBuilderError,
- PipeFailure,
- PipeSpecError,
-)
from pipelex.builder.concept.concept_spec import ConceptSpec
-from pipelex.builder.exceptions import PipelexBundleError, PipelexBundleSpecValueError
+from pipelex.builder.exceptions import PipelexBundleSpecBlueprintError
from pipelex.builder.pipe.pipe_spec_union import PipeSpecUnion
from pipelex.core.bundles.pipe_sorter import sort_pipes_by_dependencies
from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion, PipelexBundleBlueprint
@@ -72,14 +65,14 @@ class PipelexBundleSpec(StructuredContent):
validate_domain_code(code=domain)
except DomainCodeError as exc:
msg = f"Error when trying to validate pipelex bundle spec: domain '{domain}' is not a valid domain code: {exc}"
- raise PipelexBundleSpecValueError(msg) from exc
+ raise ValueError(msg) from exc
return domain
@model_validator(mode="after")
def validate_main_pipe(self) -> "PipelexBundleSpec":
if not self.pipe or (self.main_pipe not in self.pipe):
msg = f"Main pipe '{self.main_pipe}' could not be found in bundle spec"
- raise PipelexBundleError(message=msg)
+ raise ValueError(msg)
return self
def to_blueprint(self) -> PipelexBundleBlueprint:
@@ -89,12 +82,7 @@ class PipelexBundleSpec(StructuredContent):
concept = {}
for concept_code, concept_spec_or_name in self.concept.items():
if isinstance(concept_spec_or_name, ConceptSpec):
- try:
- concept[concept_code] = concept_spec_or_name.to_blueprint()
- except ValidationError as exc:
- msg = f"Failed to create concept blueprint from spec for concept code {concept_code}: {format_pydantic_validation_error(exc)}"
- concept_failure = ConceptFailure(concept_spec=concept_spec_or_name, error_message=msg)
- raise ConceptSpecError(message=msg, concept_failure=concept_failure) from exc
+ concept[concept_code] = concept_spec_or_name.to_blueprint()
else:
concept[concept_code] = ConceptBlueprint(description=concept_code, structure=concept_spec_or_name)
@@ -107,18 +95,12 @@ class PipelexBundleSpec(StructuredContent):
pipe_blueprints[pipe_code] = pipe_spec.to_blueprint()
except ValidationError as exc:
msg = f"Failed to create pipe blueprint from spec for pipe code {pipe_code}: {format_pydantic_validation_error(exc)}"
- pipe_failure = PipeFailure(pipe_spec=pipe_spec, error_message=msg)
- raise PipeSpecError(message=msg, pipe_failure=pipe_failure) from exc
+ raise PipelexBundleSpecBlueprintError(msg) from exc
# Then, sort blueprints by dependencies
- try:
- sorted_pipe_items = sort_pipes_by_dependencies(pipe_blueprints)
- except Exception as exc:
- msg = f"Failed to sort pipes by dependencies: {exc}"
- raise PipeBuilderError(msg) from exc
-
+ sorted_pipe_items = sort_pipes_by_dependencies(pipe_blueprints)
# Finally, create the ordered dict
- pipe = dict(sorted_pipe_items)
+ pipe = dict[str, PipeBlueprintUnion](sorted_pipe_items)
return PipelexBundleBlueprint(
domain=self.domain,
diff --git a/pipelex/builder/concept/concept.plx b/pipelex/builder/concept/concept.plx
index 2a11af9c4..e34cce392 100644
--- a/pipelex/builder/concept/concept.plx
+++ b/pipelex/builder/concept/concept.plx
@@ -4,37 +4,3 @@ description = "Build and process concepts for Pipelex bundles from signatures an
[concept]
ConceptStructureSpec = "A concept spec with structure but without full implementation."
ConceptSpec = "A specification for a concept including its code, description, and a structure draft as plain text."
-ConceptSpecDraft = "A specification for a concept including its code, description, and a structure draft as plain text."
-
-[pipe.build_concept_spec]
-type = "PipeSequence"
-description = "Create a ConceptSpec from a brief, existing concepts, and concept rules."
-inputs = { concept_spec_draft = "ConceptSpecDraft"}
-output = "ConceptSpec"
-steps = [
- { pipe = "spec_draft_to_structure", result = "concept_spec_structures" },
- { pipe = "spec_draft_to_concept_spec", result = "concept_spec" }
-]
-
-[pipe.spec_draft_to_structure]
-type = "PipeLLM"
-description = "Convert the ConceptSpec (with its structure draft) into a proper ConceptStructureSpec."
-inputs = { concept_spec_draft = "ConceptSpecDraft" }
-output = "ConceptStructureSpec[]"
-model = "llm_to_engineer"
-prompt = """
-Create a ConceptStructureSpec from the ConceptSpecDraft.
-Please focus only on the structure.
-
-The field "description" IS NOT a structure. It is a general description of the concept.
-If the field "structure" is empty, return an empty list.
-
-@concept_spec_draft
-"""
-
-[pipe.spec_draft_to_concept_spec]
-type = "PipeFunc"
-description = "Generate the final ConceptSpec using the spec and structure manually."
-inputs = { concept_spec_draft = "ConceptSpecDraft", concept_spec_structures = "ConceptStructureSpec"}
-output = "ConceptSpec"
-function_name = "create_concept_spec"
diff --git a/pipelex/builder/concept/concept_spec.py b/pipelex/builder/concept/concept_spec.py
index 75eeed0a4..0ebfb827e 100644
--- a/pipelex/builder/concept/concept_spec.py
+++ b/pipelex/builder/concept/concept_spec.py
@@ -8,10 +8,10 @@ from rich.text import Text
from typing_extensions import override
from pipelex import log
+from pipelex.base_exceptions import PipelexError
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint, ConceptStructureBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.exceptions import ConceptBlueprintValueError, ConceptStructureBlueprintValueError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.tools.misc.pretty import PrettyPrintable
from pipelex.tools.misc.string_utils import is_pascal_case, normalize_to_ascii, snake_to_pascal_case
@@ -26,7 +26,7 @@ class ConceptStructureSpecFieldType(StrEnum):
DATE = "date"
-class ConceptSpecError(Exception):
+class ConceptSpecError(PipelexError):
pass
@@ -94,7 +94,7 @@ class ConceptStructureSpec(StructuredContent):
def _raise_type_mismatch_error(self, expected_type_name: str, actual_type_name: str) -> None:
msg = f"default_value type mismatch: expected {expected_type_name} for type '{self.type}', but got {actual_type_name}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
def to_blueprint(self) -> ConceptStructureBlueprint:
# Convert the type enum value - self.type is already a ConceptStructureBlueprintFieldType enum
@@ -110,19 +110,6 @@ class ConceptStructureSpec(StructuredContent):
)
-class ConceptSpecDraft(StructuredContent):
- the_concept_code: str = Field(description="Concept code. Must be PascalCase.")
- description: str = Field(description="Description of the concept, in natural language.")
- structure: str = Field(
- description="Description of a dict with fieldnames as keys, and values being a dict with: description, type, required, default_value",
- )
- refines: str | None = Field(
- default=None,
- description="The native concept this concept extends (Text, Image, PDF, TextAndImages, Number, Page) "
- "in PascalCase format. Cannot be used together with 'structure'.",
- )
-
-
class ConceptSpec(StructuredContent):
"""Spec structuring a concept: a conceptual data type that can either define its own structure or refine an existing native concept.
@@ -195,7 +182,7 @@ class ConceptSpec(StructuredContent):
if refines is not None:
if not NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=refines):
msg = f"Forbidden to refine a non-native concept: '{refines}'. Refining non-native concepts will come soon."
- raise ConceptBlueprintValueError(msg)
+ raise ValueError(msg)
return refines
@model_validator(mode="before")
@@ -204,7 +191,7 @@ class ConceptSpec(StructuredContent):
if values.get("refines") and values.get("structure"):
msg = (
f"Forbidden to have refines and structure at the same time: `{values.get('refines')}` "
- f"and `{values.get('structure')}` for concept that has the description `{values.get('description')}`",
+ f"and `{values.get('structure')}` for concept that has the description `{values.get('description')}`"
)
raise ConceptSpecError(msg)
return values
diff --git a/pipelex/builder/concept/concept_spec_factory.py b/pipelex/builder/concept/concept_spec_factory.py
deleted file mode 100644
index 0ff8c8e37..000000000
--- a/pipelex/builder/concept/concept_spec_factory.py
+++ /dev/null
@@ -1,27 +0,0 @@
-from pipelex.builder.concept.concept_spec import ConceptSpec, ConceptSpecDraft, ConceptStructureSpec
-from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.system.registries.func_registry import pipe_func
-
-
-@pipe_func()
-async def create_concept_spec(working_memory: WorkingMemory) -> ConceptSpec:
- concept_spec_draft = working_memory.get_stuff_as(name="concept_spec_draft", content_type=ConceptSpecDraft)
- concept_spec_structures_stuff = working_memory.get_stuff_as_list(name="concept_spec_structures", item_type=ConceptStructureSpec)
-
- structure_dict: dict[str, ConceptStructureSpec] = {}
- for structure_item in concept_spec_structures_stuff.items:
- structure_spec = ConceptStructureSpec(
- the_field_name=structure_item.the_field_name,
- description=structure_item.description,
- type=structure_item.type,
- required=structure_item.required,
- default_value=structure_item.default_value,
- )
- structure_dict[structure_item.the_field_name] = structure_spec
-
- return ConceptSpec(
- the_concept_code=concept_spec_draft.the_concept_code,
- description=concept_spec_draft.description,
- structure=structure_dict,
- refines=concept_spec_draft.refines,
- )
diff --git a/pipelex/builder/exceptions.py b/pipelex/builder/exceptions.py
index c1313b375..a6a416f78 100644
--- a/pipelex/builder/exceptions.py
+++ b/pipelex/builder/exceptions.py
@@ -1,43 +1,5 @@
-from typing_extensions import Self, override
+from pipelex.base_exceptions import PipelexError
-from pipelex.builder.validation_error_data import (
- ConceptFailure,
- DomainFailure,
- PipeFailure,
- PipeInputErrorData,
- StaticValidationErrorData,
-)
-from pipelex.core.concepts.exceptions import ConceptDefinitionErrorData, PipelexValidationExceptionAbstract
-from pipelex.core.pipes.exceptions import PipeDefinitionErrorData
-
-class PipelexBundleSpecValueError(ValueError):
+class PipelexBundleSpecBlueprintError(PipelexError):
pass
-
-
-class PipelexBundleError(PipelexValidationExceptionAbstract):
- """Main bundle error that aggregates multiple types of errors."""
-
- def __init__(
- self: Self,
- message: str,
- static_validation_error: StaticValidationErrorData | None = None,
- domain_failures: list[DomainFailure] | None = None,
- pipe_failures: list[PipeFailure] | None = None,
- concept_failures: list[ConceptFailure] | None = None,
- concept_definition_errors: list[ConceptDefinitionErrorData] | None = None,
- pipe_definition_errors: list[PipeDefinitionErrorData] | None = None,
- pipe_input_errors: list[PipeInputErrorData] | None = None,
- ) -> None:
- self.static_validation_error = static_validation_error
- self.domain_failures = domain_failures
- self.pipe_input_errors = pipe_input_errors
- self.pipe_failures = pipe_failures
- self.concept_failures = concept_failures
- self.concept_definition_errors = concept_definition_errors
- self.pipe_definition_errors = pipe_definition_errors
- super().__init__(message)
-
- @override
- def get_concept_definition_errors(self) -> list[ConceptDefinitionErrorData]:
- return self.concept_definition_errors or []
diff --git a/pipelex/builder/flow.py b/pipelex/builder/flow.py
index b6d04d4c9..1cccb94ab 100644
--- a/pipelex/builder/flow.py
+++ b/pipelex/builder/flow.py
@@ -1,6 +1,6 @@
from pydantic import ConfigDict, Field, model_validator
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.builder.pipe.pipe_signature import PipeSignature
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
@@ -11,7 +11,7 @@ from pipelex.tools.typing.validation_utils import has_exactly_one_among_attribut
from pipelex.types import Self
-class FlowElementError(PipelexException):
+class FlowElementError(PipelexError):
"""Exception raised by FlowElement."""
diff --git a/pipelex/builder/flow_factory.py b/pipelex/builder/flow_factory.py
index dfcc43503..7fb16a37f 100644
--- a/pipelex/builder/flow_factory.py
+++ b/pipelex/builder/flow_factory.py
@@ -2,20 +2,20 @@ from pathlib import Path
from typing import Any
from pipelex import log
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.builder.builder import PipelexBundleSpec
from pipelex.builder.flow import Flow, FlowElement
from pipelex.builder.pipe.pipe_signature import PipeSignature
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.interpreter import PipelexInterpreter
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.pipes.pipe_blueprint import PipeCategory
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
-class FlowFactoryError(PipelexException):
+class FlowFactoryError(PipelexError):
"""Exception raised by FlowFactory."""
@@ -36,8 +36,7 @@ class FlowFactory:
Returns:
Flow with controllers preserved and operators as signatures.
"""
- plx_path = Path(plx_file_path) if isinstance(plx_file_path, str) else plx_file_path
- bundle_blueprint = PipelexInterpreter(file_path=plx_path).make_pipelex_bundle_blueprint()
+ bundle_blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=str(plx_file_path))
return FlowFactory.make_from_bundle_blueprint(bundle_blueprint)
@staticmethod
@@ -54,7 +53,7 @@ class FlowFactory:
if bundle_blueprint.pipe:
for pipe_code, pipe_blueprint in bundle_blueprint.pipe.items():
- if pipe_blueprint.pipe_category == AllowedPipeCategories.PIPE_CONTROLLER:
+ if PipeCategory.is_controller_by_str(pipe_blueprint.pipe_category):
# Keep controllers as-is (they are already blueprints which match spec structure)
# Type check to ensure we only assign controller blueprints
if isinstance(
diff --git a/pipelex/builder/pipe/pipe_condition_spec.py b/pipelex/builder/pipe/pipe_condition_spec.py
index 87afeb99e..00d505f9a 100644
--- a/pipelex/builder/pipe/pipe_condition_spec.py
+++ b/pipelex/builder/pipe/pipe_condition_spec.py
@@ -1,6 +1,6 @@
from typing import Literal
-from pydantic import Field
+from pydantic import Field, field_validator
from pydantic.json_schema import SkipJsonSchema
from rich.console import Group
from rich.panel import Panel
@@ -31,6 +31,11 @@ class PipeConditionSpec(PipeSpec):
outcomes: dict[str, str] = Field(..., description="Mapping `dict[str, str]` of condition to outcomes.")
default_outcome: str | SpecialOutcome = Field(description="The fallback outcome if the expression result does not match any key in outcome map.")
+ @field_validator("output", mode="after")
+ @classmethod
+ def forced_output(cls, _: str) -> str:
+ return "native.Anything"
+
@override
def rendered_pretty(self, title: str | None = None, depth: int = 0) -> PrettyPrintable:
# Get base pipe information from parent
diff --git a/pipelex/builder/pipe/pipe_design.plx b/pipelex/builder/pipe/pipe_design.plx
index e4743db63..ded034e3c 100644
--- a/pipelex/builder/pipe/pipe_design.plx
+++ b/pipelex/builder/pipe/pipe_design.plx
@@ -22,7 +22,7 @@ PipeFailure = "Details of a single pipe failure during dry run."
[pipe.detail_pipe_spec]
type = "PipeCondition"
description = "Route by signature.type to the correct spec emitter."
-inputs = { plan_draft = "PlanDraft", pipe_signature = "PipeSignature", concept_specs = "ConceptSpec" }
+inputs = { plan_draft = "PlanDraft", pipe_signature = "PipeSignature", concept_specs = "concept.ConceptSpec" }
output = "Anything"
expression = "pipe_signature.type"
default_outcome = "fail"
diff --git a/pipelex/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.py
index 65c0baafd..a048ed333 100644
--- a/pipelex/builder/pipe/pipe_extract_spec.py
+++ b/pipelex/builder/pipe/pipe_extract_spec.py
@@ -7,7 +7,6 @@ from rich.text import Text
from typing_extensions import override
from pipelex.builder.pipe.pipe_spec import PipeSpec
-from pipelex.builder.pipe.pipe_spec_exceptions import PipeExtractSpecError
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
from pipelex.tools.misc.pretty import PrettyPrintable
from pipelex.types import StrEnum
@@ -56,10 +55,10 @@ class PipeExtractSpec(PipeSpec):
def validate_extract_inputs(cls, inputs_value: dict[str, str] | None) -> dict[str, str] | None:
if inputs_value is None:
msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."
- raise PipeExtractSpecError(msg)
+ raise ValueError(msg)
if len(inputs_value) != 1:
msg = "PipeExtract must have exactly one input which must be either `Image` or `PDF`."
- raise PipeExtractSpecError(msg)
+ raise ValueError(msg)
return inputs_value
@override
diff --git a/pipelex/builder/pipe/pipe_img_gen_spec.py b/pipelex/builder/pipe/pipe_img_gen_spec.py
index 41bb9440a..788f87e96 100644
--- a/pipelex/builder/pipe/pipe_img_gen_spec.py
+++ b/pipelex/builder/pipe/pipe_img_gen_spec.py
@@ -74,7 +74,7 @@ class PipeImgGenSpec(PipeSpec):
type=self.type,
pipe_category=self.pipe_category,
img_gen_prompt=None,
- img_gen_prompt_var_name=None,
+ img_gen_prompt_var_name=base_blueprint.input_names[0] if base_blueprint.input_names else None,
model=img_gen_choice,
aspect_ratio=None,
background=None,
diff --git a/pipelex/builder/pipe/pipe_parallel_spec.py b/pipelex/builder/pipe/pipe_parallel_spec.py
index ceab29cdd..235b541fd 100644
--- a/pipelex/builder/pipe/pipe_parallel_spec.py
+++ b/pipelex/builder/pipe/pipe_parallel_spec.py
@@ -7,7 +7,6 @@ from rich.text import Text
from typing_extensions import override
from pipelex.builder.pipe.pipe_spec import PipeSpec
-from pipelex.builder.pipe.pipe_spec_exceptions import PipeParallelSpecError
from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpec
from pipelex.core.concepts.validation import validate_concept_string_or_code
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
@@ -50,11 +49,7 @@ class PipeParallelSpec(PipeSpec):
"PipeParallel requires either add_each_output to be True or combined_output to be set, "
"or both, otherwise the pipe won't output anything"
)
- raise PipeParallelSpecError(
- message=msg,
- pipe_code=self.pipe_code,
- description=self.description,
- )
+ raise ValueError(msg)
return self
@override
diff --git a/pipelex/builder/pipe/pipe_signature.py b/pipelex/builder/pipe/pipe_signature.py
index 51d8e2338..5e8b19bec 100644
--- a/pipelex/builder/pipe/pipe_signature.py
+++ b/pipelex/builder/pipe/pipe_signature.py
@@ -7,8 +7,7 @@ from rich.table import Table
from rich.text import Text
from typing_extensions import override
-from pipelex.core.pipes.exceptions import PipeBlueprintValueError
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypes
+from pipelex.core.pipes.pipe_blueprint import PipeCategory, PipeType
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.tools.misc.pretty import PrettyPrintable
@@ -31,8 +30,8 @@ class PipeSignature(StructuredContent):
"""
code: str = Field(description="Pipe code identifying the pipe. Must be snake_case.")
- type: AllowedPipeTypes | str = Field(description="Pipe type.")
- pipe_category: SkipJsonSchema[AllowedPipeCategories] = Field(description="Pipe category set according to its type.")
+ type: PipeType | str = Field(description="Pipe type.")
+ pipe_category: SkipJsonSchema[PipeCategory] = Field(description="Pipe category set according to its type.")
description: str = Field(description="What the pipe does")
inputs: dict[str, str] = Field(
description=(
@@ -57,16 +56,16 @@ class PipeSignature(StructuredContent):
type_str = values["type"]
except TypeError as exc:
msg = f"Invalid type for '{values}': could not get subscript, required for 'type'"
- raise PipeBlueprintValueError(msg) from exc
- # we need to convert the type string to the AllowedPipeTypes enum because it arrives as a str implictly converted to enum but not yet
- the_type = AllowedPipeTypes(type_str)
+ raise ValueError(msg) from exc
+ # we need to convert the type string to the PipeType enum because it arrives as a str implictly converted to enum but not yet
+ the_type = PipeType(type_str)
values["pipe_category"] = the_type.category
return values
@field_validator("type", mode="before")
@classmethod
- def validate_type(cls, type_value: str) -> AllowedPipeTypes:
- return AllowedPipeTypes(type_value)
+ def validate_type(cls, type_value: str) -> PipeType:
+ return PipeType(type_value)
@override
def rendered_pretty(self, title: str | None = None, depth: int = 0) -> PrettyPrintable:
@@ -74,7 +73,7 @@ class PipeSignature(StructuredContent):
if title:
pipe_group.renderables.append(Text(title, style="bold"))
pipe_group.renderables.append(Text.from_markup(f"Pipe Signature: [red]{self.code}[/red]\n", style="bold"))
- pipe_type = self.type.value if isinstance(self.type, AllowedPipeTypes) else str(self.type)
+ pipe_type = self.type.value if isinstance(self.type, PipeType) else str(self.type)
pipe_group.renderables.append(Text.from_markup(f"Type: [bold magenta]{pipe_type}[/bold magenta] ({self.pipe_category.value})\n"))
pipe_group.renderables.append(Text.from_markup(f"Description: [yellow italic]{self.description}[/yellow italic]\n"))
diff --git a/pipelex/builder/pipe/pipe_spec.py b/pipelex/builder/pipe/pipe_spec.py
index 8499becc2..b56e0ef40 100644
--- a/pipelex/builder/pipe/pipe_spec.py
+++ b/pipelex/builder/pipe/pipe_spec.py
@@ -9,8 +9,7 @@ from typing_extensions import override
from pipelex import log
from pipelex.core.concepts.validation import validate_concept_string_or_code
-from pipelex.core.pipes.exceptions import PipeBlueprintValueError
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypes, PipeBlueprint
+from pipelex.core.pipes.pipe_blueprint import PipeBlueprint, PipeCategory, PipeType
from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN, parse_concept_with_multiplicity
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.tools.misc.pretty import PrettyPrintable
@@ -39,12 +38,11 @@ class PipeSpec(StructuredContent):
pipe_code: str = Field(description="Unique pipe identifier. Must be snake_case.")
type: Any = Field(
description=(
- f"Pipe type. Validated at runtime, must be one of: {AllowedPipeTypes}. "
- "Examples: PipeLLM, PipeImgGen, PipeExtract, PipeSequence, PipeParallel."
+ f"Pipe type. Validated at runtime, must be one of: {PipeType}. Examples: PipeLLM, PipeImgGen, PipeExtract, PipeSequence, PipeParallel."
)
)
pipe_category: Any = Field(
- description=(f"Pipe category. Validated at runtime, must be one of: {AllowedPipeCategories}. Either 'PipeController' or 'PipeOperator'.")
+ description=(f"Pipe category. Validated at runtime, must be one of: {PipeCategory}. Either 'PipeController' or 'PipeOperator'.")
)
description: str | None = Field(description="Natural language description of the pipe's purpose and functionality.")
inputs: dict[str, str] = Field(
@@ -71,9 +69,9 @@ class PipeSpec(StructuredContent):
@field_validator("type", mode="after")
@classmethod
def validate_pipe_type(cls, value: Any) -> Any:
- if value not in AllowedPipeTypes.value_list():
- msg = f"Invalid pipe type '{value}'. Must be one of: {AllowedPipeTypes.value_list()}"
- raise PipeBlueprintValueError(msg)
+ if value not in PipeType.value_list():
+ msg = f"Invalid pipe type '{value}'. Must be one of: {PipeType.value_list()}"
+ raise ValueError(msg)
return value
@field_validator("output", mode="after")
@@ -90,22 +88,20 @@ class PipeSpec(StructuredContent):
if inputs is None:
return None
- # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
- multiplicity_pattern = MUTLIPLICITY_PATTERN
-
for input_name, concept_spec in inputs.items():
if not is_snake_case(input_name):
msg = f"Invalid input name syntax '{input_name}'. Must be in snake_case."
- raise PipeBlueprintValueError(msg)
+ raise ValueError(msg)
# Validate the concept spec format with optional multiplicity brackets
- match = re.match(multiplicity_pattern, concept_spec)
+ # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
+ match = re.match(MUTLIPLICITY_PATTERN, concept_spec)
if not match:
msg = (
f"Invalid input syntax for '{input_name}': '{concept_spec}'. "
f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."
)
- raise PipeBlueprintValueError(msg)
+ raise ValueError(msg)
# Extract the concept part (without multiplicity) and validate it
concept_string_or_code = match.group(1)
@@ -123,7 +119,7 @@ class PipeSpec(StructuredContent):
if not is_snake_case(normalized_pipe_code):
msg = f"Invalid pipe code syntax '{normalized_pipe_code}'. Must be in snake_case."
- raise PipeBlueprintValueError(msg)
+ raise ValueError(msg)
return normalized_pipe_code
def to_blueprint(self) -> PipeBlueprint:
diff --git a/pipelex/builder/pipe/pipe_spec_exceptions.py b/pipelex/builder/pipe/pipe_spec_exceptions.py
deleted file mode 100644
index 177bb5d65..000000000
--- a/pipelex/builder/pipe/pipe_spec_exceptions.py
+++ /dev/null
@@ -1,33 +0,0 @@
-from pipelex.core.pipe_errors import PipeDefinitionError
-
-
-class PipeSpecException(PipeDefinitionError):
- pass
-
-
-class PipeExtractSpecError(PipeSpecException):
- pass
-
-
-class PipeParallelSpecError(PipeSpecException):
- pass
-
-
-class PipeSequenceSpecError(PipeSpecException):
- pass
-
-
-class PipeFuncSpecError(PipeSpecException):
- pass
-
-
-class PipeImgGenSpecError(PipeSpecException):
- pass
-
-
-class PipeComposeSpecError(PipeSpecException):
- pass
-
-
-class PipeLLMSpecError(PipeSpecException):
- pass
diff --git a/pipelex/builder/runner_code.py b/pipelex/builder/runner_code.py
index e43e81def..14a1bdab4 100644
--- a/pipelex/builder/runner_code.py
+++ b/pipelex/builder/runner_code.py
@@ -89,7 +89,7 @@ import json
from typing import Any, cast
from pipelex.core.concepts.concept import Concept
-from pipelex.core.pipes.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
from pipelex.core.pipes.pipe_abstract import PipeAbstract
diff --git a/pipelex/builder/validation_error_data.py b/pipelex/builder/validation_error_data.py
deleted file mode 100644
index 05d14fcbc..000000000
--- a/pipelex/builder/validation_error_data.py
+++ /dev/null
@@ -1,71 +0,0 @@
-from pydantic import Field
-
-from pipelex.builder.concept.concept_spec import ConceptSpec
-from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec
-from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec
-from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec
-from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec
-from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec
-from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec
-from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec
-from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec
-from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec
-from pipelex.core.pipes.exceptions import StaticValidationErrorType
-from pipelex.core.stuffs.structured_content import StructuredContent
-
-# ============================================================================
-# BaseModel (StructuredContent) versions of error information
-# ============================================================================
-
-
-class StaticValidationErrorData(StructuredContent):
- """Structured data for StaticValidationError."""
-
- error_type: StaticValidationErrorType = Field(description="The type of static validation error")
- domain: str = Field(description="The domain where the error occurred")
- pipe_code: str | None = Field(None, description="The pipe code if applicable")
- variable_names: list[str] | None = Field(None, description="Variable names involved in the error")
- required_concept_codes: list[str] | None = Field(None, description="Required concept codes")
- provided_concept_code: str | None = Field(None, description="The provided concept code")
- file_path: str | None = Field(None, description="The file path where the error occurred")
- explanation: str | None = Field(None, description="Additional explanation of the error")
-
-
-class PipeInputErrorData(StructuredContent):
- """Structured data for PipeInputError."""
-
- message: str = Field(description="The error message")
- pipe_code: str | None = Field(None, description="The pipe code")
- variable_name: str | None = Field(None, description="The variable name")
- concept_code: str | None = Field(None, description="The concept code")
-
-
-class DomainFailure(StructuredContent):
- """Details of a single domain failure during dry run."""
-
- domain_code: str = Field(description="The code of the domain that failed")
- error_message: str = Field(description="The error message for this domain")
-
-
-class ConceptFailure(StructuredContent):
- """Details of a single concept failure during dry run."""
-
- concept_spec: ConceptSpec = Field(description="The failing concept spec with concept code")
- error_message: str = Field(description="The error message for this concept")
-
-
-class PipeFailure(StructuredContent):
- """Details of a single pipe failure during dry run."""
-
- pipe_spec: (
- PipeFuncSpec
- | PipeImgGenSpec
- | PipeComposeSpec
- | PipeLLMSpec
- | PipeExtractSpec
- | PipeBatchSpec
- | PipeConditionSpec
- | PipeParallelSpec
- | PipeSequenceSpec
- ) = Field(description="The failing pipe spec with pipe code")
- error_message: str = Field(description="The error message for this pipe")
diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.py
index a85cc05c7..33cbd3106 100644
--- a/pipelex/cli/commands/build_cmd.py
+++ b/pipelex/cli/commands/build_cmd.py
@@ -8,28 +8,27 @@ import typer
from posthog import tag
from pipelex import pretty_print
-from pipelex.builder.builder import PipelexBundleSpec, load_and_validate_bundle
+from pipelex.builder.builder import PipelexBundleSpec
from pipelex.builder.builder_errors import PipeBuilderError
from pipelex.builder.builder_loop import BuilderLoop
-from pipelex.builder.exceptions import PipelexBundleError
from pipelex.builder.runner_code import generate_input_memory_json_string, generate_runner_code
from pipelex.cli.error_handlers import (
ErrorContext,
handle_model_availability_error,
handle_model_choice_error,
handle_model_deck_preset_error,
- handle_validation_error,
)
from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
from pipelex.config import get_config
-from pipelex.core.pipes.exceptions import PipeInputError, PipeOperatorModelChoiceError
+from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
+from pipelex.core.pipes.inputs.exceptions import PipeInputError
from pipelex.hub import get_console, get_report_delegate, get_required_pipe, get_telemetry_manager
from pipelex.language.plx_factory import PlxFactory
-from pipelex.libraries.exceptions import LibraryLoadingError
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipelex import PACKAGE_VERSION, Pipelex
from pipelex.pipeline.exceptions import PipelineExecutionError
from pipelex.pipeline.execute import execute_pipeline
+from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventProperty
from pipelex.tools.misc.file_utils import (
@@ -109,8 +108,6 @@ def build_pipe_cmd(
) -> None:
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_PIPE)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_PIPE)
@@ -128,7 +125,7 @@ def build_pipe_cmd(
# Build the pipeline
builder_loop = BuilderLoop()
try:
- pipelex_bundle_spec = await builder_loop.build_and_fix(pipe_code=builder_pipe, inputs={"brief": prompt})
+ pipelex_bundle_spec = await builder_loop.build_and_fix(builder_pipe=builder_pipe, inputs={"brief": prompt})
except PipeBuilderError as exc:
msg = f"Builder loop: Failed to execute pipeline: {exc}."
if exc.working_memory:
@@ -197,8 +194,6 @@ def build_pipe_cmd(
save_text_to_path(text=pretty_svg, path=svg_path)
typer.secho(f"✅ Pretty SVG saved to: {svg_path}", fg=typer.colors.GREEN)
- # Load the bundle from the file we just saved to register the pipe
- _ = await load_and_validate_bundle(plx_file_path)
pipe = get_required_pipe(pipe_code=main_pipe_code)
# Generate inputs.json
@@ -327,9 +322,9 @@ def prepare_runner_cmd(
async def prepare_runner(pipe_code: str | None = None, bundle_path: str | None = None):
if bundle_path:
try:
- bundle_blueprint = await load_and_validate_bundle(bundle_path)
+ validate_bundle_result = await validate_bundle(plx_file_path=bundle_path)
if not pipe_code:
- main_pipe_code = bundle_blueprint.main_pipe
+ main_pipe_code = validate_bundle_result.blueprints[0].main_pipe
if not main_pipe_code:
typer.secho(f"Bundle '{bundle_path}' does not declare a main_pipe", fg=typer.colors.RED, err=True)
raise typer.Exit(1)
@@ -340,7 +335,7 @@ def prepare_runner_cmd(
except FileNotFoundError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- except PipelexBundleError as exc:
+ except ValidateBundleError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
except PipeInputError as exc:
@@ -382,8 +377,6 @@ def prepare_runner_cmd(
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_RUNNER)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_RUNNER)
@@ -483,9 +476,9 @@ def generate_inputs_cmd(
async def generate_inputs(pipe_code: str | None = None, bundle_path: str | None = None):
if bundle_path:
try:
- bundle_blueprint = await load_and_validate_bundle(bundle_path)
+ validate_bundle_result = await validate_bundle(plx_file_path=bundle_path)
if not pipe_code:
- main_pipe_code = bundle_blueprint.main_pipe
+ main_pipe_code = validate_bundle_result.blueprints[0].main_pipe
if not main_pipe_code:
typer.secho(f"Bundle '{bundle_path}' does not declare a main_pipe", fg=typer.colors.RED, err=True)
raise typer.Exit(1)
@@ -496,7 +489,7 @@ def generate_inputs_cmd(
except FileNotFoundError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- except PipelexBundleError as exc:
+ except ValidateBundleError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
except PipeInputError as exc:
@@ -534,8 +527,6 @@ def generate_inputs_cmd(
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_INPUTS)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_INPUTS)
@@ -578,8 +569,6 @@ def build_one_shot_cmd(
) -> None:
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_ONE_SHOT)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_ONE_SHOT)
@@ -669,8 +658,6 @@ def build_partial_cmd(
) -> None:
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_PARTIAL)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_BUILD_PARTIAL)
diff --git a/pipelex/cli/commands/doctor_cmd.py b/pipelex/cli/commands/doctor_cmd.py
index 5acda2810..5d60042b7 100644
--- a/pipelex/cli/commands/doctor_cmd.py
+++ b/pipelex/cli/commands/doctor_cmd.py
@@ -21,11 +21,12 @@ from pipelex.cli.commands.init.ui.types import InitFocus
from pipelex.cogt.exceptions import InferenceBackendLibraryError
from pipelex.cogt.model_backends.backend_library import BackendCredentialsReport, InferenceBackendLibrary
from pipelex.cogt.models.model_manager import ModelManager
-from pipelex.config import PipelexConfig, get_config
+from pipelex.config import get_config
from pipelex.core.validation import report_validation_error
from pipelex.hub import PipelexHub, get_console, set_pipelex_hub
from pipelex.kit.paths import get_kit_configs_dir
from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.configuration.configs import PipelexConfig
from pipelex.system.environment import get_optional_env
from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAME, TelemetryConfig
from pipelex.tools.misc.dict_utils import extract_vars_from_strings_recursive
diff --git a/pipelex/cli/commands/init/backends.py b/pipelex/cli/commands/init/backends.py
index e4451ef35..b05744478 100644
--- a/pipelex/cli/commands/init/backends.py
+++ b/pipelex/cli/commands/init/backends.py
@@ -62,8 +62,12 @@ def get_selected_backend_keys(backends_toml_path: str) -> list[str]:
return selected_backends
-def customize_backends_config() -> None:
- """Interactively customize which inference backends are enabled in backends.toml."""
+def customize_backends_config(is_first_time_setup: bool = False) -> None:
+ """Interactively customize which inference backends are enabled in backends.toml.
+
+ Args:
+ is_first_time_setup: Whether this is the first time backends.toml is being set up.
+ """
console = get_console()
backends_toml_path = os.path.join(config_manager.pipelex_config_dir, "inference", "backends.toml")
template_backends_path = os.path.join(str(get_kit_configs_dir()), "inference", "backends.toml")
@@ -80,9 +84,9 @@ def customize_backends_config() -> None:
# Get currently enabled backends to show user their current selection
currently_enabled = get_currently_enabled_backends(backends_toml_path, backend_options)
- # If all backends are enabled, treat this as a fresh copy (not a meaningful selection)
- # In this case, don't use currently_enabled as the default to avoid selecting everything
- if currently_enabled and len(currently_enabled) == len(backend_options):
+ # If this is first-time setup, ignore what's in the template (all enabled)
+ # and use only pipelex_inference as the default
+ if is_first_time_setup or (currently_enabled and len(currently_enabled) == len(backend_options)):
currently_enabled = []
# Load the backends.toml file
@@ -90,8 +94,8 @@ def customize_backends_config() -> None:
console.print()
# UI: Display panel and get user selection
- console.print(build_backend_selection_panel(backend_options, currently_enabled))
- selected_indices = prompt_backend_indices(console, backend_options, currently_enabled)
+ console.print(build_backend_selection_panel(backend_options, currently_enabled, is_first_time_setup))
+ selected_indices = prompt_backend_indices(console, backend_options, currently_enabled, is_first_time_setup)
# Business logic: Update TOML
update_backends_in_toml(toml_doc, selected_indices, backend_options)
diff --git a/pipelex/cli/commands/init/command.py b/pipelex/cli/commands/init/command.py
index 8fb0addf5..cc8801f6c 100644
--- a/pipelex/cli/commands/init/command.py
+++ b/pipelex/cli/commands/init/command.py
@@ -158,6 +158,7 @@ def execute_initialization(
check_routing: bool,
backends_toml_path: str,
telemetry_config_path: str,
+ is_first_time_backends_setup: bool,
):
"""Execute the initialization steps.
@@ -172,8 +173,12 @@ def execute_initialization(
check_routing: Whether routing was in focus.
backends_toml_path: Path to backends.toml file.
telemetry_config_path: Path to telemetry config file.
+ is_first_time_backends_setup: Whether backends.toml didn't exist before this run.
"""
+ # Track if backends were just copied during config initialization
+ backends_just_copied_during_config = False
+
# Step 1: Initialize config if needed
if needs_config:
# Check if backends.toml exists before copying
@@ -184,15 +189,18 @@ def execute_initialization(
# If backends.toml was just created (freshly copied), always prompt for backend selection
backends_exists_now = path_exists(backends_toml_path)
- backends_just_copied = not backends_existed_before and backends_exists_now
+ backends_just_copied_during_config = not backends_existed_before and backends_exists_now
- if backends_just_copied or (check_inference and backends_exists_now):
+ if backends_just_copied_during_config or (check_inference and backends_exists_now):
needs_inference = True
+ # Determine if this is truly a first-time setup (either tracked from before or just copied now)
+ first_time_setup = is_first_time_backends_setup or backends_just_copied_during_config
+
# Step 2: Set up inference backends if needed
if needs_inference:
console.print()
- customize_backends_config()
+ customize_backends_config(is_first_time_setup=first_time_setup)
# Automatically set up routing after backends (unless routing is the specific focus)
if not check_routing:
@@ -253,6 +261,9 @@ def init_cmd(
check_routing = focus == InitFocus.ROUTING
check_telemetry = focus in (InitFocus.ALL, InitFocus.TELEMETRY)
+ # Track if backends.toml existed before we start
+ is_first_time_backends_setup = not path_exists(backends_toml_path)
+
# Check what needs to be initialized
needs_config, needs_inference, needs_routing, needs_telemetry = determine_needs(
reset=reset,
@@ -312,6 +323,7 @@ def init_cmd(
check_routing=check_routing,
backends_toml_path=backends_toml_path,
telemetry_config_path=telemetry_config_path,
+ is_first_time_backends_setup=is_first_time_backends_setup,
)
except typer.Exit:
diff --git a/pipelex/cli/commands/init/ui/backends_ui.py b/pipelex/cli/commands/init/ui/backends_ui.py
index c7fa48774..f75fbd989 100644
--- a/pipelex/cli/commands/init/ui/backends_ui.py
+++ b/pipelex/cli/commands/init/ui/backends_ui.py
@@ -92,12 +92,15 @@ def get_currently_enabled_backends(backends_toml_path: str, backend_options: lis
return sorted(currently_enabled)
-def build_backend_selection_panel(backend_options: list[tuple[str, str]], currently_enabled: list[int] | None = None) -> Panel:
+def build_backend_selection_panel(
+ backend_options: list[tuple[str, str]], currently_enabled: list[int] | None = None, is_first_time_setup: bool = False
+) -> Panel:
"""Create a Rich Panel for backend selection with options table.
Args:
backend_options: List of tuples (backend_key, display_name).
currently_enabled: Optional list of currently enabled backend indices (0-based).
+ is_first_time_setup: Whether this is the first time backends are being set up.
Returns:
A Panel containing the backend selection interface.
@@ -108,8 +111,8 @@ def build_backend_selection_panel(backend_options: list[tuple[str, str]], curren
table.add_column(style="dim", width=15)
for idx, (_, backend_name) in enumerate(backend_options, start=1):
- # Mark currently enabled backends
- status = "[green]✓ enabled[/green]" if currently_enabled and (idx - 1) in currently_enabled else ""
+ # Mark currently enabled backends (but not for first-time setup)
+ status = "[green]✓ enabled[/green]" if currently_enabled and (idx - 1) in currently_enabled and not is_first_time_setup else ""
table.add_row(f"[{idx}]", backend_name, status)
# Add special options at the end
@@ -117,7 +120,7 @@ def build_backend_selection_panel(backend_options: list[tuple[str, str]], curren
table.add_row("[Q]", "[dim]quit - Exit without configuring[/dim]", "")
# Update description based on whether we're showing current selection
- if currently_enabled:
+ if currently_enabled and not is_first_time_setup:
# Build current selection display with numbers and names
current_items: list[str] = []
for idx in sorted(currently_enabled):
@@ -147,13 +150,16 @@ def build_backend_selection_panel(backend_options: list[tuple[str, str]], curren
)
-def prompt_backend_indices(console: Console, backend_options: list[tuple[str, str]], currently_enabled: list[int] | None = None) -> list[int]:
+def prompt_backend_indices(
+ console: Console, backend_options: list[tuple[str, str]], currently_enabled: list[int] | None = None, is_first_time_setup: bool = False
+) -> list[int]:
"""Prompt user to select backend indices with validation.
Args:
console: Rich Console instance for user interaction.
backend_options: List of available backend options.
currently_enabled: Optional list of currently enabled backend indices (0-based).
+ is_first_time_setup: Whether this is the first time backends are being set up.
Returns:
List of validated backend indices (0-based) selected by the user.
@@ -161,11 +167,12 @@ def prompt_backend_indices(console: Console, backend_options: list[tuple[str, st
Raises:
typer.Exit: If user chooses to quit.
"""
- # Determine default based on current selection or fallback to first option
- if currently_enabled:
+ # Determine default based on current selection or fallback to first option (pipelex_inference)
+ if currently_enabled and not is_first_time_setup:
default_indices = sorted(currently_enabled)
default_str = ",".join(str(i + 1) for i in default_indices)
else:
+ # For first-time setup or no current selection, default to pipelex_inference (index 0)
default_indices = [0]
default_str = "1"
diff --git a/pipelex/cli/commands/kit_cmd.py b/pipelex/cli/commands/kit_cmd.py
index 173c8286c..ef80d7c9f 100644
--- a/pipelex/cli/commands/kit_cmd.py
+++ b/pipelex/cli/commands/kit_cmd.py
@@ -6,14 +6,51 @@ import typer
from typing_extensions import Annotated
from pipelex.cli.exceptions import PipelexCLIError
-from pipelex.kit.cursor_export import export_cursor_rules, remove_cursor_rules
+from pipelex.kit.cursor_rules import remove_cursor_rules, update_cursor_rules
from pipelex.kit.index_loader import load_index
+from pipelex.kit.index_models import KitIndex
from pipelex.kit.migrations_export import export_migration_instructions
-from pipelex.kit.targets_update import build_merged_rules, remove_from_targets, update_targets
+from pipelex.kit.single_file_agent_rules import remove_from_targets, update_single_file_agent_rules
kit_app = typer.Typer(no_args_is_help=True)
+def _sync_agent_rules(
+ repo_root: Path | None,
+ cursor: bool,
+ single_files: bool,
+ dry_run: bool,
+ diff: bool,
+ backup: str | None,
+ agent_set: str | None,
+ kit_index: KitIndex | None = None,
+) -> None:
+ resolved_repo_root = repo_root if repo_root is not None else Path()
+ loaded_kit_index = load_index() if kit_index is None else kit_index
+ agent_set = agent_set or loaded_kit_index.agent_rules.default_set
+
+ if cursor:
+ typer.echo("📤 Updating Cursor rules...")
+ update_cursor_rules(resolved_repo_root, loaded_kit_index, agent_set=agent_set, dry_run=dry_run)
+
+ if single_files:
+ typer.echo("📝 Updating target files...")
+ update_single_file_agent_rules(
+ repo_root=resolved_repo_root,
+ kit_index=loaded_kit_index,
+ agent_set=agent_set,
+ targets=loaded_kit_index.agent_rules.targets,
+ dry_run=dry_run,
+ diff=diff,
+ backup=backup,
+ )
+
+ if dry_run:
+ typer.echo("✅ Dry run completed - no changes made")
+ else:
+ typer.echo("✅ Kit sync completed successfully")
+
+
@kit_app.command("rules", help="Export Pipelex Cursor rules and merge Pipelex marked sections into other agent rules files")
def agent_rules(
repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,
@@ -22,27 +59,18 @@ def agent_rules(
dry_run: Annotated[bool, typer.Option("--dry-run", help="Show what would be done without making changes")] = False,
diff: Annotated[bool, typer.Option("--diff", help="Show unified diff of changes")] = False,
backup: Annotated[str | None, typer.Option("--backup", help="Backup suffix (e.g., '.bak')")] = None,
+ agent_set: Annotated[str | None, typer.Option("--set", help="Agent rule set to sync (use 'pipelex' for Pipelex repo)")] = None,
) -> None:
try:
- if repo_root is None:
- repo_root = Path()
-
- idx = load_index()
-
- if cursor:
- typer.echo("📤 Exporting Cursor rules...")
- export_cursor_rules(repo_root, idx, dry_run=dry_run)
-
- if single_files:
- typer.echo("📝 Building merged agent documentation...")
- merged_md = build_merged_rules(idx)
- typer.echo("📝 Updating target files...")
- update_targets(repo_root, merged_md, idx.agent_rules.targets, dry_run=dry_run, diff=diff, backup=backup)
-
- if dry_run:
- typer.echo("✅ Dry run completed - no changes made")
- else:
- typer.echo("✅ Kit sync completed successfully")
+ _sync_agent_rules(
+ repo_root=repo_root,
+ cursor=cursor,
+ single_files=single_files,
+ dry_run=dry_run,
+ diff=diff,
+ backup=backup,
+ agent_set=agent_set,
+ )
except Exception as exc:
msg = f"Failed to sync kit assets for agent rules: {exc}"
diff --git a/pipelex/cli/commands/run_cmd.py b/pipelex/cli/commands/run_cmd.py
index 828cc4d82..3789e4277 100644
--- a/pipelex/cli/commands/run_cmd.py
+++ b/pipelex/cli/commands/run_cmd.py
@@ -9,23 +9,21 @@ import typer
from posthog import tag
from pipelex import log
-from pipelex.builder.builder import load_and_validate_bundle
-from pipelex.builder.exceptions import PipelexBundleError
from pipelex.cli.error_handlers import (
ErrorContext,
handle_model_availability_error,
handle_model_choice_error,
handle_model_deck_preset_error,
- handle_validation_error,
)
from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
-from pipelex.core.pipes.exceptions import PipeInputError, PipeOperatorModelChoiceError
+from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
+from pipelex.core.pipes.inputs.exceptions import PipeInputError
from pipelex.hub import get_console, get_telemetry_manager
-from pipelex.libraries.exceptions import LibraryLoadingError
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipelex import Pipelex
from pipelex.pipeline.exceptions import PipelineExecutionError
from pipelex.pipeline.execute import execute_pipeline
+from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventProperty
from pipelex.tools.misc.file_utils import get_incremental_file_path
@@ -125,9 +123,9 @@ def run_cmd(
source_description: str
if bundle_path:
try:
- bundle_blueprint = await load_and_validate_bundle(bundle_path)
+ validate_bundle_result = await validate_bundle(plx_file_path=bundle_path)
if not pipe_code:
- main_pipe_code = bundle_blueprint.main_pipe
+ main_pipe_code = validate_bundle_result.blueprints[0].main_pipe
if not main_pipe_code:
typer.secho(f"Bundle '{bundle_path}' does not declare a main_pipe", fg=typer.colors.RED, err=True)
raise typer.Exit(1)
@@ -138,7 +136,7 @@ def run_cmd(
except FileNotFoundError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- except PipelexBundleError as exc:
+ except ValidateBundleError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
except PipeInputError as exc:
@@ -200,8 +198,6 @@ def run_cmd(
# Initialize Pipelex BEFORE telemetry context to ensure proper setup
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN)
diff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.py
index e63f67754..f8b8568e3 100644
--- a/pipelex/cli/commands/show_cmd.py
+++ b/pipelex/cli/commands/show_cmd.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
+from pathlib import Path
from typing import TYPE_CHECKING, Annotated, cast
import typer
@@ -13,17 +14,24 @@ from pipelex.base_exceptions import PipelexConfigError
from pipelex.cli.error_handlers import (
ErrorContext,
handle_model_deck_preset_error,
- handle_validation_error,
)
from pipelex.cli.exceptions import PipelexCLIError
from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
from pipelex.cogt.model_backends.model_lists import ModelLister
-from pipelex.config import ConfigPaths
-from pipelex.hub import get_console, get_models_manager, get_pipe_library, get_required_pipe, get_secrets_provider, get_telemetry_manager
-from pipelex.libraries.exceptions import LibraryLoadingError
+from pipelex.hub import (
+ get_console,
+ get_library_manager,
+ get_models_manager,
+ get_pipe_library,
+ get_required_pipe,
+ get_secrets_provider,
+ get_telemetry_manager,
+ set_current_library,
+)
from pipelex.pipelex import Pipelex
from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.configuration.configs import ConfigPaths
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventName, EventProperty
from pipelex.tools.misc.package_utils import get_package_version
@@ -50,12 +58,8 @@ def do_show_config() -> None:
def do_list_pipes() -> None:
"""List all available pipes."""
- try:
- nb_pipes = get_pipe_library().pretty_list_pipes()
- get_telemetry_manager().track_event(EventName.PIPES_LIST, properties={EventProperty.NB_PIPES: nb_pipes})
- except Exception as exc:
- msg = f"Failed to list pipes: {exc}"
- raise PipelexCLIError(msg) from exc
+ nb_pipes = get_pipe_library().pretty_list_pipes()
+ get_telemetry_manager().track_event(EventName.PIPES_LIST, properties={EventProperty.NB_PIPES: nb_pipes})
def do_show_pipe(pipe_code: str) -> None:
@@ -194,11 +198,14 @@ def list_pipes_cmd() -> None:
"""
try:
Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_PIPES)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_PIPES)
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path.cwd()])
+
with get_telemetry_manager().telemetry_context():
tag(name=EventProperty.INTEGRATION, value=IntegrationMode.CLI)
tag(name=EventProperty.PIPELEX_VERSION, value=get_package_version())
@@ -219,11 +226,14 @@ def show_pipe_cmd(
"""
try:
Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_PIPE)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_PIPE)
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path.cwd()])
+
with get_telemetry_manager().telemetry_context():
tag(name=EventProperty.INTEGRATION, value=IntegrationMode.CLI)
tag(name=EventProperty.PIPELEX_VERSION, value=get_package_version())
@@ -251,8 +261,6 @@ def show_models_cmd(
"""
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_MODELS)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_MODELS)
@@ -286,8 +294,6 @@ def show_backends_cmd(
"""
try:
Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_BACKENDS)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION_BEFORE_SHOW_BACKENDS)
diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.py
index fa54b461f..26b29ddaf 100644
--- a/pipelex/cli/commands/validate_cmd.py
+++ b/pipelex/cli/commands/validate_cmd.py
@@ -2,7 +2,7 @@ from __future__ import annotations
import asyncio
from pathlib import Path
-from typing import TYPE_CHECKING, Annotated
+from typing import Annotated
import click
import typer
@@ -10,59 +10,49 @@ from posthog import tag
from rich.traceback import Traceback
from pipelex import log
-from pipelex.builder.builder_validation import validate_dry_run_bundle_blueprint
-from pipelex.builder.exceptions import PipelexBundleError
from pipelex.cli.error_handlers import (
ErrorContext,
handle_model_availability_error,
handle_model_choice_error,
handle_model_deck_preset_error,
- handle_validation_error,
+ handle_validate_bundle_error,
)
from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
-from pipelex.core.interpreter import PipelexInterpreter
-from pipelex.core.pipes.exceptions import PipeInputError, PipeOperatorModelChoiceError
-from pipelex.hub import get_console, get_library_manager, get_pipes, get_required_pipe, get_telemetry_manager
-from pipelex.libraries.exceptions import LibraryLoadingError
+from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
+from pipelex.hub import get_console, get_library_manager, get_required_pipe, get_telemetry_manager, set_current_library
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipe_run.dry_run import dry_run_pipe, dry_run_pipes
from pipelex.pipelex import Pipelex
+from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventName, EventProperty
from pipelex.tools.misc.package_utils import get_package_version
-if TYPE_CHECKING:
- from pipelex.core.pipes.pipe_abstract import PipeAbstract
-
COMMAND = "validate"
def do_validate_all_libraries_and_dry_run() -> None:
- """Validate libraries and dry-run all pipes."""
- try:
- pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION)
- except ModelDeckPresetValidatonError as model_deck_error:
- handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION)
-
try:
with get_telemetry_manager().telemetry_context():
tag(name=EventProperty.INTEGRATION, value=IntegrationMode.CLI)
tag(name=EventProperty.PIPELEX_VERSION, value=get_package_version())
tag(name=EventProperty.CLI_COMMAND, value=f"{COMMAND} all")
- pipelex_instance.validate_libraries()
- pipes = get_pipes()
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path.cwd()])
+
+ pipes = library.get_pipe_library().get_pipes()
+
get_telemetry_manager().track_event(EventName.PIPE_DRY_RUN, properties={EventProperty.NB_PIPES: len(pipes)})
+
asyncio.run(dry_run_pipes(pipes=pipes, raise_on_failure=True))
log.info("Setup sequence passed OK, config and pipelines are validated.")
except PipeOperatorModelAvailabilityError as exc:
handle_model_availability_error(exc, context=ErrorContext.VALIDATION)
except PipeOperatorModelChoiceError as exc:
handle_model_choice_error(exc, context=ErrorContext.VALIDATION)
- finally:
- pipelex_instance.teardown()
def validate_cmd(
@@ -88,9 +78,12 @@ def validate_cmd(
pipelex validate --bundle my_bundle.plx --pipe my_pipe
pipelex validate all
"""
+ pipelex_instance: Pipelex
# Check for "all" keyword
if target == "all" and not pipe and not bundle:
+ pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
do_validate_all_libraries_and_dry_run()
+ pipelex_instance.teardown()
return
# Validate mutual exclusivity
@@ -139,51 +132,26 @@ def validate_cmd(
async def validate_pipe(pipe_code: str | None = None, bundle_path: str | None = None):
if bundle_path:
- absolute_bundle_path = str(Path(bundle_path).resolve())
- if absolute_bundle_path in get_library_manager().get_loaded_plx_paths():
- bundle_blueprint = PipelexInterpreter.load_bundle_blueprint(bundle_path=bundle_path)
- if not bundle_blueprint.pipe:
- typer.secho(f"Failed to validate bundle '{bundle_path}': no pipes found in bundle", fg=typer.colors.RED, err=True)
- raise typer.Exit(1)
- pipe_codes = list(bundle_blueprint.pipe.keys())
- pipes: list[PipeAbstract] = []
- for the_pipe_code in pipe_codes:
- pipes.append(get_required_pipe(pipe_code=the_pipe_code))
- await dry_run_pipes(pipes=pipes, raise_on_failure=True)
- typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}'", fg=typer.colors.GREEN)
- return
- # When validating a bundle, load_pipe_from_bundle validates ALL pipes in the bundle
try:
- bundle_blueprint = PipelexInterpreter.load_bundle_blueprint(bundle_path=bundle_path)
- get_telemetry_manager().track_event(
- EventName.BUNDLE_DRY_RUN,
- properties={EventProperty.NB_PIPES: bundle_blueprint.nb_pipes, EventProperty.NB_CONCEPTS: bundle_blueprint.nb_concepts},
- )
- await validate_dry_run_bundle_blueprint(bundle_blueprint=bundle_blueprint)
- if not pipe_code:
- typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}'", fg=typer.colors.GREEN)
- else:
- typer.secho(f"✅ Successfully validated all pipes in bundle '{bundle_path}' (including '{pipe_code}')", fg=typer.colors.GREEN)
+ await validate_bundle(plx_file_path=bundle_path)
+ typer.secho(f"✅ Successfully validated bundle '{bundle_path}'", fg=typer.colors.GREEN)
except FileNotFoundError as exc:
get_console().print(Traceback())
typer.secho(f"Failed to load bundle '{bundle_path}':", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- except PipelexBundleError as bundle_error:
- get_console().print(Traceback())
- handle_validation_error(exc=bundle_error, context=ErrorContext.VALIDATION)
- except PipeInputError as exc:
- get_console().print(Traceback())
- typer.secho(f"\n❌ Failed to validate bundle '{bundle_path}':", fg=typer.colors.RED, err=True)
- raise typer.Exit(1) from exc
+ except ValidateBundleError as bundle_error:
+ handle_validate_bundle_error(bundle_error, bundle_path=bundle_path)
elif pipe_code:
- # Validate a single pipe by code
typer.echo(f"Validating pipe '{pipe_code}'...")
- get_telemetry_manager().track_event(
- EventName.PIPE_DRY_RUN, properties={EventProperty.PIPE_TYPE: get_required_pipe(pipe_code=pipe_code).type}
- )
- pipelex_instance.validate_libraries()
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path.cwd()])
+
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ get_telemetry_manager().track_event(EventName.PIPE_DRY_RUN, properties={EventProperty.PIPE_TYPE: pipe.type})
await dry_run_pipe(
- get_required_pipe(pipe_code=pipe_code),
+ pipe,
raise_on_failure=True,
)
typer.secho(f"✅ Successfully validated pipe '{pipe_code}'", fg=typer.colors.GREEN)
@@ -191,12 +159,8 @@ def validate_cmd(
typer.secho("Failed to validate: no pipe code or bundle specified", fg=typer.colors.RED, err=True)
raise typer.Exit(1)
- # Initialize Pipelex
- pipelex_instance: Pipelex
try:
pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.CLI)
- except LibraryLoadingError as library_loading_error:
- handle_validation_error(exc=library_loading_error, context=ErrorContext.VALIDATION)
except ModelDeckPresetValidatonError as model_deck_error:
handle_model_deck_preset_error(model_deck_error, context=ErrorContext.VALIDATION)
diff --git a/pipelex/dev_cli/__init__.py b/pipelex/cli/dev_cli/__init__.py
similarity index 100%
rename from pipelex/dev_cli/__init__.py
rename to pipelex/cli/dev_cli/__init__.py
diff --git a/pipelex/dev_cli/_dev_cli.py b/pipelex/cli/dev_cli/_dev_cli.py
similarity index 66%
rename from pipelex/dev_cli/_dev_cli.py
rename to pipelex/cli/dev_cli/_dev_cli.py
index 659b9bbc5..a2c418a48 100644
--- a/pipelex/dev_cli/_dev_cli.py
+++ b/pipelex/cli/dev_cli/_dev_cli.py
@@ -1,5 +1,6 @@
"""Main entry point for the internal development CLI."""
+import sys
from typing import Annotated
import typer
@@ -7,7 +8,8 @@ from click import Command, Context
from typer.core import TyperGroup
from typing_extensions import override
-from pipelex.dev_cli.commands.check_config_sync_cmd import LeadingConfig, check_config_sync_cmd
+from pipelex.cli.dev_cli.commands.check_config_sync_cmd import LeadingConfig, check_config_sync_cmd
+from pipelex.cli.dev_cli.commands.check_rules_sync_cmd import check_rules_sync_cmd
from pipelex.hub import get_console
from pipelex.tools.misc.package_utils import get_package_version
@@ -18,7 +20,7 @@ class PipelexDevCLI(TyperGroup):
@override
def list_commands(self, ctx: Context) -> list[str]:
"""List commands in proper order."""
- return ["check-config-sync"]
+ return ["check-config-sync", "check-rules"]
@override
def get_command(self, ctx: Context, cmd_name: str) -> Command | None:
@@ -46,6 +48,10 @@ app = typer.Typer(
@app.callback(invoke_without_command=True)
def app_callback(_ctx: typer.Context) -> None:
+ # Skip banner if --quiet or -q flag is present
+ if "--quiet" in sys.argv or "-q" in sys.argv:
+ return
+
console = get_console()
package_version = get_package_version()
console.print(
@@ -65,6 +71,16 @@ def check_config_sync_command(
LeadingConfig,
typer.Option(help="Which configuration is the leading (left) one: 'installed' (.pipelex) or 'kit' (pipelex/kit/configs)"),
] = LeadingConfig.KIT,
+ quiet: Annotated[bool, typer.Option("--quiet", "-q", help="Output only a single validation line")] = False,
) -> None:
"""Verify that .pipelex and pipelex/kit/configs are in sync."""
- check_config_sync_cmd(show_diff=show_diff, leading=leading)
+ check_config_sync_cmd(show_diff=show_diff, leading=leading, quiet=quiet)
+
+
[email protected](name="check-rules", help="Verify that installed agent rules match kit templates")
+def check_rules_command(
+ show_diff: Annotated[bool, typer.Option("--show-diff/--no-diff", help="Show differences if found")] = True,
+ quiet: Annotated[bool, typer.Option("--quiet", "-q", help="Output only a single validation line")] = False,
+) -> None:
+ """Verify that installed agent rules match kit templates."""
+ check_rules_sync_cmd(show_diff=show_diff, quiet=quiet)
diff --git a/pipelex/dev_cli/commands/__init__.py b/pipelex/cli/dev_cli/commands/__init__.py
similarity index 100%
rename from pipelex/dev_cli/commands/__init__.py
rename to pipelex/cli/dev_cli/commands/__init__.py
diff --git a/pipelex/cli/dev_cli/commands/check_config_sync_cmd.py b/pipelex/cli/dev_cli/commands/check_config_sync_cmd.py
new file mode 100644
index 000000000..254911543
--- /dev/null
+++ b/pipelex/cli/dev_cli/commands/check_config_sync_cmd.py
@@ -0,0 +1,119 @@
+"""Command to verify that .pipelex and pipelex/kit/configs are in sync."""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+from rich.panel import Panel
+
+from pipelex.hub import get_console
+from pipelex.tools.misc.diff import has_diff_dirs, make_diff_dirs_pretty
+from pipelex.types import StrEnum
+
+
+class LeadingConfig(StrEnum):
+ """Enum for leading configuration options."""
+
+ KIT = "kit"
+ INSTALLED = "installed"
+
+
+def check_config_sync_cmd(
+ show_diff: bool = True,
+ leading: LeadingConfig = LeadingConfig.INSTALLED,
+ quiet: bool = False,
+) -> None:
+ """Verify that .pipelex and pipelex/kit/configs are in sync.
+
+ Args:
+ show_diff: If True, display the differences when found
+ leading: Which configuration is the leading (left) one in the diff.
+ LeadingConfig.INSTALLED means .pipelex is the reference (default),
+ LeadingConfig.KIT means pipelex/kit/configs is the reference
+ quiet: If True, output only a single validation line (for use in Make targets)
+ """
+ console = get_console()
+
+ # Define the directories to compare
+ pipelex_dir = Path(".pipelex")
+ configs_dir = Path("pipelex/kit/configs")
+
+ # Check if both directories exist
+ if not pipelex_dir.exists():
+ if quiet:
+ console.print("[red]✗ Config sync check: FAILED[/red] - .pipelex does not exist")
+ else:
+ console.print()
+ console.print("[red]✗[/red] Directory [cyan].pipelex[/cyan] does not exist")
+ console.print()
+ sys.exit(1)
+
+ if not configs_dir.exists():
+ if quiet:
+ console.print("[red]✗ Config sync check: FAILED[/red] - pipelex/kit/configs does not exist")
+ else:
+ console.print()
+ console.print("[red]✗[/red] Directory [cyan]pipelex/kit/configs[/cyan] does not exist")
+ console.print()
+ sys.exit(1)
+
+ # Determine directory order based on leading parameter
+ match leading:
+ case LeadingConfig.INSTALLED:
+ left_dir = pipelex_dir
+ right_dir = configs_dir
+ left_label = ".pipelex"
+ right_label = "pipelex/kit/configs"
+ case LeadingConfig.KIT:
+ left_dir = configs_dir
+ right_dir = pipelex_dir
+ left_label = "pipelex/kit/configs"
+ right_label = ".pipelex"
+
+ # Check for differences
+ has_diff = has_diff_dirs(left_dir, right_dir)
+
+ if not has_diff:
+ # No differences found
+ if quiet:
+ console.print("[green]✓ Config sync check: PASSED[/green]")
+ else:
+ console.print()
+ console.print("[bold]Checking config synchronization...[/bold]")
+ console.print(f" Leading: [cyan]{left_label}[/cyan] ↔ [cyan]{right_label}[/cyan]")
+ console.print()
+ success_panel = Panel(
+ "[green]✓[/green] Directories are in sync!\n\n[dim].pipelex and pipelex/kit/configs have no differences.[/dim]",
+ title="[bold green]Config Sync Check: PASSED[/bold green]",
+ border_style="green",
+ padding=(1, 2),
+ )
+ console.print(success_panel)
+ console.print()
+ else:
+ console.print()
+ console.print("[bold]Checking config synchronization...[/bold]")
+ console.print(f" Leading: [cyan]{left_label}[/cyan] ↔ [cyan]{right_label}[/cyan]")
+ console.print()
+ error_panel = Panel(
+ "[red]✗[/red] Directories are [bold]NOT[/bold] in sync!\n\n[dim].pipelex and pipelex/kit/configs have differences.[/dim]",
+ title="[bold red]Config Sync Check: FAILED[/bold red]",
+ border_style="red",
+ padding=(1, 2),
+ )
+ console.print(error_panel)
+ console.print()
+
+ if show_diff:
+ console.print()
+ pretty_diff = make_diff_dirs_pretty(left_dir, right_dir)
+ console.print(pretty_diff)
+ console.print()
+
+ console.print("[bold yellow]Recommended Actions:[/bold yellow]")
+ console.print(" • If [cyan].pipelex[/cyan] is correct, run: [cyan]make up-kit-configs[/cyan] or simply [cyan]make ukc[/cyan]")
+ console.print(" • If [cyan]pipelex/kit/configs[/cyan] is correct, copy it to [cyan].pipelex[/cyan]")
+ console.print()
+
+ sys.exit(1)
diff --git a/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py b/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py
new file mode 100644
index 000000000..0d068a81b
--- /dev/null
+++ b/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py
@@ -0,0 +1,127 @@
+"""Command to verify installed agent rules match kit templates."""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+from rich.panel import Panel
+
+from pipelex.hub import get_console
+from pipelex.kit.index_loader import load_index
+from pipelex.kit.markers import find_span, replace_span, wrap
+from pipelex.kit.single_file_agent_rules import build_merged_rules, insert_block_with_markers, unified_diff
+
+
+def check_rules_sync_cmd(show_diff: bool = True, quiet: bool = False) -> None:
+ """Verify that installed agent rules match kit templates.
+
+ Args:
+ show_diff: If True, display the differences when found
+ quiet: If True, output only a single validation line (for use in Make targets)
+ """
+ console = get_console()
+ kit_index = load_index()
+ agent_set = "coding_standards"
+
+ missing_targets: list[Path] = []
+ mismatches: list[tuple[Path, str, str]] = []
+
+ for target in kit_index.agent_rules.targets.values():
+ # Check if this target has its own set override for the agent_set
+ target_file_list: list[str] | None = None
+ if target.sets and agent_set in target.sets:
+ target_file_list = target.sets[agent_set]
+
+ # Build merged rules for this specific target (with override if applicable)
+ merged_rules = build_merged_rules(kit_index, agent_set=agent_set, file_list=target_file_list)
+
+ target_path = Path(target.path)
+
+ if not target_path.exists():
+ missing_targets.append(target_path)
+ continue
+
+ current_content = target_path.read_text(encoding="utf-8")
+ span = find_span(current_content, target.marker_begin, target.marker_end)
+
+ if span:
+ before_markers = current_content[: span[0]]
+ after_markers = current_content[span[1] :]
+ outside_content = before_markers + after_markers
+
+ has_h1_outside = bool(re.search(r"^#\s+.+$", outside_content, flags=re.MULTILINE))
+
+ if target.heading_1 and not has_h1_outside:
+ content_with_heading = f"{target.heading_1}\n\n{merged_rules}"
+ wrapped_block = wrap(target.marker_begin, target.marker_end, content_with_heading)
+ else:
+ wrapped_block = wrap(target.marker_begin, target.marker_end, merged_rules)
+
+ expected_content = replace_span(current_content, span, wrapped_block)
+ else:
+ expected_content = insert_block_with_markers(
+ current_content,
+ merged_rules,
+ target.heading_1,
+ (target.marker_begin, target.marker_end),
+ )
+
+ if current_content != expected_content:
+ mismatches.append((target_path, current_content, expected_content))
+
+ if not missing_targets and not mismatches:
+ if quiet:
+ console.print("[green]✓ Agent rules sync check: PASSED[/green]")
+ else:
+ console.print()
+ console.print("[bold]Checking agent rules synchronization...[/bold]")
+ console.print(" Set: [cyan]coding_standards[/cyan]")
+ console.print()
+ success_panel = Panel(
+ "[green]✓[/green] Agent rules are in sync!\n\n[dim]Installed targets match templates from pipelex/kit/agent_rules.[/dim]",
+ title="[bold green]Agent Rules Sync Check: PASSED[/bold green]",
+ border_style="green",
+ padding=(1, 2),
+ )
+ console.print(success_panel)
+ console.print()
+ else:
+ console.print()
+ console.print("[bold]Checking agent rules synchronization...[/bold]")
+ console.print(" Set: [cyan]coding_standards[/cyan]")
+ console.print()
+ error_panel = Panel(
+ (
+ "[red]✗[/red] Agent rules are [bold]NOT[/bold] in sync!\n\n"
+ "[dim]Installed targets differ from templates in pipelex/kit/agent_rules.[/dim]"
+ ),
+ title="[bold red]Agent Rules Sync Check: FAILED[/bold red]",
+ border_style="red",
+ padding=(1, 2),
+ )
+ console.print(error_panel)
+ console.print()
+
+ if missing_targets:
+ console.print("[bold yellow]Missing targets:[/bold yellow]")
+ for missing_path in missing_targets:
+ console.print(f" • [cyan]{missing_path}[/cyan]")
+ console.print()
+
+ if mismatches:
+ console.print("[bold yellow]Mismatched targets:[/bold yellow]")
+ for mismatched_path, current_content, expected_content in mismatches:
+ console.print(f" • [cyan]{mismatched_path}[/cyan]")
+ if show_diff:
+ diff_output = unified_diff(current_content, expected_content, str(mismatched_path))
+ if diff_output:
+ console.print()
+ console.print(diff_output)
+ console.print()
+
+ console.print("[bold yellow]Recommended Actions:[/bold yellow]")
+ console.print(" • Run [cyan]make rules[/cyan] to sync agent rules")
+ console.print()
+ sys.exit(1)
diff --git a/pipelex/cli/error_handlers.py b/pipelex/cli/error_handlers.py
index 2e54ba2c1..140f540cb 100644
--- a/pipelex/cli/error_handlers.py
+++ b/pipelex/cli/error_handlers.py
@@ -1,16 +1,17 @@
-from typing import NoReturn
+from typing import TYPE_CHECKING, NoReturn
import typer
-from rich.syntax import Syntax
from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
-from pipelex.core.concepts.exceptions import PipelexValidationExceptionAbstract
from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
from pipelex.hub import get_console
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.types import StrEnum
from pipelex.urls import URLs
+if TYPE_CHECKING:
+ from pipelex.pipeline.validate_bundle import ValidateBundleError
+
class ErrorContext(StrEnum):
"""Context for error messages in CLI commands."""
@@ -114,61 +115,58 @@ def handle_model_deck_preset_error(exc: ModelDeckPresetValidatonError, context:
raise typer.Exit(1) from exc
-def handle_validation_error(exc: PipelexValidationExceptionAbstract, context: ErrorContext) -> NoReturn:
- """Handle and display validation errors with formatted output.
+def handle_validate_bundle_error(exc: "ValidateBundleError", bundle_path: str | None = None) -> NoReturn:
+ """Handle and display ValidateBundleError with formatted output.
Args:
- exc: The validation error exception (implements ValidationErrorDetailsProtocol)
- context: Context for the error message
+ exc: The bundle validation error exception
+ bundle_path: Optional path to the bundle file being validated
"""
console = get_console()
- console.print(f"\n[bold red]❌ {context} failed due to validation error[/bold red]\n")
- console.print(f"[bold red]Error:[/bold red] {exc}\n")
-
- # Display concept definition errors with syntax highlighting
- concept_definition_errors = exc.get_concept_definition_errors()
- if concept_definition_errors:
- console.print("[bold cyan]Concept Definition Errors:[/bold cyan]\n")
- for concept_definition_error in concept_definition_errors:
- syntax_error_data = concept_definition_error.structure_class_syntax_error_data
- if not syntax_error_data:
- continue
-
- message = concept_definition_error.message
- code = concept_definition_error.structure_class_python_code or ""
- highlight_lines: set[int] | None = None
- if syntax_error_data.lineno:
- highlight_lines = {syntax_error_data.lineno}
-
- syntax = Syntax(
- code=code,
- lexer="python",
- line_numbers=True,
- word_wrap=False,
- theme="ansi_dark",
- line_range=None,
- highlight_lines=highlight_lines,
- )
-
- # Build pretty error location
- pretty_range = ""
- if syntax_error_data.lineno and syntax_error_data.end_lineno:
- pretty_range = f"lines {syntax_error_data.lineno} to {syntax_error_data.end_lineno}"
- elif syntax_error_data.lineno:
- pretty_range = f"line {syntax_error_data.lineno}"
- if syntax_error_data.offset and syntax_error_data.end_offset:
- pretty_range += f", column {syntax_error_data.offset} to {syntax_error_data.end_offset}"
- elif syntax_error_data.offset:
- pretty_range += f", column {syntax_error_data.offset}"
-
- console.print(f"[yellow]{message}[/yellow]")
- if pretty_range:
- console.print(f"[dim]Generated code error at {pretty_range}[/dim]")
- console.print(syntax)
+ console.print("\n[bold red]❌ Bundle validation failed[/bold red]\n")
+
+ if bundle_path:
+ console.print(f"[bold cyan]Bundle:[/bold cyan] [yellow]{bundle_path}[/yellow]\n")
+ # Display pipe validation errors
+ if exc.pipe_validation_error_data:
+ console.print("[bold cyan]Pipe Validation Errors:[/bold cyan]\n")
+ for pipe_index, pipe_error in enumerate(exc.pipe_validation_error_data, 1):
+ console.print(f"[bold yellow]{pipe_index}. {pipe_error.error_type.replace('_', ' ').title()}[/bold yellow]")
+
+ # Display key identification info
+ if pipe_error.pipe_code:
+ console.print(f" [cyan]Pipe:[/cyan] [yellow]{pipe_error.pipe_code}[/yellow]")
+ if pipe_error.concept_code:
+ console.print(f" [cyan]Concept:[/cyan] [yellow]{pipe_error.concept_code}[/yellow]")
+ if pipe_error.domain:
+ console.print(f" [cyan]Domain:[/cyan] [green]{pipe_error.domain}[/green]")
+
+ # Field name if present
+ if pipe_error.field_name:
+ console.print(f" [cyan]Field:[/cyan] [yellow]{pipe_error.field_name}[/yellow]")
+
+ # Variables
+ if pipe_error.variable_names:
+ variables_str = ", ".join([f"[yellow]{v}[/yellow]" for v in pipe_error.variable_names])
+ console.print(f" [cyan]Variables:[/cyan] {variables_str}")
+
+ # Error message
+ console.print(f" [cyan]→[/cyan] {pipe_error.message}")
+
+ # Field path as secondary info
+ if pipe_error.field_path:
+ console.print(f" [dim]└─ Path: {pipe_error.field_path}[/dim]")
+
console.print()
+ # Display dry run error message
+ if exc.dry_run_error_message:
+ console.print("[bold cyan]Dry Run Error:[/bold cyan]\n")
+ console.print(f"[yellow]{exc.dry_run_error_message}[/yellow]\n")
+
+ # Display helpful tips
console.print(
- "[bold green]💡 Tip:[/bold green] Review the error message above and check your pipeline configuration. "
+ "[bold green]💡 Tip:[/bold green] Review the error messages above and check your pipeline configuration. "
"Make sure all required fields are present and correctly formatted."
)
console.print(f"[dim]Learn more: {URLs.documentation}[/dim]")
diff --git a/pipelex/cli/exceptions.py b/pipelex/cli/exceptions.py
index a64c81904..1d51c3a53 100644
--- a/pipelex/cli/exceptions.py
+++ b/pipelex/cli/exceptions.py
@@ -1,9 +1,9 @@
from click import ClickException
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class PipelexCLIError(PipelexException, ClickException):
+class PipelexCLIError(PipelexError, ClickException):
"""Raised when there's an error in CLI usage or operation."""
diff --git a/pipelex/client/exceptions.py b/pipelex/client/exceptions.py
index 36c9116a1..20128d15d 100644
--- a/pipelex/client/exceptions.py
+++ b/pipelex/client/exceptions.py
@@ -1,5 +1,5 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class ClientAuthenticationError(PipelexException):
+class ClientAuthenticationError(PipelexError):
pass
diff --git a/pipelex/client/protocol.py b/pipelex/client/protocol.py
index a38dae88d..7ca29395f 100644
--- a/pipelex/client/protocol.py
+++ b/pipelex/client/protocol.py
@@ -5,7 +5,7 @@ from pydantic import BaseModel, model_validator
from pydantic.functional_validators import SkipValidation
from typing_extensions import Annotated, runtime_checkable
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_output import DictPipeOutput
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
@@ -65,7 +65,7 @@ class ApiResponse(BaseModel):
error: str | None = None
-class PipelineRequestError(PipelexException):
+class PipelineRequestError(PipelexError):
pass
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index bff7e13ec..67b8330c3 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -1,12 +1,12 @@
+from pipelex.base_exceptions import PipelexError
from pipelex.cogt.extract.extract_setting import ExtractModelChoice
from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice
from pipelex.cogt.llm.llm_setting import LLMModelChoice
from pipelex.cogt.model_backends.model_type import ModelType
-from pipelex.system.exceptions import RootException
from pipelex.types import StrEnum
-class CogtError(RootException):
+class CogtError(PipelexError):
pass
diff --git a/pipelex/cogt/img_gen/img_gen_setting.py b/pipelex/cogt/img_gen/img_gen_setting.py
index 124f31ad0..898b728b6 100644
--- a/pipelex/cogt/img_gen/img_gen_setting.py
+++ b/pipelex/cogt/img_gen/img_gen_setting.py
@@ -7,6 +7,10 @@ from pipelex.system.configuration.config_model import ConfigModel
from pipelex.types import Self
+class ImgGenSettingValueError(ValueError):
+ pass
+
+
class ImgGenSetting(ConfigModel):
model: str
quality: Quality | None = Field(default=None, strict=False)
@@ -18,8 +22,11 @@ class ImgGenSetting(ConfigModel):
@model_validator(mode="after")
def validate_quality_or_nb_steps(self) -> Self:
if self.quality is not None and self.nb_steps is not None:
- msg = "ImgGenSetting cannot have both 'quality' and 'nb_steps' specified. Use one or the other."
- raise ValueError(msg)
+ msg = (
+ "ImgGenSetting cannot have both 'quality' and 'nb_steps' specified. Use one or the other."
+ f"Quality: {self.quality}, nb_steps: {self.nb_steps}"
+ )
+ raise ImgGenSettingValueError(msg)
return self
def desc(self) -> str:
diff --git a/pipelex/cogt/img_gen/img_gen_worker_factory.py b/pipelex/cogt/img_gen/img_gen_worker_factory.py
index fd09a846c..7336e8fe7 100644
--- a/pipelex/cogt/img_gen/img_gen_worker_factory.py
+++ b/pipelex/cogt/img_gen/img_gen_worker_factory.py
@@ -88,7 +88,7 @@ class ImgGenWorkerFactory:
inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
- case "azure_openai":
+ case "azure_rest":
from pipelex.plugins.azure_rest.azure_img_gen_worker import AzureImgGenWorker # noqa: PLC0415
img_gen_worker = AzureImgGenWorker(
diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.py
index 1ddb954ad..3abc0cf95 100644
--- a/pipelex/cogt/models/model_manager.py
+++ b/pipelex/cogt/models/model_manager.py
@@ -10,7 +10,8 @@ from pipelex.cogt.model_routing.routing_profile_loader import load_active_routin
from pipelex.cogt.models.model_deck import ModelDeck, ModelDeckBlueprint
from pipelex.cogt.models.model_deck_loader import load_model_deck_blueprint
from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
-from pipelex.config import ConfigPaths, get_config
+from pipelex.config import get_config
+from pipelex.system.configuration.configs import ConfigPaths
from pipelex.tools.misc.file_utils import find_files_in_dir
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
@@ -79,7 +80,7 @@ class ModelManager(ModelManagerAbstract):
all_models_and_possible_backends = self.inference_backend_library.get_all_models_and_possible_backends()
inference_models: dict[str, InferenceModelSpec] = {}
- for model_name, available_backends in all_models_and_possible_backends.items():
+ for model_name in all_models_and_possible_backends:
backend_match_for_model = self.routing_profile.get_backend_match_for_model(
enabled_backends=enabled_backends,
model_name=model_name,
@@ -107,15 +108,15 @@ class ModelManager(ModelManagerAbstract):
case BackendMatchingMethod.DEFAULT:
# We could not find the model spec, but it was a default match,
# so we can look for it in the other available backends
- # Use fallback_order if specified, then try remaining enabled backends
+ # Use fallback_order if specified (fallback is opt-in)
if backend_match_for_model.fallback_order:
# Try fallback_order first, then any enabled backends not in fallback_order
backends_to_try = backend_match_for_model.fallback_order + [
b for b in enabled_backends if b not in backend_match_for_model.fallback_order
]
else:
- # No fallback_order specified, use all available_backends
- backends_to_try = available_backends
+ # No fallback_order specified, skip this model (fallback is opt-in)
+ continue
for available_backend in backends_to_try:
if available_backend == matched_backend_name:
diff --git a/pipelex/cogt/templating/template_blueprint.py b/pipelex/cogt/templating/template_blueprint.py
index 4afc27280..069ad5a5a 100644
--- a/pipelex/cogt/templating/template_blueprint.py
+++ b/pipelex/cogt/templating/template_blueprint.py
@@ -4,6 +4,7 @@ from pydantic import BaseModel, Field, model_validator
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
+from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError
from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing
@@ -17,5 +18,9 @@ class TemplateBlueprint(BaseModel):
@model_validator(mode="after")
def validate_template(self) -> "TemplateBlueprint":
- check_jinja2_parsing(template_source=self.template, template_category=self.category)
+ try:
+ check_jinja2_parsing(template_source=self.template, template_category=self.category)
+ except Jinja2TemplateSyntaxError as exc:
+ msg = f"Could not parse template for TemplateBlueprint: {exc}"
+ raise ValueError(msg) from exc
return self
diff --git a/pipelex/config.py b/pipelex/config.py
index 5e839adc3..a933bb033 100644
--- a/pipelex/config.py
+++ b/pipelex/config.py
@@ -1,174 +1,5 @@
-from typing import cast
-
-import shortuuid
-from pydantic import Field, field_validator
-
-from pipelex.base_exceptions import PipelexConfigError
-from pipelex.cogt.config_cogt import Cogt
-from pipelex.cogt.model_backends.prompting_target import PromptingTarget
-from pipelex.cogt.templating.templating_style import TemplatingStyle
-from pipelex.core.pipes.exceptions import StaticValidationErrorType
from pipelex.hub import get_required_config
-from pipelex.language.plx_config import PlxConfig
-from pipelex.pipeline.track.tracker_config import TrackerConfig
-from pipelex.system.configuration.config_model import ConfigModel
-from pipelex.system.configuration.config_root import ConfigRoot
-from pipelex.tools.aws.aws_config import AwsConfig
-from pipelex.tools.log.log_config import LogConfig
-from pipelex.types import StrEnum
-
-
-class ConfigPaths:
- DEFAULT_CONFIG_DIR_PATH = "./.pipelex"
- INFERENCE_DIR_NAME = "inference"
- INFERENCE_DIR_PATH = f"{DEFAULT_CONFIG_DIR_PATH}/{INFERENCE_DIR_NAME}"
- BACKENDS_FILE_NAME = "backends.toml"
- BACKENDS_FILE_PATH = f"{INFERENCE_DIR_PATH}/{BACKENDS_FILE_NAME}"
- BACKENDS_DIR_NAME = "backends"
- BACKENDS_DIR_PATH = f"{INFERENCE_DIR_PATH}/{BACKENDS_DIR_NAME}"
- ROUTING_PROFILES_FILE_NAME = "routing_profiles.toml"
- ROUTING_PROFILES_FILE_PATH = f"{INFERENCE_DIR_PATH}/{ROUTING_PROFILES_FILE_NAME}"
- MODEL_DECKS_DIR_NAME = "deck"
- MODEL_DECKS_DIR_PATH = f"{INFERENCE_DIR_PATH}/{MODEL_DECKS_DIR_NAME}"
- BASE_DECK_FILE_NAME = "base_deck.toml"
- BASE_DECK_FILE_PATH = f"{MODEL_DECKS_DIR_PATH}/{BASE_DECK_FILE_NAME}"
- OVERRIDES_DECK_FILE_NAME = "overrides.toml"
- OVERRIDES_DECK_FILE_PATH = f"{MODEL_DECKS_DIR_PATH}/{OVERRIDES_DECK_FILE_NAME}"
-
-
-class StaticValidationReaction(StrEnum):
- RAISE = "raise"
- LOG = "log"
- IGNORE = "ignore"
-
-
-class StaticValidationConfig(ConfigModel):
- default_reaction: StaticValidationReaction = Field(strict=False)
- reactions: dict[StaticValidationErrorType, StaticValidationReaction]
-
- @field_validator("reactions", mode="before")
- @classmethod
- def validate_reactions(cls, value: dict[str, str]) -> dict[StaticValidationErrorType, StaticValidationReaction]:
- return cast(
- "dict[StaticValidationErrorType, StaticValidationReaction]",
- ConfigModel.transform_dict_str_to_enum(
- input_dict=value,
- key_enum_cls=StaticValidationErrorType,
- value_enum_cls=StaticValidationReaction,
- ),
- )
-
-
-class PipeRunConfig(ConfigModel):
- pipe_stack_limit: int
-
-
-class DryRunConfig(ConfigModel):
- apply_to_jinja2_rendering: bool
- text_gen_truncate_length: int
- nb_list_items: int
- nb_extract_pages: int
- image_urls: list[str]
- allowed_to_fail_pipes: list[str] = Field(default_factory=list)
-
- @field_validator("image_urls", mode="before")
- @classmethod
- def validate_image_urls(cls, value: list[str]) -> list[str]:
- if not value:
- msg = "dry_run_config.image_urls must be a non-empty list"
- raise PipelexConfigError(msg)
- return value
-
-
-class StructureConfig(ConfigModel):
- is_default_text_then_structure: bool
-
-
-class PromptingConfig(ConfigModel):
- default_prompting_style: TemplatingStyle
- prompting_styles: dict[str, TemplatingStyle]
-
- def get_prompting_style(self, prompting_target: PromptingTarget | None = None) -> TemplatingStyle | None:
- if prompting_target:
- return self.prompting_styles.get(prompting_target, self.default_prompting_style)
- return None
-
-
-class FeatureConfig(ConfigModel):
- is_pipeline_tracking_enabled: bool
- is_reporting_enabled: bool
-
-
-class ReportingConfig(ConfigModel):
- is_log_costs_to_console: bool
- is_generate_cost_report_file_enabled: bool
- cost_report_dir_path: str
- cost_report_base_name: str
- cost_report_extension: str
- cost_report_unit_scale: float
-
-
-class ObserverConfig(ConfigModel):
- observer_dir: str
-
-
-class ScanConfig(ConfigModel):
- excluded_dirs: frozenset[str]
-
- @field_validator("excluded_dirs", mode="before")
- @classmethod
- def validate_excluded_dirs(cls, value: list[str] | frozenset[str]) -> frozenset[str]:
- if isinstance(value, frozenset):
- return value
- return frozenset(value)
-
-
-class BuilderConfig(ConfigModel):
- default_output_dir: str
- default_bundle_file_name: str
- default_directory_base_name: str
-
-
-class Pipelex(ConfigModel):
- feature_config: FeatureConfig
- log_config: LogConfig
- aws_config: AwsConfig
-
- static_validation_config: StaticValidationConfig
- tracker_config: TrackerConfig
- structure_config: StructureConfig
- prompting_config: PromptingConfig
- plx_config: PlxConfig
-
- dry_run_config: DryRunConfig
- pipe_run_config: PipeRunConfig
- reporting_config: ReportingConfig
- observer_config: ObserverConfig
- scan_config: ScanConfig
- builder_config: BuilderConfig
-
-
-class MigrationConfig(ConfigModel):
- migration_maps: dict[str, dict[str, str]]
-
- def text_in_renaming_keys(self, category: str, text: str) -> list[tuple[str, str]]:
- renaming_map = self.migration_maps.get(category)
- if not renaming_map:
- return []
- return [(key, value) for key, value in renaming_map.items() if text in key]
-
- def text_in_renaming_values(self, category: str, text: str) -> list[tuple[str, str]]:
- renaming_map = self.migration_maps.get(category)
- if not renaming_map:
- return []
- return [(key, value) for key, value in renaming_map.items() if text in value]
-
-
-class PipelexConfig(ConfigRoot):
- session_id: str = shortuuid.uuid()
- cogt: Cogt
- pipelex: Pipelex
- migration: MigrationConfig
+from pipelex.system.configuration.configs import PipelexConfig
def get_config() -> PipelexConfig:
diff --git a/pipelex/core/bundles/exceptions.py b/pipelex/core/bundles/exceptions.py
index 706687e36..b3b9f57e0 100644
--- a/pipelex/core/bundles/exceptions.py
+++ b/pipelex/core/bundles/exceptions.py
@@ -1,2 +1,19 @@
-class PipelexBundleBlueprintValueError(ValueError):
- pass
+from pydantic import BaseModel
+
+from pipelex.types import StrEnum
+
+
+class PipelexBundleBlueprintFixableErrorType(StrEnum):
+ """Types of fixable validation errors in Pipelex bundle blueprints.
+
+ These error types represent validation issues that are actually fixed
+ in the builder loop auto-fix system.
+ """
+
+
+class PipelexBundleBlueprintValidationErrorData(BaseModel):
+ """Structured validation error data for bundle blueprint validation errors.
+
+ This model captures information about validation errors that are actually fixed
+ in the builder loop auto-fix system.
+ """
diff --git a/pipelex/core/bundles/pipe_sorter.py b/pipelex/core/bundles/pipe_sorter.py
index 6691be067..faf8532eb 100644
--- a/pipelex/core/bundles/pipe_sorter.py
+++ b/pipelex/core/bundles/pipe_sorter.py
@@ -1,7 +1,7 @@
"""Topological sorting utilities for pipe dependencies."""
from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
def sort_pipes_by_dependencies(
@@ -21,7 +21,7 @@ def sort_pipes_by_dependencies(
For PipeSequence, dependencies follow step order. For others, alphabetical order.
Raises:
- PipeDefinitionError: If circular dependencies are detected among pipes
+ PipeValidationError: If circular dependencies are detected among pipes
Example:
>>> pipes = {
@@ -50,7 +50,7 @@ def sort_pipes_by_dependencies(
return
if pipe_code in visiting:
msg = f"Circular dependency detected involving pipe: {pipe_code}"
- raise PipeDefinitionError(message=msg)
+ raise PipeValidationError(message=msg, pipe_code=pipe_code, error_type=PipeValidationErrorType.CIRCULAR_DEPENDENCY_ERROR)
if pipe_code not in pipes:
# Dependency not in this bundle, skip it
return
diff --git a/pipelex/core/bundles/pipelex_bundle_blueprint.py b/pipelex/core/bundles/pipelex_bundle_blueprint.py
index 3b0e494d4..5e046061b 100644
--- a/pipelex/core/bundles/pipelex_bundle_blueprint.py
+++ b/pipelex/core/bundles/pipelex_bundle_blueprint.py
@@ -2,7 +2,6 @@ from typing import Annotated
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
-from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValueError
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.domains.exceptions import DomainCodeError
from pipelex.core.domains.validation import validate_domain_code
@@ -46,18 +45,19 @@ class PipelexBundleBlueprint(BaseModel):
@field_validator("domain", mode="before")
@classmethod
def validate_domain_syntax(cls, domain: str) -> str:
+ # Then validate the domain code format
try:
validate_domain_code(code=domain)
except DomainCodeError as exc:
msg = f"Error when trying to validate the pipelex bundle at domain '{domain}': {exc}"
- raise PipelexBundleBlueprintValueError(msg) from exc
+ raise ValueError(msg) from exc
return domain
@model_validator(mode="after")
def validate_main_pipe(self) -> "PipelexBundleBlueprint":
if self.main_pipe and (not self.pipe or (self.main_pipe not in self.pipe)):
msg = f"Main pipe '{self.main_pipe}' could not be found in pipelex bundle at source '{self.source}' and domain '{self.domain}'"
- raise PipelexBundleBlueprintValueError(msg)
+ raise ValueError(msg)
return self
@property
diff --git a/pipelex/core/concepts/concept.py b/pipelex/core/concepts/concept.py
index bc8af884a..67f26a4bb 100644
--- a/pipelex/core/concepts/concept.py
+++ b/pipelex/core/concepts/concept.py
@@ -6,9 +6,11 @@ from pydantic import BaseModel, ConfigDict, field_validator
from pipelex import log
from pipelex.base_exceptions import PipelexUnexpectedError
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.exceptions import ConceptCodeError, ConceptStringError, ConceptValueError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.validation import validate_concept_code, validate_concept_string
from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.domains.exceptions import DomainCodeError
from pipelex.core.domains.validation import validate_domain_code
from pipelex.core.stuffs.image_field_search import search_for_nested_image_fields
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -26,25 +28,24 @@ class Concept(BaseModel):
structure_class_name: str
refines: str | None = None
- @property
- def concept_string(self) -> str:
- return f"{self.domain}.{self.code}"
-
- @classmethod
- def is_implicit_concept(cls, concept_string: str) -> bool:
- validate_concept_string(concept_string=concept_string)
- return concept_string.startswith(SpecialDomain.IMPLICIT)
-
@field_validator("code")
@classmethod
def validate_code(cls, code: str) -> str:
- validate_concept_code(concept_code=code)
+ try:
+ validate_concept_code(concept_code=code)
+ except ConceptCodeError as exc:
+ msg = f"Concept code '{code}' is not a valid concept code for concept '{cls.concept_string}'"
+ raise ConceptValueError(msg) from exc
return code
@field_validator("domain")
@classmethod
def validate_domain(cls, domain: str) -> str:
- validate_domain_code(code=domain)
+ try:
+ validate_domain_code(code=domain)
+ except DomainCodeError as exc:
+ msg = f"Domain code '{domain}' is not a valid domain code for concept '{cls.concept_string}'"
+ raise ConceptValueError(msg) from exc
return domain
@field_validator("refines", mode="before")
@@ -52,16 +53,33 @@ class Concept(BaseModel):
def validate_refines(cls, refines: str | None) -> str | None:
if refines is None:
return None
- validate_concept_string(concept_string=refines)
+ try:
+ validate_concept_string(concept_string=refines)
+ except ConceptStringError as exc:
+ msg = f"Refines '{refines}' is not a valid concept string for concept '{cls.concept_string}'"
+ raise ConceptValueError(msg) from exc
return refines
+ @property
+ def concept_string(self) -> str:
+ return f"{self.domain}.{self.code}"
+
+ @classmethod
+ def is_implicit_concept(cls, concept_string: str) -> bool:
+ try:
+ validate_concept_string(concept_string=concept_string)
+ except ConceptStringError as exc:
+ msg = f"Concept string '{concept_string}' is not a valid concept string for concept '{cls.concept_string}'"
+ raise ConceptValueError(msg) from exc
+ return concept_string.startswith(SpecialDomain.IMPLICIT)
+
@classmethod
def sentence_from_concept(cls, concept: "Concept") -> str:
return pascal_case_to_sentence(name=concept.code)
@classmethod
def is_native_concept(cls, concept: "Concept") -> bool:
- return NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=concept.concept_string) is not None
+ return NativeConceptCode.is_native_concept_string_or_code(concept_string_or_code=concept.concept_string)
@classmethod
def are_concept_compatible(cls, concept_1: "Concept", concept_2: "Concept", strict: bool = False) -> bool:
diff --git a/pipelex/core/concepts/concept_blueprint.py b/pipelex/core/concepts/concept_blueprint.py
index a9933bf17..911358f3b 100644
--- a/pipelex/core/concepts/concept_blueprint.py
+++ b/pipelex/core/concepts/concept_blueprint.py
@@ -2,9 +2,9 @@ from typing import Any
from pydantic import BaseModel, ConfigDict, field_validator, model_validator
-from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint
-from pipelex.core.concepts.exceptions import ConceptBlueprintValueError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.exceptions import NativeConceptDefinitionError
class ConceptBlueprint(BaseModel):
@@ -21,9 +21,11 @@ class ConceptBlueprint(BaseModel):
@classmethod
def validate_refines(cls, refines: str | None = None) -> str | None:
if refines is not None:
- if not NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=refines):
- msg = f"Refine '{refines}' is not a native concept and we currently can only refine native concepts"
- raise ConceptBlueprintValueError(msg)
+ try:
+ NativeConceptCode.validate_native_concept_string_or_code(concept_string_or_code=refines)
+ except NativeConceptDefinitionError as exc:
+ msg = f"Could not validate refine '{refines}': {exc}"
+ raise ValueError(msg) from exc
return refines
@model_validator(mode="before")
@@ -34,5 +36,5 @@ class ConceptBlueprint(BaseModel):
f"Forbidden to have refines and structure at the same time: `{values.get('refines')}` "
f"and `{values.get('structure')}` for concept that has the definition `{values.get('description')}`"
)
- raise ConceptBlueprintValueError(msg)
+ raise ValueError(msg)
return values
diff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.py
index db8f174af..838ba3e9b 100644
--- a/pipelex/core/concepts/concept_factory.py
+++ b/pipelex/core/concepts/concept_factory.py
@@ -3,17 +3,16 @@ from pydantic import BaseModel
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
from pipelex.core.concepts.exceptions import (
- ConceptDefinitionError,
ConceptFactoryError,
ConceptRefineError,
- ConceptStructureGeneratorError,
- StructureClassError,
+ ConceptStringError,
)
-from pipelex.core.concepts.structure_generator import StructureGenerator
-from pipelex.core.concepts.validation import is_concept_code_valid, is_concept_string_valid, validate_refine
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError
+from pipelex.core.concepts.structure_generation.generator import StructureGenerator
+from pipelex.core.concepts.validation import is_concept_code_valid, validate_concept_string_or_code
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.stuffs.text_content import TextContent
@@ -53,14 +52,11 @@ class ConceptFactory:
return normalized
@classmethod
- def make_implicit_concept(cls, concept_string: str) -> Concept:
- if not is_concept_string_valid(concept_string=concept_string):
- msg = f"Concept string '{concept_string}' is not a valid concept string"
- raise ConceptFactoryError(msg)
+ def make_implicit_concept(cls, concept_code: str) -> Concept:
return Concept(
- code=concept_string.split(".")[1],
+ code=concept_code,
domain=SpecialDomain.IMPLICIT,
- description=concept_string,
+ description=concept_code,
structure_class_name=TextContent.__name__,
)
@@ -160,13 +156,19 @@ class ConceptFactory:
concept_string_or_code: str,
concept_codes_from_the_same_domain: list[str] | None = None,
) -> DomainAndConceptCode:
- # At this point, the concept_string_or_code is already validated
+ try:
+ validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)
+ except ConceptStringError as exc:
+ msg = f"Concept string or code '{concept_string_or_code}' is not a valid concept string or code"
+ raise ConceptFactoryError(msg) from exc
+
+ if NativeConceptCode.is_native_concept_string_or_code(concept_string_or_code=concept_string_or_code):
+ natice_concept_string = NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=concept_string_or_code)
+ return DomainAndConceptCode(domain=SpecialDomain.NATIVE, concept_code=natice_concept_string.split(".")[1])
+
if "." in concept_string_or_code:
- # Is a concept string.
- parts = concept_string_or_code.rsplit(".")
- return DomainAndConceptCode(domain=parts[0], concept_code=parts[1])
- if NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=concept_string_or_code):
- return DomainAndConceptCode(domain=SpecialDomain.NATIVE, concept_code=concept_string_or_code)
+ domain_code, concept_code = concept_string_or_code.rsplit(".")
+ return DomainAndConceptCode(domain=domain_code, concept_code=concept_code)
if (
concept_codes_from_the_same_domain and concept_string_or_code in concept_codes_from_the_same_domain
): # Is a concept code from the same domain
@@ -209,19 +211,7 @@ class ConceptFactory:
ConceptFactoryError: If the refine is invalid
"""
- try:
- validate_refine(refine=refine)
- except ConceptRefineError as exc:
- msg = f"Could not validate refine '{refine}': {exc}"
- raise ConceptFactoryError(msg) from exc
-
- # Normalize the refine string (adds native domain if missing)
- normalized_refine = NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=refine)
- if normalized_refine is None:
- msg = f"Could not normalize refine '{refine}' to a native concept string"
- raise ConceptFactoryError(msg)
-
- return normalized_refine
+ return NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=refine)
@classmethod
def make_from_blueprint_or_description(
@@ -266,7 +256,7 @@ class ConceptFactory:
f"Structure class '{blueprint.structure}' set for concept '{concept_code}' in domain '{domain}' "
"is not a registered subclass of StuffContent"
)
- raise StructureClassError(msg)
+ raise ConceptFactoryError(msg)
structure_class_name = blueprint.structure
else:
# Structure is defined as a ConceptStructureBlueprint - run the structure generator and put it in the class registry
@@ -280,14 +270,8 @@ class ConceptFactory:
)
except ConceptStructureGeneratorError as exc:
msg = f"Error generating python code for structure class of concept '{concept_code}' in domain '{domain}': {exc}"
- raise ConceptDefinitionError(
+ raise ConceptFactoryError(
msg,
- domain_code=domain,
- concept_code=concept_code,
- description=blueprint.description,
- structure_class_python_code=exc.structure_class_python_code,
- structure_class_syntax_error_data=exc.syntax_error_data,
- source=blueprint.source,
) from exc
# Register the generated class
@@ -305,16 +289,12 @@ class ConceptFactory:
f"Concept '{concept_code}' in domain '{domain}' has refines but also has a structure class registered. "
"A concept cannot have both structure and refines."
)
- raise ConceptDefinitionError(
- msg, domain_code=domain, concept_code=concept_code, description=blueprint.description, source=blueprint.source
- )
+ raise ConceptFactoryError(msg)
try:
current_refine = cls.make_refine(refine=blueprint.refines)
except ConceptRefineError as exc:
msg = f"Could not validate refine '{blueprint.refines}' for concept '{concept_code}' in domain '{domain}': {exc}"
- raise ConceptDefinitionError(
- msg, domain_code=domain, concept_code=concept_code, description=blueprint.description, source=blueprint.source
- ) from exc
+ raise ConceptFactoryError(msg) from exc
structure_class_name = current_refine.split(".")[1] + "Content" if current_refine else TextContent.__name__
# Handle neither structure nor refines - check the class registry
# If there is a class, use it. structure_class_name is then the concept_code
diff --git a/pipelex/core/concepts/concept_structure_blueprint.py b/pipelex/core/concepts/concept_structure_blueprint.py
index ae85b0598..b8f9bfa71 100644
--- a/pipelex/core/concepts/concept_structure_blueprint.py
+++ b/pipelex/core/concepts/concept_structure_blueprint.py
@@ -3,7 +3,6 @@ from typing import Any
from pydantic import BaseModel, Field, model_validator
-from pipelex.core.concepts.exceptions import ConceptStructureBlueprintValueError
from pipelex.types import Self, StrEnum
@@ -35,16 +34,16 @@ class ConceptStructureBlueprint(BaseModel):
# If type is None (array), choices must not be None
if self.type is None and not self.choices:
msg = f"When type is None (array), choices must not be empty. Actual type: {self.type}, choices: {self.choices}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
# If type is "dict", key_type and value_type must not be empty
if self.type == ConceptStructureBlueprintFieldType.DICT:
if not self.key_type:
msg = f"When type is '{ConceptStructureBlueprintFieldType.DICT}', key_type must not be empty. Actual key_type: {self.key_type}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
if not self.value_type:
msg = f"When type is '{ConceptStructureBlueprintFieldType.DICT}', value_type must not be empty. Actual value_type: {self.value_type}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
# Check when default_value is not None, type is not None (except for choice fields)
if self.default_value is not None and self.type is None and not self.choices:
@@ -52,7 +51,7 @@ class ConceptStructureBlueprint(BaseModel):
f"When default_value is not None, type must be specified (unless choices are provided). Actual type: {self.type},"
f"default_value: {self.default_value}, choices: {self.choices}"
)
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
# Check default_value type is the same as type
if self.default_value is not None and self.type is not None:
@@ -62,7 +61,7 @@ class ConceptStructureBlueprint(BaseModel):
if self.default_value is not None and self.type is None and self.choices:
if self.default_value not in self.choices:
msg = f"default_value must be one of the valid choices. Got '{self.default_value}', valid choices: {self.choices}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
return self
@@ -95,4 +94,4 @@ class ConceptStructureBlueprint(BaseModel):
def _raise_type_mismatch_error(self, expected_type_name: str, actual_type_name: str) -> None:
msg = f"default_value type mismatch: expected {expected_type_name} for type '{self.type}', but got {actual_type_name}"
- raise ConceptStructureBlueprintValueError(msg)
+ raise ValueError(msg)
diff --git a/pipelex/core/concepts/exceptions.py b/pipelex/core/concepts/exceptions.py
index a9fc3b338..e5aba7c60 100644
--- a/pipelex/core/concepts/exceptions.py
+++ b/pipelex/core/concepts/exceptions.py
@@ -1,32 +1,15 @@
-from abc import ABC, abstractmethod
+from pipelex.base_exceptions import PipelexError
-from pydantic import BaseModel, Field
-from pipelex.base_exceptions import PipelexException
-from pipelex.core.exceptions import SyntaxErrorData
-
-
-class ConceptError(PipelexException):
- pass
-
-
-class ConceptBlueprintValueError(ValueError):
- pass
-
-
-class ConceptStructureBlueprintValueError(ValueError):
- pass
-
-
-class ConceptStructureValidationError(PipelexException):
+class ConceptError(PipelexError):
pass
-class ConceptFactoryError(PipelexException):
+class ConceptValueError(ValueError):
pass
-class StructureClassError(ConceptFactoryError):
+class ConceptFactoryError(PipelexError):
pass
@@ -42,61 +25,5 @@ class ConceptRefineError(ConceptError):
pass
-class ConceptLibraryConceptNotFoundError(PipelexException):
+class ConceptLibraryConceptNotFoundError(PipelexError):
pass
-
-
-class ConceptDefinitionErrorData(BaseModel):
- """Structured data for ConceptDefinitionError."""
-
- message: str = Field(description="The error message")
- domain_code: str = Field(description="The domain code")
- concept_code: str = Field(description="The concept code")
- description: str = Field(description="Description of the concept")
- structure_class_python_code: str | None = Field(None, description="Python code for the structure class if available")
- structure_class_syntax_error_data: SyntaxErrorData | None = Field(None, description="Syntax error data for the structure class if available")
- source: str | None = Field(None, description="Source of the error")
-
-
-class ConceptDefinitionError(PipelexException):
- def __init__(
- self,
- message: str,
- domain_code: str,
- concept_code: str,
- description: str,
- structure_class_python_code: str | None = None,
- structure_class_syntax_error_data: SyntaxErrorData | None = None,
- source: str | None = None,
- ):
- self.domain_code = domain_code
- self.concept_code = concept_code
- self.description = description
- self.structure_class_python_code = structure_class_python_code
- self.structure_class_syntax_error_data = structure_class_syntax_error_data
- self.source = source
- super().__init__(message)
-
- def as_structured_content(self) -> ConceptDefinitionErrorData:
- return ConceptDefinitionErrorData(
- message=str(self),
- domain_code=self.domain_code,
- concept_code=self.concept_code,
- description=self.description,
- structure_class_python_code=self.structure_class_python_code,
- structure_class_syntax_error_data=self.structure_class_syntax_error_data,
- source=self.source,
- )
-
-
-class ConceptStructureGeneratorError(PipelexException):
- def __init__(self, message: str, structure_class_python_code: str | None = None, syntax_error_data: SyntaxErrorData | None = None):
- self.structure_class_python_code = structure_class_python_code
- self.syntax_error_data = syntax_error_data
- super().__init__(message)
-
-
-class PipelexValidationExceptionAbstract(PipelexException, ABC):
- @abstractmethod
- def get_concept_definition_errors(self) -> list[ConceptDefinitionErrorData]:
- pass
diff --git a/tests/test_pipelines/__init__.py b/pipelex/core/concepts/native/__init__.py
similarity index 100%
rename from tests/test_pipelines/__init__.py
rename to pipelex/core/concepts/native/__init__.py
diff --git a/pipelex/core/concepts/concept_native.py b/pipelex/core/concepts/native/concept_native.py
similarity index 71%
rename from pipelex/core/concepts/concept_native.py
rename to pipelex/core/concepts/native/concept_native.py
index e9c2e1d21..1de58771f 100644
--- a/pipelex/core/concepts/concept_native.py
+++ b/pipelex/core/concepts/native/concept_native.py
@@ -1,11 +1,9 @@
+from pipelex.core.concepts.native.exceptions import NativeConceptDefinitionError
+from pipelex.core.concepts.validation import is_concept_string_or_code_valid
from pipelex.core.domains.domain import SpecialDomain
from pipelex.types import StrEnum
-class NativeConceptEnumError(Exception):
- pass
-
-
class NativeConceptCode(StrEnum):
DYNAMIC = "Dynamic"
TEXT = "Text"
@@ -80,26 +78,30 @@ class NativeConceptCode(StrEnum):
def values_list(cls) -> list["NativeConceptCode"]:
return list(cls)
- @classmethod
- def is_native_concept(cls, concept_code: str) -> bool:
- return concept_code in cls.values_list()
-
@classmethod
def native_concept_class_names(cls):
return [native_concept.structure_class_name for native_concept in cls]
@classmethod
- def get_validated_native_concept_string(cls, concept_string_or_code: str) -> str | None:
+ def is_native_concept_string_or_code(cls, concept_string_or_code: str) -> bool:
+ if not is_concept_string_or_code_valid(concept_string_or_code=concept_string_or_code):
+ return False
+
if "." in concept_string_or_code:
- if concept_string_or_code.count(".") > 1:
- msg = f"Trying to get a native concept with code '{concept_string_or_code}' but that is not a native concept"
- raise NativeConceptEnumError(msg)
domain_code, concept_code = concept_string_or_code.split(".", 1)
- if SpecialDomain.is_native(domain=domain_code) and concept_code in cls.values_list():
- return concept_string_or_code
- else:
- return None
- elif concept_string_or_code in cls.values_list():
- return f"{SpecialDomain.NATIVE}.{concept_string_or_code}"
+ return SpecialDomain.is_native(domain=domain_code) and concept_code in cls.values_list()
+ return concept_string_or_code in cls.values_list()
+
+ @classmethod
+ def validate_native_concept_string_or_code(cls, concept_string_or_code: str) -> None:
+ if not cls.is_native_concept_string_or_code(concept_string_or_code=concept_string_or_code):
+ msg = f"Concept string or code '{concept_string_or_code}' is not a valid native concept string or code"
+ raise NativeConceptDefinitionError(msg)
+
+ @classmethod
+ def get_validated_native_concept_string(cls, concept_string_or_code: str) -> str:
+ cls.validate_native_concept_string_or_code(concept_string_or_code=concept_string_or_code)
+ if "." in concept_string_or_code:
+ return concept_string_or_code
else:
- return None
+ return f"{SpecialDomain.NATIVE}.{concept_string_or_code}"
diff --git a/pipelex/core/concepts/native/exceptions.py b/pipelex/core/concepts/native/exceptions.py
new file mode 100644
index 000000000..4d7fe146f
--- /dev/null
+++ b/pipelex/core/concepts/native/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class NativeConceptDefinitionError(PipelexError):
+ pass
diff --git a/tests/test_pipelines/concepts/__init__.py b/pipelex/core/concepts/structure_generation/__init__.py
similarity index 100%
rename from tests/test_pipelines/concepts/__init__.py
rename to pipelex/core/concepts/structure_generation/__init__.py
diff --git a/pipelex/core/concepts/structure_generation/exceptions.py b/pipelex/core/concepts/structure_generation/exceptions.py
new file mode 100644
index 000000000..57d9b16e5
--- /dev/null
+++ b/pipelex/core/concepts/structure_generation/exceptions.py
@@ -0,0 +1,39 @@
+from pydantic import BaseModel
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+
+
+class SyntaxErrorData(BaseModel):
+ message: str
+ lineno: int | None = None
+ offset: int | None = None
+ text: str | None = None
+ end_lineno: int | None = None
+ end_offset: int | None = None
+
+ @classmethod
+ def from_syntax_error(cls, syntax_error: SyntaxError) -> "SyntaxErrorData":
+ return cls(
+ message=syntax_error.msg,
+ lineno=syntax_error.lineno,
+ offset=syntax_error.offset,
+ text=syntax_error.text,
+ end_lineno=syntax_error.end_lineno,
+ end_offset=syntax_error.end_offset,
+ )
+
+
+class StructureClassError(ConceptFactoryError):
+ pass
+
+
+class ConceptStructureGeneratorError(PipelexError):
+ def __init__(self, message: str, structure_class_python_code: str | None = None, syntax_error_data: SyntaxErrorData | None = None):
+ self.structure_class_python_code = structure_class_python_code
+ self.syntax_error_data = syntax_error_data
+ super().__init__(message)
+
+
+class ConceptStructureValidationError(PipelexError):
+ pass
diff --git a/pipelex/core/concepts/structure_generator.py b/pipelex/core/concepts/structure_generation/generator.py
similarity index 99%
rename from pipelex/core/concepts/structure_generator.py
rename to pipelex/core/concepts/structure_generation/generator.py
index 7311c4701..3abcf24b3 100644
--- a/pipelex/core/concepts/structure_generator.py
+++ b/pipelex/core/concepts/structure_generation/generator.py
@@ -6,7 +6,7 @@ from typing import Any, Literal, Optional
from pydantic import Field
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.exceptions import ConceptStructureGeneratorError, ConceptStructureValidationError, SyntaxErrorData
+from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError, ConceptStructureValidationError, SyntaxErrorData
from pipelex.core.stuffs.structured_content import StructuredContent
diff --git a/pipelex/core/concepts/validation.py b/pipelex/core/concepts/validation.py
index 8f6ff41f1..edbccdd63 100644
--- a/pipelex/core/concepts/validation.py
+++ b/pipelex/core/concepts/validation.py
@@ -1,8 +1,5 @@
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.concepts.exceptions import ConceptCodeError, ConceptRefineError, ConceptStringError
-from pipelex.core.domains.domain import SpecialDomain
-from pipelex.core.domains.exceptions import DomainCodeError
-from pipelex.core.domains.validation import is_domain_code_valid, validate_domain_code
+from pipelex.core.concepts.exceptions import ConceptCodeError, ConceptStringError
+from pipelex.core.domains.validation import is_domain_code_valid
from pipelex.tools.misc.string_utils import is_pascal_case
@@ -16,26 +13,15 @@ def validate_concept_code(concept_code: str) -> None:
raise ConceptCodeError(msg)
-def is_concept_string_or_code_valid(concept_string_or_code: str) -> bool:
- if concept_string_or_code.count(".") > 1:
- return False
-
- if concept_string_or_code.count(".") == 1:
- domain, concept_code = concept_string_or_code.split(".")
- try:
- validate_concept_code(concept_code=concept_code)
- validate_domain_code(code=domain)
- return True
- except (ConceptCodeError, DomainCodeError):
- return False
- else:
- return is_concept_code_valid(concept_code=concept_string_or_code)
+def is_concept_string_valid(concept_string: str) -> bool:
+ domain, concept_code = concept_string.split(".", 1)
+ # Validate domain
+ if not is_domain_code_valid(code=domain):
+ return False
-def validate_concept_string_or_code(concept_string_or_code: str) -> None:
- if not is_concept_string_or_code_valid(concept_string_or_code=concept_string_or_code):
- msg = f"Concept string or code '{concept_string_or_code}' is not a valid concept string or code."
- raise ConceptStringError(msg)
+ # Validate concept code
+ return is_concept_code_valid(concept_code=concept_code)
def validate_concept_string(concept_string: str) -> None:
@@ -48,39 +34,17 @@ def validate_concept_string(concept_string: str) -> None:
raise ConceptStringError(msg)
-def is_concept_string_valid(concept_string: str) -> bool:
- if "." not in concept_string or concept_string.count(".") > 1:
- return False
- domain, concept_code = concept_string.split(".", 1)
-
- # Validate domain
- if not is_domain_code_valid(code=domain):
- return False
-
- # Validate concept code
- if not is_concept_code_valid(concept_code=concept_code):
+def is_concept_string_or_code_valid(concept_string_or_code: str) -> bool:
+ if concept_string_or_code.count(".") > 1:
return False
- # Validate that if the concept code is among the native concepts, the domain MUST be native.
- if concept_code in NativeConceptCode.values_list():
- if not SpecialDomain.is_native(domain=domain):
- return False
- # Validate that if the domain is native, the concept code is a native concept
- if SpecialDomain.is_native(domain=domain):
- if concept_code not in NativeConceptCode.values_list():
- return False
- return True
-
-
-def is_refine_valid(refine: str) -> bool:
- return NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=refine) is not None
+ if concept_string_or_code.count(".") == 1:
+ return is_concept_string_valid(concept_string=concept_string_or_code)
+ else:
+ return is_concept_code_valid(concept_code=concept_string_or_code)
-def validate_refine(refine: str) -> None:
- if not is_refine_valid(refine=refine):
- msg = (
- f"Refine '{refine}' is not a valid refine. It must be referencing a native concept. "
- "More about native concepts here: "
- "https://docs.pipelex.com/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts/#native-concepts-and-their-structures"
- )
- raise ConceptRefineError(msg)
+def validate_concept_string_or_code(concept_string_or_code: str) -> None:
+ if not is_concept_string_or_code_valid(concept_string_or_code=concept_string_or_code):
+ msg = f"Concept string or code '{concept_string_or_code}' is not a valid concept string or code."
+ raise ConceptStringError(msg)
diff --git a/pipelex/core/domains/domain.py b/pipelex/core/domains/domain.py
index 66fa74e8d..0f72e0534 100644
--- a/pipelex/core/domains/domain.py
+++ b/pipelex/core/domains/domain.py
@@ -1,6 +1,6 @@
-from pydantic import BaseModel
+from pydantic import BaseModel, ConfigDict
-from pipelex.types import Self, StrEnum
+from pipelex.types import StrEnum
class SpecialDomain(StrEnum):
@@ -22,10 +22,8 @@ class SpecialDomain(StrEnum):
class Domain(BaseModel):
+ model_config = ConfigDict(extra="forbid")
+
code: str
description: str | None = None
system_prompt: str | None = None
-
- @classmethod
- def make_default(cls) -> Self:
- return cls(code=SpecialDomain.NATIVE)
diff --git a/pipelex/core/domains/domain_factory.py b/pipelex/core/domains/domain_factory.py
index 7c3fdf697..4adc0a743 100644
--- a/pipelex/core/domains/domain_factory.py
+++ b/pipelex/core/domains/domain_factory.py
@@ -2,7 +2,7 @@ from pydantic import ValidationError
from pipelex.core.domains.domain import Domain
from pipelex.core.domains.domain_blueprint import DomainBlueprint
-from pipelex.core.domains.exceptions import DomainDefinitionError
+from pipelex.core.domains.exceptions import DomainFactoryError
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
@@ -16,6 +16,5 @@ class DomainFactory:
system_prompt=blueprint.system_prompt,
)
except ValidationError as exc:
- validation_error_msg = format_pydantic_validation_error(exc)
- msg = f"Could not make domain from blueprint: {validation_error_msg}"
- raise DomainDefinitionError(message=msg, domain_code=blueprint.code, description=blueprint.description) from exc
+ msg = f"Could not make domain from blueprint: {format_pydantic_validation_error(exc)}"
+ raise DomainFactoryError(msg) from exc
diff --git a/pipelex/core/domains/exceptions.py b/pipelex/core/domains/exceptions.py
index f1d7a3c70..b6daaf5c1 100644
--- a/pipelex/core/domains/exceptions.py
+++ b/pipelex/core/domains/exceptions.py
@@ -1,13 +1,9 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class DomainCodeError(PipelexException):
+class DomainCodeError(PipelexError):
pass
-class DomainDefinitionError(PipelexException):
- def __init__(self, message: str, domain_code: str, description: str, source: str | None = None):
- self.domain_code = domain_code
- self.description = description
- self.source = source
- super().__init__(message)
+class DomainFactoryError(PipelexError):
+ pass
diff --git a/pipelex/core/exceptions.py b/pipelex/core/exceptions.py
index 37577c49e..17d7c49e0 100644
--- a/pipelex/core/exceptions.py
+++ b/pipelex/core/exceptions.py
@@ -1,72 +1,36 @@
-from pydantic import BaseModel
-from typing_extensions import override
+from pydantic import BaseModel, Field
-from pipelex.base_exceptions import PipelexException
-from pipelex.core.pipes.exceptions import StaticValidationErrorType
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
-class PipelexConfigurationError(PipelexException):
- """Raised when there are configuration issues with the PipelexInterpreter."""
+class PipesAndConceptValidationErrorData(BaseModel):
+ """Structured validation error data for Pipe/Concept validation errors.
+ This model captures validation errors raised by Pipe or Concept classes during
+ their validation (NOT blueprint validation errors).
-class SyntaxErrorData(BaseModel):
- message: str
- lineno: int | None = None
- offset: int | None = None
- text: str | None = None
- end_lineno: int | None = None
- end_offset: int | None = None
+ These errors come from:
+ - PipeAbstract and its subclasses (PipeLLM, PipeExtract, etc.)
+ - Concept validation
+ """
- @classmethod
- def from_syntax_error(cls, syntax_error: SyntaxError) -> "SyntaxErrorData":
- return cls(
- message=syntax_error.msg,
- lineno=syntax_error.lineno,
- offset=syntax_error.offset,
- text=syntax_error.text,
- end_lineno=syntax_error.end_lineno,
- end_offset=syntax_error.end_offset,
- )
+ # === Source Context ===
+ domain: str | None = Field(None, description="Domain where error occurred")
+ source: str | None = Field(None, description="Source file path")
+ # === Entity Context (what failed) ===
+ pipe_code: str | None = Field(None, description="Pipe code if error is in a pipe")
+ concept_code: str | None = Field(None, description="Concept code if error is in a concept")
+ field_name: str | None = Field(None, description="Specific field that failed")
-class StaticValidationError(Exception):
- def __init__(
- self,
- error_type: StaticValidationErrorType,
- domain: str,
- pipe_code: str | None = None,
- variable_names: list[str] | None = None,
- required_concept_codes: list[str] | None = None,
- provided_concept_code: str | None = None,
- file_path: str | None = None,
- explanation: str | None = None,
- ):
- self.error_type = error_type
- self.domain = domain
- self.pipe_code = pipe_code
- self.variable_names = variable_names
- self.required_concept_codes = required_concept_codes
- self.provided_concept_code = provided_concept_code
- self.file_path = file_path
- self.explanation = explanation
- super().__init__()
+ # === Error Classification ===
+ error_type: PipeValidationErrorType = Field(
+ description="Type of pipe/concept validation error",
+ )
- def desc(self) -> str:
- msg = f"{self.error_type} • domain='{self.domain}'"
- if self.pipe_code:
- msg += f" • pipe='{self.pipe_code}'"
- if self.variable_names:
- msg += f" • variable='{self.variable_names}'"
- if self.required_concept_codes:
- msg += f" • required_concept_codes='{self.required_concept_codes}'"
- if self.provided_concept_code:
- msg += f" • provided_concept_code='{self.provided_concept_code}'"
- if self.file_path:
- msg += f" • file='{self.file_path}'"
- if self.explanation:
- msg += f" • explanation='{self.explanation}'"
- return msg
+ # === Error Details ===
+ message: str = Field(description="Human-readable error message")
+ field_path: str = Field(description="Path to field in dot notation")
- @override
- def __str__(self) -> str:
- return self.desc()
+ # === Variable names for input/output errors ===
+ variable_names: list[str] | None = Field(None, description="Variable names (for input errors)")
diff --git a/pipelex/core/interpreter.py b/pipelex/core/interpreter.py
deleted file mode 100644
index fc67f1ced..000000000
--- a/pipelex/core/interpreter.py
+++ /dev/null
@@ -1,90 +0,0 @@
-from pathlib import Path
-
-from pydantic import BaseModel, model_validator
-
-from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.exceptions import PipelexConfigurationError
-from pipelex.tools.misc.toml_utils import TomlError, load_toml_from_content, load_toml_from_path
-from pipelex.types import Self
-
-
-class PLXDecodeError(TomlError):
- """Raised when PLX decoding fails."""
-
-
-class PipelexInterpreter(BaseModel):
- """plx -> PipelexBundleBlueprint"""
-
- file_path: Path | None = None
- file_content: str | None = None
-
- @model_validator(mode="after")
- def check_file_path_or_file_content(self) -> Self:
- """Need to check if there is at least one of file_path or file_content"""
- if self.file_path is None and self.file_content is None:
- msg = "Either file_path or file_content must be provided"
- raise PipelexConfigurationError(msg)
- return self
-
- # TODO: rethink this method
- @staticmethod
- def is_pipelex_file(file_path: Path) -> bool:
- """Check if a file is a valid Pipelex PLX file.
-
- Args:
- file_path: Path to the file to check
-
- Returns:
- True if the file is a Pipelex file, False otherwise
-
- Criteria:
- - Has .plx extension
- - Starts with "domain =" (ignoring leading whitespace)
-
- """
- # Check if it has .toml extension
- if file_path.suffix != ".plx":
- return False
-
- # Check if file exists
- if not file_path.exists() or not file_path.is_file():
- return False
-
- try:
- # Read the first few lines to check for "domain ="
- with open(file_path, encoding="utf-8") as f:
- # Read first 100 characters (should be enough to find domain)
- content = f.read(100)
- # Remove leading whitespace and check if it starts with "domain ="
- stripped_content = content.lstrip()
- return stripped_content.startswith("domain =")
- except Exception:
- # If we can't read the file, it's not a valid Pipelex file
- return False
-
- def make_pipelex_bundle_blueprint(self) -> PipelexBundleBlueprint:
- """Make a PipelexBundleBlueprint from the file_path or file_content"""
- # Load PLX content from file_path or use file_content directly.
- try:
- if self.file_path:
- blueprint_dict = load_toml_from_path(path=str(self.file_path))
- blueprint_dict.update(source=str(self.file_path))
- elif self.file_content:
- blueprint_dict = load_toml_from_content(content=self.file_content)
- else:
- msg = "Could not make PipelexBundleBlueprint: either file_path or file_content must be provided"
- raise PipelexConfigurationError(msg)
- except TomlError as exc:
- raise PLXDecodeError(message=exc.message, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from exc
- return PipelexBundleBlueprint.model_validate(blueprint_dict)
-
- @classmethod
- def load_bundle_blueprint(cls, bundle_path: str) -> PipelexBundleBlueprint:
- """Load a bundle file and return its blueprint."""
- bundle_path_obj = Path(bundle_path)
- if not bundle_path_obj.exists():
- msg = f"Bundle file not found: {bundle_path}"
- raise FileNotFoundError(msg)
-
- interpreter = cls(file_path=bundle_path_obj)
- return interpreter.make_pipelex_bundle_blueprint()
diff --git a/tests/test_pipelines/misc_tests/__init__.py b/pipelex/core/interpreter/__init__.py
similarity index 100%
rename from tests/test_pipelines/misc_tests/__init__.py
rename to pipelex/core/interpreter/__init__.py
diff --git a/pipelex/core/interpreter/exceptions.py b/pipelex/core/interpreter/exceptions.py
new file mode 100644
index 000000000..6b5c4125b
--- /dev/null
+++ b/pipelex/core/interpreter/exceptions.py
@@ -0,0 +1,19 @@
+from pipelex.base_exceptions import PipelexError
+from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
+from pipelex.tools.misc.toml_utils import TomlError
+
+
+class PipelexInterpreterError(PipelexError):
+ """Raised when PipelexInterpreter fails."""
+
+ def __init__(
+ self,
+ message: str,
+ validation_errors: list[PipelexBundleBlueprintValidationErrorData] | None = None,
+ ):
+ self.validation_errors = validation_errors or []
+ super().__init__(message)
+
+
+class PLXDecodeError(TomlError):
+ """Raised when PLX decoding fails."""
diff --git a/pipelex/core/interpreter/helpers.py b/pipelex/core/interpreter/helpers.py
new file mode 100644
index 000000000..04d108aa4
--- /dev/null
+++ b/pipelex/core/interpreter/helpers.py
@@ -0,0 +1,59 @@
+from pipelex.types import StrEnum
+
+
+class ValidationErrorScope(StrEnum):
+ PIPE = "pipe"
+ CONCEPT = "concept"
+ DOMAIN = "domain"
+ MAIN_PIPE = "main_pipe"
+ BUNDLE = "bundle"
+
+ @classmethod
+ def is_pipe_scope(cls, scope: str) -> bool:
+ try:
+ scope_enum = cls(scope)
+ except ValueError:
+ return False
+ match scope_enum:
+ case ValidationErrorScope.PIPE:
+ return True
+ case ValidationErrorScope.CONCEPT:
+ return False
+ case ValidationErrorScope.DOMAIN:
+ return False
+ case ValidationErrorScope.MAIN_PIPE:
+ return False
+ case ValidationErrorScope.BUNDLE:
+ return False
+
+
+def get_error_scope(loc: tuple[int | str, ...]) -> ValidationErrorScope:
+ """Get the scope of a validation error from its location tuple.
+
+ Args:
+ loc: Location tuple from Pydantic validation error
+
+ Returns:
+ ValidationErrorScope - defaults to BUNDLE if scope cannot be determined
+ """
+ if not loc:
+ return ValidationErrorScope.BUNDLE
+
+ scope = str(loc[0])
+
+ try:
+ scope_enum = ValidationErrorScope(scope)
+ except ValueError:
+ return ValidationErrorScope.BUNDLE
+
+ match scope_enum:
+ case ValidationErrorScope.PIPE:
+ return ValidationErrorScope.PIPE
+ case ValidationErrorScope.CONCEPT:
+ return ValidationErrorScope.CONCEPT
+ case ValidationErrorScope.DOMAIN:
+ return ValidationErrorScope.DOMAIN
+ case ValidationErrorScope.MAIN_PIPE:
+ return ValidationErrorScope.MAIN_PIPE
+ case ValidationErrorScope.BUNDLE:
+ return ValidationErrorScope.BUNDLE
diff --git a/pipelex/core/interpreter/interpreter.py b/pipelex/core/interpreter/interpreter.py
new file mode 100644
index 000000000..4f47ec237
--- /dev/null
+++ b/pipelex/core/interpreter/interpreter.py
@@ -0,0 +1,88 @@
+from pathlib import Path
+from typing import TYPE_CHECKING, Any
+
+from pydantic import BaseModel, ValidationError
+
+from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+from pipelex.core.interpreter.exceptions import PipelexInterpreterError, PLXDecodeError
+from pipelex.core.interpreter.validation_error_categorizer import categorize_blueprint_validation_error
+from pipelex.tools.misc.toml_utils import TomlError, load_toml_from_content, load_toml_from_path
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+
+if TYPE_CHECKING:
+ from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
+
+
+class PipelexInterpreter(BaseModel):
+ """plx -> PipelexBundleBlueprint"""
+
+ # TODO: rethink this method
+ @staticmethod
+ def is_pipelex_file(file_path: Path) -> bool:
+ """Check if a file is a valid Pipelex PLX file.
+
+ Args:
+ file_path: Path to the file to check
+
+ Returns:
+ True if the file is a Pipelex file, False otherwise
+
+ Criteria:
+ - Has .plx extension
+ - Starts with "domain =" (ignoring leading whitespace)
+
+ """
+ # Check if it has .toml extension
+ if file_path.suffix != ".plx":
+ return False
+
+ # Check if file exists
+ if not file_path.exists() or not file_path.is_file():
+ return False
+
+ try:
+ # Read the first few lines to check for "domain ="
+ with open(file_path, encoding="utf-8") as file:
+ # Read first 100 characters (should be enough to find domain)
+ content = file.read(100)
+ # Remove leading whitespace and check if it starts with "domain ="
+ stripped_content = content.lstrip()
+ return stripped_content.startswith("domain =")
+ except Exception:
+ # If we can't read the file, it's not a valid Pipelex file
+ return False
+
+ @classmethod
+ def make_pipelex_bundle_blueprint(cls, bundle_path: str | None = None, plx_content: str | None = None) -> PipelexBundleBlueprint:
+ blueprint_dict: dict[str, Any]
+ try:
+ if bundle_path is not None:
+ blueprint_dict = load_toml_from_path(path=bundle_path)
+ blueprint_dict.update(source=bundle_path)
+ elif plx_content is not None:
+ blueprint_dict = load_toml_from_content(content=plx_content)
+ else:
+ msg = "Either 'bundle_path' or 'plx_content' must be provided for the PipelexInterpreter to make a PipelexBundleBlueprint"
+ raise PipelexInterpreterError(msg)
+ except TomlError as exc:
+ raise PLXDecodeError(message=exc.message, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from exc
+
+ if not blueprint_dict:
+ msg = "Could not make 'PipelexBundleBlueprint': no blueprint found in the PLX file"
+ raise PipelexInterpreterError(msg)
+
+ try:
+ return PipelexBundleBlueprint.model_validate(blueprint_dict)
+ except ValidationError as exc:
+ # TODO: Move this to the validate_bundle function
+ blueprint_validation_errors: list[PipelexBundleBlueprintValidationErrorData] = []
+
+ for error in exc.errors():
+ categorized_error = categorize_blueprint_validation_error(blueprint_dict=blueprint_dict, error=error)
+ if categorized_error:
+ blueprint_validation_errors.append(categorized_error)
+
+ raise PipelexInterpreterError(
+ message=format_pydantic_validation_error(exc),
+ validation_errors=blueprint_validation_errors,
+ ) from exc
diff --git a/pipelex/core/interpreter/validation_error_categorizer.py b/pipelex/core/interpreter/validation_error_categorizer.py
new file mode 100644
index 000000000..95a8548b7
--- /dev/null
+++ b/pipelex/core/interpreter/validation_error_categorizer.py
@@ -0,0 +1,43 @@
+from typing import Any, cast
+
+from pydantic_core import ErrorDetails
+
+from pipelex import log
+from pipelex.core.bundles.exceptions import (
+ PipelexBundleBlueprintValidationErrorData,
+)
+from pipelex.core.interpreter.helpers import get_error_scope
+from pipelex.types import StrEnum
+
+
+class ErrorCatKey(StrEnum):
+ LOC = "loc"
+ MSG = "msg"
+ TYPE = "type"
+
+
+PIPELEX_BUNDLE_BLUEPRINT_DOMAIN_FIELD = "domain"
+PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD = "source"
+
+
+def categorize_blueprint_validation_error(
+ blueprint_dict: dict[str, Any],
+ error: ErrorDetails,
+) -> PipelexBundleBlueprintValidationErrorData | None:
+ """Categorize a BLUEPRINT validation error and create structured error data or return None if the error cannot be categorized.
+
+ Args:
+ blueprint_dict: The blueprint dict being validated (for context extraction)
+ error: Pydantic error from PipelexBundleBlueprint.model_validate()
+
+ Returns:
+ PipelexBundleBlueprintValidationErrorData with all relevant fields populated, or None if error cannot be categorized
+ """
+ domain = cast("str | None", blueprint_dict.get(PIPELEX_BUNDLE_BLUEPRINT_DOMAIN_FIELD)) if blueprint_dict else None
+ source = cast("str | None", blueprint_dict.get(PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD)) if blueprint_dict else None
+
+ loc = error.get(ErrorCatKey.LOC.value, ())
+ error_scope = get_error_scope(loc)
+ log.warning(f"Pipelex bundle blueprint validation error that is not categorized: {error_scope} - {source} - {domain}")
+
+ return None
diff --git a/pipelex/core/memory/exceptions.py b/pipelex/core/memory/exceptions.py
index 82793b2e0..98b54a969 100644
--- a/pipelex/core/memory/exceptions.py
+++ b/pipelex/core/memory/exceptions.py
@@ -1,12 +1,12 @@
-from pipelex.base_exceptions import PipelexException
-from pipelex.tools.misc.context_provider_abstract import ContextProviderException
+from pipelex.base_exceptions import PipelexError
+from pipelex.tools.misc.context_provider_abstract import ContextProviderError
-class WorkingMemoryFactoryError(PipelexException):
+class WorkingMemoryFactoryError(PipelexError):
pass
-class WorkingMemoryError(PipelexException):
+class WorkingMemoryError(PipelexError):
pass
@@ -14,7 +14,7 @@ class WorkingMemoryConsistencyError(WorkingMemoryError):
pass
-class WorkingMemoryVariableError(WorkingMemoryError, ContextProviderException):
+class WorkingMemoryVariableError(WorkingMemoryError, ContextProviderError):
pass
diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.py
index aad3fc46e..9a4a66cd1 100644
--- a/pipelex/core/memory/working_memory.py
+++ b/pipelex/core/memory/working_memory.py
@@ -123,6 +123,9 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):
def set_stuff(self, name: str, stuff: Stuff):
self.root[name] = stuff
+ def is_stuff_exists(self, name: str) -> bool:
+ return name in self.root or name in self.aliases
+
def add_new_stuff(self, name: str, stuff: Stuff, aliases: list[str] | None = None):
# TODO: Add unit tests for this method
if self.is_stuff_code_used(stuff_code=stuff.stuff_code):
diff --git a/pipelex/core/memory/working_memory_factory.py b/pipelex/core/memory/working_memory_factory.py
index 9df54e352..37b3c338a 100644
--- a/pipelex/core/memory/working_memory_factory.py
+++ b/pipelex/core/memory/working_memory_factory.py
@@ -1,76 +1,20 @@
-from typing import Any
-
import shortuuid
from polyfactory.factories.pydantic_factory import ModelFactory
from pydantic import BaseModel
from pipelex import log
from pipelex.client.protocol import PipelineInputs
-from pipelex.core.concepts.concept import SpecialDomain
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.concepts.validation import validate_concept_string
from pipelex.core.memory.exceptions import WorkingMemoryFactoryError
from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, StuffDict, WorkingMemory
-from pipelex.core.pipes.input_requirements import TypedNamedInputRequirement
-from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.pipes.inputs.input_requirements import TypedNamedInputRequirement
from pipelex.core.stuffs.list_content import ListContent
-from pipelex.core.stuffs.pdf_content import PDFContent
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
-from pipelex.hub import get_required_concept
class WorkingMemoryFactory(BaseModel):
- @classmethod
- def make_from_text(
- cls,
- text: str,
- concept_string: str = SpecialDomain.NATIVE + "." + NativeConceptCode.TEXT,
- name: str | None = "text",
- ) -> WorkingMemory:
- validate_concept_string(concept_string=concept_string)
- return cls.make_from_single_stuff(
- stuff=StuffFactory.make_stuff(
- concept=get_required_concept(concept_string=concept_string),
- content=TextContent(text=text),
- name=name,
- ),
- )
-
- @classmethod
- def make_from_image(
- cls,
- image_url: str,
- concept_string: str = SpecialDomain.NATIVE + "." + NativeConceptCode.IMAGE,
- name: str | None = "image",
- ) -> WorkingMemory:
- # TODO: validate that the concept is compatible with an image concept
- validate_concept_string(concept_string=concept_string)
- stuff = StuffFactory.make_stuff(
- concept=get_required_concept(concept_string=concept_string),
- content=ImageContent(url=image_url),
- name=name,
- )
- return cls.make_from_single_stuff(stuff=stuff)
-
- @classmethod
- def make_from_pdf(
- cls,
- pdf_url: str,
- concept_string: str = SpecialDomain.NATIVE + "." + NativeConceptCode.PDF,
- name: str | None = "pdf",
- ) -> WorkingMemory:
- validate_concept_string(concept_string=concept_string)
- return cls.make_from_single_stuff(
- stuff=StuffFactory.make_stuff(
- concept=get_required_concept(concept_string=concept_string),
- content=PDFContent(url=pdf_url),
- name=name,
- ),
- )
-
@classmethod
def make_from_single_stuff(cls, stuff: Stuff) -> WorkingMemory:
if not stuff.stuff_name:
@@ -103,22 +47,6 @@ class WorkingMemoryFactory(BaseModel):
aliases[MAIN_STUFF_NAME] = next(iter(stuff_dict.keys()))
return WorkingMemory(root=stuff_dict, aliases=aliases)
- @classmethod
- def make_from_strings_from_dict(cls, input_dict: dict[str, Any]) -> WorkingMemory:
- # TODO: Add unit tests for this method
- stuff_dict: StuffDict = {}
- for name, content in input_dict.items():
- if not isinstance(content, str):
- continue
- text_content = TextContent(text=content)
- stuff_dict[name] = StuffFactory.make_stuff(
- concept=get_required_concept(concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.TEXT),
- content=text_content,
- name=name,
- code="",
- )
- return WorkingMemory(root=stuff_dict)
-
@classmethod
def make_empty(cls) -> WorkingMemory:
return WorkingMemory(root={})
@@ -232,5 +160,4 @@ class WorkingMemoryFactory(BaseModel):
code=shortuuid.uuid()[:5],
)
working_memory.add_new_stuff(name=requirement.variable_name, stuff=fallback_stuff)
-
return working_memory
diff --git a/pipelex/core/pipe_errors.py b/pipelex/core/pipe_errors.py
deleted file mode 100644
index f94c6be8b..000000000
--- a/pipelex/core/pipe_errors.py
+++ /dev/null
@@ -1,37 +0,0 @@
-from pipelex.system.exceptions import RootException
-
-
-# TODO: add details from all cases raising this error
-class PipeDefinitionError(RootException):
- def __init__(
- self,
- message: str,
- domain_code: str | None = None,
- pipe_code: str | None = None,
- description: str | None = None,
- source: str | None = None,
- ):
- self.domain_code = domain_code
- self.pipe_code = pipe_code
- self.description = description
- self.source = source
- message = message + " • " + self.pipe_details()
- super().__init__(message)
-
- def pipe_details(self) -> str:
- if not self.domain_code and not self.pipe_code and not self.description and not self.source:
- return "No pipe details provided"
- details = "Pipe details:"
- if self.domain_code:
- details += f" • domain='{self.domain_code}'"
- if self.pipe_code:
- details += f" • pipe='{self.pipe_code}'"
- if self.description:
- details += f" • description='{self.description}'"
- if self.source:
- details += f" • source='{self.source}'"
- return details
-
-
-class UnexpectedPipeDefinitionError(PipeDefinitionError):
- pass
diff --git a/pipelex/core/pipes/exceptions.py b/pipelex/core/pipes/exceptions.py
index d48eff3bf..9b4d55227 100644
--- a/pipelex/core/pipes/exceptions.py
+++ b/pipelex/core/pipes/exceptions.py
@@ -1,7 +1,6 @@
-from pydantic import BaseModel, Field
from typing_extensions import override
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.cogt.extract.extract_setting import ExtractModelChoice
from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice
from pipelex.cogt.llm.llm_setting import LLMModelChoice
@@ -9,49 +8,15 @@ from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.types import StrEnum
-class PipeBlueprintValueError(ValueError):
+class PipeFactoryError(PipelexError):
pass
-class PipeInputNotFoundError(PipelexException):
+class PipeVariableMultiplicityError(ValueError):
pass
-class PipeFactoryError(PipelexException):
- pass
-
-
-class PipeInputError(PipelexException):
- def __init__(self, message: str, pipe_code: str, variable_name: str, concept_code: str | None = None):
- self.pipe_code = pipe_code
- self.variable_name = variable_name
- self.concept_code = concept_code
- super().__init__(message)
-
-
-class PipeRunInputsError(PipelexException):
- def __init__(self, message: str, pipe_code: str, missing_inputs: dict[str, str]):
- self.pipe_code = pipe_code
- self.missing_inputs = missing_inputs
- super().__init__(message)
-
-
-class PipeDefinitionErrorData(BaseModel):
- message: str = Field(description="The error message")
- domain_code: str | None = Field(None, description="The domain code")
- pipe_code: str | None = Field(None, description="The pipe code")
- description: str | None = Field(None, description="Description of the pipe")
- source: str | None = Field(None, description="Source of the error")
-
-
-class StaticValidationErrorType(StrEnum):
- MISSING_INPUT_VARIABLE = "missing_input_variable"
- EXTRANEOUS_INPUT_VARIABLE = "extraneous_input_variable"
- INADEQUATE_INPUT_CONCEPT = "inadequate_input_concept"
- TOO_MANY_CANDIDATE_INPUTS = "too_many_candidate_inputs"
-
-
-class PipeOperatorModelChoiceError(PipelexException):
+class PipeOperatorModelChoiceError(PipelexError):
def __init__(
self,
message: str,
@@ -84,3 +49,66 @@ class PipeOperatorModelChoiceError(PipelexException):
@override
def __str__(self) -> str:
return self.desc()
+
+
+class PipeValidationErrorType(StrEnum):
+ """Types of pipe validation errors.
+
+ These error types are raised during pipe validation from Pipe/Concept classes.
+ Only some are auto-fixed in the builder loop (marked below).
+ """
+
+ # Errors that are auto-fixed in builder_loop.py
+ MISSING_INPUT_VARIABLE = "missing_input_variable" # AUTO-FIXED
+ EXTRANEOUS_INPUT_VARIABLE = "extraneous_input_variable" # AUTO-FIXED
+ INPUT_REQUIREMENT_MISMATCH = "input_requirement_mismatch" # AUTO-FIXED
+ INADEQUATE_OUTPUT_CONCEPT = "inadequate_output_concept" # AUTO-FIXED
+
+ CIRCULAR_DEPENDENCY_ERROR = "circular_dependency_error"
+
+ # Errors that are raised but NOT auto-fixed (will fail validation)
+ LLM_OUTPUT_CANNOT_BE_IMAGE = "llm_output_cannot_be_image"
+ IMG_GEN_INPUT_NOT_TEXT_COMPATIBLE = "img_gen_input_not_text_compatible"
+
+ # Generic fallback for unexpected validation errors
+ UNKNOWN_VALIDATION_ERROR = "unknown_validation_error"
+
+
+class PipeValidationError(ValueError):
+ def __init__(
+ self,
+ message: str,
+ error_type: PipeValidationErrorType | None = None,
+ domain: str | None = None,
+ pipe_code: str | None = None,
+ variable_names: list[str] | None = None,
+ required_concept_codes: list[str] | None = None,
+ provided_concept_code: str | None = None,
+ file_path: str | None = None,
+ explanation: str | None = None,
+ ):
+ self.error_type = error_type
+ self.domain = domain
+ self.pipe_code = pipe_code
+ self.variable_names = variable_names
+ self.required_concept_codes = required_concept_codes
+ self.provided_concept_code = provided_concept_code
+ self.file_path = file_path
+ self.explanation = explanation
+ super().__init__(message)
+
+ def desc(self) -> str:
+ msg = f"{self.error_type} • domain='{self.domain}'"
+ if self.pipe_code:
+ msg += f" • pipe='{self.pipe_code}'"
+ if self.variable_names:
+ msg += f" • variable='{self.variable_names}'"
+ if self.required_concept_codes:
+ msg += f" • required_concept_codes='{self.required_concept_codes}'"
+ if self.provided_concept_code:
+ msg += f" • provided_concept_code='{self.provided_concept_code}'"
+ if self.file_path:
+ msg += f" • file='{self.file_path}'"
+ if self.explanation:
+ msg += f" • explanation='{self.explanation}'"
+ return msg
diff --git a/pipelex/core/pipes/handle_pipe_errors.py b/pipelex/core/pipes/handle_pipe_errors.py
new file mode 100644
index 000000000..a80979231
--- /dev/null
+++ b/pipelex/core/pipes/handle_pipe_errors.py
@@ -0,0 +1,165 @@
+from enum import Enum
+
+from pydantic import ValidationError
+from pydantic_core import ErrorDetails
+
+from pipelex.core.exceptions import PipesAndConceptValidationErrorData
+from pipelex.core.interpreter.validation_error_categorizer import ErrorCatKey
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+
+
+class ModelScope(str, Enum):
+ """Indicates which type of model is being validated."""
+
+ PIPE = "pipe"
+ CONCEPT = "concept"
+
+
+def categorize_pipe_validation_error(
+ validation_error: ValidationError,
+) -> list[PipesAndConceptValidationErrorData]:
+ """Categorize all errors from a ValidationError for Pipe instantiation.
+
+ This function determines the model scope (PIPE or CONCEPT) from the ValidationError
+ and processes all errors accordingly.
+
+ Args:
+ validation_error: Pydantic ValidationError from Pipe model validation
+
+ Returns:
+ List of PipesAndConceptValidationErrorData for all errors
+ """
+ # Determine model scope by examining field names in errors
+ # Pipe-specific fields: type, inputs, output, pipe_category
+ # Concept-specific fields: structure_class_name, refines
+ errors = validation_error.errors()
+
+ model_scope = ModelScope.PIPE
+ if errors:
+ # Check first error's field name to determine model type
+ first_error = errors[0]
+ loc = first_error.get(ErrorCatKey.LOC.value, ())
+ # TODO: Refactor how to determine model scope (use error codes)
+ if loc:
+ field_name = str(loc[0])
+ # Concept-specific fields
+ if field_name in ("structure_class_name", "refines"):
+ model_scope = ModelScope.CONCEPT
+ # Pipe-specific fields
+ elif field_name in ("type", "inputs", "output", "pipe_category"):
+ model_scope = ModelScope.PIPE
+ # Ambiguous fields - try to infer from error message
+ else:
+ error_msg = str(validation_error).lower()
+ if "concept" in error_msg and "pipe" not in error_msg:
+ model_scope = ModelScope.CONCEPT
+
+ # Process all errors
+ categorized_errors: list[PipesAndConceptValidationErrorData] = []
+ for error in errors:
+ if model_scope == ModelScope.PIPE:
+ categorized_error = _handle_pipe_errors(
+ error=error,
+ pipe_code=None, # We don't have the code at this point
+ )
+ else:
+ loc = error.get(ErrorCatKey.LOC.value, ())
+ message = error.get(ErrorCatKey.MSG.value, "Unknown validation error")
+ field_path_str = " → ".join(str(item) for item in loc)
+ unknown_field_name = str(loc[0]) if len(loc) >= 1 else None
+
+ categorized_error = PipesAndConceptValidationErrorData(
+ domain=None,
+ source=None,
+ pipe_code=None,
+ concept_code=None, # We don't have the code at this point
+ field_name=unknown_field_name,
+ error_type=PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR,
+ message=message,
+ field_path=field_path_str,
+ variable_names=None,
+ )
+ categorized_errors.append(categorized_error)
+ return categorized_errors
+
+
+def _handle_pipe_errors(
+ error: ErrorDetails,
+ pipe_code: str | None,
+) -> PipesAndConceptValidationErrorData:
+ """Handle all PIPE validation errors.
+
+ Extracts all necessary context from error and categorizes the error type.
+
+ Args:
+ error: Pydantic error details
+ pipe_code: The pipe code being validated (if known)
+
+ Returns:
+ PipesAndConceptValidationErrorData with all context populated
+ """
+ # Extract data from error
+ loc = error.get(ErrorCatKey.LOC.value, ())
+ message = error.get(ErrorCatKey.MSG.value, "Unknown validation error")
+ pydantic_type = error.get(ErrorCatKey.TYPE.value, "")
+ field_path = " → ".join(str(item) for item in loc)
+
+ # Extract field_name from loc[0] (direct field on Pipe model)
+ field_name = str(loc[0]) if len(loc) >= 1 else None
+
+ # Extract variable names from loc (for input/output errors)
+ variable_names = [str(item) for item in loc]
+
+ # Determine error type - default to UNKNOWN_VALIDATION_ERROR
+ error_type = PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
+
+ # Try to categorize based on error message patterns
+ message_lower = message.lower()
+
+ if "missing" in message_lower or "required" in pydantic_type:
+ # Could be MISSING_INPUT_VARIABLE but we can't tell for sure from generic Pydantic errors
+ # The specific error type should come from PipeValidationError exceptions instead
+ error_type = PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
+ elif "extra" in message_lower or "forbidden" in pydantic_type:
+ error_type = PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
+
+ return PipesAndConceptValidationErrorData(
+ domain=None,
+ source=None,
+ pipe_code=pipe_code,
+ concept_code=None,
+ field_name=field_name,
+ error_type=error_type,
+ message=message,
+ field_path=field_path,
+ variable_names=variable_names,
+ )
+
+
+def categorize_pipe_validation_with_libraries_error(
+ pipe_error: PipeValidationError,
+) -> PipesAndConceptValidationErrorData:
+ """Categorize a PipeValidationError with libraries and create structured error data.
+
+ Args:
+ pipe_error: PipeValidationError with libraries and structured error information
+
+ Returns:
+ PipesAndConceptValidationErrorData with all relevant fields populated
+ """
+ message = pipe_error.explanation or str(pipe_error)
+ if pipe_error.required_concept_codes and pipe_error.provided_concept_code:
+ message += f" (required: {pipe_error.required_concept_codes}, provided: {pipe_error.provided_concept_code})"
+
+ error_type = pipe_error.error_type or PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR
+ return PipesAndConceptValidationErrorData(
+ error_type=error_type,
+ domain=pipe_error.domain,
+ source=pipe_error.file_path or None,
+ pipe_code=pipe_error.pipe_code,
+ concept_code=None, # This is a pipe error, not a concept error
+ field_name=None, # Field name is not provided in PipeValidationError
+ message=message,
+ field_path=pipe_error.file_path or "",
+ variable_names=pipe_error.variable_names,
+ )
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/__init__.py b/pipelex/core/pipes/inputs/__init__.py
similarity index 100%
rename from tests/test_pipelines/misc_tests/subfolder_2/subfolder/__init__.py
rename to pipelex/core/pipes/inputs/__init__.py
diff --git a/pipelex/core/pipes/inputs/exceptions.py b/pipelex/core/pipes/inputs/exceptions.py
new file mode 100644
index 000000000..ea418c91e
--- /dev/null
+++ b/pipelex/core/pipes/inputs/exceptions.py
@@ -0,0 +1,20 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeRunInputsError(PipelexError):
+ def __init__(self, message: str, pipe_code: str, missing_inputs: dict[str, str]):
+ self.pipe_code = pipe_code
+ self.missing_inputs = missing_inputs
+ super().__init__(message)
+
+
+class PipeInputError(PipelexError):
+ def __init__(self, message: str, pipe_code: str, variable_name: str, concept_code: str | None = None):
+ self.pipe_code = pipe_code
+ self.variable_name = variable_name
+ self.concept_code = concept_code
+ super().__init__(message)
+
+
+class PipeInputNotFoundError(PipelexError):
+ pass
diff --git a/pipelex/core/pipes/input_requirements.py b/pipelex/core/pipes/inputs/input_requirements.py
similarity index 96%
rename from pipelex/core/pipes/input_requirements.py
rename to pipelex/core/pipes/inputs/input_requirements.py
index b733aeaec..5f4eb61f5 100644
--- a/pipelex/core/pipes/input_requirements.py
+++ b/pipelex/core/pipes/inputs/input_requirements.py
@@ -4,7 +4,7 @@ from pydantic import BaseModel, Field, RootModel, field_validator
from pipelex import log
from pipelex.core.concepts.concept import Concept
-from pipelex.core.pipes.exceptions import PipeInputNotFoundError
+from pipelex.core.pipes.inputs.exceptions import PipeInputNotFoundError
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -77,6 +77,9 @@ class InputRequirements(RootModel[InputRequirementsRoot]):
raise PipeInputNotFoundError(msg)
return requirement
+ def is_variable_existing(self, variable_name: str) -> bool:
+ return variable_name in self.root
+
def add_requirement(self, variable_name: str, concept: Concept, multiplicity: VariableMultiplicity | None = None):
self.root[variable_name] = InputRequirement(concept=concept, multiplicity=multiplicity)
diff --git a/pipelex/core/pipes/input_requirements_factory.py b/pipelex/core/pipes/inputs/input_requirements_factory.py
similarity index 94%
rename from pipelex/core/pipes/input_requirements_factory.py
rename to pipelex/core/pipes/inputs/input_requirements_factory.py
index d5d82a12a..ed04bf06a 100644
--- a/pipelex/core/pipes/input_requirements_factory.py
+++ b/pipelex/core/pipes/inputs/input_requirements_factory.py
@@ -1,18 +1,18 @@
import re
from typing import TYPE_CHECKING
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.concepts.validation import validate_concept_string_or_code
-from pipelex.core.pipes.input_requirements import InputRequirement, InputRequirements
+from pipelex.core.pipes.inputs.input_requirements import InputRequirement, InputRequirements
from pipelex.hub import get_required_concept
if TYPE_CHECKING:
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
-class InputRequirementsFactoryError(PipelexException):
+class InputRequirementsFactoryError(PipelexError):
pass
@@ -56,6 +56,7 @@ class InputRequirementsFactory:
Args:
domain: The domain to use for resolving concept codes without domain prefix
requirement_str: String in the format "domain.ConceptCode" or "ConceptCode" with optional "[multiplicity]"
+ library_id: The library ID to use for resolving concepts
concept_codes_from_the_same_domain: List of concept codes from the same domain for resolution
Returns:
diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.py
index f59baaf88..b5eaaad7c 100644
--- a/pipelex/core/pipes/pipe_abstract.py
+++ b/pipelex/core/pipes/pipe_abstract.py
@@ -1,19 +1,23 @@
from abc import ABC, abstractmethod
from typing import Any, final
-from pydantic import BaseModel, ConfigDict, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
-from pipelex.cogt.exceptions import ModelChoiceNotFoundError
+from pipelex import log
from pipelex.core.concepts.concept import Concept
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.pipe_blueprint import PipeCategory, PipeType
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.exceptions import PipeStackOverflowError
from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.tools.misc.string_utils import is_snake_case
+from pipelex.types import Self
class PipeAbstract(ABC, BaseModel):
@@ -34,29 +38,180 @@ class PipeAbstract(ABC, BaseModel):
@field_validator("code", mode="before")
@classmethod
def validate_pipe_code_syntax(cls, code: str) -> str:
- PipeBlueprint.validate_pipe_code_syntax(pipe_code=code)
+ if not is_snake_case(code):
+ msg = f"Invalid pipe code syntax '{code}'. Must be in snake_case."
+ raise ValueError(msg)
return code
- @abstractmethod
- def validate_output(self):
- """Validate the output for the pipe."""
+ @field_validator("type", mode="after")
+ @classmethod
+ def validate_pipe_type(cls, value: Any) -> Any:
+ if value not in PipeType.value_list():
+ msg = f"Invalid pipe type '{value}' for pipe '{cls.code}'. Must be one of: {PipeType.value_list()}"
+ raise ValueError(msg)
+ return value
+
+ @field_validator("pipe_category", mode="after")
+ @classmethod
+ def validate_pipe_category(cls, value: Any) -> Any:
+ if value not in PipeCategory.value_list():
+ msg = f"Invalid pipe category '{value}' for pipe '{cls.code}'. Must be one of: {PipeCategory.value_list()}"
+ raise ValueError(msg)
+ return value
+
+ @model_validator(mode="after")
+ def validate_pipe_category_based_on_type(self) -> Self:
+ try:
+ pipe_type = PipeType(self.type)
+ except ValueError as exc:
+ # If type is invalid, it should have been caught by the field validator
+ # but we handle it gracefully here
+ msg = f"Invalid pipe type '{self.type}' for pipe '{self.code}'. Must be one of: {PipeType.value_list()}"
+ raise ValueError(msg) from exc
+
+ if self.pipe_category != pipe_type.category:
+ msg = (
+ f"Inconsistency detected in pipe '{self.code}': pipe_category '{self.pipe_category}' "
+ f"does not match the expected category '{pipe_type.category}' for type '{self.type}'"
+ )
+ raise ValueError(msg)
+ return self
+
+ @model_validator(mode="after")
+ def validate_pipe(self) -> Self:
+ self.generic_validate_inputs_static()
+ self.generic_validate_output_static()
+ return self
@final
def validate_with_libraries(self):
- """Validate the pipe with the libraries, after the static validation"""
- try:
- self._validate_with_libraries()
- except ModelChoiceNotFoundError as exc:
- raise PipeOperatorModelChoiceError(
- message=exc.message,
- pipe_type=self.pipe_type,
- pipe_code=self.code,
- model_type=exc.model_type,
- model_choice=exc.model_choice,
- ) from exc
-
- def _validate_with_libraries(self):
- """Validate the pipe with the libraries, after the static validation"""
+ self.generic_validate_inputs_with_library()
+ self.generic_validate_output_with_library()
+
+ @final
+ def generic_validate_inputs_static(self):
+ self.validate_inputs_static()
+
+ @final
+ def generic_validate_output_static(self):
+ self.validate_output_static()
+
+ @final
+ def generic_validate_inputs_with_library(self):
+ # First validate required variables are in the inputs
+ for required_variable_name in self.required_variables():
+ if required_variable_name not in self.inputs.variables:
+ msg = f"Required variable '{required_variable_name}' is not in the inputs of pipe '{self.code}'. Current inputs: {self.inputs}"
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.MISSING_INPUT_VARIABLE,
+ domain=self.domain,
+ pipe_code=self.code,
+ variable_names=[required_variable_name],
+ )
+
+ # Then validate that all inputs are actually needed and match requirements exactly
+ the_needed_inputs = self.needed_inputs()
+
+ # Check all required variables are in the inputs and match the required InputRequirement
+ for named_input_requirement in the_needed_inputs.named_input_requirements:
+ var_name = named_input_requirement.variable_name
+
+ if var_name not in self.inputs.variables:
+ msg = f"Required variable '{var_name}' is not in the inputs of pipe '{self.code}'. Current inputs: {self.inputs}"
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.MISSING_INPUT_VARIABLE,
+ domain=self.domain,
+ pipe_code=self.code,
+ variable_names=[var_name],
+ )
+
+ # TODO: add this to the PipeController validation. (This might need to refactor a little bit how we can override the validation)
+ if PipeCategory.is_controller_by_str(self.pipe_category):
+ # Compare the essential parts of InputRequirement (concept code + multiplicity)
+ # Skip validation if the needed requirement is Dynamic or Anything (flexible output types)
+ declared_requirement = self.inputs.root[var_name]
+ needed_requirement = the_needed_inputs.root[named_input_requirement.requirement_expression or var_name]
+
+ # Allow mismatch if the needed requirement is a flexible type (Dynamic or Anything)
+ if (
+ needed_requirement.concept.code not in (NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING)
+ and declared_requirement != needed_requirement
+ ):
+ # Identify the specific mismatched field(s)
+ mismatch_details: list[str] = []
+ if declared_requirement.concept != needed_requirement.concept:
+ mismatch_details.append(f"concept: declared='{declared_requirement.concept}' vs required='{needed_requirement.concept}'")
+ if declared_requirement.multiplicity != needed_requirement.multiplicity:
+ mismatch_details.append(
+ f"multiplicity: declared='{declared_requirement.multiplicity}' vs required='{needed_requirement.multiplicity}'"
+ )
+
+ mismatch_summary = ", ".join(mismatch_details)
+ msg = (
+ f"In the pipe '{self.code}', the input variable '{var_name}' has a requirement mismatch.\n"
+ f"Mismatched field(s): {mismatch_summary}\n"
+ f"Declared: {declared_requirement}\n"
+ f"Required: {needed_requirement}"
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INPUT_REQUIREMENT_MISMATCH,
+ domain=self.domain,
+ pipe_code=self.code,
+ variable_names=[var_name],
+ )
+
+ # Check that all declared inputs are actually needed
+ for input_name in self.inputs.variables:
+ if input_name not in the_needed_inputs.required_names:
+ msg = f"Extraneous input '{input_name}' found in the inputs of pipe {self.code}"
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
+ domain=self.domain,
+ pipe_code=self.code,
+ variable_names=[input_name],
+ )
+
+ self.validate_inputs_with_library()
+
+ @final
+ def generic_validate_output_with_library(self):
+ self.validate_output_with_library()
+
+ @abstractmethod
+ def validate_inputs_with_library(self):
+ pass
+
+ @abstractmethod
+ def validate_inputs_static(self):
+ pass
+
+ @abstractmethod
+ def validate_output_with_library(self):
+ pass
+
+ @abstractmethod
+ def validate_output_static(self):
+ pass
+
+ async def validate_before_run(
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ ): ...
+
+ async def validate_after_run(
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ ): ...
@abstractmethod
def required_variables(self) -> set[str]:
@@ -82,31 +237,6 @@ class PipeAbstract(ABC, BaseModel):
"""
- def pipe_dependencies(self) -> set[str]:
- """Return the pipes that are dependencies of the pipe.
- - PipeBatch: The pipe that is being batched
- - PipeCondition: The pipes in the outcome_map
- - PipeSequence: The pipes in the steps
- """
- return set()
-
- def concept_dependencies(self) -> list[Concept]:
- required_concepts: list[Concept] = [self.output]
- required_concepts.extend(self.inputs.concepts)
- required_concepts.append(self.output)
- return required_concepts
-
- @abstractmethod
- async def run_pipe(
- self,
- job_metadata: JobMetadata,
- working_memory: WorkingMemory,
- pipe_run_params: PipeRunParams,
- output_name: str | None = None,
- print_intermediate_outputs: bool | None = False,
- ) -> PipeOutput:
- pass
-
def monitor_pipe_stack(self, pipe_run_params: PipeRunParams):
pipe_stack = pipe_run_params.pipe_stack
limit = pipe_run_params.pipe_stack_limit
@@ -130,5 +260,60 @@ class PipeAbstract(ABC, BaseModel):
arrow = "[yellow]→[/yellow]"
return f"{indent}{pipe_type_label} {pipe_code_label} {arrow} {concept_code_label}"
+ @abstractmethod
+ async def _run_pipe(
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ ) -> PipeOutput:
+ pass
+
+ @final
+ async def run_pipe(
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ ) -> PipeOutput:
+ pipe_run_params.push_pipe_to_stack(pipe_code=self.code)
+ self.monitor_pipe_stack(pipe_run_params=pipe_run_params)
+
+ updated_metadata = JobMetadata(
+ pipe_job_ids=[self.code],
+ )
+ job_metadata.update(updated_metadata=updated_metadata)
+
+ # check we have the required inputs in the working memory
+ missing_inputs: dict[str, str] = {}
+ for required_stuff_name, requirement in self.needed_inputs().items:
+ if not working_memory.is_stuff_exists(name=required_stuff_name):
+ missing_inputs[required_stuff_name] = requirement.concept.code
+ if missing_inputs:
+ raise PipeRunInputsError(
+ message=f"Missing required inputs for pipe '{self.code}': {missing_inputs}", pipe_code=self.code, missing_inputs=missing_inputs
+ )
+
+ pipe_run_info = self._format_pipe_run_info(pipe_run_params=pipe_run_params)
+ if pipe_run_params.run_mode == PipeRunMode.LIVE:
+ log.info(pipe_run_info)
+
+ await self.validate_before_run(
+ job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ )
+
+ pipe_output = await self._run_pipe(
+ job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ )
+
+ await self.validate_after_run(
+ job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ )
+
+ pipe_run_params.pop_pipe_from_stack(pipe_code=self.code)
+ return pipe_output
+
PipeAbstractType = type[PipeAbstract]
diff --git a/pipelex/core/pipes/pipe_blueprint.py b/pipelex/core/pipes/pipe_blueprint.py
index 701979416..26a1a24f7 100644
--- a/pipelex/core/pipes/pipe_blueprint.py
+++ b/pipelex/core/pipes/pipe_blueprint.py
@@ -1,18 +1,17 @@
import re
-from typing import Any
+from abc import ABC
+from typing import Any, final
-from pydantic import BaseModel, ConfigDict, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.concepts.validation import validate_concept_string_or_code
-from pipelex.core.pipes.exceptions import PipeBlueprintValueError
from pipelex.core.pipes.validation import validate_input_name
-from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN, parse_concept_with_multiplicity
-from pipelex.tools.misc.string_utils import is_snake_case
-from pipelex.types import StrEnum
+from pipelex.core.pipes.variable_multiplicity import MUTLIPLICITY_PATTERN, PipeVariableMultiplicityError, parse_concept_with_multiplicity
+from pipelex.types import Self, StrEnum
-class AllowedPipeCategories(StrEnum):
+class PipeCategory(StrEnum):
PIPE_OPERATOR = "PipeOperator"
PIPE_CONTROLLER = "PipeController"
@@ -23,9 +22,9 @@ class AllowedPipeCategories(StrEnum):
@property
def is_controller(self) -> bool:
match self:
- case AllowedPipeCategories.PIPE_CONTROLLER:
+ case PipeCategory.PIPE_CONTROLLER:
return True
- case AllowedPipeCategories.PIPE_OPERATOR:
+ case PipeCategory.PIPE_OPERATOR:
return False
@classmethod
@@ -37,7 +36,7 @@ class AllowedPipeCategories(StrEnum):
return False
-class AllowedPipeTypes(StrEnum):
+class PipeType(StrEnum):
# Pipe Operators
PIPE_FUNC = "PipeFunc"
PIPE_IMG_GEN = "PipeImgGen"
@@ -55,29 +54,29 @@ class AllowedPipeTypes(StrEnum):
return list(cls)
@property
- def category(self) -> AllowedPipeCategories:
+ def category(self) -> PipeCategory:
match self:
- case AllowedPipeTypes.PIPE_FUNC:
- return AllowedPipeCategories.PIPE_OPERATOR
- case AllowedPipeTypes.PIPE_IMG_GEN:
- return AllowedPipeCategories.PIPE_OPERATOR
- case AllowedPipeTypes.PIPE_COMPOSE:
- return AllowedPipeCategories.PIPE_OPERATOR
- case AllowedPipeTypes.PIPE_LLM:
- return AllowedPipeCategories.PIPE_OPERATOR
- case AllowedPipeTypes.PIPE_EXTRACT:
- return AllowedPipeCategories.PIPE_OPERATOR
- case AllowedPipeTypes.PIPE_BATCH:
- return AllowedPipeCategories.PIPE_CONTROLLER
- case AllowedPipeTypes.PIPE_CONDITION:
- return AllowedPipeCategories.PIPE_CONTROLLER
- case AllowedPipeTypes.PIPE_PARALLEL:
- return AllowedPipeCategories.PIPE_CONTROLLER
- case AllowedPipeTypes.PIPE_SEQUENCE:
- return AllowedPipeCategories.PIPE_CONTROLLER
-
-
-class PipeBlueprint(BaseModel):
+ case PipeType.PIPE_FUNC:
+ return PipeCategory.PIPE_OPERATOR
+ case PipeType.PIPE_IMG_GEN:
+ return PipeCategory.PIPE_OPERATOR
+ case PipeType.PIPE_COMPOSE:
+ return PipeCategory.PIPE_OPERATOR
+ case PipeType.PIPE_LLM:
+ return PipeCategory.PIPE_OPERATOR
+ case PipeType.PIPE_EXTRACT:
+ return PipeCategory.PIPE_OPERATOR
+ case PipeType.PIPE_BATCH:
+ return PipeCategory.PIPE_CONTROLLER
+ case PipeType.PIPE_CONDITION:
+ return PipeCategory.PIPE_CONTROLLER
+ case PipeType.PIPE_PARALLEL:
+ return PipeCategory.PIPE_CONTROLLER
+ case PipeType.PIPE_SEQUENCE:
+ return PipeCategory.PIPE_CONTROLLER
+
+
+class PipeBlueprint(ABC, BaseModel):
model_config = ConfigDict(extra="forbid")
source: str | None = None
pipe_category: Any = Field(exclude=True) # Technical field for Union discrimination, not user-facing
@@ -86,6 +85,14 @@ class PipeBlueprint(BaseModel):
inputs: dict[str, str] | None = None
output: str
+ @property
+ def nb_inputs(self) -> int:
+ return len(self.inputs) if self.inputs else 0
+
+ @property
+ def input_names(self) -> list[str]:
+ return list(self.inputs.keys()) if self.inputs else []
+
@property
def pipe_dependencies(self) -> set[str]:
"""Return the set of pipe codes that this pipe depends on.
@@ -110,66 +117,97 @@ class PipeBlueprint(BaseModel):
"""
return None
- @field_validator("inputs", mode="after")
- @classmethod
- def validate_inputs(cls, inputs: dict[str, str] | None) -> dict[str, str] | None:
- if inputs is None:
- return None
-
- # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
- multiplicity_pattern = MUTLIPLICITY_PATTERN
-
- for input_name, concept_spec in inputs.items():
- validate_input_name(input_name)
-
- # Validate the concept spec format with optional multiplicity brackets
- match = re.match(multiplicity_pattern, concept_spec)
- if not match:
- msg = (
- f"Invalid input syntax for '{input_name}': '{concept_spec}'. "
- f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."
- )
- raise PipeBlueprintValueError(msg)
-
- # Extract the concept part (without multiplicity) and validate it
- concept_string_or_code = match.group(1)
- validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)
-
- return inputs
-
@field_validator("type", mode="after")
@classmethod
def validate_pipe_type(cls, value: Any) -> Any:
- """Validate that the pipe type is one of the allowed values."""
- if value not in AllowedPipeTypes.value_list():
- msg = f"Invalid pipe type '{value}'. Must be one of: {AllowedPipeTypes.value_list()}"
- raise PipeBlueprintValueError(msg)
+ if value not in PipeType.value_list():
+ msg = f"Invalid pipe type '{value}'. Must be one of: {PipeType.value_list()}"
+ raise ValueError(msg)
return value
@field_validator("pipe_category", mode="after")
@classmethod
def validate_pipe_category(cls, value: Any) -> Any:
- """Validate that the pipe category is one of the allowed values."""
- if value not in AllowedPipeCategories.value_list():
- msg = f"Invalid pipe category '{value}'. Must be one of: {AllowedPipeCategories.value_list()}"
- raise PipeBlueprintValueError(msg)
+ if value not in PipeCategory.value_list():
+ msg = f"Invalid pipe category '{value}'. Must be one of: {PipeCategory.value_list()}"
+ raise ValueError(msg)
return value
- @field_validator("output", mode="before")
- @classmethod
- def validate_output(cls, output: str) -> str:
+ @model_validator(mode="after")
+ def validate_pipe_category_based_on_type(self) -> Self:
+ try:
+ pipe_type = PipeType(self.type)
+ except ValueError as exc:
+ # If type is invalid, it should have been caught by the field validator
+ # but we handle it gracefully here
+ msg = f"Invalid pipe type '{self.type}'. Must be one of: {PipeType.value_list()}"
+ raise ValueError(msg) from exc
+
+ if self.pipe_category != pipe_type.category:
+ msg = (
+ f"Inconsistency detected: pipe_category '{self.pipe_category}' does not match the "
+ f"expected category '{pipe_type.category}' for type '{self.type}'"
+ )
+ raise ValueError(msg)
+ return self
+
+ @model_validator(mode="after")
+ def validate_inputs_blueprint(self) -> Self:
+ self.generic_validate_inputs()
+ self.generic_validate_output()
+ return self
+
+ def validate_inputs(self):
+ pass
+
+ def validate_output(self):
+ pass
+
+ @final
+ def generic_validate_inputs(self):
+ if self.inputs:
+ for input_name, concept_spec in self.inputs.items():
+ validate_input_name(input_name)
+
+ # Validate the concept spec format with optional multiplicity brackets
+ # Pattern allows: ConceptName, domain.ConceptName, ConceptName[], ConceptName[N]
+ match = re.match(MUTLIPLICITY_PATTERN, concept_spec)
+ if not match:
+ msg = (
+ f"Invalid input syntax for '{input_name}': '{concept_spec}'. "
+ f"Expected format: 'ConceptName', 'ConceptName[]', or 'ConceptName[N]' where N is an integer."
+ )
+ raise ValueError(msg)
+
+ # Extract the concept part (without multiplicity) and validate it
+ concept_string_or_code = match.group(1)
+ try:
+ validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)
+ except ConceptStringError as exc:
+ msg = f"Invalid concept string or code '{concept_string_or_code}' when trying to validate the input of a pipe blueprint: {exc}"
+ raise ValueError(msg) from exc
+
+ # Check that every input_name is unique
+ input_names = list(self.inputs.keys())
+ if len(input_names) != len(set(input_names)):
+ duplicates = [name for name in input_names if input_names.count(name) > 1]
+ msg = f"Duplicate input names found: {duplicates}. Input names must be unique."
+ raise ValueError(msg)
+
+ self.validate_inputs()
+
+ @final
+ def generic_validate_output(self):
# Strip multiplicity brackets before validating
- output_parse_result = parse_concept_with_multiplicity(output)
+ try:
+ output_parse_result = parse_concept_with_multiplicity(self.output)
+ except PipeVariableMultiplicityError as exc:
+ msg = f"Invalid concept specification syntax: '{self.output}'. {exc}"
+ raise ValueError(msg) from exc
try:
validate_concept_string_or_code(concept_string_or_code=output_parse_result.concept)
except ConceptStringError as exc:
msg = f"Invalid concept string '{output_parse_result.concept}' when trying to validate the output of a pipe blueprint: {exc}"
- raise PipeBlueprintValueError(msg) from exc
- return output
+ raise ValueError(msg) from exc
- @classmethod
- def validate_pipe_code_syntax(cls, pipe_code: str) -> str:
- if not is_snake_case(pipe_code):
- msg = f"Invalid pipe code syntax '{pipe_code}'. Must be in snake_case."
- raise PipeBlueprintValueError(msg)
- return pipe_code
+ self.validate_output()
diff --git a/pipelex/core/pipes/pipe_factory.py b/pipelex/core/pipes/pipe_factory.py
index e2dabc5d6..5e1a52943 100644
--- a/pipelex/core/pipes/pipe_factory.py
+++ b/pipelex/core/pipes/pipe_factory.py
@@ -9,11 +9,11 @@ from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
PipeBlueprintType = TypeVar("PipeBlueprintType", bound="PipeBlueprint", contravariant=True)
-PipeType = TypeVar("PipeType", bound="PipeAbstract", covariant=True)
+PipeAbstractType = TypeVar("PipeAbstractType", bound="PipeAbstract", covariant=True)
@runtime_checkable
-class PipeFactoryProtocol(Protocol[PipeBlueprintType, PipeType]):
+class PipeFactoryProtocol(Protocol[PipeBlueprintType, PipeAbstractType]):
@classmethod
def make_from_blueprint(
cls,
@@ -21,7 +21,7 @@ class PipeFactoryProtocol(Protocol[PipeBlueprintType, PipeType]):
pipe_code: str,
blueprint: PipeBlueprintType,
concept_codes_from_the_same_domain: list[str] | None = None,
- ) -> PipeType: ...
+ ) -> PipeAbstractType: ...
class PipeFactory(PipeFactoryProtocol[PipeBlueprint, PipeAbstract]):
diff --git a/pipelex/core/pipes/validation.py b/pipelex/core/pipes/validation.py
index 64aef7d29..d52c26014 100644
--- a/pipelex/core/pipes/validation.py
+++ b/pipelex/core/pipes/validation.py
@@ -1,4 +1,3 @@
-from pipelex.core.pipes.exceptions import PipeBlueprintValueError
from pipelex.tools.misc.string_utils import is_snake_case
@@ -59,7 +58,7 @@ def validate_input_name(input_name: str) -> None:
input_name: The input name to validate
Raises:
- PipeBlueprintValueError: If the input name is invalid
+ ValueError: If the input name is invalid
"""
if not is_valid_input_name(input_name):
@@ -69,4 +68,4 @@ def validate_input_name(input_name: str) -> None:
"Nested field access is allowed using dots (e.g., 'my_input.field_name'), "
"where each part must also be in snake_case."
)
- raise PipeBlueprintValueError(msg)
+ raise ValueError(msg)
diff --git a/pipelex/core/pipes/variable_multiplicity.py b/pipelex/core/pipes/variable_multiplicity.py
index 88d3a07eb..b9d6ff3e1 100644
--- a/pipelex/core/pipes/variable_multiplicity.py
+++ b/pipelex/core/pipes/variable_multiplicity.py
@@ -4,6 +4,8 @@ import re
from pydantic import BaseModel, Field
+from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
+
VariableMultiplicity = bool | int
MUTLIPLICITY_PATTERN = r"^([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)?)(?:\[(\d*)\])?$"
@@ -88,7 +90,7 @@ def parse_concept_with_multiplicity(concept_spec: str) -> MultiplicityParseResul
f"'domain.ConceptName', 'domain.ConceptName[]', or 'domain.ConceptName[N]' "
f"where concept and domain names must start with a letter or underscore."
)
- raise ValueError(msg)
+ raise PipeVariableMultiplicityError(msg)
concept = match.group(1)
bracket_content = match.group(2)
@@ -105,3 +107,43 @@ def parse_concept_with_multiplicity(concept_spec: str) -> MultiplicityParseResul
multiplicity = int(bracket_content)
return MultiplicityParseResult(concept=concept, multiplicity=multiplicity)
+
+
+def format_concept_with_multiplicity(concept_code_or_string: str, multiplicity: VariableMultiplicity | None) -> str:
+ """Format a concept code or string with multiplicity notation.
+
+ This is the reverse operation of parse_concept_with_multiplicity.
+
+ Args:
+ concept_code_or_string: The concept code or string (e.g., "ConceptName" or "domain.ConceptName")
+ multiplicity: The multiplicity value:
+ - None: single item (no brackets)
+ - True: variable-length list (empty brackets [])
+ - int: fixed-length list (brackets with number [N])
+
+ Returns:
+ Formatted concept specification string with multiplicity notation
+
+ Examples:
+ >>> format_concept_with_multiplicity("Text", None)
+ "Text"
+ >>> format_concept_with_multiplicity("Text", True)
+ "Text[]"
+ >>> format_concept_with_multiplicity("Text", 3)
+ "Text[3]"
+ >>> format_concept_with_multiplicity("domain.Text", None)
+ "domain.Text"
+ >>> format_concept_with_multiplicity("domain.Text", True)
+ "domain.Text[]"
+ >>> format_concept_with_multiplicity("domain.Text", 5)
+ "domain.Text[5]"
+ """
+ if multiplicity is None:
+ # Single item - no brackets
+ return concept_code_or_string
+ elif multiplicity is True:
+ # Variable-length list - empty brackets
+ return f"{concept_code_or_string}[]"
+ else:
+ # Fixed-length list - brackets with number
+ return f"{concept_code_or_string}[{multiplicity}]"
diff --git a/pipelex/core/stuffs/exceptions.py b/pipelex/core/stuffs/exceptions.py
index 12e763462..60078562d 100644
--- a/pipelex/core/stuffs/exceptions.py
+++ b/pipelex/core/stuffs/exceptions.py
@@ -1,7 +1,7 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class StuffArtefactError(PipelexException):
+class StuffArtefactError(PipelexError):
pass
@@ -9,7 +9,7 @@ class StuffArtefactReservedFieldError(StuffArtefactError):
pass
-class StuffError(PipelexException):
+class StuffError(PipelexError):
pass
diff --git a/pipelex/core/stuffs/stuff_factory.py b/pipelex/core/stuffs/stuff_factory.py
index 7a2a43c6f..084cf1572 100644
--- a/pipelex/core/stuffs/stuff_factory.py
+++ b/pipelex/core/stuffs/stuff_factory.py
@@ -3,12 +3,11 @@ from typing import Any, cast
import shortuuid
from pydantic import BaseModel, ValidationError, field_validator
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.client.protocol import StuffContentOrData
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_library import ConceptLibraryConceptNotFoundError
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.validation import validate_concept_string
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.structured_content import StructuredContent
@@ -16,10 +15,11 @@ from pipelex.core.stuffs.stuff import DictStuff, Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import get_class_registry, get_concept_library, get_native_concept, get_required_concept
+from pipelex.libraries.concept.concept_library import ConceptLibraryConceptNotFoundError
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
-class StuffFactoryError(PipelexException):
+class StuffFactoryError(PipelexError):
pass
@@ -196,7 +196,9 @@ class StuffFactory:
content_class_name = type(first_item).__name__
# Check if it's a native concept
- if "Content" in content_class_name and NativeConceptCode.is_native_concept(concept_code=content_class_name.split("Content")[0]):
+ if "Content" in content_class_name and NativeConceptCode.is_native_concept_string_or_code(
+ concept_string_or_code=content_class_name.split("Content")[0]
+ ):
concept = get_native_concept(native_concept=NativeConceptCode(content_class_name.split("Content")[0]))
else:
try:
@@ -223,7 +225,9 @@ class StuffFactory:
content_class_name = stuff_content_or_data.__class__.__name__
# Check if it's a native concept
- if "Content" in content_class_name and NativeConceptCode.is_native_concept(concept_code=content_class_name.split("Content")[0]):
+ if "Content" in content_class_name and NativeConceptCode.is_native_concept_string_or_code(
+ concept_string_or_code=content_class_name.split("Content")[0]
+ ):
# It's a native concept like TextContent, ImageContent, etc.
concept = get_native_concept(native_concept=NativeConceptCode(content_class_name.split("Content")[0]))
else:
@@ -288,7 +292,9 @@ class StuffFactory:
raise StuffFactoryError(msg)
# Check if it's a native concept
- if "Content" in content_class_name and NativeConceptCode.is_native_concept(concept_code=content_class_name.split("Content")[0]):
+ if "Content" in content_class_name and NativeConceptCode.is_native_concept_string_or_code(
+ concept_string_or_code=content_class_name.split("Content")[0]
+ ):
concept = get_native_concept(native_concept=NativeConceptCode(content_class_name.split("Content")[0]))
else:
try:
@@ -487,7 +493,7 @@ class StuffFactory:
raise StuffFactoryError(msg)
-class StuffContentFactoryError(PipelexException):
+class StuffContentFactoryError(PipelexError):
pass
diff --git a/pipelex/dev_cli/commands/check_config_sync_cmd.py b/pipelex/dev_cli/commands/check_config_sync_cmd.py
deleted file mode 100644
index b43a7f70d..000000000
--- a/pipelex/dev_cli/commands/check_config_sync_cmd.py
+++ /dev/null
@@ -1,107 +0,0 @@
-"""Command to verify that .pipelex and pipelex/kit/configs are in sync."""
-
-from __future__ import annotations
-
-import sys
-from pathlib import Path
-
-from rich.panel import Panel
-
-from pipelex.hub import get_console
-from pipelex.tools.misc.diff import has_diff_dirs, make_diff_dirs_pretty
-from pipelex.types import StrEnum
-
-
-class LeadingConfig(StrEnum):
- """Enum for leading configuration options."""
-
- KIT = "kit"
- INSTALLED = "installed"
-
-
-def check_config_sync_cmd(
- show_diff: bool = True,
- leading: LeadingConfig = LeadingConfig.INSTALLED,
-) -> None:
- """Verify that .pipelex and pipelex/kit/configs are in sync.
-
- Args:
- show_diff: If True, display the differences when found
- leading: Which configuration is the leading (left) one in the diff.
- LeadingConfig.INSTALLED means .pipelex is the reference (default),
- LeadingConfig.KIT means pipelex/kit/configs is the reference
- """
- console = get_console()
-
- # Define the directories to compare
- pipelex_dir = Path(".pipelex")
- configs_dir = Path("pipelex/kit/configs")
-
- # Check if both directories exist
- if not pipelex_dir.exists():
- console.print()
- console.print("[red]✗[/red] Directory [cyan].pipelex[/cyan] does not exist")
- console.print()
- sys.exit(1)
-
- if not configs_dir.exists():
- console.print()
- console.print("[red]✗[/red] Directory [cyan]pipelex/kit/configs[/cyan] does not exist")
- console.print()
- sys.exit(1)
-
- # Determine directory order based on leading parameter
- match leading:
- case LeadingConfig.INSTALLED:
- left_dir = pipelex_dir
- right_dir = configs_dir
- left_label = ".pipelex"
- right_label = "pipelex/kit/configs"
- case LeadingConfig.KIT:
- left_dir = configs_dir
- right_dir = pipelex_dir
- left_label = "pipelex/kit/configs"
- right_label = ".pipelex"
-
- # Check for differences
- console.print()
- console.print("[bold]Checking config synchronization...[/bold]")
- console.print(f" Leading: [cyan]{left_label}[/cyan] ↔ [cyan]{right_label}[/cyan]")
- console.print()
-
- has_diff = has_diff_dirs(left_dir, right_dir)
-
- if not has_diff:
- # No differences found
- success_panel = Panel(
- "[green]✓[/green] Directories are in sync!\n\n[dim].pipelex and pipelex/kit/configs have no differences.[/dim]",
- title="[bold green]Config Sync Check: PASSED[/bold green]",
- border_style="green",
- padding=(1, 2),
- )
- console.print(success_panel)
- console.print()
- sys.exit(0)
-
- # Differences found
- error_panel = Panel(
- "[red]✗[/red] Directories are [bold]NOT[/bold] in sync!\n\n[dim].pipelex and pipelex/kit/configs have differences.[/dim]",
- title="[bold red]Config Sync Check: FAILED[/bold red]",
- border_style="red",
- padding=(1, 2),
- )
- console.print(error_panel)
- console.print()
-
- if show_diff:
- console.print()
- pretty_diff = make_diff_dirs_pretty(left_dir, right_dir)
- console.print(pretty_diff)
- console.print()
-
- console.print("[bold yellow]Recommended Actions:[/bold yellow]")
- console.print(" • If [cyan].pipelex[/cyan] is correct, run: [cyan]make up-kit-configs[/cyan] or simply [cyan]make ukc[/cyan]")
- console.print(" • If [cyan]pipelex/kit/configs[/cyan] is correct, copy it to [cyan].pipelex[/cyan]")
- console.print()
-
- sys.exit(1)
diff --git a/pipelex/exceptions.py b/pipelex/exceptions.py
index dbc7cb79c..2323d8358 100644
--- a/pipelex/exceptions.py
+++ b/pipelex/exceptions.py
@@ -1,23 +1,14 @@
from pipelex.client.exceptions import ClientAuthenticationError
-from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValueError
from pipelex.core.concepts.exceptions import (
- ConceptBlueprintValueError,
ConceptCodeError,
- ConceptDefinitionError,
- ConceptDefinitionErrorData,
ConceptError,
ConceptFactoryError,
ConceptLibraryConceptNotFoundError,
ConceptRefineError,
ConceptStringError,
- ConceptStructureBlueprintValueError,
- ConceptStructureGeneratorError,
- ConceptStructureValidationError,
- PipelexValidationExceptionAbstract,
- StructureClassError,
)
-from pipelex.core.domains.exceptions import DomainCodeError, DomainDefinitionError
-from pipelex.core.exceptions import PipelexConfigurationError, StaticValidationError, SyntaxErrorData
+from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError, ConceptStructureValidationError, StructureClassError
+from pipelex.core.domains.exceptions import DomainCodeError
from pipelex.core.memory.exceptions import (
WorkingMemoryConsistencyError,
WorkingMemoryError,
@@ -27,15 +18,11 @@ from pipelex.core.memory.exceptions import (
WorkingMemoryVariableError,
)
from pipelex.core.pipes.exceptions import (
- PipeBlueprintValueError,
- PipeDefinitionErrorData,
PipeFactoryError,
- PipeInputError,
- PipeInputNotFoundError,
PipeOperatorModelChoiceError,
- PipeRunInputsError,
- StaticValidationErrorType,
+ PipeValidationError,
)
+from pipelex.core.pipes.inputs.exceptions import PipeInputNotFoundError
from pipelex.core.stuffs.exceptions import (
StuffArtefactError,
StuffArtefactReservedFieldError,
@@ -44,25 +31,17 @@ from pipelex.core.stuffs.exceptions import (
StuffError,
)
from pipelex.libraries.exceptions import (
- ConceptLibraryError,
ConceptLoadingError,
- DomainLibraryError,
DomainLoadingError,
LibraryError,
LibraryLoadingError,
- LibraryLoadingErrorData,
- PipeLibraryError,
- PipeLibraryPipeNotFoundError,
PipeLoadingError,
)
+from pipelex.libraries.pipe.exceptions import PipeNotFoundError
from pipelex.pipe_controllers.exceptions import PipeControllerError, PipeControllerOutputConceptMismatchError
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipe_run.exceptions import BatchParamsError, PipeRouterError, PipeRunError, PipeRunParamsError
from pipelex.pipeline.exceptions import (
- DryRunError,
- DryRunMissingInputsError,
- DryRunMissingPipesError,
- DryRunTemplatingError,
PipeExecutionError,
PipelineExecutionError,
PipeStackOverflowError,
@@ -74,22 +53,17 @@ from pipelex.system.exceptions import (
CredentialsError,
FatalError,
NestedKeyConflictError,
- ToolException,
+ ToolError,
TracebackMessageError,
)
__all__ = [
# from pipelex.client.exceptions
"ClientAuthenticationError",
- # from pipelex.core.bundles.exceptions
- "PipelexBundleBlueprintValueError",
# from pipelex.core.domains.exceptions
"DomainCodeError",
- "DomainDefinitionError",
# from pipelex.core.concepts.exceptions
"ConceptError",
- "ConceptBlueprintValueError",
- "ConceptStructureBlueprintValueError",
"ConceptStructureValidationError",
"ConceptFactoryError",
"StructureClassError",
@@ -97,21 +71,15 @@ __all__ = [
"ConceptStringError",
"ConceptRefineError",
"ConceptLibraryConceptNotFoundError",
- "ConceptDefinitionErrorData",
- "ConceptDefinitionError",
"ConceptStructureGeneratorError",
- "PipelexValidationExceptionAbstract",
# from pipelex.libraries.exceptions
"LibraryError",
- "LibraryLoadingErrorData",
"LibraryLoadingError",
- "DomainLibraryError",
- "ConceptLibraryError",
"ConceptLoadingError",
"DomainLoadingError",
- "PipeLibraryError",
- "PipeLibraryPipeNotFoundError",
"PipeLoadingError",
+ # from pipelex.libraries.pipe.exceptions
+ "PipeNotFoundError",
# from pipelex.pipe_controllers.exceptions
"PipeControllerError",
"PipeControllerOutputConceptMismatchError",
@@ -123,14 +91,9 @@ __all__ = [
"PipeRouterError",
"PipeRunError",
# from pipelex.core.pipes.exceptions
- "PipeDefinitionErrorData",
- "PipeBlueprintValueError",
"PipeInputNotFoundError",
"PipeFactoryError",
"PipeOperatorModelChoiceError",
- "PipeInputError",
- "StaticValidationErrorType",
- "PipeRunInputsError",
# from pipelex.core.stuffs.exceptions
"StuffArtefactError",
"StuffArtefactReservedFieldError",
@@ -138,9 +101,7 @@ __all__ = [
"StuffContentTypeError",
"StuffContentValidationError",
# from pipelex.core.exceptions
- "PipelexConfigurationError",
- "SyntaxErrorData",
- "StaticValidationError",
+ "PipeValidationError",
# from pipelex.core.memory.exceptions
"WorkingMemoryConsistencyError",
"WorkingMemoryError",
@@ -150,16 +111,12 @@ __all__ = [
"WorkingMemoryVariableError",
# pipelex.pipeline.exceptions
"PipeStackOverflowError",
- "DryRunTemplatingError",
- "DryRunMissingPipesError",
- "DryRunMissingInputsError",
- "DryRunError",
"PipelineExecutionError",
"PipeExecutionError",
# from pipelex.pipeline.track.exceptions
"JobHistoryError",
# from pipelex.system.exceptions
- "ToolException",
+ "ToolError",
"NestedKeyConflictError",
"CredentialsError",
"TracebackMessageError",
diff --git a/pipelex/hub.py b/pipelex/hub.py
index 95678c784..983363518 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -1,4 +1,5 @@
import sys
+from contextvars import ContextVar
from typing import ClassVar, Optional
from kajson.class_registry_abstract import ClassRegistryAbstract
@@ -15,13 +16,14 @@ from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.models.model_deck import ModelDeck
from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
from pipelex.core.concepts.concept import Concept
-from pipelex.core.concepts.concept_library_abstract import ConceptLibraryAbstract
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import Domain
-from pipelex.core.domains.domain_library_abstract import DomainLibraryAbstract
from pipelex.core.pipes.pipe_abstract import PipeAbstract
-from pipelex.core.pipes.pipe_library_abstract import PipeLibraryAbstract
+from pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstract
+from pipelex.libraries.domain.domain_library_abstract import DomainLibraryAbstract
+from pipelex.libraries.library import Library
from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
+from pipelex.libraries.pipe.pipe_library_abstract import PipeLibraryAbstract
from pipelex.observer.observer_protocol import ObserverProtocol
from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
from pipelex.pipeline.pipeline import Pipeline
@@ -62,11 +64,11 @@ class PipelexHub:
self._content_generator: ContentGeneratorProtocol | None = None
# pipelex
+ self._library_manager: LibraryManagerAbstract | None = None
self._domain_library: DomainLibraryAbstract | None = None
self._concept_library: ConceptLibraryAbstract | None = None
self._pipe_library: PipeLibraryAbstract | None = None
self._pipe_router: PipeRouterProtocol | None = None
- self._library_manager: LibraryManagerAbstract | None = None
# pipeline
self._pipeline_tracker: PipelineTrackerProtocol | None = None
@@ -180,9 +182,6 @@ class PipelexHub:
def set_pipeline_manager(self, pipeline_manager: PipelineManagerAbstract):
self._pipeline_manager = pipeline_manager
- def set_library_manager(self, library_manager: LibraryManagerAbstract):
- self._library_manager = library_manager
-
def set_observer(self, observer: ObserverProtocol):
self._observer = observer
@@ -268,18 +267,24 @@ class PipelexHub:
# pipelex
def get_required_domain_library(self) -> DomainLibraryAbstract:
+ if self._library_manager is not None:
+ return self._library_manager.get_current_library().domain_library
if self._domain_library is None:
msg = "DomainLibrary is not initialized"
raise RuntimeError(msg)
return self._domain_library
def get_required_concept_library(self) -> ConceptLibraryAbstract:
+ if self._library_manager is not None:
+ return self._library_manager.get_current_library().concept_library
if self._concept_library is None:
msg = "ConceptLibrary is not initialized"
raise RuntimeError(msg)
return self._concept_library
def get_required_pipe_library(self) -> PipeLibraryAbstract:
+ if self._library_manager is not None:
+ return self._library_manager.get_current_library().pipe_library
if self._pipe_library is None:
msg = "PipeLibrary is not initialized"
raise RuntimeError(msg)
@@ -303,17 +308,20 @@ class PipelexHub:
raise RuntimeError(msg)
return self._pipeline_manager
- def get_required_library_manager(self) -> LibraryManagerAbstract:
+ def get_library_manager(self) -> LibraryManagerAbstract:
if self._library_manager is None:
- msg = "Library manager is not set. You must initialize Pipelex first."
+ msg = "LibraryManager is not initialized"
raise RuntimeError(msg)
return self._library_manager
- def get_observer(self) -> ObserverProtocol:
- if self._observer is None:
- msg = "Observer is not set. You must initialize Pipelex first."
- raise RuntimeError(msg)
- return self._observer
+ def set_library_manager(self, library_manager: LibraryManagerAbstract):
+ self._library_manager = library_manager
+
+ def get_library(self) -> Library:
+ if self._library_manager is not None:
+ return self._library_manager.get_current_library()
+ msg = "Library is not initialized"
+ raise RuntimeError(msg)
# Shorthand functions for accessing the singleton
@@ -404,6 +412,31 @@ def get_secret(secret_id: str) -> str:
return get_secrets_provider().get_secret(secret_id=secret_id)
+# libraries
+
+
+_library_id: ContextVar[str | None] = ContextVar("library_id", default=None)
+
+
+def set_current_library(library_id: str) -> None:
+ """Set the library_id for the current async context."""
+ _library_id.set(library_id)
+
+
+def get_current_library() -> str:
+ """Get the library_id from the current async context."""
+ library_id = _library_id.get()
+ if library_id is None:
+ msg = "No current library set. Must call set_current_library() first."
+ raise RuntimeError(msg)
+ return library_id
+
+
+def teardown_current_library() -> None:
+ """Teardown the library_id for the current async context."""
+ _library_id.set(None)
+
+
def get_required_domain(domain: str) -> Domain:
return get_pipelex_hub().get_required_domain_library().get_required_domain(domain=domain)
@@ -429,11 +462,11 @@ def get_optional_pipe(pipe_code: str) -> PipeAbstract | None:
def get_concept_library() -> ConceptLibraryAbstract:
- return get_pipelex_hub().get_required_concept_library()
+ return get_pipelex_hub().get_library().concept_library
def get_required_concept(concept_string: str) -> Concept:
- return get_pipelex_hub().get_required_concept_library().get_required_concept(concept_string=concept_string)
+ return get_pipelex_hub().get_library().concept_library.get_required_concept(concept_string=concept_string)
def get_pipe_router() -> PipeRouterProtocol:
@@ -453,11 +486,11 @@ def get_pipeline(pipeline_run_id: str) -> Pipeline:
def get_library_manager() -> LibraryManagerAbstract:
- return get_pipelex_hub().get_required_library_manager()
+ return get_pipelex_hub().get_library_manager()
-def get_observer() -> ObserverProtocol:
- return get_pipelex_hub().get_observer()
+def get_library() -> Library:
+ return get_pipelex_hub().get_library()
def get_native_concept(native_concept: NativeConceptCode) -> Concept:
diff --git a/pipelex/kit/agent_rules/codex_commands.md b/pipelex/kit/agent_rules/codex_commands.md
new file mode 100644
index 000000000..244eb3592
--- /dev/null
+++ b/pipelex/kit/agent_rules/codex_commands.md
@@ -0,0 +1,62 @@
+# Codex Cloud Commands
+
+## Linting
+
+ After making code changes, you must always lint using `make check`.
+
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
+
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
+
+ Always fix any issues reported by these tools before proceeding.
+
+## Running Tests in Codex Cloud
+
+ To test everything that can be tested from within the Codex Cloud sandbox, run this:
+
+ ```bash
+ make codex-tests
+ # It's equivalent to running pytest with `-m "(dry_runnable or not inference) and not (pipelex_api or codex_disabled)"`
+ # If some test fails, re-run it with `-s -vv` to see more details
+ ```
+
+---
+
+## Prerequisites for running command lines: activate virtual environment
+
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
+
+ Do this:
+
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
+
+ or do that:
+
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
+
+ (adapt the above command to the OS and available virtual environment name)
+
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
+
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
+
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
+
diff --git a/pipelex/kit/agent_rules/commands.md b/pipelex/kit/agent_rules/commands.md
index 83ef8cd48..996c24265 100644
--- a/pipelex/kit/agent_rules/commands.md
+++ b/pipelex/kit/agent_rules/commands.md
@@ -1,4 +1,91 @@
# Commands
- - When you want to run commands such as `python`, `pytest` or any of our CLI such as `pipelex` or `cocode`, ALWAYS use the current obvious virtual env. If the installaton is standard, the venv is named `.venv` so always check that first.
-
\ No newline at end of file
+## Linting
+
+ After making code changes, you must always lint using `make check`.
+
+ ```bash
+ make check
+ # If the current system doesn't have the `make` command, lookup the "check" target in the Makefile and run the command manually.
+ ```
+
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fast Python linter
+ - Mypy: Static type checker
+
+ Always fix any issues reported by these tools before proceeding.
+
+## Running Tests
+
+ After you're finished making code changes, you must always run tests using `make test-xdist`.
+
+ ```bash
+ make test-xdist
+ # If the current system doesn't have the `make` command, lookup the "test-xdist" target in the Makefile and run the command manually.
+ # If some test failes, re-run it with `-s -vv` to see more details
+ ```
+
+## Running Tests with Prints
+
+ If anything went wrong, you can run the tests with prints to see the error:
+
+ ```bash
+ make test-with-prints
+ # If the current system doesn't have the `make` command, lookup the "test-with-prints" target in the Makefile and run the command manually.
+ ```
+
+## Running specific Tests
+
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
+
+## Running Last Failed Tests
+
+ To rerun only the tests that failed in the previous run, use:
+
+ ```bash
+ make tp TEST=LF
+ # or with any test target
+ make test TEST=LF
+ make t TEST=LF
+ ```
+ Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
+
+---
+
+## Prerequisites for running command lines: activate virtual environment
+
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
+
+ Do this:
+
+ ```bash
+ source .venv/bin/activate
+ pytest -s -v -k test_render_jinja2_from_text
+ pipelex validate all
+ ```
+
+ or do that:
+
+ ```bash
+ .venv/bin/python -m pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate all
+ ```
+
+ (adapt the above command to the OS and available virtual environment name)
+
+ For standard installations, the virtual environment is named `.venv`. Always check this first:
+
+ ```bash
+ # Activate the virtual environment (standard installation)
+ source .venv/bin/activate # On macOS/Linux
+ # or
+ .venv\Scripts\activate # On Windows
+ ```
+
+ If the installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
diff --git a/pipelex/kit/agent_rules/docs.md b/pipelex/kit/agent_rules/docs.md
index 67c5674b2..440d5c534 100644
--- a/pipelex/kit/agent_rules/docs.md
+++ b/pipelex/kit/agent_rules/docs.md
@@ -1,7 +1,5 @@
# Writing Docs
-Write docs and answer questions about writing docs.
-
We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
## MkDocs Markdown Requirements
diff --git a/pipelex/kit/agent_rules/llms.md b/pipelex/kit/agent_rules/llms.md
index e8b5979ed..fbe088d9c 100644
--- a/pipelex/kit/agent_rules/llms.md
+++ b/pipelex/kit/agent_rules/llms.md
@@ -49,7 +49,7 @@ Presets are meant to record the choice of an llm with its hyper parameters (temp
Examples:
```toml
llm_to_engineer = { model = "base-claude", temperature = 1 }
-llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }
+llm_to_extract_invoice = { model = "claude-4.5-sonnet", temperature = 0.1, max_tokens = "auto" }
```
The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:
diff --git a/pipelex/kit/agent_rules/pytest_standards.md b/pipelex/kit/agent_rules/pytest_standards.md
index 8e06a0db8..230a039c0 100644
--- a/pipelex/kit/agent_rules/pytest_standards.md
+++ b/pipelex/kit/agent_rules/pytest_standards.md
@@ -1,8 +1,8 @@
-# Writing unit tests
+# Writing tests
## Unit test generalities
-NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.
+NEVER USE unittest.mock. YOU MUST USE pytest-mock instead.
### Test file structure
@@ -13,9 +13,8 @@ NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.
- `tests/e2e/` - for end-to-end tests that test complete workflows
- `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)
- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
-- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
-- Always put test inside Test classes.
-- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes: 1 TestClass per module.
### Markers
@@ -57,45 +56,23 @@ class TestFooBar:
# Test implementation
```
-- Never more than 1 class per test file.
+- Never more than 1 class per test module.
- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.
### Test Data Organization
- If it's not already there, create a `test_data.py` file in the proper test directory
-- Define test cases using `StuffBlueprint`:
-
-```python
-class TestCases:
- CASE_BLUEPRINT_1 = StuffBlueprint(
- name="test_case_1",
- concept_code="domain.ConceptName1",
- value="test_value"
- )
- CASE_BLUEPRINT_2 = StuffBlueprint(
- name="test_case_2",
- concept_code="domain.ConceptName2",
- value="test_value"
- )
-
- CASE_BLUEPRINTS: ClassVar[list[tuple[str, str]]] = [ # topic, blueprint"
- ("topic1", CASE_BLUEPRINT_1),
- ("topic2", CASE_BLUEPRINT_2),
- ]
-```
-
-Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
-Also note that we provide a topic for the test case, which is purely for convenience.
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
## Best Practices for Testing
-- Whenever possible, use strong asserts to test value, not just type and presence.
+- Use strong asserts: test value, not just type and presence.
- Use parametrize for multiple test cases
- Test both success and failure cases
- Verify working memory state
- Check output structure and content
- Use meaningful test case names
-- Include docstrings explaining test purpose but not on top of the file
+- Include docstrings explaining test purpose but not on top of the file and not on top of the class.
- Log outputs for debugging
-- Generate reports for cost tracking
diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.md
index 06db00371..eba03917d 100644
--- a/pipelex/kit/agent_rules/python_standards.md
+++ b/pipelex/kit/agent_rules/python_standards.md
@@ -14,6 +14,10 @@ This document outlines the core coding standards, best practices, and quality co
- When defining enums related to string values, always inherit from `StrEnum`
- Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
- As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
## Imports
@@ -76,17 +80,17 @@ This document outlines the core coding standards, best practices, and quality co
## Factory Pattern
- - Use Factory Pattern for object creation when dealing with multiple implementations
+ - Use factory pattern for object creation when dealing with multiple implementations
- Our factory methods are named `make_from_...` and such
## Error Handling
- Always catch exceptions at the place where you can add useful context to it.
- Use try/except blocks with specific exceptions
- - Convert third-party exceptions to our custom ones
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
- NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
- - NEVER raise generic exceptions like ValueError or TypeError, create new error classes and raise them instead
- - Always add `from exc` to the exception
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
```python
try:
@@ -96,8 +100,6 @@ This document outlines the core coding standards, best practices, and quality co
raise PipelexSetupError(msg) from exc
```
- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.
-
## Documentation
1. **Docstring Format**
@@ -123,48 +125,3 @@ This document outlines the core coding standards, best practices, and quality co
Provides methods for resizing, converting, and optimizing images.
"""
```
-
-## Code Quality Checks
-
-### Linting and Type Checking
-
-Before finalizing a task, run:
-```bash
-make fix-unused-imports
-make check
-```
-
-This runs multiple code quality tools:
-- Pyright: Static type checking
-- Ruff: Fast Python linter
-- Mypy: Static type checker
-
-Always fix any issues reported by these tools before proceeding.
-
-### Running Tests
-
-1. **Quick Test Run** (no LLM/image generation):
- ```bash
- make tp
- ```
- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)`
-
-2. **Specific Tests**:
- ```bash
- make tp TEST=TestClassName
- # or
- make tp TEST=test_function_name
- ```
- Note: Matches names starting with the provided string.
-
-**Important**: Never run `make ti`, `make test-inference`, `make te`, `make test-extract`, `make tg`, or `make test-img-gen` - these use costly inference.
-
-## Pipelines
-
-- Always validate pipelines after creation/edit with `make validate`.
- Iterate if there are errors.
-
-## Project Structure
-
-- **Tests**: `tests/` directory
-- **Documentation**: `docs/` directory
diff --git a/pipelex/kit/agent_rules/run_pipelex.md b/pipelex/kit/agent_rules/run_pipelex.md
index 42d696587..2f146d388 100644
--- a/pipelex/kit/agent_rules/run_pipelex.md
+++ b/pipelex/kit/agent_rules/run_pipelex.md
@@ -1,20 +1,5 @@
# Guide to execute a pipeline and write example code
-## Prerequisites: Virtual Environment
-
-**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST activate the appropriate Python virtual environment. Without proper venv activation, these commands will not work.
-
-For standard installations, the virtual environment is named `.venv`. Always check this first:
-
-```bash
-# Activate the virtual environment (standard installation)
-source .venv/bin/activate # On macOS/Linux
-# or
-.venv\Scripts\activate # On Windows
-```
-
-If your installation uses a different venv name or location, activate that one instead. All subsequent `pipelex` and `pytest` commands assume the venv is active.
-
## Example to execute a pipeline with text output
```python
diff --git a/pipelex/kit/agent_rules/tdd.md b/pipelex/kit/agent_rules/tdd.md
index 0f41c71e8..a7800b1bf 100644
--- a/pipelex/kit/agent_rules/tdd.md
+++ b/pipelex/kit/agent_rules/tdd.md
@@ -5,7 +5,6 @@ This document outlines our test-driven development (TDD) process and the tools a
## TDD Cycle
1. **Write a Test First**
-[pytest.mdc](pytest.mdc)
2. **Write the Code**
- Implement the minimum amount of code needed to pass the test
@@ -13,12 +12,8 @@ This document outlines our test-driven development (TDD) process and the tools a
- Keep it simple - don't write more than needed
3. **Run Linting and Type Checking**
-[coding_standards.mdc](coding_standards.mdc)
-4. **Refactor if needed**
-If the code needs refactoring, with the best practices [coding_standards.mdc](coding_standards.mdc)
+4. **Validate tests**
-5. **Validate tests**
-
-Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
diff --git a/pipelex/kit/agent_rules/write_pipelex.md b/pipelex/kit/agent_rules/write_pipelex.md
index ca54a8d01..2a4b17711 100644
--- a/pipelex/kit/agent_rules/write_pipelex.md
+++ b/pipelex/kit/agent_rules/write_pipelex.md
@@ -69,7 +69,7 @@ output = "ConceptName"
The pipes will all have at least this base definition.
- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).
So If you have this error:
-`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
+`PipeValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •
variable='['invoice']'``
That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.
diff --git a/pipelex/kit/configs/inference/backends.toml b/pipelex/kit/configs/inference/backends.toml
index 52fe3cbfc..2f002afc8 100644
--- a/pipelex/kit/configs/inference/backends.toml
+++ b/pipelex/kit/configs/inference/backends.toml
@@ -71,7 +71,7 @@ api_key = "${OPENAI_API_KEY}"
[vertexai]
display_name = "Google Vertex AI"
-enabled = true
+enabled = false # This is the only one we disable beacuse setting it up requires internet access just to get credentials so it fails in CI sandboxes
gcp_project_id = "${GCP_PROJECT_ID}"
gcp_location = "${GCP_LOCATION}"
gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
diff --git a/pipelex/kit/configs/inference/backends/anthropic.toml b/pipelex/kit/configs/inference/backends/anthropic.toml
index 6b9c9128c..a3a485379 100644
--- a/pipelex/kit/configs/inference/backends/anthropic.toml
+++ b/pipelex/kit/configs/inference/backends/anthropic.toml
@@ -98,3 +98,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 100
costs = { input = 1.0, output = 5.0 }
+
+["claude-4.5-opus"]
+model_id = "claude-opus-4-5-20251101"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 5.0, output = 25.0 }
diff --git a/pipelex/kit/configs/inference/backends/azure_openai.toml b/pipelex/kit/configs/inference/backends/azure_openai.toml
index 6054b124d..ee04f7ad6 100644
--- a/pipelex/kit/configs/inference/backends/azure_openai.toml
+++ b/pipelex/kit/configs/inference/backends/azure_openai.toml
@@ -97,24 +97,43 @@ model_id = "gpt-5-mini-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.25, output = 2.0 }
+constraints = ["temperature_must_be_1"]
[gpt-5-nano]
model_id = "gpt-5-nano-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.05, output = 0.4 }
+constraints = ["temperature_must_be_1"]
[gpt-5-chat]
model_id = "gpt-5-chat-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
[gpt-5]
model_id = "gpt-5-2025-08-07"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+# --- GPT-5.1 Series -------------------------------------------------------------
+["gpt-5.1"]
+model_id = "gpt-5.1-2025-11-13"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+["gpt-5.1-chat"]
+model_id = "gpt-5.1-chat-2025-11-13"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
################################################################################
# IMAGE GENERATION MODELS
@@ -122,6 +141,7 @@ costs = { input = 1.25, output = 10.0 }
# --- OpenAI Image Generation --------------------------------------------------
[gpt-image-1]
+sdk = "azure_rest"
model_type = "img_gen"
model_id = "gpt-image-1-2025-04-15"
inputs = ["text"]
diff --git a/pipelex/kit/configs/inference/backends/bedrock.toml b/pipelex/kit/configs/inference/backends/bedrock.toml
index cea91344d..c4ab17657 100644
--- a/pipelex/kit/configs/inference/backends/bedrock.toml
+++ b/pipelex/kit/configs/inference/backends/bedrock.toml
@@ -109,3 +109,12 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 100
costs = { input = 1.0, output = 5.0 }
+
+["claude-4.5-opus"]
+sdk = "bedrock_anthropic"
+model_id = "global.anthropic.claude-opus-4-5-20251101-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 5.0, output = 25.0 }
diff --git a/pipelex/kit/configs/inference/backends/google.toml b/pipelex/kit/configs/inference/backends/google.toml
index ff0040a91..0d39dacb1 100644
--- a/pipelex/kit/configs/inference/backends/google.toml
+++ b/pipelex/kit/configs/inference/backends/google.toml
@@ -59,3 +59,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 3000
costs = { input = 0.10, output = 0.40 }
+
+# --- Gemini 3.0 Series ----------------------------------------
+["gemini-3.0-pro"]
+model_id = "gemini-3-pro-preview"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 2, output = 12.0 }
diff --git a/pipelex/kit/configs/inference/backends/openai.toml b/pipelex/kit/configs/inference/backends/openai.toml
index 485333254..7fcf0afed 100644
--- a/pipelex/kit/configs/inference/backends/openai.toml
+++ b/pipelex/kit/configs/inference/backends/openai.toml
@@ -157,6 +157,13 @@ outputs = ["text"]
costs = { input = 1.25, output = 10.0 }
constraints = ["temperature_must_be_1"]
+# --- GPT-5.1 Series -------------------------------------------------------------
+["gpt-5.1"]
+model_id = "gpt-5.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+
################################################################################
# IMAGE GENERATION MODELS
################################################################################
diff --git a/pipelex/kit/configs/inference/backends/pipelex_inference.toml b/pipelex/kit/configs/inference/backends/pipelex_inference.toml
index 655105a47..a7dbdfbc3 100644
--- a/pipelex/kit/configs/inference/backends/pipelex_inference.toml
+++ b/pipelex/kit/configs/inference/backends/pipelex_inference.toml
@@ -85,6 +85,18 @@ inputs = ["text", "images"]
outputs = ["text"]
costs = { input = 1.25, output = 10.00 }
+["gpt-5.1"]
+model_id = "pipelex/gpt-5.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
+["gpt-5.1-chat"]
+model_id = "pipelex/gpt-5.1-chat"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
# --- Claude LLMs --------------------------------------------------------------
["claude-4-sonnet"]
model_id = "pipelex/claude-4-sonnet"
@@ -110,32 +122,45 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 1, output = 5 }
+["claude-4.5-opus"]
+model_id = "pipelex/claude-4.5-opus"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 5, output = 25 }
+
# --- Gemini LLMs --------------------------------------------------------------
["gemini-2.0-flash"]
-model_id = "gemini/gemini-2.0-flash"
+model_id = "pipelex/gemini-2.0-flash"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.10, output = 0.40 }
["gemini-2.5-pro"]
-model_id = "gemini/gemini-2.5-pro"
+model_id = "pipelex/gemini-2.5-pro"
inputs = ["text", "images"]
outputs = ["text", "structured"]
max_prompt_images = 3000
costs = { input = 1.25, output = 10.0 }
["gemini-2.5-flash"]
-model_id = "gemini/gemini-2.5-flash"
+model_id = "pipelex/gemini-2.5-flash"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.30, output = 2.50 }
["gemini-2.5-flash-lite"]
-model_id = "gemini/gemini-2.5-flash-lite"
+model_id = "pipelex/gemini-2.5-flash-lite"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.10, output = 0.40 }
+["gemini-3.0-pro"]
+model_id = "pipelex/gemini-3.0-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 2, output = 12.0 }
+
# --- XAI LLMs --------------------------------------------------------------
[grok-3]
diff --git a/pipelex/kit/configs/inference/routing_profiles.toml b/pipelex/kit/configs/inference/routing_profiles.toml
index fa3f1d42a..0910ff285 100644
--- a/pipelex/kit/configs/inference/routing_profiles.toml
+++ b/pipelex/kit/configs/inference/routing_profiles.toml
@@ -94,10 +94,6 @@ default = "openai"
description = "Use Pipelex Inference backend for all its supported models"
default = "pipelex_inference"
-[profiles.all_vertexai]
-description = "Use Vertex AI backend for all its supported models"
-default = "vertexai"
-
[profiles.all_xai]
description = "Use xAI backend for all its supported models"
default = "xai"
diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.toml
index 4a164cd80..9e4f9cbb6 100644
--- a/pipelex/kit/configs/pipelex.toml
+++ b/pipelex/kit/configs/pipelex.toml
@@ -44,6 +44,7 @@ excluded_dirs = [
]
[pipelex.builder_config]
+fix_loop_max_attempts = 10
# Uncomment to change where the generated pipelines are saved
# default_output_dir = "."
# default_bundle_file_name = "bundle"
diff --git a/pipelex/kit/cursor_export.py b/pipelex/kit/cursor_rules.py
similarity index 64%
rename from pipelex/kit/cursor_export.py
rename to pipelex/kit/cursor_rules.py
index ebc8b38d8..a89082fc7 100644
--- a/pipelex/kit/cursor_export.py
+++ b/pipelex/kit/cursor_rules.py
@@ -6,6 +6,7 @@ from typing import Any
import typer
import yaml
+from pipelex.kit.exceptions import KitError
from pipelex.kit.index_models import KitIndex
from pipelex.kit.paths import get_kit_agents_dir
@@ -24,40 +25,50 @@ def _iter_agent_files(agents_dir: Traversable) -> Iterable[tuple[str, str]]:
yield child.name, child.read_text(encoding="utf-8")
-def _front_matter_for(name: str, idx: KitIndex) -> dict[str, Any]:
+def _front_matter_for(name: str, kit_index: KitIndex) -> dict[str, Any]:
"""Build front-matter for a specific file.
Args:
name: Filename (e.g., "pytest_standards.md")
- idx: Kit index configuration
+ kit_index: Kit index configuration
Returns:
Merged front-matter dictionary
"""
- base = idx.agent_rules.cursor.front_matter.copy()
+ base = kit_index.agent_rules.cursor.front_matter.copy()
key = name.removesuffix(".md")
- if key in idx.agent_rules.cursor.files:
- base |= idx.agent_rules.cursor.files[key].front_matter
+ if key in kit_index.agent_rules.cursor.files:
+ base |= kit_index.agent_rules.cursor.files[key].front_matter
# Remove globs if it's an empty list
if "globs" in base and base["globs"] == []:
del base["globs"]
return base
-def export_cursor_rules(repo_root: Path, idx: KitIndex, dry_run: bool = False) -> None:
- """Export agent markdown files to Cursor .mdc files with YAML front-matter.
+def update_cursor_rules(repo_root: Path, kit_index: KitIndex, agent_set: str, dry_run: bool = False) -> None:
+ """Update Cursor .mdc rule files from agent markdown with YAML front-matter.
Args:
repo_root: Repository root directory
- idx: Kit index configuration
+ kit_index: Kit index configuration
+ agent_set: Agent set to limit exported files
dry_run: If True, only print what would be done
"""
agents_dir = get_kit_agents_dir()
out_dir = repo_root / ".cursor" / "rules"
out_dir.mkdir(parents=True, exist_ok=True)
+ allowed_files: set[str] | None = None
+ rules_set = kit_index.agent_rules.sets.get(agent_set)
+ if not rules_set:
+ msg = f"Agent set '{agent_set}' not found in index.toml. Available sets: {list(kit_index.agent_rules.sets.keys())}"
+ raise KitError(msg)
+ allowed_files = set(rules_set)
+
for fname, body in _iter_agent_files(agents_dir):
- fm = _front_matter_for(fname, idx)
+ if fname not in allowed_files:
+ continue
+ fm = _front_matter_for(fname, kit_index)
yaml_block = "---\n" + yaml.safe_dump(fm, sort_keys=False).rstrip() + "\n---\n"
mdc = yaml_block + body
out_path = out_dir / (fname.removesuffix(".md") + ".mdc")
@@ -65,8 +76,13 @@ def export_cursor_rules(repo_root: Path, idx: KitIndex, dry_run: bool = False) -
if dry_run:
typer.echo(f"[DRY] write {out_path}")
else:
- out_path.write_text(mdc, encoding="utf-8")
- typer.echo(f"✅ Exported {out_path}")
+ # Only write and print if content changed
+ existing_content = out_path.read_text(encoding="utf-8") if out_path.exists() else None
+ if existing_content != mdc:
+ out_path.write_text(mdc, encoding="utf-8")
+ typer.echo(f"✅ Exported {out_path}")
+ else:
+ typer.echo(f"⚪ Unchanged {out_path}")
def remove_cursor_rules(repo_root: Path, dry_run: bool = False) -> None:
diff --git a/pipelex/kit/exceptions.py b/pipelex/kit/exceptions.py
new file mode 100644
index 000000000..5f11b96f8
--- /dev/null
+++ b/pipelex/kit/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class KitError(PipelexError):
+ pass
diff --git a/pipelex/kit/index.toml b/pipelex/kit/index.toml
index b05cd6e95..5c420bc28 100644
--- a/pipelex/kit/index.toml
+++ b/pipelex/kit/index.toml
@@ -65,28 +65,37 @@ globs = ["*.plx", "*.toml"]
path = "AGENTS.md"
marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"
marker_end = "<!-- END_PIPELEX_RULES -->"
-heading_1 = "# Pipelex Rules"
+heading_1 = "# Pipelex Coding Rules"
+
+[agent_rules.targets.agents.sets]
+coding_standards = [
+ "codex_commands.md",
+ "python_standards.md",
+ "pytest_standards.md",
+ "docs.md",
+ "tdd.md",
+]
[agent_rules.targets.claude]
path = "CLAUDE.md"
marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"
marker_end = "<!-- END_PIPELEX_RULES -->"
-heading_1 = "# Pipelex Rules"
+heading_1 = "# Pipelex Coding Rules"
[agent_rules.targets.github_copilot]
path = ".github/copilot-instructions.md"
marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"
marker_end = "<!-- END_PIPELEX_RULES -->"
-heading_1 = "# Pipelex Rules"
+heading_1 = "# Pipelex Coding Rules"
[agent_rules.targets.windsurf]
path = ".windsurfrules.md"
marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"
marker_end = "<!-- END_PIPELEX_RULES -->"
-heading_1 = "# Pipelex Rules"
+heading_1 = "# Pipelex Coding Rules"
[agent_rules.targets.blackbox]
path = ".blackboxrules"
marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"
marker_end = "<!-- END_PIPELEX_RULES -->"
-heading_1 = "# Pipelex Rules"
+heading_1 = "# Pipelex Coding Rules"
diff --git a/pipelex/kit/index_loader.py b/pipelex/kit/index_loader.py
index b5130220a..9dccad7d8 100644
--- a/pipelex/kit/index_loader.py
+++ b/pipelex/kit/index_loader.py
@@ -1,13 +1,13 @@
from pydantic import ValidationError
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.kit.index_models import KitIndex
from pipelex.kit.paths import get_kit_root
from pipelex.tools.misc.toml_utils import load_toml_from_path
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
-class KitIndexLoadingError(PipelexException):
+class KitIndexLoadingError(PipelexError):
pass
diff --git a/pipelex/kit/index_models.py b/pipelex/kit/index_models.py
index 9e910ac4a..8c32ac30b 100644
--- a/pipelex/kit/index_models.py
+++ b/pipelex/kit/index_models.py
@@ -2,7 +2,7 @@ from typing import Any
from pydantic import Field
-from pipelex.config import ConfigModel
+from pipelex.system.configuration.config_model import ConfigModel
class CursorFileOverride(ConfigModel):
@@ -25,6 +25,9 @@ class Target(ConfigModel):
marker_begin: str = Field(description="Beginning marker for content insertion")
marker_end: str = Field(description="Ending marker for content insertion")
heading_1: str | None = Field(default=None, description="Main title (H1) to add when inserting into empty file or file with no H1 headings")
+ sets: dict[str, list[str]] | None = Field(
+ default=None, description="Target-specific named sets overrides for agent_rules files (e.g., coding_standards, pipelex_language, all)"
+ )
class AgentRules(ConfigModel):
diff --git a/pipelex/kit/migrations/migrate_0.11.0_0.12.x.md b/pipelex/kit/migrations/migrate_0.11.0_0.12.x.md
index 4877f2b6d..17722a736 100644
--- a/pipelex/kit/migrations/migrate_0.11.0_0.12.x.md
+++ b/pipelex/kit/migrations/migrate_0.11.0_0.12.x.md
@@ -931,8 +931,8 @@ from pipelex.core.pipes.pipe_input_factory import PipeInputSpecFactory
**After:**
```python
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
```
#### Comprehensive OCR → Extract Renaming
@@ -1219,7 +1219,7 @@ The following replacements can be done with find/replace tools:
- Remove `config_folder_path` parameters from `Pipelex.make()`
- `from pipelex.core.stuffs.stuff_content import` → Update to specific module imports
- `from pipelex.pipe_works` → `from pipelex.pipe_run`
-- `from pipelex.core.pipes.pipe_input` → `from pipelex.core.pipes.input_requirements`
+- `from pipelex.core.pipes.pipe_input` → `from pipelex.core.pipes.inputs.input_requirements`
- `from pipelex.cogt.ocr` → `from pipelex.cogt.extract`
- `PipeInputSpec` → `InputRequirements`
- `ConceptProviderAbstract` → `ConceptLibraryAbstract`
diff --git a/pipelex/kit/targets_update.py b/pipelex/kit/single_file_agent_rules.py
similarity index 72%
rename from pipelex/kit/targets_update.py
rename to pipelex/kit/single_file_agent_rules.py
index c4038b81f..862185485 100644
--- a/pipelex/kit/targets_update.py
+++ b/pipelex/kit/single_file_agent_rules.py
@@ -47,37 +47,44 @@ def _demote_headings(md_content: str, levels: int) -> str:
return re.sub(pattern, demote_match, md_content, flags=re.MULTILINE)
-# TODO: fix bug which makes it not idempotent (because heading 1 gets deleted)
-def build_merged_rules(idx: KitIndex, agent_set: str | None = None) -> str:
+def build_merged_rules(kit_index: KitIndex, agent_set: str | None = None, file_list: list[str] | None = None) -> str:
"""Build merged agent documentation from ordered files.
Args:
- idx: Kit index configuration
- agent_set: Name of the agent set to use (defaults to idx.agents.default_set)
+ kit_index: Kit index configuration
+ agent_set: Name of the agent set to use (defaults to kit_index.agents.default_set)
+ file_list: Optional explicit list of files to merge (overrides agent_set lookup)
Returns:
Merged markdown content with demoted headings
"""
agents_dir = get_kit_agents_dir()
- if agent_set is None:
- agent_set = idx.agent_rules.default_set
+ # If explicit file list provided, use it
+ if file_list is not None:
+ files_to_merge = file_list
+ else:
+ # Otherwise, look up the agent set
+ if agent_set is None:
+ agent_set = kit_index.agent_rules.default_set
+
+ if agent_set not in kit_index.agent_rules.sets:
+ msg = f"Agent set '{agent_set}' not found in index.toml. Available sets: {list(kit_index.agent_rules.sets.keys())}"
+ raise ValueError(msg)
- if agent_set not in idx.agent_rules.sets:
- msg = f"Agent set '{agent_set}' not found in index.toml. Available sets: {list(idx.agent_rules.sets.keys())}"
- raise ValueError(msg)
+ files_to_merge = kit_index.agent_rules.sets[agent_set]
parts: list[str] = []
- for name in idx.agent_rules.sets[agent_set]:
+ for name in files_to_merge:
md = _read_agent_file(agents_dir, name)
- demoted = _demote_headings(md, idx.agent_rules.demote)
+ demoted = _demote_headings(md, kit_index.agent_rules.demote)
parts.append(demoted.rstrip())
return ("\n\n".join(parts)).strip() + "\n"
-def _insert_block_with_markers(target_md: str, block_md: str, main_title: str | None, markers: tuple[str, str]) -> str:
+def insert_block_with_markers(target_md: str, block_md: str, main_title: str | None, markers: tuple[str, str]) -> str:
"""Insert block into target markdown using marker-based logic.
Args:
@@ -110,7 +117,7 @@ def _insert_block_with_markers(target_md: str, block_md: str, main_title: str |
return target_md.rstrip() + "\n\n" + wrapped_block + "\n"
-def _diff(before: str, after: str, path: str) -> str:
+def unified_diff(before: str, after: str, path: str) -> str:
"""Generate unified diff between before and after.
Args:
@@ -131,37 +138,61 @@ def _diff(before: str, after: str, path: str) -> str:
)
-def update_targets(
+def update_single_file_agent_rules(
repo_root: Path,
- merged_rules: str,
+ kit_index: KitIndex,
+ agent_set: str,
targets: dict[str, Target],
dry_run: bool,
diff: bool,
backup: str | None,
) -> None:
- """Update target files with merged agent documentation.
+ """Update single-file agent rules targets with merged agent documentation.
Args:
repo_root: Repository root directory
- merged_rules: Merged markdown content to insert
+ kit_index: Kit index configuration
+ agent_set: Default agent set to use (can be overridden by target.sets)
targets: Dictionary of target file configurations keyed by ID
dry_run: If True, only print what would be done
diff: If True, show unified diff
backup: Backup suffix (e.g., ".bak"), or None for no backup
"""
for target in targets.values():
+ # Check if this target has its own set override for the given agent_set
+ target_file_list: list[str] | None = None
+ if target.sets and agent_set in target.sets:
+ target_file_list = target.sets[agent_set]
+
+ # Build merged rules for this specific target (with override if applicable)
+ merged_rules = build_merged_rules(kit_index, agent_set=agent_set, file_list=target_file_list)
+
target_path = repo_root / target.path
before = target_path.read_text(encoding="utf-8") if target_path.exists() else ""
span = find_span(before, target.marker_begin, target.marker_end)
if span:
- # Markers exist - replace content between them
- wrapped_block = wrap(target.marker_begin, target.marker_end, merged_rules)
+ # Markers exist - need to check if we should preserve heading_1
+ # Check if the entire file (outside markers) has an H1 heading
+ before_markers = before[: span[0]]
+ after_markers = before[span[1] :]
+ outside_content = before_markers + after_markers
+
+ h1_pattern = r"^#\s+.+$"
+ has_h1_outside = bool(re.search(h1_pattern, outside_content, flags=re.MULTILINE))
+
+ # Add heading_1 if it's configured and there's no H1 outside markers
+ if target.heading_1 and not has_h1_outside:
+ content_with_heading = f"{target.heading_1}\n\n{merged_rules}"
+ wrapped_block = wrap(target.marker_begin, target.marker_end, content_with_heading)
+ else:
+ wrapped_block = wrap(target.marker_begin, target.marker_end, merged_rules)
+
after = replace_span(before, span, wrapped_block)
else:
# No markers - insert with markers
- after = _insert_block_with_markers(
+ after = insert_block_with_markers(
before,
merged_rules,
target.heading_1,
@@ -171,10 +202,11 @@ def update_targets(
if dry_run:
typer.echo(f"[DRY] update {target_path}")
if diff:
- diff_output = _diff(before, after, str(target_path))
+ diff_output = unified_diff(before, after, str(target_path))
if diff_output:
typer.echo(diff_output)
- else:
+ # Only write and print if content changed
+ elif before != after:
if backup and target_path.exists():
backup_path = target_path.with_suffix(target_path.suffix + backup)
backup_path.write_text(before, encoding="utf-8")
@@ -185,9 +217,11 @@ def update_targets(
typer.echo(f"✅ Updated {target_path}")
if diff:
- diff_output = _diff(before, after, str(target_path))
+ diff_output = unified_diff(before, after, str(target_path))
if diff_output:
typer.echo(diff_output)
+ else:
+ typer.echo(f"⚪ Unchanged {target_path}")
def remove_from_targets(
@@ -264,7 +298,7 @@ def remove_from_targets(
if dry_run:
typer.echo(f"[DRY] remove marked section from {target_path}")
if diff:
- diff_output = _diff(before, after, str(target_path))
+ diff_output = unified_diff(before, after, str(target_path))
if diff_output:
typer.echo(diff_output)
else:
@@ -277,6 +311,6 @@ def remove_from_targets(
typer.echo(f"✅ Removed marked section from {target_path}")
if diff:
- diff_output = _diff(before, after, str(target_path))
+ diff_output = unified_diff(before, after, str(target_path))
if diff_output:
typer.echo(diff_output)
diff --git a/pipelex/language/plx_factory.py b/pipelex/language/plx_factory.py
index c32d290d2..02582ef94 100644
--- a/pipelex/language/plx_factory.py
+++ b/pipelex/language/plx_factory.py
@@ -7,12 +7,13 @@ from tomlkit import array, document, inline_table, table
from tomlkit import string as tomlkit_string
from pipelex import log
-from pipelex.config import PlxConfig, get_config
+from pipelex.config import get_config
from pipelex.tools.misc.json_utils import remove_none_values_from_dict
from pipelex.types import StrEnum
if TYPE_CHECKING:
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+ from pipelex.system.configuration.configs import PlxConfig
class SectionKey(StrEnum):
diff --git a/pipelex/libraries/__init__.py b/pipelex/libraries/__init__.py
index 8b1378917..e69de29bb 100644
--- a/pipelex/libraries/__init__.py
+++ b/pipelex/libraries/__init__.py
@@ -1 +0,0 @@
-
diff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/__init__.py b/pipelex/libraries/concept/__init__.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_batch/__init__.py
rename to pipelex/libraries/concept/__init__.py
diff --git a/pipelex/core/concepts/concept_library.py b/pipelex/libraries/concept/concept_library.py
similarity index 70%
rename from pipelex/core/concepts/concept_library.py
rename to pipelex/libraries/concept/concept_library.py
index 22ae29144..4c34b9ee2 100644
--- a/pipelex/core/concepts/concept_library.py
+++ b/pipelex/libraries/concept/concept_library.py
@@ -1,18 +1,14 @@
-from typing import Any
-
-from pydantic import Field, RootModel
+from pydantic import Field, RootModel, model_validator
from typing_extensions import override
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_library_abstract import ConceptLibraryAbstract
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.concepts.exceptions import ConceptLibraryConceptNotFoundError
-from pipelex.core.concepts.validation import is_concept_code_valid, is_concept_string_valid
+from pipelex.core.concepts.exceptions import ConceptLibraryConceptNotFoundError, ConceptStringError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.concepts.validation import is_concept_string_valid, validate_concept_string_or_code
from pipelex.core.domains.domain import SpecialDomain
-from pipelex.core.stuffs.image_content import ImageContent
-from pipelex.hub import get_class_registry
-from pipelex.libraries.exceptions import ConceptLibraryError
+from pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstract
+from pipelex.libraries.concept.exceptions import ConceptLibraryError
from pipelex.types import Self
ConceptLibraryRoot = dict[str, Concept]
@@ -21,30 +17,41 @@ ConceptLibraryRoot = dict[str, Concept]
class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
root: ConceptLibraryRoot = Field(default_factory=dict)
- def validate_with_libraries(self):
- """Validates that the each refine concept code in the refines array of each concept in the library exists in the library"""
+ @model_validator(mode="after")
+ def validation_static(self):
for concept in self.root.values():
if concept.refines and concept.refines not in self.root:
msg = f"Concept '{concept.code}' refines '{concept.refines}' but no concept with the code '{concept.refines}' exists"
raise ConceptLibraryError(msg)
+ return self
@override
def setup(self):
- all_native_concepts = ConceptFactory.make_all_native_concepts()
- self.add_concepts(concepts=all_native_concepts)
+ pass
@override
- def reset(self):
+ def teardown(self):
self.root = {}
- self.setup()
@override
- def teardown(self):
- self.root = {}
+ def reset(self):
+ self.teardown()
+ self.setup()
+ # TODO: Rethink the make_empty of libraries. It doesn't makes sense to call the setup inside the make_empty method.
@classmethod
def make_empty(cls) -> Self:
- return cls(root={})
+ library = cls(root={})
+ library.setup()
+ return library
+
+ @classmethod
+ def make_empty_with_native_concepts(cls) -> Self:
+ library = cls(root={})
+ library.setup()
+ all_native_concepts = ConceptFactory.make_all_native_concepts()
+ library.add_concepts(concepts=all_native_concepts)
+ return library
@override
def list_concepts(self) -> list[Concept]:
@@ -85,14 +92,16 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
If it is not native and doesnt have a domain, it should raise an error
"""
if Concept.is_implicit_concept(concept_string=concept_string):
- return ConceptFactory.make_implicit_concept(concept_string=concept_string)
+ return ConceptFactory.make_implicit_concept(concept_code=concept_string.split(".")[1])
+
if not is_concept_string_valid(concept_string=concept_string):
msg = f"Concept string '{concept_string}' is not a valid concept string"
raise ConceptLibraryError(msg)
+
the_concept = self.get_optional_concept(concept_string=concept_string)
if not the_concept:
msg = f"Concept '{concept_string}' not found in the library"
- raise ConceptLibraryConceptNotFoundError(msg)
+ raise ConceptLibraryError(msg)
return the_concept
@override
@@ -107,31 +116,19 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
"""Create all native concepts from the hardcoded data"""
return [self.get_native_concept(native_concept=native_concept) for native_concept in NativeConceptCode.values_list()]
- @override
- def get_class(self, concept_code: str) -> type[Any] | None:
- return get_class_registry().get_class(concept_code)
-
- @override
- def is_image_concept(self, concept: Concept) -> bool:
- """Check if the concept is an image concept.
- It is an image concept if its structure class is a subclass of ImageContent
- or if it refines the native Image concept.
- """
- pydantic_model = self.get_class(concept_code=concept.structure_class_name)
- is_image_class = bool(pydantic_model and issubclass(pydantic_model, ImageContent))
- refines_image = self.is_compatible(
- tested_concept=concept,
- wanted_concept=self.get_native_concept(native_concept=NativeConceptCode.IMAGE),
- strict=True,
- )
- return is_image_class or refines_image
-
@override
def get_required_concept_from_concept_string_or_code(self, concept_string_or_code: str, search_domains: list[str] | None = None) -> Concept:
- if "." in concept_string_or_code:
+ try:
+ validate_concept_string_or_code(concept_string_or_code=concept_string_or_code)
+ except ConceptStringError as exc:
+ msg = f"Could not validate concept string or code '{concept_string_or_code}': {exc}"
+ raise ConceptLibraryError(msg) from exc
+
+ if NativeConceptCode.is_native_concept_string_or_code(concept_string_or_code=concept_string_or_code):
+ native_concept_string = NativeConceptCode.get_validated_native_concept_string(concept_string_or_code=concept_string_or_code)
+ return self.get_native_concept(native_concept=NativeConceptCode(native_concept_string.split(".")[1]))
+ elif "." in concept_string_or_code:
return self.get_required_concept(concept_string=concept_string_or_code)
- elif NativeConceptCode.is_native_concept(concept_code=concept_string_or_code):
- return self.get_native_concept(native_concept=NativeConceptCode(concept_string_or_code))
else:
found_concepts: list[Concept] = []
if search_domains is None:
@@ -159,15 +156,5 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
raise ConceptLibraryConceptNotFoundError(msg)
return found_concepts[0]
- @override
- def search_for_concept_in_domains(self, concept_code: str, search_domains: list[str]) -> Concept | None:
- if not is_concept_code_valid(concept_code=concept_code):
- msg = f"Concept code '{concept_code}' is not a valid concept code"
- raise ConceptLibraryError(msg)
- for domain in search_domains:
- if found_concept := self.get_required_concept(
- concept_string=ConceptFactory.make_concept_string_with_domain(domain=domain, concept_code=concept_code),
- ):
- return found_concept
-
- return None
+ def is_concept_exists(self, concept_string: str) -> bool:
+ return concept_string in self.root
diff --git a/pipelex/core/concepts/concept_library_abstract.py b/pipelex/libraries/concept/concept_library_abstract.py
similarity index 76%
rename from pipelex/core/concepts/concept_library_abstract.py
rename to pipelex/libraries/concept/concept_library_abstract.py
index d34b65677..9ad5b7101 100644
--- a/pipelex/core/concepts/concept_library_abstract.py
+++ b/pipelex/libraries/concept/concept_library_abstract.py
@@ -1,8 +1,7 @@
from abc import ABC, abstractmethod
-from typing import Any
from pipelex.core.concepts.concept import Concept
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
class ConceptLibraryAbstract(ABC):
@@ -46,18 +45,6 @@ class ConceptLibraryAbstract(ABC):
def teardown(self) -> None:
pass
- @abstractmethod
- def is_image_concept(self, concept: Concept) -> bool:
- pass
-
- @abstractmethod
- def search_for_concept_in_domains(self, concept_code: str, search_domains: list[str]) -> Concept | None:
- pass
-
- @abstractmethod
- def get_class(self, concept_code: str) -> type[Any] | None:
- pass
-
@abstractmethod
def get_native_concept(self, native_concept: NativeConceptCode) -> Concept:
pass
diff --git a/pipelex/libraries/concept/exceptions.py b/pipelex/libraries/concept/exceptions.py
new file mode 100644
index 000000000..01972b31c
--- /dev/null
+++ b/pipelex/libraries/concept/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class ConceptLibraryError(PipelexError):
+ pass
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/__init__.py b/pipelex/libraries/domain/__init__.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/__init__.py
rename to pipelex/libraries/domain/__init__.py
diff --git a/pipelex/core/domains/domain_library.py b/pipelex/libraries/domain/domain_library.py
similarity index 61%
rename from pipelex/core/domains/domain_library.py
rename to pipelex/libraries/domain/domain_library.py
index 39e370fee..aaad77bad 100644
--- a/pipelex/core/domains/domain_library.py
+++ b/pipelex/libraries/domain/domain_library.py
@@ -2,52 +2,55 @@ from pydantic import RootModel
from typing_extensions import override
from pipelex.core.domains.domain import Domain
-from pipelex.core.domains.domain_library_abstract import DomainLibraryAbstract
-from pipelex.libraries.exceptions import DomainLibraryError
+from pipelex.libraries.domain.domain_library_abstract import DomainLibraryAbstract
+from pipelex.libraries.domain.exceptions import DomainLibraryError
from pipelex.types import Self
DomainLibraryRoot = dict[str, Domain]
class DomainLibrary(RootModel[DomainLibraryRoot], DomainLibraryAbstract):
- def validate_with_libraries(self):
+ def setup(self):
pass
- def reset(self):
+ @override
+ def teardown(self):
self.root = {}
+ def reset(self):
+ self.teardown()
+ self.setup()
+
@classmethod
def make_empty(cls) -> Self:
return cls(root={})
+ @override
+ def get_domain(self, domain: str) -> Domain | None:
+ return self.root.get(domain)
+
def add_domain(self, domain: Domain):
domain_code = domain.code
- if existing_domain := self.root.get(domain_code):
- # merge the new domain with the existing one
- self.root[domain_code] = existing_domain.model_copy(update=domain.model_dump())
- else:
- self.root[domain_code] = domain
+ if domain_code in self.root:
+ msg = f"Trying to add domain '{domain_code}' to domain library but it already exists"
+ raise DomainLibraryError(msg)
+ self.root[domain_code] = domain
def add_domains(self, domains: list[Domain]):
for domain in domains:
self.add_domain(domain=domain)
def remove_domain_by_code(self, domain_code: str) -> None:
- if domain_code in self.root:
- del self.root[domain_code]
-
- @override
- def get_domain(self, domain: str) -> Domain | None:
- return self.root.get(domain)
+ if domain_code not in self.root:
+ msg = f"Trying to remove domain '{domain_code}' from domain library but it does not exist"
+ raise DomainLibraryError(msg)
+ del self.root[domain_code]
@override
def get_required_domain(self, domain: str) -> Domain:
+ """Get a domain by code from this library, raising an error if not found."""
the_domain = self.get_domain(domain=domain)
if not the_domain:
msg = f"Domain '{domain}' not found. Check for typos and make sure it is declared in a pipeline library."
raise DomainLibraryError(msg)
return the_domain
-
- @override
- def teardown(self) -> None:
- self.root = {}
diff --git a/pipelex/core/domains/domain_library_abstract.py b/pipelex/libraries/domain/domain_library_abstract.py
similarity index 71%
rename from pipelex/core/domains/domain_library_abstract.py
rename to pipelex/libraries/domain/domain_library_abstract.py
index 5de1b5726..4102b4605 100644
--- a/pipelex/core/domains/domain_library_abstract.py
+++ b/pipelex/libraries/domain/domain_library_abstract.py
@@ -6,11 +6,11 @@ from pipelex.core.domains.domain import Domain
class DomainLibraryAbstract(ABC):
@abstractmethod
def get_domain(self, domain: str) -> Domain | None:
- pass
+ """Get a domain by code from this library."""
@abstractmethod
def get_required_domain(self, domain: str) -> Domain:
- pass
+ """Get a domain by code from this library, raising an error if not found."""
@abstractmethod
def teardown(self) -> None:
diff --git a/pipelex/libraries/domain/exceptions.py b/pipelex/libraries/domain/exceptions.py
new file mode 100644
index 000000000..ee2dfb1a7
--- /dev/null
+++ b/pipelex/libraries/domain/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class DomainLibraryError(PipelexError):
+ pass
diff --git a/pipelex/libraries/exceptions.py b/pipelex/libraries/exceptions.py
index 968a5c5e8..b1090c632 100644
--- a/pipelex/libraries/exceptions.py
+++ b/pipelex/libraries/exceptions.py
@@ -1,63 +1,46 @@
-from pydantic import BaseModel, Field
-from typing_extensions import override
+from pipelex.base_exceptions import PipelexError
+from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
+from pipelex.core.exceptions import PipesAndConceptValidationErrorData
-from pipelex.base_exceptions import PipelexException
-from pipelex.core.concepts.exceptions import ConceptDefinitionError, ConceptDefinitionErrorData, PipelexValidationExceptionAbstract
-from pipelex.core.pipes.exceptions import PipeDefinitionErrorData
-
-class LibraryError(PipelexException):
+class LibraryError(PipelexError):
pass
-class LibraryLoadingErrorData(BaseModel):
- """Structured data for LibraryLoadingError."""
+class LibraryLoadingError(LibraryError):
+ """Error raised when loading library components fails.
- message: str = Field(description="The main error message")
- concept_definition_errors: list[ConceptDefinitionErrorData] | None = Field(None, description="List of concept definition errors")
- pipe_definition_errors: list[PipeDefinitionErrorData] | None = Field(None, description="List of pipe definition errors")
+ This error aggregates TWO types of validation errors:
+ 1. Blueprint validation errors (from PipelexBundleBlueprint.model_validate())
+ - Stored in: blueprint_validation_errors
+ - Example: PIPE_SEQUENCE_OUTPUT_MISMATCH
+ 2. Pipe/Concept validation errors (from Pipe or Concept class validation)
+ - Stored in: pipe_concept_validation_errors
+ - Example: MISSING_INPUT_VARIABLE, INPUT_REQUIREMENT_MISMATCH
-class LibraryLoadingError(LibraryError, PipelexValidationExceptionAbstract):
- """Error raised when loading library components fails."""
+ Also handles:
+ - Factory errors (Domain, Concept, Pipe)
+ - Interpreter errors (blueprint parsing)
+ """
def __init__(
self,
message: str,
- concept_definition_errors: list[ConceptDefinitionErrorData] | None = None,
- pipe_definition_errors: list[PipeDefinitionErrorData] | None = None,
+ blueprint_validation_errors: list[PipelexBundleBlueprintValidationErrorData] | None = None,
+ pipe_concept_validation_errors: list[PipesAndConceptValidationErrorData] | None = None,
):
- self.concept_definition_errors = concept_definition_errors
- self.pipe_definition_errors = pipe_definition_errors
+ """Initialize LibraryLoadingError.
+
+ Args:
+ message: Error message
+ blueprint_validation_errors: Blueprint validation errors (from PipelexBundleBlueprint validation)
+ pipe_concept_validation_errors: Pipe/Concept validation errors (from Pipe/Concept validation)
+ """
+ self.blueprint_validation_errors = blueprint_validation_errors or []
+ self.pipe_concept_validation_errors = pipe_concept_validation_errors or []
super().__init__(message)
- def as_structured_content(self) -> LibraryLoadingErrorData:
- return LibraryLoadingErrorData(
- message=str(self),
- concept_definition_errors=self.concept_definition_errors,
- pipe_definition_errors=self.pipe_definition_errors,
- )
-
- @override
- def get_concept_definition_errors(self) -> list[ConceptDefinitionErrorData]:
- return self.concept_definition_errors or []
-
-
-class DomainLibraryError(LibraryError):
- pass
-
-
-class ConceptLibraryError(LibraryError):
- pass
-
-
-class PipeLibraryError(LibraryError):
- pass
-
-
-class PipeLibraryPipeNotFoundError(PipeLibraryError):
- pass
-
class DomainLoadingError(LibraryLoadingError):
def __init__(self, message: str, domain_code: str, description: str, source: str | None = None):
@@ -69,19 +52,31 @@ class DomainLoadingError(LibraryLoadingError):
class ConceptLoadingError(LibraryLoadingError):
def __init__(
- self, message: str, concept_definition_error: ConceptDefinitionError, concept_code: str, description: str, source: str | None = None
+ self,
+ message: str,
+ concept_code: str,
+ description: str,
+ source: str | None = None,
+ original_error: Exception | None = None,
):
- self.concept_definition_error = concept_definition_error
self.concept_code = concept_code
self.description = description
self.source = source
+ self.original_error = original_error
super().__init__(message)
class PipeLoadingError(LibraryLoadingError):
- def __init__(self, message: str, pipe_definition_error: PipeDefinitionErrorData, pipe_code: str, description: str, source: str | None = None):
- self.pipe_definition_error = pipe_definition_error
+ def __init__(
+ self,
+ message: str,
+ pipe_code: str,
+ description: str,
+ source: str | None = None,
+ original_error: Exception | None = None,
+ ):
self.pipe_code = pipe_code
self.description = description
self.source = source
+ self.original_error = original_error
super().__init__(message)
diff --git a/pipelex/libraries/library.py b/pipelex/libraries/library.py
new file mode 100644
index 000000000..aeed8bbc8
--- /dev/null
+++ b/pipelex/libraries/library.py
@@ -0,0 +1,80 @@
+from pydantic import BaseModel
+
+from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.libraries.concept.concept_library import ConceptLibrary
+from pipelex.libraries.concept.exceptions import ConceptLibraryError
+from pipelex.libraries.domain.domain_library import DomainLibrary
+from pipelex.libraries.exceptions import LibraryError
+from pipelex.libraries.pipe.exceptions import PipeLibraryError
+from pipelex.libraries.pipe.pipe_library import PipeLibrary
+from pipelex.pipe_controllers.pipe_controller import PipeController
+
+
+class Library(BaseModel):
+ """A Library bundles together domain, concept, and pipe libraries for a specific context.
+
+ This represents a complete set of Pipelex definitions (domains, concepts, pipes)
+ that can be loaded and used together, typically for a single pipeline run.
+
+ Limitations: It lacks the Func Registry library and Class Registry library
+
+ Each Library (except BASE) inherits native concepts and base pipes from the BASE library.
+ """
+
+ domain_library: DomainLibrary
+ concept_library: ConceptLibrary
+ pipe_library: PipeLibrary
+
+ def get_domain_library(self) -> DomainLibrary:
+ return self.domain_library
+
+ def get_concept_library(self) -> ConceptLibrary:
+ return self.concept_library
+
+ def get_pipe_library(self) -> PipeLibrary:
+ return self.pipe_library
+
+ def teardown(self) -> None:
+ self.pipe_library.teardown()
+ self.concept_library.teardown()
+ self.domain_library.teardown()
+
+ def validate_library(self) -> None:
+ self.validate_pipe_library_with_libraries()
+ self.validate_concept_library_with_libraries()
+ self.validate_domain_library_with_libraries()
+
+ def validate_pipe_library_with_libraries(self) -> None:
+ for pipe in self.pipe_library.get_pipes():
+ # Validate concept dependencies exist
+ # Note: This should NEVER fail as concepts are validated during pipe construction via get_required_concept()
+ # TODO: Make this non mandatory in production, or a test
+ for concept in [pipe.output, *pipe.inputs.concepts]:
+ try:
+ self.concept_library.is_concept_exists(concept_string=concept.concept_string)
+ except ConceptLibraryError as concept_error:
+ msg = (
+ f"INTERNAL ERROR: Pipe '{pipe.code}' references concept '{concept.concept_string}' "
+ f"which doesn't exist in the concept library. This should be impossible as concepts are "
+ f"validated during pipe construction (via get_required_concept() in pipe factories). "
+ f"This indicates a bug in the system. Original error: {concept_error}"
+ )
+ raise PipelexUnexpectedError(msg) from concept_error
+
+ # Validate pipe dependencies exist for pipe controllers
+ if isinstance(pipe, PipeController):
+ for sub_pipe_code in pipe.pipe_dependencies():
+ try:
+ self.pipe_library.get_required_pipe(pipe_code=sub_pipe_code)
+ except PipeLibraryError as pipe_error:
+ msg = f"Error validating pipe '{pipe.code}' dependency pipe '{sub_pipe_code}' because of: {pipe_error}"
+ raise LibraryError(msg) from pipe_error
+
+ for pipe in self.pipe_library.root.values():
+ pipe.validate_with_libraries()
+
+ def validate_concept_library_with_libraries(self) -> None:
+ pass
+
+ def validate_domain_library_with_libraries(self) -> None:
+ pass
diff --git a/pipelex/libraries/library_factory.py b/pipelex/libraries/library_factory.py
new file mode 100644
index 000000000..40d176539
--- /dev/null
+++ b/pipelex/libraries/library_factory.py
@@ -0,0 +1,16 @@
+from pydantic import BaseModel
+
+from pipelex.libraries.concept.concept_library import ConceptLibrary
+from pipelex.libraries.domain.domain_library import DomainLibrary
+from pipelex.libraries.library import Library
+from pipelex.libraries.pipe.pipe_library import PipeLibrary
+
+
+class LibraryFactory(BaseModel):
+ @classmethod
+ def make_empty(cls) -> Library:
+ concept_library = ConceptLibrary.make_empty_with_native_concepts()
+ pipe_library = PipeLibrary.make_empty()
+ domain_library = DomainLibrary.make_empty()
+
+ return Library(domain_library=domain_library, concept_library=concept_library, pipe_library=pipe_library)
diff --git a/pipelex/libraries/library_ids.py b/pipelex/libraries/library_ids.py
new file mode 100644
index 000000000..1eb743edb
--- /dev/null
+++ b/pipelex/libraries/library_ids.py
@@ -0,0 +1,12 @@
+from pipelex.types import StrEnum
+
+
+class SpecialLibraryId(StrEnum):
+ """Special library identifiers.
+
+ UNTITLED: The untitled/default library
+ TEST: The test library
+ """
+
+ UNTITLED = "untitled"
+ TEST = "test"
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 8f4f98585..85f2f267e 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -1,97 +1,69 @@
+import uuid
from pathlib import Path
-from typing import ClassVar
+from typing import TYPE_CHECKING
-from pydantic import ValidationError
from typing_extensions import override
from pipelex import log
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_library import ConceptLibrary
-from pipelex.core.concepts.exceptions import ConceptDefinitionError
from pipelex.core.domains.domain import Domain
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.core.domains.domain_factory import DomainFactory
-from pipelex.core.domains.domain_library import DomainLibrary
-from pipelex.core.domains.exceptions import DomainDefinitionError
-from pipelex.core.interpreter import PipelexInterpreter
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.exceptions import PipeDefinitionErrorData
+from pipelex.core.interpreter.exceptions import PipelexInterpreterError
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.core.pipes.pipe_factory import PipeFactory
-from pipelex.core.pipes.pipe_library import PipeLibrary
from pipelex.core.stuffs.structured_content import StructuredContent
-from pipelex.core.validation import report_validation_error
+from pipelex.hub import get_current_library
from pipelex.libraries.exceptions import (
- ConceptLibraryError,
- ConceptLoadingError,
- DomainLoadingError,
LibraryError,
LibraryLoadingError,
- PipeLibraryError,
- PipeLoadingError,
)
+from pipelex.libraries.library import Library
+from pipelex.libraries.library_factory import LibraryFactory
from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
from pipelex.libraries.library_utils import (
- find_plx_files_in_dir,
- get_pipelex_package_dir_for_imports,
- get_pipelex_plx_files_from_package,
+ get_pipelex_plx_files_from_dirs,
)
-from pipelex.system.configuration.config_loader import config_manager
from pipelex.system.registries.class_registry_utils import ClassRegistryUtils
from pipelex.system.registries.func_registry_utils import FuncRegistryUtils
-from pipelex.types import StrEnum
-
-class LibraryComponent(StrEnum):
- CONCEPT = "concept"
- PIPE = "pipe"
-
- @property
- def error_class(self) -> type[LibraryError]:
- match self:
- case LibraryComponent.CONCEPT:
- return ConceptLibraryError
- case LibraryComponent.PIPE:
- return PipeLibraryError
+if TYPE_CHECKING:
+ from pipelex.core.concepts.concept import Concept
+ from pipelex.core.domains.domain import Domain
class LibraryManager(LibraryManagerAbstract):
- allowed_root_attributes: ClassVar[list[str]] = [
- "domain",
- "description",
- "system_prompt",
- ]
-
- def __init__(
- self,
- domain_library: DomainLibrary,
- concept_library: ConceptLibrary,
- pipe_library: PipeLibrary,
- ):
- self.domain_library = domain_library
- self.concept_library = concept_library
- self.pipe_library = pipe_library
+ def __init__(self):
+ # UNTITLED library is the fallback library for all others
+ self._libraries: dict[str, Library] = {}
self.loaded_plx_paths: list[str] = []
- @override
- def validate_libraries(self):
- log.verbose("LibraryManager validating libraries")
-
- self.concept_library.validate_with_libraries()
- self.pipe_library.validate_with_libraries()
- self.domain_library.validate_with_libraries()
+ ############################################################
+ # Manager lifecycle
+ ############################################################
+ def generate_library_id(self) -> str:
+ return str(uuid.uuid4())
@override
def setup(self) -> None:
- self.concept_library.setup()
+ pass
@override
- def teardown(self) -> None:
- self.pipe_library.teardown()
- self.concept_library.teardown()
- self.domain_library.teardown()
+ def teardown(self, library_id: str | None = None) -> None:
+ if library_id:
+ if library_id not in self._libraries:
+ msg = f"Trying to teardown a library that does not exist: '{library_id}'"
+ raise LibraryError(msg)
+ library = self._libraries[library_id]
+ library.teardown()
+ del self._libraries[library_id]
+ return
+
+ for library in self._libraries.values():
+ library.teardown()
+ self._libraries = {}
@override
def reset(self) -> None:
@@ -99,200 +71,81 @@ class LibraryManager(LibraryManagerAbstract):
self.setup()
@override
- def get_loaded_plx_paths(self) -> list[str]:
- return self.loaded_plx_paths
+ def open_library(self, library_id: str | None = None) -> tuple[str, Library]:
+ if not library_id:
+ library_id = self.generate_library_id()
- def _get_pipelex_plx_files_from_dirs(self, dirs: set[Path]) -> list[Path]:
- """Get all valid Pipelex PLX files from the given directories."""
- all_plx_paths: list[Path] = []
- seen_files: set[str] = set() # Track by absolute path to avoid duplicates
-
- for dir_path in dirs:
- if not dir_path.exists():
- log.verbose(f"Directory does not exist, skipping: {dir_path}")
- continue
-
- # Find all .plx files in the directory, excluding problematic directories
- plx_files = find_plx_files_in_dir(
- dir_path=str(dir_path),
- pattern="*.plx",
- is_recursive=True,
- )
-
- # Filter to only include valid Pipelex files
- for plx_file in plx_files:
- absolute_path = str(plx_file.resolve())
-
- # Skip if already seen
- if absolute_path in seen_files:
- log.verbose(f"Skipping duplicate PLX file: {plx_file}")
- continue
+ if library_id in self._libraries:
+ the_library = self._libraries[library_id]
+ else:
+ the_library = LibraryFactory.make_empty()
+ self._libraries[library_id] = the_library
- if PipelexInterpreter.is_pipelex_file(plx_file):
- all_plx_paths.append(plx_file)
- seen_files.add(absolute_path)
- else:
- log.verbose(f"Skipping non-Pipelex PLX file: {plx_file}")
+ return library_id, the_library
- return all_plx_paths
+ ############################################################
+ # Public library accessors
+ ############################################################
@override
- def load_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> list[PipeAbstract]:
- """Load a blueprint."""
- # Create and load domain
- try:
- domain = self._load_domain_from_blueprint(blueprint)
- except DomainDefinitionError as pipe_def_error:
- msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
- raise DomainLoadingError(
- message=msg, domain_code=pipe_def_error.domain_code, description=pipe_def_error.description, source=pipe_def_error.source
- ) from pipe_def_error
- self.domain_library.add_domain(domain=domain)
-
- # Create and load concepts
- try:
- concepts = self._load_concepts_from_blueprint(blueprint)
- except ConceptDefinitionError as pipe_def_error:
- msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
- raise ConceptLoadingError(
- message=msg,
- concept_definition_error=pipe_def_error,
- concept_code=pipe_def_error.concept_code,
- description=pipe_def_error.description,
- source=pipe_def_error.source,
- ) from pipe_def_error
- self.concept_library.add_concepts(concepts=concepts)
-
- # Create and load pipes
- try:
- pipes = self._load_pipes_from_blueprint(blueprint)
- except PipeDefinitionError as pipe_def_error:
- msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
- raise PipeLoadingError(
- message=msg,
- pipe_definition_error=PipeDefinitionErrorData(
- message=pipe_def_error.message,
- domain_code=pipe_def_error.domain_code,
- pipe_code=pipe_def_error.pipe_code,
- description=pipe_def_error.description,
- source=pipe_def_error.source,
- ),
- pipe_code=pipe_def_error.pipe_code or "",
- description=pipe_def_error.description or "",
- source=pipe_def_error.source,
- ) from pipe_def_error
- self.pipe_library.add_pipes(pipes=pipes)
-
- return pipes
+ def get_library(self, library_id: str) -> Library:
+ if library_id not in self._libraries:
+ msg = f"Library '{library_id}' does not exist"
+ raise LibraryError(msg)
+ return self._libraries[library_id]
@override
- def remove_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
- if blueprint.pipe is not None:
- self.pipe_library.remove_pipes_by_codes(pipe_codes=list(blueprint.pipe.keys()))
-
- # Remove concepts (they may depend on domain)
- if blueprint.concept is not None:
- concept_codes_to_remove = [
- ConceptFactory.make_concept_string_with_domain(domain=blueprint.domain, concept_code=concept_code)
- for concept_code in blueprint.concept
- ]
- self.concept_library.remove_concepts_by_concept_strings(concept_strings=concept_codes_to_remove)
-
- self.domain_library.remove_domain_by_code(domain_code=blueprint.domain)
-
- def _load_domain_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> Domain:
- return DomainFactory.make_from_blueprint(
- blueprint=DomainBlueprint(
- source=blueprint.source,
- code=blueprint.domain,
- description=blueprint.description or "",
- system_prompt=blueprint.system_prompt,
- main_pipe=blueprint.main_pipe,
- ),
- )
-
- def _load_concepts_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> list[Concept]:
- if blueprint.concept is None:
- return []
-
- concepts: list[Concept] = []
- for concept_code, concept_blueprint_or_description in blueprint.concept.items():
- concept = ConceptFactory.make_from_blueprint_or_description(
- domain=blueprint.domain,
- concept_code=concept_code,
- concept_codes_from_the_same_domain=list(blueprint.concept.keys()),
- concept_blueprint_or_description=concept_blueprint_or_description,
- )
- concepts.append(concept)
- return concepts
+ def get_current_library(self) -> Library:
+ library_id = get_current_library()
+ if library_id not in self._libraries:
+ msg = f"No current library set. Library '{library_id}' does not exist"
+ raise LibraryError(msg)
+ return self._libraries[library_id]
- def _load_pipes_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> list[PipeAbstract]:
- pipes: list[PipeAbstract] = []
- if blueprint.pipe is not None:
- for pipe_name, pipe_blueprint in blueprint.pipe.items():
- pipe = PipeFactory.make_from_blueprint(
- domain=blueprint.domain,
- pipe_code=pipe_name,
- blueprint=pipe_blueprint,
- concept_codes_from_the_same_domain=list(blueprint.concept.keys()) if blueprint.concept else None,
- )
- pipes.append(pipe)
- return pipes
-
- def _import_pipelex_modules_directly(self) -> None:
- """Import pipelex modules to register @pipe_func decorated functions.
-
- This ensures critical pipelex functions are registered regardless of how pipelex
- is installed (wheel, source, relative path, etc.).
- """
- import pipelex.builder # noqa: PLC0415 - intentional local import
-
- log.verbose("Registering @pipe_func functions from pipelex.builder")
- functions_count = FuncRegistryUtils.register_pipe_funcs_from_package("pipelex.builder", pipelex.builder)
- log.verbose(f"Registered {functions_count} @pipe_func functions from pipelex.builder")
+ ############################################################
+ # Private methods
+ ############################################################
@override
def load_libraries(
self,
+ library_id: str,
library_dirs: list[Path] | None = None,
library_file_paths: list[Path] | None = None,
) -> None:
- # Collect directories to scan (user project directories)
- user_dirs: set[Path] = set()
- if library_dirs:
- user_dirs.update(library_dirs)
- else:
- user_dirs.add(Path(config_manager.local_root_dir))
+ # Ensure libraries exist for this library_id
+ if library_id not in self._libraries:
+ msg = f"Trying to load a library that does not exist: '{library_id}'"
+ raise LibraryError(msg)
+
+ if not library_dirs:
+ library_dirs = [Path()]
+
+ all_dirs: list[Path] = []
+ all_plx_paths: list[Path] = []
+ all_dirs.extend(library_dirs)
+ all_plx_paths.extend(get_pipelex_plx_files_from_dirs(set(library_dirs)))
- valid_plx_paths: list[Path]
if library_file_paths:
- valid_plx_paths = library_file_paths
- else:
- # Get PLX files from user directories
- user_plx_paths: list[Path] = self._get_pipelex_plx_files_from_dirs(user_dirs)
-
- # Get PLX files from pipelex package using importlib.resources
- # This works reliably in all installation modes (wheel, source, relative)
- pipelex_plx_paths: list[Path] = get_pipelex_plx_files_from_package()
-
- # Combine and deduplicate
- all_plx_paths = user_plx_paths + pipelex_plx_paths
- seen_absolute_paths: set[str] = set()
- valid_plx_paths = []
- for plx_path in all_plx_paths:
- try:
- absolute_path = str(plx_path.resolve())
- except (OSError, RuntimeError):
- # For paths that can't be resolved (e.g., in zipped packages), use string representation
- absolute_path = str(plx_path)
-
- if absolute_path not in seen_absolute_paths:
- valid_plx_paths.append(plx_path)
- seen_absolute_paths.add(absolute_path)
-
- # Import modules to load them into sys.modules (but don't register classes yet)
+ all_plx_paths.extend(library_file_paths)
+
+ # Combine and deduplicate
+ seen_absolute_paths: set[str] = set()
+ valid_plx_paths: list[Path] = []
+ for plx_path in all_plx_paths:
+ try:
+ absolute_path = str(plx_path.resolve())
+ except (OSError, RuntimeError):
+ # For paths that can't be resolved (e.g., in zipped packages), use string representation
+ absolute_path = str(plx_path)
+
+ if absolute_path not in seen_absolute_paths:
+ valid_plx_paths.append(plx_path)
+ seen_absolute_paths.add(absolute_path)
+
+ # Import modules and register in global registries
# Import from user directories
- for library_dir in user_dirs:
+ for library_dir in all_dirs:
# Only import files that contain StructuredContent subclasses (uses AST pre-check)
ClassRegistryUtils.import_modules_in_folder(
folder_path=str(library_dir),
@@ -303,112 +156,134 @@ class LibraryManager(LibraryManagerAbstract):
folder_path=str(library_dir),
)
- # Import from pipelex package
- # Always directly import critical builder modules first (works in all installation modes)
- log.verbose("About to import pipelex.builder modules for @pipe_func registration")
- self._import_pipelex_modules_directly()
-
- # Verify critical functions were registered
- from pipelex.system.registries.func_registry import func_registry # noqa: PLC0415 - intentional local import
-
- critical_functions = ["create_concept_spec", "assemble_pipelex_bundle_spec"]
- for func_name in critical_functions:
- if func_registry.has_function(func_name):
- log.verbose(f"✓ Function '{func_name}' successfully registered")
- else:
- log.error(f"✗ Function '{func_name}' NOT registered - this will cause errors!")
-
- # Then try filesystem-based scanning if package is accessible (for completeness)
- pipelex_pkg_dir = get_pipelex_package_dir_for_imports()
- if pipelex_pkg_dir:
- log.verbose(f"Additionally scanning pipelex package filesystem: {pipelex_pkg_dir}")
- ClassRegistryUtils.import_modules_in_folder(
- folder_path=str(pipelex_pkg_dir),
- base_class_names=[StructuredContent.__name__],
- )
- FuncRegistryUtils.register_funcs_in_folder(
- folder_path=str(pipelex_pkg_dir),
- )
-
# Auto-discover and register all StructuredContent classes from sys.modules
- num_registered = ClassRegistryUtils.auto_register_all_subclasses(base_class=StructuredContent)
- log.verbose(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")
+ num_registered = ClassRegistryUtils.auto_register_all_subclasses(
+ base_class=StructuredContent,
+ )
+ log.debug(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")
- # Parse all blueprints first
- blueprints: list[PipelexBundleBlueprint] = []
- for plx_file_path in valid_plx_paths:
- try:
- blueprint = PipelexInterpreter(file_path=plx_file_path).make_pipelex_bundle_blueprint()
- except FileNotFoundError as file_not_found_error:
- msg = f"Could not find PLX bundle at '{plx_file_path}'"
- raise LibraryLoadingError(msg) from file_not_found_error
- except PipeDefinitionError as pipe_def_error:
- msg = f"Could not load PLX bundle from '{plx_file_path}' because of: {pipe_def_error}"
- raise LibraryLoadingError(msg) from pipe_def_error
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
- msg = f"Could not load PLX bundle from '{plx_file_path}' because of: {validation_error_msg}"
- raise LibraryLoadingError(msg) from validation_error
- blueprint.source = str(plx_file_path)
- blueprints.append(blueprint)
+ # Load PLX files into the specific library
- self.loaded_plx_paths.extend([str(plx_file_path) for plx_file_path in valid_plx_paths])
+ self._load_plx_files_into_library(library_id=library_id, valid_plx_paths=valid_plx_paths)
+
+ @override
+ def load_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
+ """Load domains, concepts, and pipes from a list of blueprints.
+
+ Args:
+ library_id: The ID of the library to load into
+ blueprints: List of parsed PLX blueprints to load
+
+ Returns:
+ List of all pipes that were loaded
+ """
+ library = self.get_library(library_id=library_id)
+ all_pipes: list[PipeAbstract] = []
# Load all domains first
all_domains: list[Domain] = []
for blueprint in blueprints:
- try:
- domain = self._load_domain_from_blueprint(blueprint)
- except DomainDefinitionError as domain_def_error:
- msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {domain_def_error}"
- raise LibraryLoadingError(msg) from domain_def_error
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
- msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"
- raise LibraryLoadingError(msg) from validation_error
+ domain = DomainFactory.make_from_blueprint(
+ blueprint=DomainBlueprint(
+ source=blueprint.source,
+ code=blueprint.domain,
+ description=blueprint.description or "",
+ system_prompt=blueprint.system_prompt,
+ ),
+ )
all_domains.append(domain)
- self.domain_library.add_domains(domains=all_domains)
+ library.domain_library.add_domains(domains=all_domains)
# Load all concepts second
all_concepts: list[Concept] = []
for blueprint in blueprints:
- try:
- concepts = self._load_concepts_from_blueprint(blueprint)
- except ConceptDefinitionError as concept_def_error:
- msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {concept_def_error}"
- raise LibraryLoadingError(
- msg,
- concept_definition_errors=[concept_def_error.as_structured_content()],
- ) from concept_def_error
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
- msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"
- raise LibraryLoadingError(msg) from validation_error
- all_concepts.extend(concepts)
- self.concept_library.add_concepts(concepts=all_concepts)
+ if blueprint.concept is not None:
+ concepts: list[Concept] = []
+ for concept_code, concept_blueprint_or_description in blueprint.concept.items():
+ concept = ConceptFactory.make_from_blueprint_or_description(
+ domain=blueprint.domain,
+ concept_code=concept_code,
+ concept_codes_from_the_same_domain=list(blueprint.concept.keys()),
+ concept_blueprint_or_description=concept_blueprint_or_description,
+ )
+ concepts.append(concept)
+ all_concepts.extend(concepts)
+ library.concept_library.add_concepts(concepts=all_concepts)
# Load all pipes third
- all_pipes: list[PipeAbstract] = []
for blueprint in blueprints:
+ pipes: list[PipeAbstract] = []
+ if blueprint.pipe is not None:
+ for pipe_name, pipe_blueprint in blueprint.pipe.items():
+ pipe = PipeFactory.make_from_blueprint(
+ domain=blueprint.domain,
+ pipe_code=pipe_name,
+ blueprint=pipe_blueprint,
+ concept_codes_from_the_same_domain=list(blueprint.concept.keys()) if blueprint.concept else None,
+ )
+ pipes.append(pipe)
+ all_pipes.extend(pipes)
+
+ library.pipe_library.add_pipes(pipes=all_pipes)
+
+ library.validate_library()
+ return all_pipes
+
+ ############################################################
+ # Private helper methods
+ ############################################################
+
+ def _load_plx_files_into_library(self, library_id: str, valid_plx_paths: list[Path]) -> None:
+ """Load PLX files into a specific library.
+
+ This method:
+ 1. Parses blueprints from PLX files
+ 2. Loads blueprints into the specified library
+
+ Args:
+ library_id: The ID of the library to load into
+ valid_plx_paths: List of PLX file paths to load
+ """
+ blueprints: list[PipelexBundleBlueprint] = []
+ for plx_file_path in valid_plx_paths:
try:
- pipes = self._load_pipes_from_blueprint(blueprint)
- except PipeDefinitionError as pipe_def_error:
- msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=str(plx_file_path))
+ blueprint.source = str(plx_file_path)
+ except FileNotFoundError as file_not_found_error:
+ msg = f"Could not find PLX bundle at '{plx_file_path}'"
+ raise LibraryLoadingError(msg) from file_not_found_error
+ except PipelexInterpreterError as interpreter_error:
+ # Forward BLUEPRINT validation errors from interpreter
+ msg = f"Could not load PLX bundle from '{plx_file_path}' because of: {interpreter_error.message}"
raise LibraryLoadingError(
- msg,
- pipe_definition_errors=[
- PipeDefinitionErrorData(
- message=pipe_def_error.message,
- domain_code=pipe_def_error.domain_code,
- pipe_code=pipe_def_error.pipe_code,
- description=pipe_def_error.description,
- source=pipe_def_error.source,
- )
- ],
- ) from pipe_def_error
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
- msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"
- raise LibraryLoadingError(msg) from validation_error
- all_pipes.extend(pipes)
- self.pipe_library.add_pipes(pipes=all_pipes)
+ message=msg,
+ blueprint_validation_errors=interpreter_error.validation_errors,
+ ) from interpreter_error
+ blueprints.append(blueprint)
+
+ self.loaded_plx_paths.extend([str(plx_file_path) for plx_file_path in valid_plx_paths])
+ self.load_from_blueprints(library_id=library_id, blueprints=blueprints)
+
+ def _remove_pipes_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
+ library = self.get_current_library()
+ if blueprint.pipe is not None:
+ library.pipe_library.remove_pipes_by_codes(pipe_codes=list(blueprint.pipe.keys()))
+
+ def _remove_concepts_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
+ library = self.get_current_library()
+ if blueprint.concept is not None:
+ concept_codes_to_remove = [
+ ConceptFactory.make_concept_string_with_domain(domain=blueprint.domain, concept_code=concept_code)
+ for concept_code in blueprint.concept
+ ]
+ library.concept_library.remove_concepts_by_concept_strings(concept_strings=concept_codes_to_remove)
+
+ @override
+ def _remove_from_blueprint(self, library_id: str, blueprint: PipelexBundleBlueprint) -> None:
+ self._remove_pipes_from_blueprint(blueprint=blueprint)
+ self._remove_concepts_from_blueprint(blueprint=blueprint)
+
+ @override
+ def _remove_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> None:
+ for blueprint in blueprints:
+ self._remove_from_blueprint(library_id=library_id, blueprint=blueprint)
diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.py
index b3b6b3b83..a6e2061ae 100644
--- a/pipelex/libraries/library_manager_abstract.py
+++ b/pipelex/libraries/library_manager_abstract.py
@@ -1,9 +1,13 @@
from abc import ABC, abstractmethod
from pathlib import Path
+from typing import TYPE_CHECKING
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.pipes.pipe_abstract import PipeAbstract
+if TYPE_CHECKING:
+ from pipelex.libraries.library import Library
+
class LibraryManagerAbstract(ABC):
@abstractmethod
@@ -11,7 +15,7 @@ class LibraryManagerAbstract(ABC):
pass
@abstractmethod
- def teardown(self) -> None:
+ def teardown(self, library_id: str | None = None) -> None:
pass
@abstractmethod
@@ -19,21 +23,34 @@ class LibraryManagerAbstract(ABC):
pass
@abstractmethod
- def validate_libraries(self) -> None:
- pass
+ def open_library(self, library_id: str | None = None) -> tuple[str, "Library"]:
+ """Open a library with the given library_id. Creates it if it doesn't exist. If no library_id is provided, it creates one."""
+
+ @abstractmethod
+ def get_library(self, library_id: str) -> "Library":
+ """Get the Library object for a specific library_id."""
+
+ @abstractmethod
+ def get_current_library(self) -> "Library":
+ """Get the Library object for the current library."""
@abstractmethod
- def get_loaded_plx_paths(self) -> list[str]:
+ def load_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
pass
@abstractmethod
- def load_libraries(self, library_dirs: list[Path] | None = None, library_file_paths: list[Path] | None = None) -> None:
+ def _remove_from_blueprint(self, library_id: str, blueprint: PipelexBundleBlueprint) -> None:
pass
@abstractmethod
- def load_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> list[PipeAbstract]:
+ def _remove_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> None:
pass
@abstractmethod
- def remove_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
+ def load_libraries(
+ self,
+ library_id: str,
+ library_dirs: list[Path] | None = None,
+ library_file_paths: list[Path] | None = None,
+ ) -> None:
pass
diff --git a/pipelex/libraries/library_manager_factory.py b/pipelex/libraries/library_manager_factory.py
deleted file mode 100644
index 47631d1df..000000000
--- a/pipelex/libraries/library_manager_factory.py
+++ /dev/null
@@ -1,34 +0,0 @@
-from pipelex.core.concepts.concept_library import ConceptLibrary
-from pipelex.core.domains.domain_library import DomainLibrary
-from pipelex.core.pipes.pipe_library import PipeLibrary
-from pipelex.libraries.library_manager import LibraryManager
-
-
-class LibraryManagerFactory:
- """Factory for creating LibraryManager instances."""
-
- @classmethod
- def make_empty(cls) -> "LibraryManager":
- domain_library = DomainLibrary.make_empty()
- concept_library = ConceptLibrary.make_empty()
- pipe_library = PipeLibrary.make_empty()
-
- return LibraryManager(
- domain_library=domain_library,
- concept_library=concept_library,
- pipe_library=pipe_library,
- )
-
- @classmethod
- def make(
- cls,
- domain_library: DomainLibrary,
- concept_library: ConceptLibrary,
- pipe_library: PipeLibrary,
- ) -> "LibraryManager":
- """Create a LibraryManager with provided libraries."""
- return LibraryManager(
- domain_library=domain_library,
- concept_library=concept_library,
- pipe_library=pipe_library,
- )
diff --git a/pipelex/libraries/library_utils.py b/pipelex/libraries/library_utils.py
index 40c58a6c2..1dad303fd 100644
--- a/pipelex/libraries/library_utils.py
+++ b/pipelex/libraries/library_utils.py
@@ -1,12 +1,10 @@
-"""Utility functions for library management."""
-
from importlib.abc import Traversable
from importlib.resources import files
from pathlib import Path
from pipelex import log
from pipelex.config import get_config
-from pipelex.core.interpreter import PipelexInterpreter
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
from pipelex.tools.misc.file_utils import find_files_in_dir
@@ -89,3 +87,29 @@ def find_plx_files_in_dir(dir_path: str, pattern: str, is_recursive: bool) -> li
filtered_files.append(file_path)
return filtered_files
+
+
+def get_pipelex_plx_files_from_dirs(dirs: set[Path]) -> list[Path]:
+ """Get all valid Pipelex PLX files from the given directories."""
+ all_plx_paths: list[Path] = []
+
+ for dir_path in dirs:
+ if not dir_path.exists():
+ log.debug(f"Directory does not exist, skipping: {dir_path}")
+ continue
+
+ # Find all .plx files in the directory, excluding problematic directories
+ plx_files = find_plx_files_in_dir(
+ dir_path=str(dir_path),
+ pattern="*.plx",
+ is_recursive=True,
+ )
+
+ # Filter to only include valid Pipelex files
+ for plx_file in plx_files:
+ if PipelexInterpreter.is_pipelex_file(plx_file):
+ all_plx_paths.append(plx_file)
+ else:
+ log.debug(f"Skipping non-Pipelex PLX file: {plx_file}")
+
+ return all_plx_paths
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/__init__.py b/pipelex/libraries/pipe/__init__.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_parallel/__init__.py
rename to pipelex/libraries/pipe/__init__.py
diff --git a/pipelex/libraries/pipe/exceptions.py b/pipelex/libraries/pipe/exceptions.py
new file mode 100644
index 000000000..b463b893b
--- /dev/null
+++ b/pipelex/libraries/pipe/exceptions.py
@@ -0,0 +1,9 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeLibraryError(PipelexError):
+ pass
+
+
+class PipeNotFoundError(PipeLibraryError):
+ pass
diff --git a/pipelex/core/pipes/pipe_library.py b/pipelex/libraries/pipe/pipe_library.py
similarity index 74%
rename from pipelex/core/pipes/pipe_library.py
rename to pipelex/libraries/pipe/pipe_library.py
index 9d404a8d5..44a9fe071 100644
--- a/pipelex/core/pipes/pipe_library.py
+++ b/pipelex/libraries/pipe/pipe_library.py
@@ -6,11 +6,9 @@ from rich.table import Table
from typing_extensions import override
from pipelex import pretty_print
-from pipelex.core.concepts.exceptions import ConceptError, ConceptLibraryConceptNotFoundError
from pipelex.core.pipes.pipe_abstract import PipeAbstract
-from pipelex.core.pipes.pipe_library_abstract import PipeLibraryAbstract
-from pipelex.hub import get_concept_library
-from pipelex.libraries.exceptions import PipeLibraryError, PipeLibraryPipeNotFoundError
+from pipelex.libraries.pipe.exceptions import PipeLibraryError, PipeNotFoundError
+from pipelex.libraries.pipe.pipe_library_abstract import PipeLibraryAbstract
from pipelex.types import Self
PipeLibraryRoot = dict[str, PipeAbstract]
@@ -18,27 +16,23 @@ PipeLibraryRoot = dict[str, PipeAbstract]
class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
@override
- def validate_with_libraries(self):
- concept_library = get_concept_library()
- for pipe in self.root.values():
- pipe.validate_output()
- try:
- for concept in pipe.concept_dependencies():
- try:
- concept_library.get_required_concept(concept_string=concept.concept_string)
- except ConceptError as concept_error:
- msg = f"Error validating pipe '{pipe.code}' dependency concept '{concept.concept_string}' because of: {concept_error}"
- raise PipeLibraryError(msg) from concept_error
- for pipe_code in pipe.pipe_dependencies():
- self.get_required_pipe(pipe_code=pipe_code)
- pipe.validate_with_libraries()
- except (ConceptLibraryConceptNotFoundError, PipeLibraryPipeNotFoundError) as not_found_error:
- msg = f"Missing dependency for pipe '{pipe.code}': {not_found_error}"
- raise PipeLibraryError(msg) from not_found_error
+ def setup(self):
+ pass
+
+ @override
+ def teardown(self):
+ self.root = {}
+
+ @override
+ def reset(self):
+ self.teardown()
+ self.setup()
@classmethod
def make_empty(cls) -> Self:
- return cls(root={})
+ library = cls(root=PipeLibraryRoot())
+ library.setup()
+ return library
@override
def add_new_pipe(self, pipe: PipeAbstract):
@@ -65,7 +59,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
the_pipe = self.get_optional_pipe(pipe_code=pipe_code)
if not the_pipe:
msg = f"Pipe '{pipe_code}' not found. Check for typos and make sure it is declared in plx file in an imported package."
- raise PipeLibraryPipeNotFoundError(msg)
+ raise PipeNotFoundError(msg)
return the_pipe
@override
@@ -86,11 +80,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
del self.root[pipe_code]
@override
- def teardown(self) -> None:
- self.root = {}
-
- @override
- def pretty_list_pipes(self) -> int:
+ def pretty_list_pipes(self) -> None:
def _format_concept_code(concept_code: str | None, current_domain: str) -> str:
"""Format concept code by removing domain prefix if it matches current domain."""
if not concept_code:
@@ -146,4 +136,3 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
}
pretty_print(table)
- return len(pipes)
diff --git a/pipelex/core/pipes/pipe_library_abstract.py b/pipelex/libraries/pipe/pipe_library_abstract.py
similarity index 84%
rename from pipelex/core/pipes/pipe_library_abstract.py
rename to pipelex/libraries/pipe/pipe_library_abstract.py
index a33fc5f29..12e035400 100644
--- a/pipelex/core/pipes/pipe_library_abstract.py
+++ b/pipelex/libraries/pipe/pipe_library_abstract.py
@@ -5,9 +5,18 @@ from pipelex.core.pipes.pipe_abstract import PipeAbstract
class PipeLibraryAbstract(ABC):
@abstractmethod
- def validate_with_libraries(self) -> None:
+ def setup(self) -> None:
pass
+ @abstractmethod
+ def teardown(self) -> None:
+ pass
+
+ @abstractmethod
+ def reset(self) -> None:
+ self.teardown()
+ self.setup()
+
@abstractmethod
def get_required_pipe(self, pipe_code: str) -> PipeAbstract:
pass
@@ -28,11 +37,7 @@ class PipeLibraryAbstract(ABC):
pass
@abstractmethod
- def teardown(self) -> None:
- pass
-
- @abstractmethod
- def pretty_list_pipes(self) -> int:
+ def pretty_list_pipes(self) -> None:
pass
@abstractmethod
diff --git a/pipelex/observer/local_observer.py b/pipelex/observer/local_observer.py
index e0428ecc0..317235257 100644
--- a/pipelex/observer/local_observer.py
+++ b/pipelex/observer/local_observer.py
@@ -26,8 +26,8 @@ class LocalObserver(ObserverProtocol):
}
file_path = os.path.join(self.storage_dir, f"{event_type}.jsonl")
- with open(file_path, "a", encoding="utf-8") as f:
- f.write(kajson.dumps(payload) + "\n")
+ with open(file_path, "a", encoding="utf-8") as file:
+ file.write(kajson.dumps(payload) + "\n")
@override
async def observe_before_run(self, payload: PayloadType) -> None:
diff --git a/pipelex/pipe_controllers/batch/exceptions.py b/pipelex/pipe_controllers/batch/exceptions.py
new file mode 100644
index 000000000..b6f63b547
--- /dev/null
+++ b/pipelex/pipe_controllers/batch/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeBatchFactoryError(PipelexError):
+ pass
diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.py
index eeadab176..8469b1344 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch.py
@@ -2,22 +2,19 @@ import asyncio
from typing import TYPE_CHECKING, Any, Literal, cast
import shortuuid
-from pydantic import model_validator
from typing_extensions import override
from pipelex.config import get_config
-from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
-from pipelex.core.pipes.exceptions import PipeInputError, PipeInputNotFoundError, PipeRunInputsError
-from pipelex.core.pipes.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_pipeline_tracker, get_required_pipe
from pipelex.pipe_controllers.pipe_controller import PipeController
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunMode, PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.types import Self
if TYPE_CHECKING:
from collections.abc import Coroutine
@@ -32,37 +29,6 @@ class PipeBatch(PipeController):
branch_pipe_code: str
batch_params: BatchParams
- @override
- def pipe_dependencies(self) -> set[str]:
- return {self.branch_pipe_code}
-
- @override
- def validate_output(self):
- pass
-
- @model_validator(mode="after")
- def validate_required_variables(self) -> Self:
- # Skip for now
- return self
-
- @override
- def _validate_with_libraries(self):
- self._validate_required_variables()
-
- def _validate_required_variables(self) -> Self:
- # Now check that the required variables ARE in the inputs of the pipe
- required_variables = self.required_variables()
- for variable_name in required_variables:
- if variable_name == self.batch_params.input_item_stuff_name:
- continue
- if variable_name not in self.inputs.root:
- msg = (
- f"Input '{variable_name}' required by branch pipe '{self.branch_pipe_code}' "
- f"of PipeBatch '{self.code}', is not listed in its inputs"
- )
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=variable_name, concept_code=None)
- return self
-
@override
def required_variables(self) -> set[str]:
required_variables: set[str] = set()
@@ -80,24 +46,46 @@ class PipeBatch(PipeController):
return self.inputs
@override
- def _validate_inputs_in_memory(self, working_memory: WorkingMemory) -> None:
- try:
- required_concept_code = self.inputs.get_required_input_requirement(variable_name=self.batch_params.input_list_stuff_name).concept.code
- except PipeInputNotFoundError as exc:
+ def pipe_dependencies(self) -> set[str]:
+ return {self.branch_pipe_code}
+
+ @override
+ def validate_inputs_static(self):
+ pass
+
+ @override
+ def validate_inputs_with_library(self):
+ pass
+
+ @override
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
+ pass
+
+ @override
+ async def validate_before_run(
+ self, job_metadata: JobMetadata, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None = None
+ ) -> None:
+ batch_params = pipe_run_params.batch_params or self.batch_params or BatchParams.make_default()
+ input_list_stuff_name = batch_params.input_list_stuff_name
+ if not self.inputs.is_variable_existing(variable_name=input_list_stuff_name):
+ msg = f"Batch input list named '{input_list_stuff_name}' is not in PipeBatch '{self.code}' input requirements: {self.inputs}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
+
+ if not working_memory.is_stuff_exists(input_list_stuff_name):
+ msg = f"Input list stuff '{input_list_stuff_name}' required by this PipeBatch '{self.code}' not found in working memory"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
+
+ input_stuff = working_memory.get_stuff(input_list_stuff_name)
+ if not isinstance(input_stuff.content, ListContent):
msg = (
- f"Batch input list named '{self.batch_params.input_list_stuff_name}' is not in "
- f"PipeBatch '{self.code}' input requirements: {self.inputs}"
+ f"Input list stuff '{input_list_stuff_name}' of PipeBatch '{self.code}' must be ListContent, "
+ f"got {input_stuff.stuff_name or 'unnamed'} = {type(input_stuff.content)}. stuff: {input_stuff}"
)
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=self.batch_params.input_list_stuff_name, concept_code=None) from exc
-
- required_stuff_name = self.batch_params.input_list_stuff_name
- try:
- working_memory.get_stuff(required_stuff_name)
- except WorkingMemoryStuffNotFoundError as exc:
- variable_name: str = exc.variable_name or required_stuff_name
- missing_inputs: dict[str, str] = {variable_name: exc.concept_code or required_concept_code}
- msg = f"Missing required inputs for pipe '{self.code}': {missing_inputs}"
- raise PipeRunInputsError(message=msg, pipe_code=self.code, missing_inputs=missing_inputs) from exc
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
async def _run_batch_pipe(
self,
@@ -110,34 +98,15 @@ class PipeBatch(PipeController):
batch_params = pipe_run_params.batch_params or self.batch_params or BatchParams.make_default()
input_item_stuff_name = batch_params.input_item_stuff_name
input_list_stuff_name = batch_params.input_list_stuff_name
- try:
- input_requirement = self.inputs.get_required_input_requirement(input_list_stuff_name)
- except PipeInputNotFoundError as exc:
- msg = f"Batch input item list named '{input_list_stuff_name}' is not in this PipeBatch '{self.code}' input requirements: {self.inputs}"
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=input_list_stuff_name, concept_code=None) from exc
if pipe_run_params.final_stuff_code:
method_name = "dry_run_pipe" if pipe_run_params.run_mode == PipeRunMode.DRY else "_run_controller_pipe"
pipe_run_params.final_stuff_code = None
pipe_run_params.push_pipe_layer(pipe_code=self.branch_pipe_code)
- try:
- input_stuff = working_memory.get_stuff(batch_params.input_list_stuff_name)
- except WorkingMemoryStuffNotFoundError as exc:
- msg = (
- f"Input list stuff '{batch_params.input_list_stuff_name}' required by this PipeBatch '{self.code}' not found in working memory: {exc}"
- )
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=batch_params.input_list_stuff_name, concept_code=None) from exc
- input_stuff_code = input_stuff.stuff_code
- input_content = input_stuff.content
- if not isinstance(input_content, ListContent):
- msg = (
- f"Input of PipeBatch '{self.code}' must be ListContent, "
- f"got {input_stuff.stuff_name or 'unnamed'} = {type(input_content)}. stuff: {input_stuff}"
- )
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=batch_params.input_list_stuff_name, concept_code=None)
- input_content = cast("ListContent[StuffContent]", input_content)
+ input_stuff = working_memory.get_stuff(input_list_stuff_name)
+ input_content = cast("ListContent[StuffContent]", input_stuff.content)
# TODO: Make commented code work when inputing images named "a.b.c"
sub_pipe = get_required_pipe(pipe_code=self.branch_pipe_code)
@@ -153,10 +122,10 @@ class PipeBatch(PipeController):
branch_output_item_codes.append(branch_output_item_code)
if nb_history_items_limit and branch_index >= nb_history_items_limit:
break
- branch_input_item_code = f"{input_stuff_code}-branch-{branch_index}"
+ branch_input_item_code = f"{input_stuff.stuff_code}-branch-{branch_index}"
item_input_stuff = StuffFactory.make_stuff(
code=branch_input_item_code,
- concept=input_requirement.concept,
+ concept=self.inputs.get_required_input_requirement(input_list_stuff_name).concept,
content=item,
name=input_item_stuff_name,
)
@@ -166,7 +135,7 @@ class PipeBatch(PipeController):
required_variables = sub_pipe.required_variables()
required_stuffs = branch_memory.get_existing_stuffs(names=required_variables)
- required_stuffs = [required_stuff for required_stuff in required_stuffs if required_stuff.stuff_code != input_stuff_code]
+ required_stuffs = [required_stuff for required_stuff in required_stuffs if required_stuff.stuff_code != input_stuff.stuff_code]
required_stuff_lists.append(required_stuffs)
branch_pipe_run_params = pipe_run_params.deep_copy_with_final_stuff_code(final_stuff_code=branch_output_item_code)
@@ -257,7 +226,6 @@ class PipeBatch(PipeController):
pipe_run_params: PipeRunParams,
output_name: str | None = None,
) -> PipeOutput:
- """Run a pipe in batch mode for each item in the input list."""
return await self._run_batch_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
@@ -273,7 +241,6 @@ class PipeBatch(PipeController):
pipe_run_params: PipeRunParams,
output_name: str | None = None,
) -> PipeOutput:
- """Dry run a pipe in batch mode for each item in the input list."""
return await self._run_batch_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
diff --git a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
index 2c537a4d0..9f31a06cc 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
@@ -17,3 +17,11 @@ class PipeBatchBlueprint(PipeBlueprint):
def pipe_dependencies(self) -> set[str]:
"""Return the set containing the branch pipe code."""
return {self.branch_pipe_code}
+
+ @override
+ def validate_inputs(self):
+ pass
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_controllers/batch/pipe_batch_factory.py b/pipelex/pipe_controllers/batch/pipe_batch_factory.py
index b48ee8ca1..52e2724f2 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch_factory.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch_factory.py
@@ -1,10 +1,13 @@
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
+from pipelex.pipe_controllers.batch.exceptions import PipeBatchFactoryError
from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
from pipelex.pipe_run.pipe_run_params import BatchParams
@@ -21,31 +24,43 @@ class PipeBatchFactory(PipeFactoryProtocol[PipeBatchBlueprint, PipeBatch]):
concept_codes_from_the_same_domain: list[str] | None = None,
) -> PipeBatch:
# Parse output to strip multiplicity brackets
- output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ try:
+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ except PipeVariableMultiplicityError as exc:
+ msg = f"Error parsing concept with multiplicity for PipeBatch: {exc}"
+ raise PipeBatchFactoryError(msg) from exc
- output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
+ try:
+ output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
+ domain=domain,
+ concept_string_or_code=output_parse_result.concept,
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ except ConceptFactoryError as exc:
+ msg = f"Error making domain and concept code for PipeBatch: {exc}"
+ raise PipeBatchFactoryError(msg) from exc
+
+ inputs = InputRequirementsFactory.make_from_blueprint(
domain=domain,
- concept_string_or_code=output_parse_result.concept,
+ blueprint=blueprint.inputs or {},
concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
)
+ output = get_required_concept(
+ concept_string=ConceptFactory.make_concept_string_with_domain(
+ domain=output_domain_and_code.domain,
+ concept_code=output_domain_and_code.concept_code,
+ ),
+ )
+ batch_params = BatchParams.make_batch_params(
+ input_list_name=blueprint.input_list_name,
+ input_item_name=blueprint.input_item_name,
+ )
return PipeBatch(
domain=domain,
code=pipe_code,
description=blueprint.description,
- inputs=InputRequirementsFactory.make_from_blueprint(
- domain=domain,
- blueprint=blueprint.inputs or {},
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- ),
- output=get_required_concept(
- concept_string=ConceptFactory.make_concept_string_with_domain(
- domain=output_domain_and_code.domain,
- concept_code=output_domain_and_code.concept_code,
- ),
- ),
+ inputs=inputs,
+ output=output,
branch_pipe_code=blueprint.branch_pipe_code,
- batch_params=BatchParams.make_batch_params(
- input_list_name=blueprint.input_list_name,
- input_item_name=blueprint.input_item_name,
- ),
+ batch_params=batch_params,
)
diff --git a/pipelex/pipe_controllers/condition/exceptions.py b/pipelex/pipe_controllers/condition/exceptions.py
index f5198718b..e1d4f7cf3 100644
--- a/pipelex/pipe_controllers/condition/exceptions.py
+++ b/pipelex/pipe_controllers/condition/exceptions.py
@@ -1,5 +1,5 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class PipeConditionError(PipelexException):
+class PipeConditionFactoryError(PipelexError):
pass
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index aae5ffe2b..056788446 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -1,43 +1,35 @@
from typing import Literal
import shortuuid
-from pydantic import model_validator
from typing_extensions import override
from pipelex import log
from pipelex.cogt.templating.template_category import TemplateCategory
-from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.exceptions import StaticValidationError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.exceptions import PipeInputError, StaticValidationErrorType
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import get_content_generator, get_optional_pipe, get_pipe_router, get_pipeline_tracker, get_required_pipe
-from pipelex.pipe_controllers.condition.exceptions import PipeConditionError
from pipelex.pipe_controllers.condition.pipe_condition_details import PipeConditionDetails
from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
from pipelex.pipe_controllers.pipe_controller import PipeController
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.pipeline.exceptions import DryRunMissingInputsError, DryRunMissingPipesError, DryRunTemplatingError
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.jinja2.jinja2_errors import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
-from pipelex.tools.typing.validation_utils import has_exactly_one_among_attributes_from_list
-from pipelex.types import Self
ConditionOutcomeMap = dict[str, str | SpecialOutcome]
class PipeCondition(PipeController):
type: Literal["PipeCondition"] = "PipeCondition"
- expression_template: str | None = None
- expression: str | None = None
+ expression: str
outcome_map: ConditionOutcomeMap
default_outcome: str | SpecialOutcome
add_alias_from_expression_to: str | None = None
@@ -49,57 +41,19 @@ class PipeCondition(PipeController):
codes.add(self.default_outcome)
return codes - set(SpecialOutcome.value_list())
- #########################################################################################
- # Validation
- #########################################################################################
- @override
- def validate_output(self):
- """Validate the output for the pipe condition.
- The output of the pipe condition should match the output of all the conditional pipes, and the default pipe.
- """
- for pipe_code in self.mapped_pipe_codes:
- pipe = get_required_pipe(pipe_code=pipe_code)
- if self.output.concept_string not in (
- pipe.output.concept_string,
- NativeConceptCode.DYNAMIC.concept_string,
- NativeConceptCode.ANYTHING.concept_string,
- ):
- msg = (
- f"The output concept code '{self.output.concept_string}' of the pipe '{self.code}' is "
- f"not matching the output concept code '{pipe.output.concept_string}' of the pipe '{pipe_code}'"
- )
- raise PipeConditionError(msg)
-
- @model_validator(mode="after")
- def validate_expression(self) -> Self:
- if not has_exactly_one_among_attributes_from_list(self, attributes_list=["expression_template", "expression"]):
- msg = "PipeCondition should have exactly one of expression_template or expression"
- raise PipeDefinitionError(message=msg, domain_code=self.domain, pipe_code=self.code, description=self.description)
- return self
-
def _make_pipe_condition_details(self, evaluated_expression: str, chosen_pipe_code: str) -> PipeConditionDetails:
return PipeConditionDetails(
code=shortuuid.uuid()[:5],
- test_expression=self.expression or self.applied_expression_template,
+ test_expression=self.expression,
outcomes=self.outcome_map,
default_pipe_code=self.default_outcome,
evaluated_expression=evaluated_expression,
chosen_pipe_code=chosen_pipe_code,
)
- @property
- def applied_expression_template(self) -> str:
- if self.expression_template:
- return self.expression_template
- elif self.expression:
- return "{{ " + self.expression + " }}"
- else:
- msg = "No expression or expression_template provided"
- raise PipeDefinitionError(message=msg, domain_code=self.domain, pipe_code=self.code, description=self.description)
-
- #########################################################################################
- # Inputs
- #########################################################################################
+ @override
+ def pipe_dependencies(self) -> set[str]:
+ return self.mapped_pipe_codes
@override
def required_variables(self) -> set[str]:
@@ -107,7 +61,7 @@ class PipeCondition(PipeController):
# Variables from the expression/expression_template
expression_required_variables = detect_jinja2_required_variables(
template_category=TemplateCategory.EXPRESSION,
- template_source=self.applied_expression_template,
+ template_source=self.expression,
)
required_variables.update(expression_required_variables)
@@ -116,13 +70,6 @@ class PipeCondition(PipeController):
required_variables.update(get_required_pipe(pipe_code=pipe_code).required_variables())
return required_variables
- def _validate_required_variables(self) -> Self:
- for required_variable_name in self.required_variables():
- if required_variable_name not in self.inputs.variables:
- msg = f"Required variable '{required_variable_name}' is not in the inputs of pipe {self.code}"
- raise PipeDefinitionError(message=msg, domain_code=self.domain, pipe_code=self.code, description=self.description)
- return self
-
@override
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
if visited_pipes is None:
@@ -140,7 +87,7 @@ class PipeCondition(PipeController):
# 1. Add the variables from the expression/expression_template
required_variables = detect_jinja2_required_variables(
template_category=TemplateCategory.EXPRESSION,
- template_source=self.applied_expression_template,
+ template_source=self.expression,
)
for var_name in required_variables:
@@ -162,71 +109,65 @@ class PipeCondition(PipeController):
for input_name, requirement in pipe_needed_inputs.items:
needed_inputs.add_requirement(variable_name=input_name, concept=requirement.concept)
-
return needed_inputs
- @model_validator(mode="after")
- def validate_inputs(self) -> Self:
- if not self.outcome_map:
- msg = f"Pipe'{self.code}'(PipeCondition) must have at least one mapping in outcomes"
- raise PipeDefinitionError(message=msg, domain_code=self.domain, pipe_code=self.code, description=self.description)
-
- # Skip validation during model creation - it will be done in validate_with_libraries()
- return self
+ @override
+ def validate_inputs_static(self):
+ pass
- def _validate_inputs(self):
- """Validate that the inputs declared for this PipeCondition match what is actually needed."""
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
+ @override
+ def validate_inputs_with_library(self):
+ pass
- the_needed_inputs = self.needed_inputs()
+ @override
+ def validate_output_static(self):
+ pass
- # Check all required variables are in the inputs
- for named_input_requirement in the_needed_inputs.named_input_requirements:
- if named_input_requirement.variable_name not in self.inputs.variables:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[named_input_requirement.variable_name],
+ @override
+ def validate_output_with_library(self):
+ """Validate the output for the pipe condition.
+ The output of the pipe condition should match the output of all the conditional pipes, and the default pipe.
+ """
+ for pipe_code in self.mapped_pipe_codes:
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ if self.output.concept_string not in (
+ pipe.output.concept_string,
+ NativeConceptCode.DYNAMIC.concept_string,
+ NativeConceptCode.ANYTHING.concept_string,
+ ):
+ msg = (
+ f"The output concept code '{self.output.concept_string}' of the pipe '{self.code}' is not "
+ f"matching the output concept code '{pipe.output.concept_string}' of the pipe '{pipe_code}'"
)
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # Check that all declared inputs are actually needed
- for input_name in self.inputs.variables:
- if input_name not in the_needed_inputs.required_names:
- extraneous_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
domain=self.domain,
pipe_code=self.code,
- variable_names=[input_name],
+ provided_concept_code=pipe.output.concept_string,
+ required_concept_codes=[self.output.concept_string],
)
- match reactions.get(StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(extraneous_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise extraneous_input_var_error
-
- @override
- def _validate_with_libraries(self):
- """Perform full validation after all libraries are loaded.
- This is called after all pipes and concepts are available.
- """
- self._validate_inputs()
- self._validate_required_variables()
- @override
- def pipe_dependencies(self) -> set[str]:
- return self.mapped_pipe_codes
+ # TODO: Restore this validation. The problem lies with needed_inputs that construct Anything concepts.
+ # @override
+ # async def validate_before_run(
+ # self,
+ # job_metadata: JobMetadata,
+ # working_memory: WorkingMemory,
+ # pipe_run_params: PipeRunParams,
+ # output_name: str | None = None,
+ # ):
+ # evaluated_expression = await get_content_generator().make_templated_text(
+ # context=working_memory.generate_context(),
+ # template=self.expression,
+ # template_category=TemplateCategory.EXPRESSION,
+ # )
+ # if not evaluated_expression or evaluated_expression == "None":
+ # error_msg = f"PipeCondition '{self.code}': Conditional expression returned no result"
+ # raise PipeRunError(
+ # message=error_msg,
+ # run_mode=pipe_run_params.run_mode,
+ # )
async def _evaluate_expression(
self,
@@ -239,27 +180,13 @@ class PipeCondition(PipeController):
Returns:
The evaluated expression
-
- Raises:
- PipeConditionError: If expression evaluation fails or no matching pipe is found
"""
- content_generator = get_content_generator()
-
- # Evaluate the expression using templating
- evaluated_expression = await content_generator.make_templated_text(
+ evaluated_expression = await get_content_generator().make_templated_text(
context=working_memory.generate_context(),
- template=self.applied_expression_template,
+ template=self.expression,
template_category=TemplateCategory.EXPRESSION,
)
- # Validate the evaluated expression
- if not evaluated_expression or evaluated_expression == "None":
- error_msg = f"Conditional expression returned no result in pipe {self.code}:"
- error_msg += f"\n\nExpression: {self.applied_expression_template}"
- raise PipeConditionError(error_msg)
- log.verbose(f"evaluated_expression: '{evaluated_expression}'")
-
- # Add alias if configured
log.verbose(f"add_alias: {evaluated_expression} -> {self.add_alias_from_expression_to}")
if self.add_alias_from_expression_to:
working_memory.add_alias(
@@ -294,13 +221,13 @@ class PipeCondition(PipeController):
if SpecialOutcome.is_fail(outcome):
msg = f"PipeCondition '{self.code}' failed with outcome: {outcome}. Evaluated expression: {evaluated_expression}"
- raise PipeConditionError(message=msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
chosen_pipe = get_required_pipe(pipe_code=outcome)
# Create condition details for tracking
condition_details = self._make_pipe_condition_details(
- evaluated_expression=self.applied_expression_template,
+ evaluated_expression=self.expression,
chosen_pipe_code=chosen_pipe.code,
)
@@ -314,14 +241,14 @@ class PipeCondition(PipeController):
pipe_condition_path_str = ".".join(pipe_condition_path)
error_details = f"PipeCondition '{pipe_condition_path_str}', required_variables: {required_variables}, missing: '{exc.variable_name}'"
msg = f"Some required stuff(s) not found: {error_details}"
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=exc.variable_name, concept_code=None) from exc
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
# Track condition steps
for required_stuff in required_stuffs:
get_pipeline_tracker().add_condition_step(
from_stuff=required_stuff,
to_condition=condition_details,
- condition_expression=self.expression or self.applied_expression_template,
+ condition_expression=self.expression,
pipe_layer=pipe_run_params.pipe_layers,
comment="PipeCondition required for condition",
)
@@ -359,7 +286,6 @@ class PipeCondition(PipeController):
Validates that all required inputs are present, expression is valid, and target pipes exist.
"""
log.verbose(f"PipeCondition: dry run controller pipe: {self.code}")
-
# 1. Validate that all required inputs are present in the working memory
needed_inputs = self.needed_inputs()
missing_input_names: list[str] = []
@@ -370,33 +296,25 @@ class PipeCondition(PipeController):
if missing_input_names:
log.error(f"Dry run failed: missing required inputs: {missing_input_names}")
- raise DryRunMissingInputsError(
+ raise PipeRunError(
message=f"Dry run failed for pipe '{self.code}' (PipeCondition): missing required inputs: {', '.join(missing_input_names)}",
- pipe_type=self.__class__.__name__,
- pipe_code=self.code,
- missing_inputs=missing_input_names,
+ run_mode=pipe_run_params.run_mode,
)
# 2. Validate that the expression template is valid
try:
required_variables = detect_jinja2_required_variables(
template_category=TemplateCategory.EXPRESSION,
- template_source=self.applied_expression_template,
+ template_source=self.expression,
)
log.verbose(f"Expression template is valid, requires variables: {required_variables}")
except Jinja2DetectVariablesError as exc:
log.error(f"Dry run failed: could not detect required variables from expression template: {exc}")
msg = (
f"Dry run failed for pipe '{self.code}' (PipeCondition): could not detect required variables "
- f"from expression template: {exc}\nTemplate:\n'{self.applied_expression_template}'"
+ f"from expression template: {exc}\nTemplate:\n'{self.expression}'"
)
- raise DryRunTemplatingError(
- message=msg,
- pipe_type=self.__class__.__name__,
- pipe_code=self.code,
- template_category=TemplateCategory.EXPRESSION,
- template=self.applied_expression_template,
- ) from exc
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
# 3. Validate that all values in the outcomes map (appart from special outcomes) do exist as pipe codes
all_pipe_codes = set(self.outcome_map.values())
@@ -411,7 +329,7 @@ class PipeCondition(PipeController):
f"Dry run failed for PipeCondition '{self.code}': missing pipes: {', '.join(missing_pipes)}. "
f"Pipe map: {self.outcome_map}, default: {self.default_outcome}"
)
- raise DryRunMissingPipesError(message=msg, pipe_type=self.__class__.__name__, pipe_code=self.code, missing_pipes=missing_pipes)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
# Here, it should launch the dry run of all the pipes in the outcomes map
for pipe_code in self.mapped_pipe_codes:
diff --git a/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py b/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py
index e8940d992..ff66a247e 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py
@@ -1,10 +1,12 @@
from typing import Literal
-from pydantic import Field
+from pydantic import Field, field_validator, model_validator
from typing_extensions import override
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
+from pipelex.tools.typing.validation_utils import has_exactly_one_among_attributes_from_list
+from pipelex.types import Self
OutcomeMap = dict[str, str]
@@ -29,3 +31,26 @@ class PipeConditionBlueprint(PipeBlueprint):
if self.default_outcome:
pipe_codes.add(self.default_outcome)
return pipe_codes - set(SpecialOutcome.value_list())
+
+ @field_validator("outcomes", mode="after")
+ @classmethod
+ def validate_outcome_map(cls, outcomes: OutcomeMap) -> OutcomeMap:
+ if not outcomes:
+ msg = f"PipeConditionBlueprint must have at least one mapping in outcomes, got: {outcomes}"
+ raise ValueError(msg)
+ return outcomes
+
+ @model_validator(mode="after")
+ def validate_expression_and_expression_template(self) -> Self:
+ if not has_exactly_one_among_attributes_from_list(self, attributes_list=["expression_template", "expression"]):
+ msg = "PipeCondition should have exactly one of 'expression_template' or 'expression'"
+ raise ValueError(msg)
+ return self
+
+ @override
+ def validate_inputs(self):
+ pass
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_controllers/condition/pipe_condition_factory.py b/pipelex/pipe_controllers/condition/pipe_condition_factory.py
index 05c5c7fcf..4858b62f4 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition_factory.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition_factory.py
@@ -1,10 +1,13 @@
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory, InputRequirementsFactoryError
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
+from pipelex.pipe_controllers.condition.exceptions import PipeConditionFactoryError
from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
@@ -20,30 +23,55 @@ class PipeConditionFactory(PipeFactoryProtocol[PipeConditionBlueprint, PipeCondi
concept_codes_from_the_same_domain: list[str] | None = None,
) -> PipeCondition:
# Parse output to strip multiplicity brackets
- output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ try:
+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ except PipeVariableMultiplicityError as exc:
+ msg = f"Error parsing concept with multiplicity for PipeCondition: {exc}"
+ raise PipeConditionFactoryError(msg) from exc
- output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
- domain=domain,
- concept_string_or_code=output_parse_result.concept,
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ try:
+ output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
+ domain=domain,
+ concept_string_or_code=output_parse_result.concept,
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ except ConceptFactoryError as exc:
+ msg = f"Error making domain and concept code for PipeCondition: {exc}"
+ raise PipeConditionFactoryError(msg) from exc
+
+ try:
+ inputs = InputRequirementsFactory.make_from_blueprint(
+ domain=domain,
+ blueprint=blueprint.inputs or {},
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ except InputRequirementsFactoryError as exc:
+ msg = f"Error making input requirements for PipeCondition: {exc}"
+ raise PipeConditionFactoryError(msg) from exc
+
+ output = get_required_concept(
+ concept_string=ConceptFactory.make_concept_string_with_domain(
+ domain=output_domain_and_code.domain,
+ concept_code=output_domain_and_code.concept_code,
+ ),
)
+
+ # Compute expression from expression_template or expression in blueprint
+ if blueprint.expression_template:
+ expression = blueprint.expression_template
+ elif blueprint.expression:
+ expression = "{{ " + blueprint.expression + " }}"
+ else:
+ msg = "PipeCondition must have either expression_template or expression"
+ raise PipeConditionFactoryError(msg)
+
return PipeCondition(
domain=domain,
code=pipe_code,
description=blueprint.description,
- inputs=InputRequirementsFactory.make_from_blueprint(
- domain=domain,
- blueprint=blueprint.inputs or {},
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- ),
- output=get_required_concept(
- concept_string=ConceptFactory.make_concept_string_with_domain(
- domain=output_domain_and_code.domain,
- concept_code=output_domain_and_code.concept_code,
- ),
- ),
- expression_template=blueprint.expression_template,
- expression=blueprint.expression,
+ inputs=inputs,
+ output=output,
+ expression=expression,
outcome_map=blueprint.outcomes,
default_outcome=blueprint.default_outcome,
add_alias_from_expression_to=blueprint.add_alias_from_expression_to,
diff --git a/pipelex/pipe_controllers/exceptions.py b/pipelex/pipe_controllers/exceptions.py
index cae5a0f45..af29b7448 100644
--- a/pipelex/pipe_controllers/exceptions.py
+++ b/pipelex/pipe_controllers/exceptions.py
@@ -1,7 +1,7 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class PipeControllerError(PipelexException):
+class PipeControllerError(PipelexError):
pass
diff --git a/pipelex/pipe_controllers/parallel/exceptions.py b/pipelex/pipe_controllers/parallel/exceptions.py
new file mode 100644
index 000000000..8428cf8b2
--- /dev/null
+++ b/pipelex/pipe_controllers/parallel/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeParallelFactoryError(PipelexError):
+ pass
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index 83c381d5b..e31fa1022 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -5,23 +5,21 @@ from pydantic import field_validator, model_validator
from typing_extensions import override
from pipelex import log
-from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept import Concept
-from pipelex.core.exceptions import StaticValidationError, StaticValidationErrorType
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.exceptions import PipeInputError, PipeInputNotFoundError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.exceptions import PipeValidationError
+from pipelex.core.pipes.inputs.exceptions import PipeInputNotFoundError
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_pipeline_tracker, get_required_pipe
+from pipelex.libraries.pipe.exceptions import PipeNotFoundError
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_controllers.sub_pipe import SubPipe
-from pipelex.pipe_run.exceptions import PipeRunParamsError
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.pipeline.exceptions import DryRunMissingInputsError
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.types import Self
@@ -46,13 +44,13 @@ class PipeParallel(PipeController):
for sub_pipe in parallel_sub_pipes:
if not sub_pipe.output_name:
msg = f"PipeParallel '{cls.code}' sub-pipe '{sub_pipe.pipe_code}' output name not specified"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
if sub_pipe.output_name in seen_output_names:
msg = (
f"PipeParallel '{cls.code}' sub-pipe '{sub_pipe.pipe_code}' output name '{sub_pipe.output_name}' "
"is already used by another sub-pipe"
)
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
seen_output_names.add(sub_pipe.output_name)
return parallel_sub_pipes
@@ -86,9 +84,7 @@ class PipeParallel(PipeController):
f"Batch input item named '{sub_pipe.batch_params.input_item_stuff_name}' is not "
f"in this Parallel Pipe '{self.code}' input requirements: {pipe_needed_inputs}"
)
- raise PipeInputError(
- message=msg, pipe_code=self.code, variable_name=sub_pipe.batch_params.input_item_stuff_name, concept_code=None
- ) from exc
+ raise PipeValidationError(message=msg) from exc
needed_inputs.add_requirement(
variable_name=sub_pipe.batch_params.input_list_stuff_name,
concept=requirement.concept,
@@ -103,66 +99,29 @@ class PipeParallel(PipeController):
return needed_inputs
@model_validator(mode="after")
- def validate_inputs(self) -> Self:
+ def validate_fields_add_each_output_and_combined_output(self) -> Self:
# Validate that either add_each_output or combined_output is set
if not self.add_each_output and not self.combined_output:
msg = f"PipeParallel'{self.code}'requires either add_each_output or combined_output to be set"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
return self
@override
- def validate_output(self):
+ def validate_inputs_static(self):
pass
- def _validate_inputs(self):
- """Validate that the inputs declared for this PipeParallel match what is actually needed."""
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
-
- the_needed_inputs = self.needed_inputs()
-
- # Check all required variables are in the inputs
- for named_input_requirement in the_needed_inputs.named_input_requirements:
- if named_input_requirement.variable_name not in self.inputs.variables:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[named_input_requirement.variable_name],
- )
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # Check that all declared inputs are actually needed
- for input_name in self.inputs.variables:
- if input_name not in the_needed_inputs.required_names:
- extraneous_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[input_name],
- )
- match reactions.get(StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(extraneous_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise extraneous_input_var_error
+ @override
+ def validate_inputs_with_library(self):
+ pass
@override
- def _validate_with_libraries(self):
- """Perform full validation after all libraries are loaded.
- This is called after all pipes and concepts are available.
- """
- self._validate_inputs()
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
+ pass
@override
def pipe_dependencies(self) -> set[str]:
@@ -176,10 +135,6 @@ class PipeParallel(PipeController):
pipe_run_params: PipeRunParams,
output_name: str | None = None,
) -> PipeOutput:
- """Run a list of pipes in parallel."""
- if not self.add_each_output and not self.combined_output:
- msg = "PipeParallel requires either add_each_output or combined_output to be set"
- raise PipeDefinitionError(msg)
if pipe_run_params.final_stuff_code:
log.verbose(f"PipeBatch.run_pipe() final_stuff_code: {pipe_run_params.final_stuff_code}")
pipe_run_params.final_stuff_code = None
@@ -208,19 +163,19 @@ class PipeParallel(PipeController):
sub_pipe_output_name = self.parallel_sub_pipes[output_index].output_name
if not sub_pipe_output_name:
msg = "PipeParallel requires a result specified for each parallel sub pipe"
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
if self.add_each_output:
working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)
output_stuff_content_items.append(output_stuff.content)
if sub_pipe_output_name in output_stuffs:
# TODO: check that at the blueprint / factory level
msg = f"PipeParallel requires unique output names for each parallel sub pipe, but {sub_pipe_output_name} is already used"
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
output_stuffs[sub_pipe_output_name] = output_stuff
if sub_pipe_output_name in output_stuff_contents:
# TODO: check that at the blueprint / factory level
msg = f"PipeParallel requires unique output names for each parallel sub pipe, but {sub_pipe_output_name} is already used"
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
output_stuff_contents[sub_pipe_output_name] = output_stuff.content
log.verbose(f"PipeParallel '{self.code}': output_stuff_contents[{sub_pipe_output_name}]: {output_stuff_contents[sub_pipe_output_name]}")
@@ -260,7 +215,7 @@ class PipeParallel(PipeController):
log.verbose(f"PipeParallel: dry run controller pipe: {self.code}")
if pipe_run_params.run_mode != PipeRunMode.DRY:
msg = f"PipeSequence._dry_run_controller_pipe() called with run_mode = {pipe_run_params.run_mode} in pipe {self.code}"
- raise PipeRunParamsError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
# 1. Validate that all required inputs are present in the working memory
needed_inputs = self.needed_inputs()
@@ -270,22 +225,16 @@ class PipeParallel(PipeController):
missing_input_names.append(named_input_requirement.variable_name)
if missing_input_names:
- msg = f"Dry run failed: missing required inputs: {missing_input_names}"
- log.error(f"Dry run failed: missing required inputs: {missing_input_names}")
- raise DryRunMissingInputsError(
- message=msg,
- pipe_type=self.__class__.__name__,
- pipe_code=self.code,
- missing_inputs=missing_input_names,
- )
+ msg = f"Dry run failed for pipe '{self.code}' (PipeParallel): missing required inputs: {missing_input_names}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
# 2. Validate that all sub-pipes exist
for sub_pipe in self.parallel_sub_pipes:
try:
get_required_pipe(pipe_code=sub_pipe.pipe_code)
- except Exception as exc:
- msg = f"PipeParallel'{self.code}'sub-pipe '{sub_pipe.pipe_code}' not found"
- raise PipeDefinitionError(msg) from exc
+ except PipeNotFoundError as exc:
+ msg = f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe.pipe_code}' not found"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
# 3. Run all sub-pipes in dry mode
tasks: list[Coroutine[Any, Any, PipeOutput]] = []
@@ -312,7 +261,7 @@ class PipeParallel(PipeController):
if not sub_pipe_output_name:
sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code
msg = f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe_code}' output name not specified"
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
if self.add_each_output:
working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)
@@ -322,7 +271,7 @@ class PipeParallel(PipeController):
msg = (
f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe_code}' duplicate output name '{sub_pipe_output_name}'"
)
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
output_stuffs[sub_pipe_output_name] = output_stuff
output_stuff_contents[sub_pipe_output_name] = output_stuff.content
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
index 92dc881c6..9dacf01d1 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py
@@ -4,7 +4,6 @@ from pydantic import field_validator, model_validator
from typing_extensions import override
from pipelex.core.concepts.validation import is_concept_string_or_code_valid
-from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
from pipelex.types import Self
@@ -29,7 +28,7 @@ class PipeParallelBlueprint(PipeBlueprint):
if combined_output:
if not is_concept_string_or_code_valid(concept_string_or_code=combined_output):
msg = f"Combined output '{combined_output}' is not a valid concept string or code"
- raise PipeDefinitionError(message=msg)
+ raise ValueError(msg)
return combined_output
@model_validator(mode="after")
@@ -39,5 +38,13 @@ class PipeParallelBlueprint(PipeBlueprint):
"PipeParallel requires either add_each_output to be True or combined_output to be set, "
"or both, otherwise the pipe won't output anything"
)
- raise PipeDefinitionError(message=msg, description=self.description, source=self.source)
+ raise ValueError(msg)
return self
+
+ @override
+ def validate_inputs(self):
+ pass
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
index fdfffe546..4464ccd35 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py
@@ -3,11 +3,11 @@ from typing import TYPE_CHECKING
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
+from pipelex.pipe_controllers.parallel.exceptions import PipeParallelFactoryError
from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.pipe_controllers.sub_pipe_factory import SubPipeFactory
@@ -30,9 +30,7 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalle
for sub_pipe_blueprint in blueprint.parallels:
if not sub_pipe_blueprint.result:
msg = f"Unexpected error in pipe '{pipe_code}': PipeParallel requires a result specified for each parallel sub pipe"
- raise PipeDefinitionError(
- message=msg, domain_code=domain, pipe_code=pipe_code, description=blueprint.description, source=blueprint.source
- )
+ raise PipeParallelFactoryError(message=msg)
sub_pipe = SubPipeFactory.make_from_blueprint(sub_pipe_blueprint, concept_codes_from_the_same_domain=concept_codes_from_the_same_domain)
parallel_sub_pipes.append(sub_pipe)
if not blueprint.add_each_output and not blueprint.combined_output:
@@ -40,8 +38,8 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalle
f"Unexpected error in pipe '{pipe_code}': PipeParallel requires either add_each_output to be True or combined_output to be set, "
"or both, otherwise the pipe won't output anything"
)
- raise PipeDefinitionError(
- message=msg, domain_code=domain, pipe_code=pipe_code, description=blueprint.description, source=blueprint.source
+ raise PipeParallelFactoryError(
+ message=msg,
)
# Parse output to strip multiplicity brackets
@@ -67,21 +65,24 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalle
concept_string_or_code=output_parse_result.concept,
concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
)
+ inputs = InputRequirementsFactory.make_from_blueprint(
+ domain=domain,
+ blueprint=blueprint.inputs or {},
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+
+ output = get_required_concept(
+ concept_string=ConceptFactory.make_concept_string_with_domain(
+ domain=output_domain_and_code.domain,
+ concept_code=output_domain_and_code.concept_code,
+ ),
+ )
return PipeParallel(
domain=domain,
code=pipe_code,
description=blueprint.description,
- inputs=InputRequirementsFactory.make_from_blueprint(
- domain=domain,
- blueprint=blueprint.inputs or {},
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- ),
- output=get_required_concept(
- concept_string=ConceptFactory.make_concept_string_with_domain(
- domain=output_domain_and_code.domain,
- concept_code=output_domain_and_code.concept_code,
- ),
- ),
+ inputs=inputs,
+ output=output,
parallel_sub_pipes=parallel_sub_pipes,
add_each_output=blueprint.add_each_output or False,
combined_output=combined_output,
diff --git a/pipelex/pipe_controllers/pipe_controller.py b/pipelex/pipe_controllers/pipe_controller.py
index 8fd00d651..c7ee46a30 100644
--- a/pipelex/pipe_controllers/pipe_controller.py
+++ b/pipelex/pipe_controllers/pipe_controller.py
@@ -1,12 +1,9 @@
from abc import abstractmethod
-from typing import Literal
+from typing import Literal, final
from typing_extensions import override
-from pipelex import log
-from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipes.exceptions import PipeRunInputsError
from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.pipe_run_params import PipeRunMode, PipeRunParams
@@ -20,21 +17,17 @@ class PipeController(PipeAbstract):
def class_name(self) -> str:
return self.__class__.__name__
- def _validate_inputs_in_memory(self, working_memory: WorkingMemory) -> None:
- missing_inputs: dict[str, str] = {}
- for required_stuff_name, requirement in self.needed_inputs().items:
- try:
- working_memory.get_stuff(required_stuff_name)
- except WorkingMemoryStuffNotFoundError as exc:
- variable_name: str = exc.variable_name or required_stuff_name
- missing_inputs[variable_name] = exc.concept_code or requirement.concept.code
- if missing_inputs:
- raise PipeRunInputsError(
- message=f"Missing required inputs for pipe '{self.code}': {missing_inputs}", pipe_code=self.code, missing_inputs=missing_inputs
- )
+ @abstractmethod
+ def pipe_dependencies(self) -> set[str]:
+ """Return the pipes that are dependencies of the pipe.
+ - PipeBatch: The pipe that is being batched
+ - PipeCondition: The pipes in the outcome_map
+ - PipeSequence: The pipes in the steps
+ """
+ @final
@override
- async def run_pipe(
+ async def _run_pipe(
self,
job_metadata: JobMetadata,
working_memory: WorkingMemory,
@@ -42,21 +35,6 @@ class PipeController(PipeAbstract):
output_name: str | None = None,
print_intermediate_outputs: bool | None = False,
) -> PipeOutput:
- pipe_run_params.push_pipe_to_stack(pipe_code=self.code)
- self.monitor_pipe_stack(pipe_run_params=pipe_run_params)
-
- updated_metadata = JobMetadata(
- pipe_job_ids=[self.code],
- )
- job_metadata.update(updated_metadata=updated_metadata)
-
- # check we have the required inputs in the working memory
- self._validate_inputs_in_memory(working_memory=working_memory)
-
- pipe_run_info = self._format_pipe_run_info(pipe_run_params=pipe_run_params)
- # log.info(pipe_run_info)
- if pipe_run_params.run_mode == PipeRunMode.LIVE:
- log.info(pipe_run_info)
match pipe_run_params.run_mode:
case PipeRunMode.LIVE:
pipe_output = await self._run_controller_pipe(
@@ -72,8 +50,6 @@ class PipeController(PipeAbstract):
pipe_run_params=pipe_run_params,
output_name=output_name,
)
-
- pipe_run_params.pop_pipe_from_stack(pipe_code=self.code)
return pipe_output
@abstractmethod
diff --git a/pipelex/pipe_controllers/sequence/exceptions.py b/pipelex/pipe_controllers/sequence/exceptions.py
index bfafd40b9..12d1c8f57 100644
--- a/pipelex/pipe_controllers/sequence/exceptions.py
+++ b/pipelex/pipe_controllers/sequence/exceptions.py
@@ -1,6 +1,9 @@
-class PipeSequenceBlueprintError(Exception):
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeSequenceValueError(ValueError):
pass
-class PipeSequenceError(Exception):
+class PipeSequenceFactoryError(PipelexError):
pass
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index 043e3ce00..677ee0844 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -1,31 +1,76 @@
from typing import Literal
-from pydantic import model_validator
+from pydantic import field_validator
from typing_extensions import override
-from pipelex import log
-from pipelex.config import StaticValidationReaction, get_config
-from pipelex.core.exceptions import StaticValidationError, StaticValidationErrorType
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipes.exceptions import PipeInputError, PipeInputNotFoundError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.exceptions import PipeInputNotFoundError
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import get_concept_library, get_required_pipe
from pipelex.pipe_controllers.exceptions import PipeControllerOutputConceptMismatchError
from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
from pipelex.pipe_controllers.pipe_controller import PipeController
+from pipelex.pipe_controllers.sequence.exceptions import PipeSequenceValueError
from pipelex.pipe_controllers.sub_pipe import SubPipe
-from pipelex.pipe_run.exceptions import PipeRunParamsError
+from pipelex.pipe_run.exceptions import PipeRunError, PipeRunParamsError
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.types import Self
class PipeSequence(PipeController):
type: Literal["PipeSequence"] = "PipeSequence"
sequential_sub_pipes: list[SubPipe]
+ @override
+ def required_variables(self) -> set[str]:
+ return set()
+
+ @field_validator("sequential_sub_pipes", mode="after")
+ @classmethod
+ def validate_sequential_sub_pipes(cls, value: list[SubPipe]) -> list[SubPipe]:
+ if not value:
+ msg = f"PipeSequence '{cls.code}' requires at least one sub-pipe"
+ raise ValueError(msg)
+ return value
+
+ @override
+ def validate_inputs_static(self):
+ pass
+
+ @override
+ def validate_inputs_with_library(self):
+ pass
+
+ @override
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
+ """Validate the output for the pipe sequence.
+ The output of the pipe sequence should match the output of the last step.
+ """
+ last_step_pipe_code = self.sequential_sub_pipes[-1].pipe_code
+ last_step_pipe = get_required_pipe(pipe_code=last_step_pipe_code)
+ last_step_output_concept = last_step_pipe.output
+ if not get_concept_library().is_compatible(tested_concept=last_step_output_concept, wanted_concept=self.output):
+ msg = (
+ f"PipeSequence concept mismatch: the output concept '{last_step_output_concept.concept_string}' "
+ f"of the last step '{last_step_pipe_code}' of sequence pipe '{self.code}' "
+ f"is not compatible with the output concept '{self.output.concept_string}' of the sequence."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
+ domain=self.domain,
+ pipe_code=self.code,
+ provided_concept_code=last_step_output_concept.concept_string,
+ required_concept_codes=[self.output.concept_string],
+ )
+
@override
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
if visited_pipes is None:
@@ -62,9 +107,7 @@ class PipeSequence(PipeController):
f"Batch input item named '{sequential_sub_pipe.batch_params.input_item_stuff_name}' is not "
f"in this PipeSequence '{self.code}' input requirements: {sub_pipe_needed_inputs}"
)
- raise PipeInputError(
- message=msg, pipe_code=self.code, variable_name=sequential_sub_pipe.batch_params.input_item_stuff_name, concept_code=None
- ) from exc
+ raise PipeSequenceValueError(msg) from exc
needed_inputs.add_requirement(
variable_name=sequential_sub_pipe.batch_params.input_list_stuff_name,
concept=requirement.concept,
@@ -83,93 +126,6 @@ class PipeSequence(PipeController):
generated_outputs.add(sequential_sub_pipe.output_name)
return needed_inputs
- @override
- def required_variables(self) -> set[str]:
- return set()
-
- @override
- def validate_output(self):
- """Validate the output for the pipe sequence.
- The output of the pipe sequence should match the output of the last step.
- """
- last_step_output_pipe = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code)
- concept_of_last_step = last_step_output_pipe.output
- if not get_concept_library().is_compatible(tested_concept=concept_of_last_step, wanted_concept=self.output):
- msg = f"""PipeSequence concept mismatch:
-the output concept '{concept_of_last_step.concept_string}' of the last step '{self.sequential_sub_pipes[-1].pipe_code}'
-of sequence pipe '{self.code}' is not compatible with the output concept '{self.output.concept_string}' of the sequence.
-"""
- raise PipeControllerOutputConceptMismatchError(
- message=msg, tested_concept=concept_of_last_step.concept_string, wanted_concept=self.output.concept_string
- )
-
- @model_validator(mode="after")
- def validate_inputs(self) -> Self:
- if len(self.sequential_sub_pipes) == 0:
- msg = f"Pipe'{self.code}'(PipeSequence) must have at least 1 step"
- raise ValueError(msg)
- return self
-
- def _validate_output_multiplicity_support(self, pipe_run_params: PipeRunParams) -> None:
- """Validate that the pipe supports the requested output multiplicity."""
- if pipe_run_params.is_multiple_output_required:
- msg = f"{self.__class__.__name__} does not support multiple outputs, got output_multiplicity = {pipe_run_params.output_multiplicity}"
- raise PipeRunParamsError(msg)
-
- def _validate_inputs(self):
- """Validate that the inputs declared for this PipeSequence match what is actually needed."""
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
-
- the_needed_inputs = self.needed_inputs()
-
- # Check all required variables are in the inputs
- for named_input_requirement in the_needed_inputs.named_input_requirements:
- if named_input_requirement.variable_name not in self.inputs.variables:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[named_input_requirement.variable_name],
- required_concept_codes=[named_input_requirement.concept.code],
- )
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # Check that all declared inputs are actually needed
- for input_name in self.inputs.variables:
- if input_name not in the_needed_inputs.required_names:
- log.verbose(f"the_needed_inputs.required_names: {the_needed_inputs.required_names}")
- extraneous_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[input_name],
- )
- match reactions.get(
- StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
- default_reaction,
- ):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(extraneous_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise extraneous_input_var_error
-
- @override
- def _validate_with_libraries(self):
- """Perform full validation after all libraries are loaded.
- This is called after all pipes and concepts are available.
- """
- self._validate_inputs()
-
@override
def pipe_dependencies(self) -> set[str]:
return {sub_pipe.pipe_code for sub_pipe in self.sequential_sub_pipes}
@@ -183,7 +139,9 @@ of sequence pipe '{self.code}' is not compatible with the output concept '{self.
output_name: str | None = None,
) -> PipeOutput:
pipe_run_params.push_pipe_layer(pipe_code=self.code)
- self._validate_output_multiplicity_support(pipe_run_params)
+ if pipe_run_params.is_multiple_output_required:
+ msg = f"{self.__class__.__name__} does not support multiple outputs, got output_multiplicity = {pipe_run_params.output_multiplicity}"
+ raise PipeRunParamsError(msg)
evolving_memory = working_memory
@@ -215,7 +173,7 @@ of sequence pipe '{self.code}' is not compatible with the output concept '{self.
) -> PipeOutput:
if not pipe_run_params.run_mode.is_dry:
msg = f"PipeSequence._dry_run_controller_pipe() called with run_mode = {pipe_run_params.run_mode} in pipe {self.code}"
- raise PipeRunParamsError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
# Verify the output of this pipe is matching the output of the last step.
concept_of_last_step = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code).output
# if self.output.concept_string != concept_string_of_last_step:
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py b/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py
index 4df7f4344..e51dde706 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py
@@ -1,5 +1,6 @@
from typing import Literal
+from pydantic import field_validator
from typing_extensions import override
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
@@ -25,3 +26,19 @@ class PipeSequenceBlueprint(PipeBlueprint):
For sequences, the order of steps matters, so we preserve it.
"""
return [step.pipe for step in self.steps]
+
+ @field_validator("steps", mode="before")
+ @classmethod
+ def validate_steps(cls, steps: list[SubPipeBlueprint]) -> list[SubPipeBlueprint]:
+ if not steps:
+ msg = "PipeSequence must have at least 1 step"
+ raise ValueError(msg)
+ return steps
+
+ @override
+ def validate_inputs(self):
+ pass
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py
index bbde1cc13..3b0acd0d8 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py
@@ -1,10 +1,13 @@
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
+from pipelex.pipe_controllers.sequence.exceptions import PipeSequenceFactoryError
from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
from pipelex.pipe_controllers.sub_pipe_factory import SubPipeFactory
@@ -20,13 +23,28 @@ class PipeSequenceFactory(PipeFactoryProtocol[PipeSequenceBlueprint, PipeSequenc
blueprint: PipeSequenceBlueprint,
concept_codes_from_the_same_domain: list[str] | None = None,
) -> PipeSequence:
- # Parse output to strip multiplicity brackets
- output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ # Parse output to strip multiplicity brackets``
+ try:
+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ except PipeVariableMultiplicityError as exc:
+ msg = f"Error parsing concept with multiplicity for PipeSequence: {exc}"
+ raise PipeSequenceFactoryError(msg) from exc
- output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
- domain=domain,
- concept_string_or_code=output_parse_result.concept,
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ try:
+ output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
+ domain=domain,
+ concept_string_or_code=output_parse_result.concept,
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ except ConceptFactoryError as exc:
+ msg = f"Error making domain and concept code for PipeSequence: {exc}"
+ raise PipeSequenceFactoryError(msg) from exc
+
+ output = get_required_concept(
+ concept_string=ConceptFactory.make_concept_string_with_domain(
+ domain=output_domain_and_code.domain,
+ concept_code=output_domain_and_code.concept_code,
+ ),
)
return PipeSequence(
domain=domain,
@@ -37,12 +55,7 @@ class PipeSequenceFactory(PipeFactoryProtocol[PipeSequenceBlueprint, PipeSequenc
blueprint=blueprint.inputs or {},
concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
),
- output=get_required_concept(
- concept_string=ConceptFactory.make_concept_string_with_domain(
- domain=output_domain_and_code.domain,
- concept_code=output_domain_and_code.concept_code,
- ),
- ),
+ output=output,
sequential_sub_pipes=[
SubPipeFactory.make_from_blueprint(blueprint=step, concept_codes_from_the_same_domain=concept_codes_from_the_same_domain)
for step in blueprint.steps
diff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.py
index d86159da8..20d26dea4 100644
--- a/pipelex/pipe_controllers/sub_pipe.py
+++ b/pipelex/pipe_controllers/sub_pipe.py
@@ -3,7 +3,7 @@ from pydantic import BaseModel
from pipelex import log
from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipes.exceptions import PipeInputError, PipeInputNotFoundError
+from pipelex.core.pipes.inputs.exceptions import PipeInputError, PipeInputNotFoundError
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.list_content import ListContent
diff --git a/pipelex/pipe_controllers/sub_pipe_blueprint.py b/pipelex/pipe_controllers/sub_pipe_blueprint.py
index e26e6d342..499e07e48 100644
--- a/pipelex/pipe_controllers/sub_pipe_blueprint.py
+++ b/pipelex/pipe_controllers/sub_pipe_blueprint.py
@@ -1,6 +1,5 @@
from pydantic import BaseModel, ConfigDict, model_validator
-from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_list
from pipelex.types import Self
@@ -19,17 +18,17 @@ class SubPipeBlueprint(BaseModel):
def validate_multiple_output(self) -> Self:
if has_more_than_one_among_attributes_from_list(self, attributes_list=["nb_output", "multiple_output"]):
msg = "PipeStepBlueprint should have no more than '1' of nb_output or multiple_output"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
return self
@model_validator(mode="after")
def validate_batch_params(self) -> Self:
if self.batch_over and not self.batch_as:
msg = f"In pipe '{self.pipe}': When 'batch_over' is specified, 'batch_as' must also be provided"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
if self.batch_as and not self.batch_over:
msg = f"In pipe '{self.pipe}': When 'batch_as' is specified, 'batch_over' must also be provided"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
return self
diff --git a/pipelex/pipe_operators/compose/exceptions.py b/pipelex/pipe_operators/compose/exceptions.py
new file mode 100644
index 000000000..a97b10757
--- /dev/null
+++ b/pipelex/pipe_operators/compose/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeComposeFactoryError(PipelexError):
+ pass
diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.py
index 5072a52c8..f3c37ed71 100644
--- a/pipelex/pipe_operators/compose/pipe_compose.py
+++ b/pipelex/pipe_operators/compose/pipe_compose.py
@@ -1,6 +1,6 @@
from typing import Any, Literal
-from pydantic import ConfigDict, model_validator
+from pydantic import ConfigDict
from typing_extensions import override
from pipelex import log
@@ -11,11 +11,10 @@ from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.config import get_config
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
@@ -26,10 +25,8 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError
-from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing
+from pipelex.tools.jinja2.jinja2_errors import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
-from pipelex.types import Self
class PipeComposeOutput(PipeOutput):
@@ -49,36 +46,25 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
category: TemplateCategory = TemplateCategory.BASIC
extra_context: dict[str, Any] | None = None
- @model_validator(mode="after")
- def validate_template(self) -> Self:
- try:
- check_jinja2_parsing(template_source=self.template, template_category=self.category)
- except Jinja2TemplateSyntaxError as exc:
- msg = f"Could not parse template for PipeCompose '{self.code}: {exc}"
- raise PipeDefinitionError(msg) from exc
- return self
-
- @model_validator(mode="after")
- def validate_inputs(self) -> Self:
- self._validate_required_variables()
- return self
-
- def _validate_required_variables(self) -> Self:
- """This method checks that all required variables are in the inputs"""
- required_variables = self.required_variables()
- for required_variable_name in required_variables:
- if required_variable_name not in self.inputs.variables:
- msg = f"Required variable '{required_variable_name}' is not in the inputs of pipe {self.code}"
- raise PipeDefinitionError(msg)
- return self
-
- @override
- def validate_output(self):
- pass
+ @property
+ def desc(self) -> str:
+ return f"Jinja2 included template, prompting style {self.templating_style}"
@override
- def _validate_with_libraries(self):
- pass
+ def required_variables(self) -> set[str]:
+ try:
+ required_variables = detect_jinja2_required_variables(
+ template_category=self.category,
+ template_source=self.template,
+ )
+ except Jinja2DetectVariablesError as exc:
+ msg = f"Error detecting required variables for PipeCompose: {exc}"
+ raise ValueError(msg) from exc
+ return {
+ variable_name
+ for variable_name in required_variables
+ if not variable_name.startswith("_") and variable_name not in ("preliminary_text", "place_holder")
+ }
@override
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
@@ -87,21 +73,21 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
needed_inputs.add_requirement(variable_name=input_name, concept=requirement.concept)
return needed_inputs
- @property
- def desc(self) -> str:
- return f"Jinja2 included template, prompting style {self.templating_style}"
+ @override
+ def validate_inputs_static(self):
+ pass
@override
- def required_variables(self) -> set[str]:
- required_variables = detect_jinja2_required_variables(
- template_category=self.category,
- template_source=self.template,
- )
- return {
- variable_name
- for variable_name in required_variables
- if not variable_name.startswith("_") and variable_name not in ("preliminary_text", "place_holder")
- }
+ def validate_inputs_with_library(self):
+ pass
+
+ @override
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
+ pass
@override
async def _run_operator_pipe(
diff --git a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py b/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
index 28927c757..13492ac56 100644
--- a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
+++ b/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
@@ -1,9 +1,15 @@
from typing import Any, Literal
+from typing_extensions import override
+
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_preprocessor import preprocess_template
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError
+from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing
+from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
class PipeComposeBlueprint(PipeBlueprint):
@@ -37,3 +43,31 @@ class PipeComposeBlueprint(PipeBlueprint):
return self.template.extra_context
else:
return None
+
+ @override
+ def validate_inputs(self):
+ preprocessed_template = preprocess_template(self.template_source)
+ try:
+ check_jinja2_parsing(
+ template_source=preprocessed_template,
+ template_category=self.template_category,
+ )
+ except Jinja2TemplateSyntaxError as exc:
+ msg = f"Could not parse template for PipeCompose: {exc}"
+ raise ValueError(msg) from exc
+ required_variables = {
+ variable_name
+ for variable_name in detect_jinja2_required_variables(
+ template_category=self.template_category,
+ template_source=preprocessed_template,
+ )
+ if not variable_name.startswith("_") and variable_name not in ("preliminary_text", "place_holder")
+ }
+ for required_variable_name in required_variables:
+ if required_variable_name not in self.input_names:
+ msg = f"Required variable '{required_variable_name}' is not in the inputs of PipeCompose."
+ raise ValueError(msg)
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_operators/compose/pipe_compose_factory.py b/pipelex/pipe_operators/compose/pipe_compose_factory.py
index 435772e26..5c5b1b86a 100644
--- a/pipelex/pipe_operators/compose/pipe_compose_factory.py
+++ b/pipelex/pipe_operators/compose/pipe_compose_factory.py
@@ -1,15 +1,17 @@
from typing_extensions import override
-from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.template_preprocessor import preprocess_template
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+from pipelex.core.pipes.exceptions import PipeVariableMultiplicityError
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
+from pipelex.pipe_operators.compose.exceptions import PipeComposeFactoryError
from pipelex.pipe_operators.compose.pipe_compose import PipeCompose
from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
+from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError
from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing
@@ -24,19 +26,30 @@ class PipeComposeFactory(PipeFactoryProtocol[PipeComposeBlueprint, PipeCompose])
concept_codes_from_the_same_domain: list[str] | None = None,
) -> PipeCompose:
preprocessed_template = preprocess_template(blueprint.template_source)
- check_jinja2_parsing(
- template_source=preprocessed_template,
- template_category=blueprint.template_category,
- )
+ try:
+ check_jinja2_parsing(
+ template_source=preprocessed_template,
+ template_category=blueprint.template_category,
+ )
+ except Jinja2TemplateSyntaxError as exc:
+ msg = f"Error parsing Jinja2 template for PipeCompose: {exc}"
+ raise PipeComposeFactoryError(msg) from exc
# Parse output to strip multiplicity brackets
- output_parse_result = parse_concept_with_multiplicity(blueprint.output)
-
- output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
- domain=domain,
- concept_string_or_code=output_parse_result.concept,
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- )
+ try:
+ output_parse_result = parse_concept_with_multiplicity(blueprint.output)
+ except PipeVariableMultiplicityError as exc:
+ msg = f"Error parsing concept with multiplicity for PipeCompose: {exc}"
+ raise PipeComposeFactoryError(msg) from exc
+ try:
+ output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_string_or_code(
+ domain=domain,
+ concept_string_or_code=output_parse_result.concept,
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ except ConceptFactoryError as exc:
+ msg = f"Error making domain and concept code for PipeCompose: {exc}"
+ raise PipeComposeFactoryError(msg) from exc
return PipeCompose(
domain=domain,
code=pipe_code,
@@ -57,22 +70,3 @@ class PipeComposeFactory(PipeFactoryProtocol[PipeComposeBlueprint, PipeCompose])
category=blueprint.template_category,
extra_context=blueprint.extra_context,
)
-
- @classmethod
- def make_pipe_compose_from_template_str(
- cls,
- domain: str,
- template_str: str,
- inputs: InputRequirements | None = None,
- ) -> PipeCompose:
- preprocessed_template = preprocess_template(template_str)
- check_jinja2_parsing(
- template_source=preprocessed_template,
- template_category=TemplateCategory.LLM_PROMPT,
- )
- return PipeCompose(
- domain=domain,
- code="adhoc_pipe_compose_from_template_str",
- template=preprocessed_template,
- inputs=inputs or InputRequirementsFactory.make_empty(),
- )
diff --git a/pipelex/pipe_operators/exceptions.py b/pipelex/pipe_operators/exceptions.py
index 50ef07c1d..692a21fde 100644
--- a/pipelex/pipe_operators/exceptions.py
+++ b/pipelex/pipe_operators/exceptions.py
@@ -1,8 +1,8 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-class PipeOperatorModelAvailabilityError(PipelexException):
+class PipeOperatorModelAvailabilityError(PipelexError):
def __init__(
self,
message: str,
diff --git a/pipelex/pipe_operators/extract/exceptions.py b/pipelex/pipe_operators/extract/exceptions.py
new file mode 100644
index 000000000..4a0e9d670
--- /dev/null
+++ b/pipelex/pipe_operators/extract/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeExtractFactoryError(PipelexError):
+ pass
diff --git a/pipelex/pipe_operators/extract/pipe_extract.py b/pipelex/pipe_operators/extract/pipe_extract.py
index c12bfaa97..db570dc35 100644
--- a/pipelex/pipe_operators/extract/pipe_extract.py
+++ b/pipelex/pipe_operators/extract/pipe_extract.py
@@ -1,21 +1,20 @@
from typing import Literal
from pydantic import model_validator
-from typing_extensions import override
+from typing_extensions import Self, override
from pipelex import log
from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
+from pipelex.cogt.exceptions import ModelChoiceNotFoundError
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
from pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSetting
from pipelex.cogt.models.model_deck_check import check_extract_choice_with_deck
-from pipelex.config import StaticValidationReaction, get_config
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.exceptions import StaticValidationError, StaticValidationErrorType
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements import InputRequirements
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
@@ -24,17 +23,16 @@ from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import (
- get_concept_library,
get_content_generator,
get_model_deck,
get_native_concept,
)
from pipelex.pipe_operators.pipe_operator import PipeOperator
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.pdf.pypdfium2_renderer import pypdfium2_renderer
-from pipelex.types import Self
class PipeExtractOutput(PipeOutput):
@@ -52,101 +50,50 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
image_stuff_name: str | None = None
pdf_stuff_name: str | None = None
+ @override
+ def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
+ return self.inputs
+
+ @override
+ def required_variables(self) -> set[str]:
+ return set(self.inputs.required_names)
+
@model_validator(mode="after")
- def validate_inputs(self) -> Self:
- self._validate_inputs()
+ def validate_fields(self) -> Self:
+ if self.image_stuff_name is None and self.pdf_stuff_name is None:
+ msg = "For PipeExtract you must provide either a pdf or an image or a concept that refines one of them"
+ raise ValueError(msg)
return self
@override
- def _validate_with_libraries(self):
- self._validate_inputs()
+ def validate_inputs_static(self):
if self.extract_choice:
- check_extract_choice_with_deck(extract_choice=self.extract_choice)
+ try:
+ check_extract_choice_with_deck(extract_choice=self.extract_choice)
+ except ModelChoiceNotFoundError as exc:
+ msg = f"Extract choice '{self.extract_choice}' was not found in the model deck"
+ raise ValueError(msg) from exc
@override
- def required_variables(self) -> set[str]:
- return set(self.inputs.required_names)
+ def validate_inputs_with_library(self):
+ pass
@override
- def validate_output(self):
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
if self.output != get_native_concept(native_concept=NativeConceptCode.PAGE):
msg = f"PipeExtract output should be a Page concept, but is {self.output.concept_string}"
- raise PipeDefinitionError(msg)
-
- def _validate_inputs(self):
- concept_library = get_concept_library()
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
-
- # check that we have exactly one input
- nb_inputs = self.inputs.nb_inputs
- if nb_inputs > 1:
- too_many_candidate_inputs_error = StaticValidationError(
- error_type=StaticValidationErrorType.TOO_MANY_CANDIDATE_INPUTS,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=self.inputs.variables,
- explanation="Only one image or pdf can be provided for OCR",
- )
- match reactions.get(StaticValidationErrorType.TOO_MANY_CANDIDATE_INPUTS, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(too_many_candidate_inputs_error.desc())
- case StaticValidationReaction.RAISE:
- raise too_many_candidate_inputs_error
- elif nb_inputs < 1:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- explanation="For OCR you must provide either a pdf or an image or a concept that refines one of them",
- )
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # We have confirmed right above that we have exactly one input
- # get input_name, requirement from single item in inputs
- input_name, requirement = self.inputs.items[0]
- log.verbose(f"Validating input '{input_name}' with concept code '{requirement.concept.code}'")
- if concept_library.is_compatible(
- tested_concept=requirement.concept,
- wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),
- strict=True,
- ):
- self.image_stuff_name = input_name
- elif concept_library.is_compatible(
- tested_concept=requirement.concept,
- wanted_concept=get_native_concept(native_concept=NativeConceptCode.PDF),
- strict=True,
- ):
- self.pdf_stuff_name = input_name
- else:
- inadequate_input_concept_error = StaticValidationError(
- error_type=StaticValidationErrorType.INADEQUATE_INPUT_CONCEPT,
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
domain=self.domain,
pipe_code=self.code,
- variable_names=[input_name],
- provided_concept_code=requirement.concept.code,
- explanation="For PipeExtract you must provide either a pdf or an image or a concept that refines one of them",
+ provided_concept_code=self.output.concept_string,
+ required_concept_codes=[NativeConceptCode.PAGE.concept_string],
)
- match reactions.get(StaticValidationErrorType.INADEQUATE_INPUT_CONCEPT, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(inadequate_input_concept_error.desc())
- case StaticValidationReaction.RAISE:
- raise inadequate_input_concept_error
-
- @override
- def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
- return self.inputs
@override
async def _run_operator_pipe(
@@ -167,9 +114,6 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
elif self.pdf_stuff_name:
pdf_stuff = working_memory.get_stuff_as_pdf(name=self.pdf_stuff_name)
pdf_uri = pdf_stuff.url
- else:
- msg = "PipeExtract should have a non-None image_stuff_name or pdf_stuff_name"
- raise PipeDefinitionError(msg)
extract_choice: ExtractModelChoice = self.extract_choice or get_model_deck().extract_choice_default
extract_setting: ExtractSetting = get_model_deck().get_extract_setting(extract_choice=extract_choice)
@@ -264,7 +208,7 @@ class PipeExtract(PipeOperator[PipeExtractOutput]):
log.verbose(f"PipeExtract: dry run operator pipe: {self.code}")
if pipe_run_params.run_mode != PipeRunMode.DRY:
msg = f"Running pipe '{self.code}' (PipeExtract) _dry_run_operator_pipe() in non-dry mode is not allowed."
- raise PipeDefinitionError(msg)
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
content_generator_dry = ContentGeneratorDry()
return await self._run_operator_pipe(
diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
index 752dde075..4dfca3667 100644
--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
@@ -1,6 +1,5 @@
from typing import Literal
-from pydantic import field_validator
from typing_extensions import override
from pipelex.cogt.extract.extract_setting import ExtractModelChoice
@@ -17,7 +16,23 @@ class PipeExtractBlueprint(PipeBlueprint):
page_views_dpi: int | None = None
@override
- @field_validator("output", mode="before")
- @classmethod
- def validate_output(cls, output: str) -> str:
- return "Page[]"
+ def validate_inputs(self):
+ nb_inputs = self.nb_inputs
+ msg = (
+ "Exactly one input must be provided for the PipeExtract, and it must be a pdf or an image or a concept that refines one of them."
+ f"{nb_inputs} inputs were provided."
+ )
+ if self.inputs is None:
+ raise ValueError(msg)
+ if nb_inputs > 1:
+ msg = f"Too many inputs provided for PipeExtract: {self.input_names}. Only one input is allowed."
+ raise ValueError(msg)
+ if nb_inputs < 1:
+ msg = f"Missing input provided for PipeExtract: {self.input_names}. Only one input is allowed."
+ raise ValueError(msg)
+
+ @override
+ def validate_output(self):
+ if self.output != "Page[]":
+ msg = f"PipeExtract output should be a 'native.Page[]' concept, but is {self.output}"
+ raise ValueError(msg)
diff --git a/pipelex/pipe_operators/extract/pipe_extract_factory.py b/pipelex/pipe_operators/extract/pipe_extract_factory.py
index 6a3c9fad3..a105fdb7d 100644
--- a/pipelex/pipe_operators/extract/pipe_extract_factory.py
+++ b/pipelex/pipe_operators/extract/pipe_extract_factory.py
@@ -2,10 +2,12 @@ from typing_extensions import override
from pipelex.config import get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
-from pipelex.hub import get_required_concept
+from pipelex.hub import get_concept_library, get_native_concept, get_required_concept
+from pipelex.pipe_operators.extract.exceptions import PipeExtractFactoryError
from pipelex.pipe_operators.extract.pipe_extract import PipeExtract
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
@@ -29,6 +31,42 @@ class PipeExtractFactory(PipeFactoryProtocol[PipeExtractBlueprint, PipeExtract])
concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
)
+ image_stuff_name = None
+ pdf_stuff_name = None
+ concept_library = get_concept_library()
+
+ if blueprint.inputs is None:
+ msg = "For PipeExtract you must provide either a pdf or an image or a concept that refines one of them"
+ raise PipeExtractFactoryError(msg)
+ inputs = InputRequirementsFactory.make_from_blueprint(
+ domain=domain,
+ blueprint=blueprint.inputs or {},
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+ # Already validated above that we have exactly one input
+ input_name = blueprint.input_names[0]
+ input_requirement = inputs.get_required_input_requirement(input_name)
+
+ if concept_library.is_compatible(
+ tested_concept=input_requirement.concept,
+ wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),
+ strict=True,
+ ):
+ image_stuff_name = input_name
+ elif concept_library.is_compatible(
+ tested_concept=input_requirement.concept,
+ wanted_concept=get_native_concept(native_concept=NativeConceptCode.PDF),
+ strict=True,
+ ):
+ pdf_stuff_name = input_name
+ else:
+ msg = (
+ f"The input concept {input_requirement.concept.concept_string} is not compatible "
+ f"with the required concept {get_native_concept(native_concept=NativeConceptCode.IMAGE).concept_string} or "
+ f"{get_native_concept(native_concept=NativeConceptCode.PDF).concept_string}"
+ )
+ raise PipeExtractFactoryError(msg)
+
return PipeExtract(
domain=domain,
code=pipe_code,
@@ -39,12 +77,10 @@ class PipeExtractFactory(PipeFactoryProtocol[PipeExtractBlueprint, PipeExtract])
concept_code=output_domain_and_code.concept_code,
),
),
- inputs=InputRequirementsFactory.make_from_blueprint(
- domain=domain,
- blueprint=blueprint.inputs or {},
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- ),
+ inputs=inputs,
extract_choice=blueprint.model,
+ image_stuff_name=image_stuff_name,
+ pdf_stuff_name=pdf_stuff_name,
should_include_images=blueprint.page_images or False,
should_caption_images=blueprint.page_image_captions or False,
should_include_page_views=blueprint.page_views or False,
diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.py
index a1f86e9d1..8446b7501 100644
--- a/pipelex/pipe_operators/func/pipe_func.py
+++ b/pipelex/pipe_operators/func/pipe_func.py
@@ -8,16 +8,15 @@ from pipelex import log
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements import InputRequirements, TypedNamedInputRequirement
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements, TypedNamedInputRequirement
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
from pipelex.pipe_operators.pipe_operator import PipeOperator
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.pipeline.exceptions import DryRunMissingInputsError
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.system.registries.func_registry import func_registry
@@ -30,34 +29,46 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
type: Literal["PipeFunc"] = "PipeFunc"
function_name: str
+ @override
+ def required_variables(self) -> set[str]:
+ return set()
+
+ @override
+ def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
+ return self.inputs
+
@field_validator("function_name", mode="before")
@classmethod
def validate_function_name(cls, function_name: str) -> str:
function = func_registry.get_function(function_name)
if not function:
msg = f"Function '{function_name}' not found in registry"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
return_type = get_type_hints(function).get("return")
if return_type is None:
msg = f"Function '{function_name}' has no return type annotation"
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
if not issubclass(return_type, StuffContent):
msg = f"Function '{function_name}' return type {return_type} is not a subclass of StuffContent"
- raise PipeDefinitionError(msg)
+ raise TypeError(msg)
return function_name
@override
- def required_variables(self) -> set[str]:
- return set()
+ def validate_inputs_static(self):
+ pass
@override
- def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
- return self.inputs
+ def validate_inputs_with_library(self):
+ pass
@override
- def validate_output(self):
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
pass
@override
@@ -125,23 +136,20 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
if not working_memory.get_optional_stuff(named_input_requirement.variable_name):
missing_input_names.append(named_input_requirement.variable_name)
if missing_input_names:
- msg = f"Dry run failed for PipeFunc (function '{self.function_name}'): missing required inputs: {missing_input_names}"
- log.error(f"Dry run failed: missing required inputs: {missing_input_names}")
- raise DryRunMissingInputsError(
- message=msg,
- pipe_type=self.__class__.__name__,
- pipe_code=self.code,
- missing_inputs=missing_input_names,
- )
+ msg = f"Dry run failed for pipe '{self.code}' (PipeFunc): missing required inputs: {missing_input_names}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
return_type = get_type_hints(function).get("return")
if return_type is None:
- msg = f"Function '{self.function_name}' has no return type annotation"
- raise PipeDefinitionError(msg)
+ msg = f"Dry run failed for pipe '{self.code}' (PipeFunc): function '{self.function_name}' has no return type annotation"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
if not issubclass(return_type, StuffContent):
- msg = f"Function '{self.function_name}' return type {return_type} is not a subclass of StuffContent"
- raise PipeDefinitionError(msg)
+ msg = (
+ f"Dry run failed for pipe '{self.code}' (PipeFunc): "
+ f"function '{self.function_name}' return type {return_type} is not a subclass of StuffContent"
+ )
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode)
# TODO: Support PipeFunc returning with multiplicity. Create an equivalent of TypedNamedInputRequirement for outputs.
requirement = TypedNamedInputRequirement(
diff --git a/pipelex/pipe_operators/func/pipe_func_blueprint.py b/pipelex/pipe_operators/func/pipe_func_blueprint.py
index 5e7e83fa4..4df43116e 100644
--- a/pipelex/pipe_operators/func/pipe_func_blueprint.py
+++ b/pipelex/pipe_operators/func/pipe_func_blueprint.py
@@ -1,6 +1,7 @@
from typing import Literal
from pydantic import Field
+from typing_extensions import override
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
@@ -10,4 +11,10 @@ class PipeFuncBlueprint(PipeBlueprint):
pipe_category: Literal["PipeOperator"] = "PipeOperator"
function_name: str = Field(description="The name of the function to call.")
- # TODO: validate function_name
+ @override
+ def validate_inputs(self):
+ pass
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_operators/func/pipe_func_factory.py b/pipelex/pipe_operators/func/pipe_func_factory.py
index 75eb47b3b..785804d21 100644
--- a/pipelex/pipe_operators/func/pipe_func_factory.py
+++ b/pipelex/pipe_operators/func/pipe_func_factory.py
@@ -1,7 +1,7 @@
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
diff --git a/pipelex/pipe_operators/img_gen/exceptions.py b/pipelex/pipe_operators/img_gen/exceptions.py
new file mode 100644
index 000000000..e6656ab3c
--- /dev/null
+++ b/pipelex/pipe_operators/img_gen/exceptions.py
@@ -0,0 +1,5 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipeImgGenRunError(PipelexError):
+ pass
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen.py b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
index 491068d84..129e3eb0a 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen.py
@@ -1,7 +1,7 @@
from typing import TYPE_CHECKING, Literal
-from pydantic import Field, field_validator, model_validator
-from typing_extensions import override
+from pydantic import Field, model_validator
+from typing_extensions import Self, override
from pipelex import log
from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
@@ -10,16 +10,14 @@ from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background,
from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting
from pipelex.cogt.models.model_deck_check import check_img_gen_choice_with_deck
-from pipelex.config import StaticValidationReaction, get_config
+from pipelex.config import get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.exceptions import StaticValidationError, StaticValidationErrorType
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError, UnexpectedPipeDefinitionError
-from pipelex.core.pipes.exceptions import PipeInputError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.exceptions import StuffContentTypeError
@@ -27,13 +25,13 @@ from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_concept_library, get_content_generator, get_model_deck, get_native_concept
+from pipelex.pipe_operators.img_gen.exceptions import PipeImgGenRunError
from pipelex.pipe_operators.pipe_operator import PipeOperator
from pipelex.pipe_run.exceptions import PipeRunParamsError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_apply
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.types import Self
if TYPE_CHECKING:
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -55,9 +53,6 @@ class PipeImgGenOutput(PipeOutput):
return the_urls
-DEFAULT_PROMPT_VAR_NAME = "prompt"
-
-
class PipeImgGen(PipeOperator[PipeImgGenOutput]):
type: Literal["PipeImgGen"] = "PipeImgGen"
img_gen_prompt: str | None = None
@@ -65,6 +60,7 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
img_gen_choice: ImgGenModelChoice | None = None
+ # One-time settings (not in ImgGenSetting)
aspect_ratio: AspectRatio | None = Field(default=None, strict=False)
is_raw: bool | None = None
seed: int | Literal["auto"] | None = None
@@ -72,127 +68,98 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
output_format: OutputFormat | None = Field(default=None, strict=False)
output_multiplicity: VariableMultiplicity
- @field_validator("img_gen_prompt_var_name")
- @classmethod
- def validate_input_var_name_not_provided_as_attribute(cls, value: str | None) -> str | None:
- if value is not None:
- msg = "img_gen_prompt_var_name must be None before input validation"
- raise PipeDefinitionError(msg)
- return value
+ @override
+ def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
+ needed_inputs = InputRequirementsFactory.make_empty()
+ if not self.img_gen_prompt:
+ input_name, requirement = self.inputs.items[0] # We know there is only one input because of the validation
+ needed_inputs.add_requirement(variable_name=input_name, concept=requirement.concept)
+ return needed_inputs
+
+ @override
+ def required_variables(self) -> set[str]:
+ if self.img_gen_prompt_var_name:
+ return {self.img_gen_prompt_var_name}
+ return set()
@model_validator(mode="after")
- def validate_inputs(self) -> Self:
- self._validate_inputs()
+ def validate_fields(self) -> Self:
+ # Either we have img_gen_prompt or img_gen_prompt_var_name, but not both
+ if self.img_gen_prompt_var_name is not None and self.img_gen_prompt is not None:
+ msg = "Either 'img_gen_prompt' or 'img_gen_prompt_var_name' must be provided, but not both"
+ raise ValueError(msg)
+
+ # If img_gen_prompt is None, validate that inputs are properly configured
+ if self.img_gen_prompt is None:
+ # Must have exactly one input
+ if not self.inputs or len(self.inputs.items) != 1:
+ msg = "When 'img_gen_prompt' is not provided, there must be exactly one input"
+ raise ValueError(msg)
+
+ # The img_gen_prompt_var_name must match the key of the single input
+ input_name, _ = self.inputs.items[0]
+ if self.img_gen_prompt_var_name != input_name:
+ msg = (
+ f"When 'img_gen_prompt' is not provided, 'img_gen_prompt_var_name' must match the input name. "
+ f"Expected '{input_name}', got '{self.img_gen_prompt_var_name}'"
+ )
+ raise ValueError(msg)
+
return self
@override
- def _validate_with_libraries(self):
- self._validate_inputs()
+ def validate_inputs_static(self):
if self.img_gen_choice:
check_img_gen_choice_with_deck(img_gen_choice=self.img_gen_choice)
@override
- def validate_output(self):
- if not get_concept_library().is_compatible(
- tested_concept=self.output,
- wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),
+ def validate_inputs_with_library(self):
+ concept_library = get_concept_library()
+ input_name = self.inputs.variables[0]
+ input_requirement = self.inputs.get_required_input_requirement(input_name)
+
+ if not concept_library.is_compatible(
+ tested_concept=input_requirement.concept,
+ wanted_concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
strict=True,
):
msg = (
- f"The output of a ImgGen pipe must be compatible with the Image concept. "
- f"In the pipe '{self.code}' the output is '{self.output.concept_string}'"
+ f"The input of a PipeImgGen must be compatible with the Text concept (or refine it). "
+ f"Input '{input_name}' has concept '{input_requirement.concept.concept_string}'. "
+ "Image generation requires a text prompt. Use native.Text or a concept that refines it."
)
- raise PipeDefinitionError(msg)
-
- @override
- def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
- needed_inputs = InputRequirementsFactory.make_empty()
- if self.img_gen_prompt:
- needed_inputs.add_requirement(
- variable_name="img_gen_prompt",
- concept=ConceptFactory.make_native_concept(
- native_concept_code=NativeConceptCode.TEXT,
- ),
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.IMG_GEN_INPUT_NOT_TEXT_COMPATIBLE,
+ pipe_code=self.code,
+ variable_names=[input_name],
+ required_concept_codes=["native.Text"],
+ provided_concept_code=input_requirement.concept.concept_string,
)
- else:
- for input_name, requirement in self.inputs.items:
- needed_inputs.add_requirement(variable_name=input_name, concept=requirement.concept)
- return needed_inputs
@override
- def required_variables(self) -> set[str]:
- if self.img_gen_prompt_var_name:
- return {self.img_gen_prompt_var_name}
- return {DEFAULT_PROMPT_VAR_NAME}
+ def validate_output_static(self):
+ pass
- def _validate_inputs(self):
- concept_library = get_concept_library()
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
- # check that we have either an img_gen_prompt passed as attribute or as a single text input
- nb_inputs = self.inputs.nb_inputs
- if self.img_gen_prompt:
- if nb_inputs > 0:
- msg = "There must be no inputs if img_gen_prompt is provided"
- raise PipeDefinitionError(msg)
- # we're good with the prompt provided as attribute
- return
-
- if nb_inputs > 1:
- too_many_candidate_inputs_error = StaticValidationError(
- error_type=StaticValidationErrorType.TOO_MANY_CANDIDATE_INPUTS,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=self.inputs.variables,
- explanation="Only one text input can be provided for image gen prompt",
- )
- match reactions.get(StaticValidationErrorType.TOO_MANY_CANDIDATE_INPUTS, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(too_many_candidate_inputs_error.desc())
- case StaticValidationReaction.RAISE:
- raise too_many_candidate_inputs_error
- elif nb_inputs < 1:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- explanation="You must provide an image gen prompt either as attribute of the pipe or as a single text input",
- )
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # We have confirmed right above that we have exactly one input
- # get input_name, requirement from single item in inputs
- input_name, requirement = self.inputs.items[0]
- if concept_library.is_compatible(
- tested_concept=requirement.concept,
- wanted_concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ @override
+ def validate_output_with_library(self):
+ if not get_concept_library().is_compatible(
+ tested_concept=self.output,
+ wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),
+ strict=True,
):
- self.img_gen_prompt_var_name = input_name
- else:
- inadequate_input_concept_error = StaticValidationError(
- error_type=StaticValidationErrorType.INADEQUATE_INPUT_CONCEPT,
+ msg = (
+ f"The output of a ImgGen pipe must be compatible with the Image concept. "
+ f"In the pipe '{self.code}' the output is '{self.output.concept_string}'"
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
domain=self.domain,
pipe_code=self.code,
- variable_names=[input_name],
- provided_concept_code=requirement.concept.code,
- explanation="For PipeImgGen you must provide a text input or a concept that refines text",
+ provided_concept_code=self.output.concept_string,
+ required_concept_codes=[NativeConceptCode.IMAGE.concept_string],
)
- match reactions.get(StaticValidationErrorType.INADEQUATE_INPUT_CONCEPT, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(inadequate_input_concept_error.desc())
- case StaticValidationReaction.RAISE:
- raise inadequate_input_concept_error
@override
async def _run_operator_pipe(
@@ -211,6 +178,7 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
)
applied_output_multiplicity = multiplicity_resolution.resolved_multiplicity
+ # TODO: move this to the factory
log.verbose("Getting image generation prompt from context")
if self.img_gen_prompt:
img_gen_prompt_text = self.img_gen_prompt
@@ -218,19 +186,23 @@ class PipeImgGen(PipeOperator[PipeImgGenOutput]):
try:
img_gen_prompt_text = working_memory.get_stuff_as_str(stuff_name)
except WorkingMemoryStuffNotFoundError as stuff_not_found_error:
- msg = f"Could not find a valid image generation prompt named '{stuff_name}' in the working_memory: {stuff_not_found_error}"
- raise PipeInputError(
- message=msg,
- pipe_code=self.code,
- variable_name=stuff_name,
- concept_code=None,
- ) from stuff_not_found_error
+ msg = (
+ f"While runnning the PipeImgGen '{self.code}' an error occurred: "
+ f"Could not find a valid image generation prompt named '{stuff_name}' in the working_memory: {stuff_not_found_error}"
+ )
+ raise PipeImgGenRunError(message=msg) from stuff_not_found_error
except StuffContentTypeError as stuff_content_type_error:
- msg = f"The image generation prompt named '{stuff_name}' is not of the right type: {stuff_content_type_error}"
- raise PipeInputError(message=msg, pipe_code=self.code, variable_name=stuff_name, concept_code=None) from stuff_content_type_error
+ msg = (
+ f"While runnning the PipeImgGen '{self.code}' an error occurred: "
+ f"The image generation prompt named '{stuff_name}' is not of the right type: {stuff_content_type_error}"
+ )
+ raise PipeImgGenRunError(message=msg) from stuff_content_type_error
else:
- msg = "You must provide an image gen prompt either as attribute of the pipe or as a single text input"
- raise UnexpectedPipeDefinitionError(msg)
+ msg = (
+ f"While runnning the PipeImgGen '{self.code}' an error occurred: "
+ "You must provide an image gen prompt either as attribute of the pipe or as a single text input"
+ )
+ raise PipeImgGenRunError(msg)
img_gen_config = get_config().cogt.img_gen_config
img_gen_param_defaults = img_gen_config.img_gen_param_defaults
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py
index 402bc7e2e..62866ecfe 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py
@@ -1,6 +1,7 @@
from typing import Literal
from pydantic import Field
+from typing_extensions import override
from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, OutputFormat
from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice
@@ -21,3 +22,24 @@ class PipeImgGenBlueprint(PipeBlueprint):
seed: int | Literal["auto"] | None = None
background: Background | None = Field(default=None, strict=False)
output_format: OutputFormat | None = Field(default=None, strict=False)
+
+ @override
+ def validate_inputs(self):
+ # check that we have either an img_gen_prompt passed as attribute or as a single text input
+ if not self.inputs:
+ if not self.img_gen_prompt:
+ msg = "If no inputs are provided, you must provide an 'img_gen_prompt' as attribute."
+ raise ValueError(msg)
+
+ if self.inputs and self.img_gen_prompt:
+ msg = "You must provide either an 'img_gen_prompt' as attribute or as a single text input, but not both"
+ raise ValueError(msg)
+
+ nb_inputs = self.nb_inputs
+ if nb_inputs > 1:
+ msg = f"Too many inputs provided for PipeImgGen: {self.input_names}. Only one input is allowed."
+ raise ValueError(msg)
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py b/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py
index 130c6ae14..558ff0fae 100644
--- a/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py
+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen_factory.py
@@ -1,7 +1,7 @@
from typing_extensions import override
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.hub import get_required_concept
@@ -31,15 +31,27 @@ class PipeImgGenFactory(PipeFactoryProtocol[PipeImgGenBlueprint, PipeImgGen]):
concept_string_or_code=output_parse_result.concept,
concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
)
+ inputs = InputRequirementsFactory.make_from_blueprint(
+ domain=domain,
+ blueprint=blueprint.inputs or {},
+ concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
+ )
+
+ img_gen_prompt = blueprint.img_gen_prompt
+ img_gen_prompt_var_name = blueprint.img_gen_prompt_var_name
+
+ # If we have inputs, that means that the prompt is in the inputs.
+ # The blueprint already validated that there is only 1 input.
+ if blueprint.inputs:
+ img_gen_prompt_var_name = blueprint.input_names[0]
+ else:
+ img_gen_prompt = blueprint.img_gen_prompt
+
return PipeImgGen(
domain=domain,
code=pipe_code,
description=blueprint.description,
- inputs=InputRequirementsFactory.make_from_blueprint(
- domain=domain,
- blueprint=blueprint.inputs or {},
- concept_codes_from_the_same_domain=concept_codes_from_the_same_domain,
- ),
+ inputs=inputs,
output=get_required_concept(
concept_string=ConceptFactory.make_concept_string_with_domain(
domain=output_domain_and_code.domain,
@@ -47,8 +59,8 @@ class PipeImgGenFactory(PipeFactoryProtocol[PipeImgGenBlueprint, PipeImgGen]):
),
),
output_multiplicity=final_multiplicity,
- img_gen_prompt=blueprint.img_gen_prompt,
- img_gen_prompt_var_name=blueprint.img_gen_prompt_var_name,
+ img_gen_prompt=img_gen_prompt,
+ img_gen_prompt_var_name=img_gen_prompt_var_name,
img_gen_choice=blueprint.model,
aspect_ratio=blueprint.aspect_ratio,
is_raw=blueprint.is_raw,
diff --git a/pipelex/pipe_operators/llm/exceptions.py b/pipelex/pipe_operators/llm/exceptions.py
new file mode 100644
index 000000000..039717bc2
--- /dev/null
+++ b/pipelex/pipe_operators/llm/exceptions.py
@@ -0,0 +1,9 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class LLMPromptBlueprintValueError(ValueError):
+ pass
+
+
+class PipeLLMFactoryError(PipelexError):
+ pass
diff --git a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py b/pipelex/pipe_operators/llm/llm_prompt_blueprint.py
index 80ce1b08a..c7ba93f2c 100644
--- a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py
+++ b/pipelex/pipe_operators/llm/llm_prompt_blueprint.py
@@ -3,7 +3,6 @@ from typing import TYPE_CHECKING, Any
from pydantic import BaseModel
from pipelex import log
-from pipelex.cogt.exceptions import LLMPromptSpecError
from pipelex.cogt.image.prompt_image_factory import PromptImageFactory
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
@@ -11,8 +10,9 @@ from pipelex.cogt.templating.template_preprocessor import preprocess_template
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.hub import get_content_generator
+from pipelex.pipe_operators.llm.exceptions import LLMPromptBlueprintValueError
from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
-from pipelex.tools.misc.context_provider_abstract import ContextProviderAbstract, ContextProviderException
+from pipelex.tools.misc.context_provider_abstract import ContextProviderAbstract, ContextProviderError
from pipelex.tools.misc.dict_utils import substitute_nested_in_context
if TYPE_CHECKING:
@@ -25,9 +25,6 @@ class LLMPromptBlueprint(BaseModel):
prompt_blueprint: TemplateBlueprint | None = None
user_images: list[str] | None = None
- def validate_with_libraries(self):
- pass
-
def required_variables(self) -> set[str]:
required_variables: set[str] = set()
if self.user_images:
@@ -51,7 +48,6 @@ class LLMPromptBlueprint(BaseModel):
template_source=system_prompt_template_source,
)
)
-
return {
variable_name
for variable_name in required_variables
@@ -88,7 +84,7 @@ class LLMPromptBlueprint(BaseModel):
for image_item in prompt_image_content: # pyright: ignore[reportUnknownVariableType]
if not isinstance(image_item, ImageContent):
msg = f"Item of '{user_image_name}' is of type '{type(image_item).__name__}', it should be ImageContent" # pyright: ignore[reportUnknownArgumentType]
- raise LLMPromptSpecError(msg)
+ raise LLMPromptBlueprintValueError(msg)
user_image = PromptImageFactory.make_prompt_image(url=image_item.url, base_64_str=image_item.base_64)
prompt_user_images[user_image_name] = user_image
else:
@@ -96,8 +92,8 @@ class LLMPromptBlueprint(BaseModel):
f"User image '{user_image_name}' is of type '{type(prompt_image_content).__name__}', "
"it should be ImageContent or list of ImageContent"
)
- raise LLMPromptSpecError(msg)
- except ContextProviderException:
+ raise LLMPromptBlueprintValueError(msg)
+ except ContextProviderError:
# If single image failed, try to get as a collection (list or tuple)
try:
image_collection = context_provider.get_typed_object_or_attribute(name=user_image_name, wanted_type=None)
@@ -111,10 +107,10 @@ class LLMPromptBlueprint(BaseModel):
msg = (
f"Could not find a valid user image or image collection named '{user_image_name}' from the provided context_provider"
)
- raise LLMPromptSpecError(msg)
- except ContextProviderException as exc:
+ raise LLMPromptBlueprintValueError(msg)
+ except ContextProviderError as exc:
msg = f"Could not find a valid user image named '{user_image_name}' from the provided context_provider: {exc}"
- raise LLMPromptSpecError(msg) from exc
+ raise LLMPromptBlueprintValueError(msg) from exc
############################################################
# User text
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index 52a65ad71..4a1d23e14 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -1,6 +1,6 @@
from typing import Literal, cast
-from pydantic import model_validator
+from pydantic import ValidationError, model_validator
from typing_extensions import override
from pipelex import log
@@ -13,15 +13,14 @@ from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
from pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSetting, LLMSettingChoices
from pipelex.cogt.models.model_deck_check import check_llm_choice_with_deck
from pipelex.cogt.templating.template_category import TemplateCategory
-from pipelex.config import StaticValidationReaction, get_config
+from pipelex.config import get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
-from pipelex.core.exceptions import StaticValidationError, StaticValidationErrorType
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements import InputRequirements
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.list_content import ListContent
@@ -39,13 +38,14 @@ from pipelex.hub import (
from pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprint
from pipelex.pipe_operators.llm.pipe_llm_blueprint import StructuringMethod
from pipelex.pipe_operators.pipe_operator import PipeOperator
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_params import (
PipeRunParamKey,
PipeRunParams,
output_multiplicity_to_apply,
)
-from pipelex.pipeline.exceptions import PipeRunError
from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
from pipelex.tools.typing.structure_printer import StructurePrinter
from pipelex.types import Self
@@ -61,11 +61,6 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
structuring_method: StructuringMethod | None = None
output_multiplicity: VariableMultiplicity | None = None
- @model_validator(mode="after")
- def _validate_inputs(self) -> Self:
- self.validate_inputs()
- return self
-
@model_validator(mode="after")
def validate_output_concept_consistency(self) -> Self:
if self.structuring_method is not None and self.output.structure_class_name == NativeConceptCode.TEXT:
@@ -73,28 +68,47 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
f"Output concept '{self.output.code}' is considered a Text concept, "
f"so it cannot be structured. Maybe you forgot to add '{NativeConceptCode.TEXT}' to the class registry?"
)
- raise PipeDefinitionError(msg)
+ raise ValueError(msg)
return self
@override
- def _validate_with_libraries(self):
- self.validate_inputs()
- self.llm_prompt_spec.validate_with_libraries()
+ def validate_inputs_static(self):
if self.llm_choices:
for llm_choice in self.llm_choices.list_choice_strings():
check_llm_choice_with_deck(llm_choice=llm_choice)
@override
- def validate_output(self):
+ def validate_inputs_with_library(self):
+ pass
+
+ @override
+ def validate_output_static(self):
+ pass
+
+ @override
+ def validate_output_with_library(self):
+ # TODO: generalize because there are other concepts PipeLLM can't generate, not just images,
+ # and PipeLLM is not the only one with this kind of constraints
+
+ # Allow Dynamic output concept as it's flexible and can represent anything
+ if NativeConceptCode.is_dynamic_concept(concept_code=self.output.code):
+ return
+
if get_concept_library().is_compatible(
tested_concept=self.output,
wanted_concept=get_native_concept(native_concept=NativeConceptCode.IMAGE),
):
msg = (
- f"The output of a LLM pipe cannot be compatible with the Image concept. In the "
- f"pipe '{self.code}' the output is '{self.output.concept_string}'"
+ f"The output of the PipeLLM '{self.code}' cannot be compatible with the Image concept. "
+ f"The output concept is '{self.output.concept_string}'. "
+ "Use a PipeImgGen if you want to generate images. You can use a PipeLLM to generate the prompt for a PipeImgGen."
+ )
+ raise PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.LLM_OUTPUT_CANNOT_BE_IMAGE,
+ pipe_code=self.code,
+ provided_concept_code=self.output.concept_string,
)
- raise PipeDefinitionError(msg)
@override
def needed_inputs(self, visited_pipes: set[str] | None = None) -> InputRequirements:
@@ -110,56 +124,6 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
"""Required variables are the variables that are used in the current prompt template or system prompt"""
return {variable_name for variable_name in self.llm_prompt_spec.required_variables() if not variable_name.startswith("_")}
- def validate_inputs(self):
- static_validation_config = get_config().pipelex.static_validation_config
- default_reaction = static_validation_config.default_reaction
- reactions = static_validation_config.reactions
- # Those are the variables required in the prompt template or system prompt
- required_variables = self.required_variables()
-
- # 1: Check that all the required variables are actually in the inputs
- for required_variable_name in required_variables:
- if required_variable_name not in self.needed_inputs().variables:
- missing_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.MISSING_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[required_variable_name],
- )
- match reactions.get(StaticValidationErrorType.MISSING_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(missing_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise missing_input_var_error
-
- # 2: Check that all inputs are in the required variables
- for input_name, requirement in self.needed_inputs().items:
- if input_name not in required_variables:
- explanation: str | None = None
- if get_concept_library().is_image_concept(concept=requirement.concept):
- # We have an exraneous image input, the user probably forgot to add it into the prompt template
- explanation = (
- f"You have provided an image input named '{input_name}', but it is not referenced in the prompt template. "
- "Please add it to the prompt template."
- )
-
- extraneous_input_var_error = StaticValidationError(
- error_type=StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
- domain=self.domain,
- pipe_code=self.code,
- variable_names=[input_name],
- explanation=explanation,
- )
- match reactions.get(StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE, default_reaction):
- case StaticValidationReaction.IGNORE:
- pass
- case StaticValidationReaction.LOG:
- log.error(extraneous_input_var_error.desc())
- case StaticValidationReaction.RAISE:
- raise extraneous_input_var_error
-
@override
async def _run_operator_pipe(
self,
@@ -259,8 +223,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
)
except LLMCompletionError as exc:
location = self._format_error_location(pipe_run_params=pipe_run_params)
- msg = f"Error generating text with LLM {location}: {exc}"
- raise PipeRunError(msg) from exc
+ error_details = self._format_llm_error(exc)
+ msg = f"Error generating text with LLM {location}: {error_details}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
the_content = TextContent(
text=generated_text,
@@ -373,8 +338,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
)
except LLMCompletionError as exc:
location = self._format_error_location(pipe_run_params=pipe_run_params)
- msg = f"Error generating list of objects with text then object {location}: {exc}"
- raise PipeRunError(msg) from exc
+ error_details = self._format_llm_error(exc)
+ msg = f"Error generating list of objects with text then object {location}: {error_details}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
else:
# We're generating a list of objects directly
method_desc = "object_direct"
@@ -389,8 +355,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
)
except LLMCompletionError as exc:
location = self._format_error_location(pipe_run_params=pipe_run_params)
- msg = f"Error generating list of objects with direct method {location}: {exc}"
- raise PipeRunError(msg) from exc
+ error_details = self._format_llm_error(exc)
+ msg = f"Error generating list of objects with direct method {location}: {error_details}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
the_content = ListContent(items=generated_objects)
else:
@@ -413,8 +380,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
)
except LLMCompletionError as exc:
location = self._format_error_location(pipe_run_params=pipe_run_params)
- msg = f"Error generating single object with text then object {location}: {exc}"
- raise PipeRunError(msg) from exc
+ error_details = self._format_llm_error(exc)
+ msg = f"Error generating single object with text then object {location}: {error_details}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
else:
# We're generating a single object directly
method_desc = "object_direct"
@@ -428,8 +396,9 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
)
except LLMCompletionError as exc:
location = self._format_error_location(pipe_run_params=pipe_run_params)
- msg = f"Error generating single object with direct method {location}: {exc}"
- raise PipeRunError(msg) from exc
+ error_details = self._format_llm_error(exc)
+ msg = f"Error generating single object with direct method {location}: {error_details}"
+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode) from exc
the_content = generated_object
return the_content
@@ -437,6 +406,17 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):
def _format_error_location(self, pipe_run_params: PipeRunParams) -> str:
return f"in pipe '{pipe_run_params.pipe_stack_str}'"
+ def _format_llm_error(self, exc: LLMCompletionError) -> str:
+ """Format an LLMCompletionError, extracting and formatting any ValidationError in the chain."""
+ error_details = str(exc)
+ current_exc: BaseException | None = exc
+ while current_exc is not None:
+ if isinstance(current_exc, ValidationError):
+ error_details = format_pydantic_validation_error(current_exc)
+ break
+ current_exc = current_exc.__cause__
+ return error_details
+
@override
async def _dry_run_operator_pipe(
self,
diff --git a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
index 78868772a..9aeef6590 100644
--- a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
+++ b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
@@ -1,7 +1,13 @@
from typing import Literal
+from typing_extensions import override
+
from pipelex.cogt.llm.llm_setting import LLMModelChoice
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_preprocessor import preprocess_template
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+from pipelex.tools.jinja2.jinja2_errors import Jinja2DetectVariablesError
+from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
from pipelex.types import StrEnum
@@ -21,3 +27,53 @@ class PipeLLMBlueprint(PipeBlueprint):
prompt: str | None = None
structuring_method: StructuringMethod | None = None
+
+ @override
+ def validate_inputs(self):
+ # Get all required variables from prompt and system_prompt
+ required_variables: set[str] = set()
+
+ if self.prompt:
+ preprocessed_template = preprocess_template(self.prompt)
+ try:
+ required_variables.update(
+ detect_jinja2_required_variables(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=preprocessed_template,
+ )
+ )
+ except Jinja2DetectVariablesError as exc:
+ msg = f"Could not detect required variables in prompt for PipeLLM: {exc}"
+ raise ValueError(msg) from exc
+
+ if self.system_prompt:
+ preprocessed_system_template = preprocess_template(self.system_prompt)
+ try:
+ required_variables.update(
+ detect_jinja2_required_variables(
+ template_category=TemplateCategory.LLM_PROMPT,
+ template_source=preprocessed_system_template,
+ )
+ )
+ except Jinja2DetectVariablesError as exc:
+ msg = f"Could not detect required variables in system prompt for PipeLLM: {exc}"
+ raise ValueError(msg) from exc
+ # Filter out internal variables that start with underscore and special variables
+ # TODO: replace magic strings by StrEnum and also, make this check clearer and more readable
+ required_variables = {var for var in required_variables if not var.startswith("_") and var not in ("preliminary_text", "place_holder")}
+
+ # Check that all required variables are in inputs
+ input_names: set[str] = set(self.inputs.keys()) if self.inputs else set()
+ missing_variables: set[str] = required_variables - input_names
+
+ if missing_variables:
+ missing_vars_str = ", ".join(sorted(missing_variables))
+ msg = (
+ f"Missing input variable(s) in prompt template: {missing_vars_str}. "
+ "These variables are used in the prompt but not declared in inputs."
+ )
+ raise ValueError(msg)
+
+ @override
+ def validate_output(self):
+ pass
diff --git a/pipelex/pipe_operators/llm/pipe_llm_factory.py b/pipelex/pipe_operators/llm/pipe_llm_factory.py
index 6a39ae224..d09b842b9 100644
--- a/pipelex/pipe_operators/llm/pipe_llm_factory.py
+++ b/pipelex/pipe_operators/llm/pipe_llm_factory.py
@@ -1,3 +1,4 @@
+from pydantic import ValidationError
from typing_extensions import override
from pipelex.cogt.llm.llm_setting import LLMSettingChoices
@@ -5,12 +6,12 @@ from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.core.pipe_errors import PipeDefinitionError
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.variable_multiplicity import make_variable_multiplicity, parse_concept_with_multiplicity
from pipelex.hub import get_native_concept, get_optional_domain, get_required_concept
+from pipelex.pipe_operators.llm.exceptions import PipeLLMFactoryError
from pipelex.pipe_operators.llm.llm_prompt_blueprint import LLMPromptBlueprint
from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
@@ -38,12 +39,12 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):
template=system_prompt,
category=TemplateCategory.LLM_PROMPT,
)
- except Jinja2TemplateSyntaxError as exc:
+ except ValidationError as exc:
error_msg = (
- f"Template syntax error in system prompt for pipe '{pipe_code}' "
+ f"Template syntax error in system prompt for pipe '{pipe_code}'"
f"in domain '{domain}': {exc}. Template source:\n{blueprint.system_prompt}"
)
- raise PipeDefinitionError(error_msg) from exc
+ raise PipeLLMFactoryError(error_msg) from exc
user_text_jinja2_blueprint: TemplateBlueprint | None = None
if blueprint.prompt:
@@ -56,7 +57,7 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):
error_msg = (
f"Template syntax error in user prompt for pipe '{pipe_code}' in domain '{domain}': {exc}. Template source:\n{blueprint.prompt}"
)
- raise PipeDefinitionError(error_msg) from exc
+ raise PipeLLMFactoryError(error_msg) from exc
user_images: list[str] = []
if blueprint.inputs:
diff --git a/pipelex/pipe_operators/pipe_operator.py b/pipelex/pipe_operators/pipe_operator.py
index 7f2fb56ac..1b9f171b8 100644
--- a/pipelex/pipe_operators/pipe_operator.py
+++ b/pipelex/pipe_operators/pipe_operator.py
@@ -1,9 +1,8 @@
from abc import abstractmethod
-from typing import TYPE_CHECKING, Generic, Literal, TypeVar
+from typing import TYPE_CHECKING, Generic, Literal, TypeVar, final
from typing_extensions import override
-from pipelex import log
from pipelex.cogt.exceptions import ModelNotFoundError, ModelWaterfallError
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_abstract import PipeAbstract
@@ -16,6 +15,7 @@ from pipelex.pipeline.job_metadata import JobMetadata
if TYPE_CHECKING:
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
+
PipeOperatorOutputType = TypeVar("PipeOperatorOutputType", bound=PipeOutput)
@@ -26,27 +26,15 @@ class PipeOperator(PipeAbstract, Generic[PipeOperatorOutputType]):
def class_name(self) -> str:
return self.__class__.__name__
+ @final
@override
- async def run_pipe(
+ async def _run_pipe(
self,
job_metadata: JobMetadata,
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
- print_intermediate_outputs: bool | None = False,
) -> PipeOutput:
- pipe_run_params.push_pipe_to_stack(pipe_code=self.code)
- self.monitor_pipe_stack(pipe_run_params=pipe_run_params)
-
- updated_metadata = JobMetadata(
- pipe_job_ids=[self.code],
- )
- job_metadata.update(updated_metadata=updated_metadata)
-
- pipe_run_info = self._format_pipe_run_info(pipe_run_params=pipe_run_params)
- # log.info(pipe_run_info)
- if pipe_run_params.run_mode == PipeRunMode.LIVE:
- log.info(pipe_run_info)
try:
match pipe_run_params.run_mode:
case PipeRunMode.LIVE:
@@ -95,8 +83,6 @@ class PipeOperator(PipeAbstract, Generic[PipeOperatorOutputType]):
model_handle=model_not_found_error.model_handle,
fallback_list=None,
) from model_not_found_error
- pipe_run_params.pop_pipe_from_stack(pipe_code=self.code)
-
return pipe_output
@abstractmethod
diff --git a/pipelex/pipe_run/dry_run.py b/pipelex/pipe_run/dry_run.py
index cf2adce06..e95d272a2 100644
--- a/pipelex/pipe_run/dry_run.py
+++ b/pipelex/pipe_run/dry_run.py
@@ -1,14 +1,12 @@
-import asyncio
-import functools
import time
-from concurrent.futures import ThreadPoolExecutor
from pydantic import BaseModel
from pipelex import log
+from pipelex.base_exceptions import PipelexError
from pipelex.config import get_config
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.input_requirements import InputRequirements, TypedNamedInputRequirement
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements, TypedNamedInputRequirement
from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.text_content import TextContent
@@ -20,7 +18,7 @@ from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.types import StrEnum
-class DryRunError(Exception):
+class DryRunError(PipelexError):
"""Raised when a dry run fails due to missing inputs or other validation issues."""
@@ -44,12 +42,8 @@ class DryRunOutput(BaseModel):
async def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> DryRunOutput:
- """Dry run a single pipe directly without parallelization."""
- allowed_to_fail_pipes = get_config().pipelex.dry_run_config.allowed_to_fail_pipes
- # TODO: fail and raise properly
try:
needed_inputs_for_factory = _convert_to_working_memory_format(needed_inputs_spec=pipe.needed_inputs())
-
working_memory = WorkingMemoryFactory.make_for_dry_run(needed_inputs=needed_inputs_for_factory)
pipe.validate_with_libraries()
await pipe.run_pipe(
@@ -58,7 +52,7 @@ async def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> Dr
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),
)
except PipeStackOverflowError as exc:
- if pipe.code in allowed_to_fail_pipes:
+ if pipe.code in get_config().pipelex.dry_run_config.allowed_to_fail_pipes:
error_message = f"Allowed to fail dry run for pipe '{pipe.code}': {exc}"
return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.FAILURE, error_message=error_message)
elif raise_on_failure:
@@ -70,7 +64,7 @@ async def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> Dr
return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.SUCCESS)
-async def dry_run_pipes(pipes: list[PipeAbstract], run_in_parallel: bool = True, raise_on_failure: bool = True) -> dict[str, DryRunOutput]:
+async def dry_run_pipes(pipes: list[PipeAbstract], raise_on_failure: bool = True) -> dict[str, DryRunOutput]:
"""Dry run pipes with optional parallelization.
Args:
@@ -94,37 +88,8 @@ async def dry_run_pipes(pipes: list[PipeAbstract], run_in_parallel: bool = True,
results: dict[str, DryRunOutput] = {}
allowed_to_fail_pipes = get_config().pipelex.dry_run_config.allowed_to_fail_pipes
- if run_in_parallel:
-
- def run_pipe_in_thread(pipe: PipeAbstract) -> DryRunOutput:
- """Parallel execution using ThreadPoolExecutor"""
- loop = asyncio.new_event_loop()
- asyncio.set_event_loop(loop)
- try:
- return loop.run_until_complete(dry_run_pipe(pipe, raise_on_failure=raise_on_failure))
- finally:
- loop.close()
-
- with ThreadPoolExecutor() as executor:
- futures = [asyncio.get_running_loop().run_in_executor(executor, functools.partial(run_pipe_in_thread, pipe)) for pipe in pipes]
- for future in asyncio.as_completed(futures):
- try:
- output = await future
- results[output.pipe_code] = output
- except Exception as exc:
- # If raise_on_failure is True, re-raise the first exception encountered
- # Otherwise, this shouldn't happen as dry_run_pipe should return a DryRunOutput
- if raise_on_failure:
- # Cancel remaining futures
- for f in futures:
- if not f.done():
- f.cancel()
- raise
- # This path shouldn't normally be reached, but handle it gracefully
- log.error(f"Unexpected exception in dry run: {exc}")
- else:
- for pipe in pipes:
- results[pipe.code] = await dry_run_pipe(pipe, raise_on_failure=raise_on_failure)
+ for pipe in pipes:
+ results[pipe.code] = await dry_run_pipe(pipe, raise_on_failure=raise_on_failure)
successful_pipes: list[str] = []
failed_pipes: list[str] = []
diff --git a/pipelex/pipe_run/exceptions.py b/pipelex/pipe_run/exceptions.py
index a468f4a7c..cd1a0dc44 100644
--- a/pipelex/pipe_run/exceptions.py
+++ b/pipelex/pipe_run/exceptions.py
@@ -1,20 +1,22 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-class PipeRunParamsError(PipelexException):
+class PipeRunParamsError(PipelexError):
pass
-class BatchParamsError(PipelexException):
+class BatchParamsError(PipelexError):
pass
-class PipeRunError(PipelexException):
- pass
+class PipeRunError(PipelexError):
+ def __init__(self, message: str, run_mode: PipeRunMode):
+ self.run_mode = run_mode
+ super().__init__(message)
-class PipeRouterError(PipelexException):
+class PipeRouterError(PipelexError):
def __init__(
self,
message: str,
diff --git a/pipelex/pipe_run/pipe_router_protocol.py b/pipelex/pipe_run/pipe_router_protocol.py
index dab56a97a..daf3d8b43 100644
--- a/pipelex/pipe_run/pipe_router_protocol.py
+++ b/pipelex/pipe_run/pipe_router_protocol.py
@@ -1,10 +1,10 @@
+from abc import abstractmethod
from typing import Protocol
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.observer.observer_protocol import ObserverProtocol, PayloadKey, PayloadType
from pipelex.pipe_run.exceptions import PipeRouterError, PipeRunError
from pipelex.pipe_run.pipe_job import PipeJob
-from pipelex.pipeline.exceptions import DryRunMissingInputsError
class PipeRouterProtocol(Protocol):
@@ -52,16 +52,6 @@ class PipeRouterProtocol(Protocol):
try:
pipe_output = await self._run_pipe_job(pipe_job)
- except DryRunMissingInputsError as exc:
- await self._after_failing_run(pipe_job, exc)
- raise PipeRouterError(
- message=exc.message,
- run_mode=pipe_job.pipe_run_params.run_mode,
- pipe_code=pipe_job.pipe.code,
- output_name=pipe_job.output_name,
- pipe_stack=pipe_job.pipe_run_params.pipe_stack,
- missing_inputs=exc.missing_inputs,
- ) from exc
except PipeRunError as exc:
await self._after_failing_run(pipe_job, exc)
raise PipeRouterError(
@@ -76,6 +66,7 @@ class PipeRouterProtocol(Protocol):
return pipe_output
+ @abstractmethod
async def _run_pipe_job(
self,
pipe_job: PipeJob,
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index 76e75d5a8..0f736eddb 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -25,14 +25,12 @@ from pipelex.cogt.exceptions import (
from pipelex.cogt.inference.inference_manager import InferenceManager
from pipelex.cogt.models.model_manager import ModelManager
from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
-from pipelex.config import ConfigPaths, PipelexConfig, get_config
-from pipelex.core.concepts.concept_library import ConceptLibrary
-from pipelex.core.domains.domain_library import DomainLibrary
-from pipelex.core.pipes.pipe_library import PipeLibrary
+from pipelex.config import get_config
from pipelex.core.registry_models import CoreRegistryModels
from pipelex.core.validation import report_validation_error
from pipelex.hub import PipelexHub, set_pipelex_hub
-from pipelex.libraries.library_manager_factory import LibraryManagerFactory
+from pipelex.libraries.library_manager import LibraryManager
+from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
from pipelex.observer.local_observer import LocalObserver
from pipelex.observer.multi_observer import MultiObserver
from pipelex.observer.observer_protocol import ObserverProtocol
@@ -49,7 +47,8 @@ from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocol
from pipelex.system.configuration.config_loader import config_manager
from pipelex.system.configuration.config_root import ConfigRoot
-from pipelex.system.environment import get_optional_env
+from pipelex.system.configuration.configs import ConfigPaths, PipelexConfig
+from pipelex.system.environment import is_env_var_truthy
from pipelex.system.registries.func_registry import func_registry
from pipelex.system.runtime import IntegrationMode, runtime_manager
from pipelex.system.telemetry.observer_telemetry import ObserverTelemetry
@@ -108,25 +107,11 @@ class Pipelex(metaclass=MetaSingleton):
self.plugin_manager = PluginManager()
self.pipelex_hub.set_plugin_manager(self.plugin_manager)
- # pipelex libraries
- domain_library = DomainLibrary.make_empty()
- concept_library = ConceptLibrary.make_empty()
- pipe_library = PipeLibrary.make_empty()
- self.pipelex_hub.set_domain_library(domain_library=domain_library)
- self.pipelex_hub.set_concept_library(concept_library=concept_library)
- self.pipelex_hub.set_pipe_library(pipe_library=pipe_library)
-
- self.library_manager = LibraryManagerFactory.make(
- domain_library=domain_library,
- concept_library=concept_library,
- pipe_library=pipe_library,
- )
- self.pipelex_hub.set_library_manager(library_manager=self.library_manager)
-
self.reporting_delegate: ReportingProtocol | None = None
self.telemetry_manager: TelemetryManagerAbstract | None = None
# pipeline
self.pipeline_tracker: PipelineTrackerProtocol | None = None
+ self.library_manager: LibraryManagerAbstract | None = None
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} init done")
@@ -173,6 +158,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
telemetry_config: TelemetryConfig | None = None,
telemetry_manager: TelemetryManagerAbstract | None = None,
observers: dict[str, ObserverProtocol] | None = None,
+ library_manager: LibraryManagerAbstract | None = None,
**kwargs: Any,
):
if kwargs:
@@ -186,6 +172,9 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.pipelex_hub.set_secrets_provider(secrets_provider=secrets_provider)
self.pipelex_hub.set_storage_provider(storage_provider)
+ self.library_manager = library_manager or LibraryManager()
+ self.pipelex_hub.set_library_manager(library_manager=self.library_manager)
+
# cogt
self.plugin_manager.setup()
@@ -242,6 +231,9 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.inference_manager = inference_manager or InferenceManager()
self.pipelex_hub.set_inference_manager(self.inference_manager)
+ self.library_manager = library_manager or LibraryManager()
+ self.pipelex_hub.set_library_manager(library_manager=self.library_manager)
+
# reporting
if get_config().pipelex.feature_config.is_reporting_enabled:
self.reporting_delegate = reporting_delegate or ReportingManager()
@@ -277,9 +269,9 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.telemetry_manager = TelemetryManagerNoOp()
log.debug("Telemetry is disabled because telemetry_mode is set to 'off'")
case TelemetryMode.ANONYMOUS | TelemetryMode.IDENTIFIED:
- if telemetry_config.respect_dnt and (dnt := get_optional_env(DO_NOT_TRACK_ENV_VAR_KEY)) and dnt.lower() not in ["false", "0"]:
+ if telemetry_config.respect_dnt and is_env_var_truthy(DO_NOT_TRACK_ENV_VAR_KEY):
self.telemetry_manager = TelemetryManagerNoOp()
- log.debug(f"Telemetry is disabled by env var 'DO_NOT_TRACK' which is set to {dnt}")
+ log.debug(f"Telemetry is disabled by env var '{DO_NOT_TRACK_ENV_VAR_KEY}'")
else:
self.telemetry_manager = telemetry_manager or TelemetryManager(telemetry_config=telemetry_config)
else:
@@ -297,26 +289,11 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.pipelex_hub.set_observer(observer=multi_observer)
self.pipelex_hub.set_pipe_router(pipe_router or PipeRouter(observer=multi_observer))
- # pipeline
self.pipeline_tracker.setup()
self.pipeline_manager.setup()
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")
- def setup_libraries(self):
- self.library_manager.setup()
- self.library_manager.load_libraries()
- log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup libraries done")
-
- def validate_libraries(self):
- try:
- self.library_manager.validate_libraries()
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
- msg = f"Could not validate libraries because of: {validation_error_msg}"
- raise PipelexSetupError(msg) from validation_error
- log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} validate libraries done")
-
def teardown(self):
# pipelex
self.pipeline_manager.teardown()
@@ -324,7 +301,6 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.pipeline_tracker.teardown()
if self.telemetry_manager:
self.telemetry_manager.teardown()
- self.library_manager.teardown()
# cogt
self.inference_manager.teardown()
@@ -418,7 +394,6 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
observers=observers,
**kwargs,
)
- pipelex_instance.setup_libraries()
pipelex_instance.models_manager.validate_model_deck()
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} ready")
return pipelex_instance
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index 19b3fc8fa..b33d21d07 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -22,6 +22,7 @@ excluded_dirs = [
]
[pipelex.builder_config]
+fix_loop_max_attempts = 10
default_output_dir = "."
default_bundle_file_name = "bundle"
default_directory_base_name = "pipeline"
@@ -204,13 +205,13 @@ gemini = { tag_style = "xml" }
is_default_text_then_structure = false # turn this to true to get better results: generates text before structuring
####################################################################################################
-# Static validation config
+# Validation error config
####################################################################################################
-[pipelex.static_validation_config]
+[pipelex.validation_error_config]
default_reaction = "raise"
-[pipelex.static_validation_config.reactions]
+[pipelex.validation_error_config.reactions]
# default_reaction is on raise
# missing_input_variable = "log"
# extraneous_input_variable = "log"
diff --git a/pipelex/pipeline/exceptions.py b/pipelex/pipeline/exceptions.py
index 06804320c..4c300783b 100644
--- a/pipelex/pipeline/exceptions.py
+++ b/pipelex/pipeline/exceptions.py
@@ -1,14 +1,12 @@
-from pipelex.base_exceptions import PipelexException
-from pipelex.cogt.templating.template_category import TemplateCategory
-from pipelex.pipe_run.exceptions import PipeRunError
+from pipelex.base_exceptions import PipelexError
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-class PipeExecutionError(PipelexException):
+class PipeExecutionError(PipelexError):
pass
-class PipelineExecutionError(PipelexException):
+class PipelineExecutionError(PipelexError):
def __init__(
self,
message: str,
@@ -24,41 +22,7 @@ class PipelineExecutionError(PipelexException):
super().__init__(message)
-class DryRunError(PipeRunError):
- """Raised when a dry run fails due to missing inputs or other validation issues."""
-
- def __init__(self, message: str, pipe_type: str, pipe_code: str | None = None):
- self.pipe_type = pipe_type
- self.pipe_code = pipe_code
- super().__init__(message)
-
-
-class DryRunMissingInputsError(DryRunError):
- """Raised when a dry run fails due to missing inputs or other validation issues."""
-
- def __init__(self, message: str, pipe_type: str, pipe_code: str, missing_inputs: list[str] | None = None):
- self.missing_inputs = missing_inputs or []
- super().__init__(message, pipe_type, pipe_code)
-
-
-class DryRunMissingPipesError(DryRunError):
- """Raised when a dry run fails due to missing pipes or other validation issues."""
-
- def __init__(self, message: str, pipe_type: str, pipe_code: str, missing_pipes: list[str] | None = None):
- self.missing_pipes = missing_pipes or []
- super().__init__(message, pipe_type, pipe_code)
-
-
-class DryRunTemplatingError(DryRunError):
- """Raised when a dry run fails due to templating issues."""
-
- def __init__(self, message: str, pipe_type: str, pipe_code: str, template_category: TemplateCategory, template: str):
- self.template_category = template_category
- self.template = template
- super().__init__(message, pipe_type, pipe_code)
-
-
-class PipeStackOverflowError(PipelexException):
+class PipeStackOverflowError(PipelexError):
def __init__(self, message: str, limit: int, pipe_stack: list[str]):
self.limit = limit
self.pipe_stack = pipe_stack
diff --git a/pipelex/pipeline/execute.py b/pipelex/pipeline/execute.py
index fbe430e4f..9637f38e4 100644
--- a/pipelex/pipeline/execute.py
+++ b/pipelex/pipeline/execute.py
@@ -1,3 +1,4 @@
+from pathlib import Path
from typing import TYPE_CHECKING
from pipelex.client.protocol import PipelineInputs
@@ -11,6 +12,8 @@ from pipelex.hub import (
get_report_delegate,
get_required_pipe,
get_telemetry_manager,
+ set_current_library,
+ teardown_current_library,
)
from pipelex.pipe_run.exceptions import PipeRouterError
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
@@ -22,7 +25,7 @@ from pipelex.pipe_run.pipe_run_params import (
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError
from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.pipeline.validate_plx import validate_plx
+from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.system.environment import get_optional_env
from pipelex.system.telemetry.events import EventName, EventProperty, Outcome
@@ -32,6 +35,8 @@ if TYPE_CHECKING:
async def execute_pipeline(
+ library_id: str | None = None,
+ library_path: str | None = None,
pipe_code: str | None = None,
plx_content: str | None = None,
inputs: PipelineInputs | WorkingMemory | None = None,
@@ -49,6 +54,10 @@ async def execute_pipeline(
Parameters
----------
+ library_id:
+ The library ID to use for the pipeline execution. If not provided, the library_id will be set to the pipeline run ID.
+ library_path:
+ Path to the library directory to load.
pipe_code:
The code identifying the pipeline to execute.
plx_content:
@@ -79,12 +88,24 @@ async def execute_pipeline(
msg = "Either pipe_code or plx_content must be provided to the API execute_pipeline."
raise ValueError(msg)
+ pipeline = get_pipeline_manager().add_new_pipeline()
+ pipeline_run_id = pipeline.pipeline_run_id
+
+ if not library_id:
+ library_id = pipeline_run_id
+
+ library_manager = get_library_manager()
+ set_current_library(library_id=library_id)
+ library_manager.open_library(library_id=library_id)
+
pipe: PipeAbstract | None = None
blueprint: PipelexBundleBlueprint | None = None
if plx_content:
- blueprint, _ = await validate_plx(plx_content=plx_content, remove_after_validation=False)
-
+ validate_bundle_result = await validate_bundle(plx_content=plx_content)
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=validate_bundle_result.blueprints)
+ # For now, we only support one blueprint when given a plx_content. So blueprints is of length 1.
+ blueprint = validate_bundle_result.blueprints[0]
if pipe_code:
pipe = get_required_pipe(pipe_code=pipe_code)
elif blueprint.main_pipe:
@@ -93,6 +114,10 @@ async def execute_pipeline(
msg = "No pipe code or main pipe in the PLX content provided to the API execute_pipeline."
raise PipeExecutionError(message=msg)
elif pipe_code:
+ if library_path:
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(library_path)])
+ else:
+ library_manager.load_libraries(library_id=library_id)
pipe = get_required_pipe(pipe_code=pipe_code)
else:
msg = "Either provide pipe_code or plx_content to the API execute_pipeline. 'pipe_code' must be provided when 'plx_content' is None"
@@ -119,8 +144,7 @@ async def execute_pipeline(
else:
pipe_run_mode = PipeRunMode.LIVE
- pipeline = get_pipeline_manager().add_new_pipeline()
- get_report_delegate().open_registry(pipeline_run_id=pipeline.pipeline_run_id)
+ get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
job_metadata = JobMetadata(
pipeline_run_id=pipeline.pipeline_run_id,
@@ -157,8 +181,9 @@ async def execute_pipeline(
pipe_stack=pipe_job.pipe_run_params.pipe_stack,
) from exc
finally:
- if plx_content and blueprint is not None:
- get_library_manager().remove_from_blueprint(blueprint=blueprint)
+ library = get_library_manager().get_library(library_id=library_id)
+ library.teardown()
+ teardown_current_library()
properties = {
EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
EventProperty.PIPE_TYPE: pipe.pipe_type,
diff --git a/pipelex/pipeline/pipeline_manager.py b/pipelex/pipeline/pipeline_manager.py
index 8c323a23b..791e99925 100644
--- a/pipelex/pipeline/pipeline_manager.py
+++ b/pipelex/pipeline/pipeline_manager.py
@@ -1,7 +1,7 @@
from pydantic import Field, RootModel
from typing_extensions import override
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
from pipelex.pipeline.pipeline import Pipeline
from pipelex.pipeline.pipeline_factory import PipelineFactory
from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract
@@ -9,7 +9,7 @@ from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract
PipelineManagerRoot = dict[str, Pipeline]
-class PipelineManagerNotFoundError(PipelexException):
+class PipelineManagerNotFoundError(PipelexError):
pass
diff --git a/pipelex/pipeline/start.py b/pipelex/pipeline/start.py
index 93b0debd2..655b2616a 100644
--- a/pipelex/pipeline/start.py
+++ b/pipelex/pipeline/start.py
@@ -1,44 +1,69 @@
import asyncio
+from pathlib import Path
+from typing import TYPE_CHECKING
from pipelex.client.protocol import PipelineInputs
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import (
+ get_library_manager,
get_pipe_router,
get_pipeline_manager,
get_report_delegate,
get_required_pipe,
+ get_telemetry_manager,
+ set_current_library,
)
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-from pipelex.pipe_run.pipe_run_params import VariableMultiplicity
+from pipelex.pipe_run.pipe_run_params import (
+ FORCE_DRY_RUN_MODE_ENV_KEY,
+ VariableMultiplicity,
+)
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.exceptions import PipeExecutionError
from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.system.environment import get_optional_env
+from pipelex.system.telemetry.events import EventName, EventProperty
+
+if TYPE_CHECKING:
+ from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+ from pipelex.core.pipes.pipe_abstract import PipeAbstract
async def start_pipeline(
- pipe_code: str,
+ library_id: str | None = None,
+ library_path: str | None = None,
+ pipe_code: str | None = None,
+ plx_content: str | None = None,
inputs: PipelineInputs | WorkingMemory | None = None,
output_name: str | None = None,
output_multiplicity: VariableMultiplicity | None = None,
dynamic_output_concept_code: str | None = None,
- pipe_run_mode: PipeRunMode = PipeRunMode.LIVE,
+ pipe_run_mode: PipeRunMode | None = None,
search_domains: list[str] | None = None,
) -> tuple[str, asyncio.Task[PipeOutput]]:
"""Start a pipeline in the background.
This function mirrors *execute_pipeline* but returns immediately with the
- ``pipeline_run_id`` instead of waiting for the pipe run to complete. The
- actual execution is scheduled on the current event-loop using
+ ``pipeline_run_id`` and a task instead of waiting for the pipe run to complete.
+ The actual execution is scheduled on the current event-loop using
:pyfunc:`asyncio.create_task`.
Parameters
----------
+ library_id:
+ The library ID to use for the pipeline execution. If not provided, the library_id will be set to the pipeline run ID.
+ library_path:
+ Path to the library directory to load.
pipe_code:
The code identifying the pipeline to execute.
+ plx_content:
+ Content of the pipeline bundle to execute.
inputs:
- Optional pipeline inputs or working memory to pass to the pipe.
+ Inputs passed to the pipeline.
output_name:
Name of the output slot to write to.
output_multiplicity:
@@ -46,7 +71,10 @@ async def start_pipeline(
dynamic_output_concept_code:
Override the dynamic output concept code.
pipe_run_mode:
- Pipe run mode: ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``.
+ Pipe run mode: if specified, it must be ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``.
+ If not specified, the pipe run mode is inferred from the environment variable
+ ``PIPELEX_FORCE_DRY_RUN_MODE``. If the environment variable is not set,
+ the pipe run mode is ``PipeRunMode.LIVE``.
search_domains:
List of domains to search for pipes.
@@ -57,7 +85,44 @@ async def start_pipeline(
can be awaited to get the pipe output.
"""
- pipe = get_required_pipe(pipe_code=pipe_code)
+ if not plx_content and not pipe_code:
+ msg = "Either pipe_code or plx_content must be provided to the API start_pipeline."
+ raise ValueError(msg)
+
+ pipeline = get_pipeline_manager().add_new_pipeline()
+ pipeline_run_id = pipeline.pipeline_run_id
+
+ if not library_id:
+ library_id = pipeline_run_id
+
+ library_manager = get_library_manager()
+ set_current_library(library_id=library_id)
+ library_manager.open_library(library_id=library_id)
+
+ pipe: PipeAbstract | None = None
+ blueprint: PipelexBundleBlueprint | None = None
+
+ if plx_content:
+ validate_bundle_result = await validate_bundle(plx_content=plx_content)
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=validate_bundle_result.blueprints)
+ # For now, we only support one blueprint when given a plx_content. So blueprints is of length 1.
+ blueprint = validate_bundle_result.blueprints[0]
+ if pipe_code:
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ elif blueprint.main_pipe:
+ pipe = get_required_pipe(pipe_code=blueprint.main_pipe)
+ else:
+ msg = "No pipe code or main pipe in the PLX content provided to the API start_pipeline."
+ raise PipeExecutionError(message=msg)
+ elif pipe_code:
+ if library_path:
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(library_path)])
+ else:
+ library_manager.load_libraries(library_id=library_id)
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ else:
+ msg = "Either provide pipe_code or plx_content to the API start_pipeline. 'pipe_code' must be provided when 'plx_content' is None"
+ raise PipeExecutionError(message=msg)
search_domains = search_domains or []
if pipe.domain not in search_domains:
@@ -74,8 +139,13 @@ async def start_pipeline(
search_domains=search_domains,
)
- pipeline = get_pipeline_manager().add_new_pipeline()
- get_report_delegate().open_registry(pipeline_run_id=pipeline.pipeline_run_id)
+ if pipe_run_mode is None:
+ if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):
+ pipe_run_mode = PipeRunMode(run_mode_from_env)
+ else:
+ pipe_run_mode = PipeRunMode.LIVE
+
+ get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
job_metadata = JobMetadata(
pipeline_run_id=pipeline.pipeline_run_id,
@@ -87,9 +157,6 @@ async def start_pipeline(
pipe_run_mode=pipe_run_mode,
)
- if working_memory:
- working_memory.pretty_print_summary()
-
pipe_job = PipeJobFactory.make_pipe_job(
pipe=pipe,
pipe_run_params=pipe_run_params,
@@ -98,6 +165,12 @@ async def start_pipeline(
output_name=output_name,
)
+ properties = {
+ EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
+ EventProperty.PIPE_TYPE: pipe.pipe_type,
+ }
+ get_telemetry_manager().track_event(event_name=EventName.PIPELINE_EXECUTE, properties=properties)
+
# Launch execution without awaiting the result.
task: asyncio.Task[PipeOutput] = asyncio.create_task(get_pipe_router().run(pipe_job))
diff --git a/pipelex/pipeline/track/exceptions.py b/pipelex/pipeline/track/exceptions.py
index 8e042d45a..73732cba3 100644
--- a/pipelex/pipeline/track/exceptions.py
+++ b/pipelex/pipeline/track/exceptions.py
@@ -1,5 +1,5 @@
-from pipelex.base_exceptions import PipelexException
+from pipelex.base_exceptions import PipelexError
-class JobHistoryError(PipelexException):
+class JobHistoryError(PipelexError):
pass
diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.py
new file mode 100644
index 000000000..aa0ddbd1f
--- /dev/null
+++ b/pipelex/pipeline/validate_bundle.py
@@ -0,0 +1,131 @@
+from pydantic import BaseModel, ValidationError
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
+from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+from pipelex.core.exceptions import PipesAndConceptValidationErrorData
+from pipelex.core.interpreter.exceptions import PipelexInterpreterError
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.pipes.exceptions import PipeValidationError
+from pipelex.core.pipes.handle_pipe_errors import (
+ categorize_pipe_validation_error,
+ categorize_pipe_validation_with_libraries_error,
+)
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.core.validation import report_validation_error
+from pipelex.hub import get_library_manager, set_current_library
+from pipelex.pipe_run.dry_run import DryRunError, DryRunOutput, dry_run_pipes
+
+
+class ValidateBundleError(PipelexError):
+ """Raised when bundle validation fails.
+
+ This error aggregates validation errors from different stages:
+ - Blueprint validation errors (from interpreter)
+ - Pipe validation errors (from PipeValidationError exceptions)
+ - Pipe/Concept instantiation errors (from Pydantic ValidationError during factory instantiation)
+ - Dry run errors
+
+ All errors are categorized and stored in their respective lists.
+ """
+
+ def __init__(
+ self,
+ message: str,
+ pipelex_bundle_blueprint_validation_errors: list[PipelexBundleBlueprintValidationErrorData] | None = None,
+ pipe_validation_errors: list[PipesAndConceptValidationErrorData] | None = None,
+ pipe_concept_instantiation_errors: list[PipesAndConceptValidationErrorData] | None = None,
+ dry_run_error_message: str | None = None,
+ ):
+ # Blueprint validation errors (e.g., PIPE_SEQUENCE_OUTPUT_MISMATCH)
+ self.pipelex_bundle_blueprint_validation_errors = pipelex_bundle_blueprint_validation_errors or []
+
+ # Pipe validation errors from PipeValidationError exceptions
+ # (e.g., MISSING_INPUT_VARIABLE, EXTRANEOUS_INPUT_VARIABLE, INPUT_REQUIREMENT_MISMATCH, INADEQUATE_OUTPUT_CONCEPT)
+ self.pipe_validation_errors = pipe_validation_errors or []
+
+ # Pipe/Concept instantiation errors from Pydantic ValidationError
+ # These occur during factory instantiation of Pipe or Concept classes
+ # TODO: Currently not caught, but structure is prepared for future implementation
+ self.pipe_concept_instantiation_errors = pipe_concept_instantiation_errors or []
+
+ # Dry run errors
+ self.dry_run_error_message = dry_run_error_message
+
+ super().__init__(message)
+
+ @property
+ def pipe_validation_error_data(self) -> list[PipesAndConceptValidationErrorData]:
+ """Backwards compatibility: combine pipe validation and instantiation errors.
+
+ This property provides the old interface for accessing all pipe/concept validation errors.
+ """
+ # TODO: refactor so we don't need this anymore?
+ return self.pipe_validation_errors + self.pipe_concept_instantiation_errors
+
+
+class ValidateBundleResult(BaseModel):
+ blueprints: list[PipelexBundleBlueprint]
+ pipes: list[PipeAbstract]
+ dry_run_result: dict[str, DryRunOutput]
+
+
+async def validate_bundle(
+ plx_content: str | None = None, blueprints: list[PipelexBundleBlueprint] | None = None, plx_file_path: str | None = None
+) -> ValidateBundleResult:
+ provided_params = sum([blueprints is not None, plx_content is not None, plx_file_path is not None])
+ if provided_params == 0:
+ msg = "At least one of blueprints, plx_content, or plx_file_path must be provided to validate_bundle"
+ raise ValidateBundleError(message=msg)
+ if provided_params > 1:
+ msg = "Only one of blueprints, plx_content, or plx_file_path can be provided to validate_bundle, not multiple"
+ raise ValidateBundleError(message=msg)
+
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+
+ loaded_pipes: list[PipeAbstract] | None = None
+ loaded_blueprints: list[PipelexBundleBlueprint] | None = None
+ try:
+ if blueprints is not None:
+ loaded_blueprints = blueprints
+ loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints)
+
+ elif plx_content is not None:
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(plx_content=plx_content)
+ loaded_blueprints = [blueprint]
+ loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=[blueprint])
+
+ else: # plx_file_path is not None
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=plx_file_path)
+ loaded_blueprints = [blueprint]
+ loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=[blueprint])
+
+ dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
+ except PipelexInterpreterError as interpreter_error:
+ raise ValidateBundleError(
+ message=interpreter_error.message,
+ pipelex_bundle_blueprint_validation_errors=interpreter_error.validation_errors,
+ ) from interpreter_error
+ except PipeValidationError as pipe_error:
+ pipe_error_data = categorize_pipe_validation_with_libraries_error(pipe_error=pipe_error)
+ raise ValidateBundleError(
+ message=f"Pipe validation failed: {pipe_error}",
+ pipe_validation_errors=[pipe_error_data],
+ ) from pipe_error
+ except ValidationError as validation_error:
+ pipe_validation_errors = categorize_pipe_validation_error(validation_error=validation_error)
+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)
+ msg = f"Could not load blueprints because of: {validation_error_msg}"
+ raise ValidateBundleError(
+ message=msg,
+ pipe_validation_errors=pipe_validation_errors,
+ ) from validation_error
+ except DryRunError as dry_run_error:
+ raise ValidateBundleError(
+ message=dry_run_error.message,
+ dry_run_error_message=dry_run_error.message,
+ ) from dry_run_error
+
+ return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result=dry_run_results)
diff --git a/pipelex/pipeline/validate_plx.py b/pipelex/pipeline/validate_plx.py
deleted file mode 100644
index d90f6da43..000000000
--- a/pipelex/pipeline/validate_plx.py
+++ /dev/null
@@ -1,70 +0,0 @@
-from pydantic import ValidationError
-
-from pipelex import log
-from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.exceptions import PipelexConfigurationError
-from pipelex.core.interpreter import PipelexInterpreter, PLXDecodeError
-from pipelex.core.pipes.pipe_abstract import PipeAbstract
-from pipelex.hub import get_library_manager
-from pipelex.libraries.exceptions import ConceptLoadingError, DomainLoadingError, PipeLibraryError, PipeLoadingError
-from pipelex.pipe_run.dry_run import DryRunError, dry_run_pipes
-
-
-async def validate_plx(plx_content: str, remove_after_validation: bool = True) -> tuple[PipelexBundleBlueprint, list[PipeAbstract]]:
- """Validate PLX content.
-
- This function:
- 1. Parses PLX content into a bundle blueprint
- 2. Loads pipes from the blueprint
- 3. Runs static validation and dry runs all pipes
-
- Args:
- plx_content: The PLX content to validate
- remove_after_validation: Whether to remove the blueprint from the library manager after validation
-
- Returns:
- Tuple of (blueprint, pipes): The blueprint and the loaded pipes
-
- Raises:
- PipelexConfigurationError: For interpreter configuration errors
- PLXDecodeError: For PLX parsing/decoding errors
- ValidationError: For Pydantic validation errors in blueprint
- DomainLoadingError: For domain loading errors
- ConceptLoadingError: For concept loading errors
- PipeLoadingError: For pipe loading errors
- PipeLibraryError: For pipe library validation errors
- DryRunError: For dry run validation errors
- """
- library_manager = get_library_manager()
- blueprint: PipelexBundleBlueprint | None = None
-
- try:
- converter = PipelexInterpreter(file_content=plx_content)
- blueprint = converter.make_pipelex_bundle_blueprint()
-
- pipes = library_manager.load_from_blueprint(blueprint=blueprint)
-
- for pipe in pipes:
- pipe.validate_with_libraries()
- await dry_run_pipes(pipes=[pipe], raise_on_failure=True)
-
- if remove_after_validation:
- library_manager.remove_from_blueprint(blueprint=blueprint)
- return blueprint, pipes
- except (
- PipelexConfigurationError,
- PLXDecodeError,
- ValidationError,
- DomainLoadingError,
- ConceptLoadingError,
- PipeLoadingError,
- PipeLibraryError,
- DryRunError,
- ):
- if blueprint is not None:
- try:
- library_manager.remove_from_blueprint(blueprint=blueprint)
- except Exception as cleanup_error:
- log.error(f"Error during cleanup after validation failure: {cleanup_error}")
-
- raise
diff --git a/pipelex/plugins/anthropic/anthropic_factory.py b/pipelex/plugins/anthropic/anthropic_factory.py
index 3ea373d41..45cab7124 100644
--- a/pipelex/plugins/anthropic/anthropic_factory.py
+++ b/pipelex/plugins/anthropic/anthropic_factory.py
@@ -9,7 +9,6 @@ from openai.types.chat import (
ChatCompletionSystemMessageParam,
)
-from pipelex import log
from pipelex.cogt.exceptions import CogtError
from pipelex.cogt.image.prompt_image import (
PromptImage,
@@ -24,7 +23,7 @@ from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.config import get_config
-from pipelex.plugins.plugin import Plugin
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.tools.misc.base_64_utils import load_binary_as_base64_async
from pipelex.tools.misc.filetype_utils import detect_file_type_from_base64
from pipelex.types import StrEnum
@@ -130,7 +129,6 @@ class AnthropicFactory:
text_block_param: TextBlockParam = {"type": "text", "text": user_content_txt}
message: MessageParam
if prepped_user_images is not None:
- log.verbose(prepped_user_images)
images_block_params: list[ImageBlockParam] = []
for prepped_image in prepped_user_images:
image_block_param_in_loop: ImageBlockParam
diff --git a/pipelex/plugins/anthropic/anthropic_llm_worker.py b/pipelex/plugins/anthropic/anthropic_llm_worker.py
index 855e35e0e..3414b6b6d 100644
--- a/pipelex/plugins/anthropic/anthropic_llm_worker.py
+++ b/pipelex/plugins/anthropic/anthropic_llm_worker.py
@@ -5,6 +5,7 @@ from anthropic import AsyncAnthropic, AsyncAnthropicBedrock, omit
from typing_extensions import override
from pipelex import log
+from pipelex.base_exceptions import PipelexError
from pipelex.cogt.exceptions import LLMCompletionError, SdkTypeError
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_utils import (
@@ -31,7 +32,7 @@ class AnthropicExtraField(StrEnum):
CLAUDE_4_TOKENS_LIMIT = "claude_4_tokens_limit"
-class AnthropicLLMWorkerError(Exception):
+class AnthropicLLMWorkerError(PipelexError):
"""Base exception for Anthropic LLM Worker errors."""
diff --git a/pipelex/plugins/azure_rest/azure_img_gen_worker.py b/pipelex/plugins/azure_rest/azure_img_gen_worker.py
index 9bf94e449..c032c12a9 100644
--- a/pipelex/plugins/azure_rest/azure_img_gen_worker.py
+++ b/pipelex/plugins/azure_rest/azure_img_gen_worker.py
@@ -30,7 +30,7 @@ class AzureImgGenWorker(ImgGenWorkerAbstract):
):
super().__init__(inference_model=inference_model, reporting_delegate=reporting_delegate)
- if plugin.sdk != "azure_openai":
+ if plugin.sdk != "azure_rest":
msg = f"Plugin '{plugin}' is not supported for image generation"
raise NotImplementedError(msg)
self.plugin = plugin
diff --git a/pipelex/plugins/google/google_llm_worker.py b/pipelex/plugins/google/google_llm_worker.py
index 363d6d79d..3198e1cd9 100644
--- a/pipelex/plugins/google/google_llm_worker.py
+++ b/pipelex/plugins/google/google_llm_worker.py
@@ -7,6 +7,7 @@ from google.genai import types
from typing_extensions import override
from pipelex import log
+from pipelex.base_exceptions import PipelexError
from pipelex.cogt.exceptions import LLMCompletionError
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen
@@ -22,7 +23,7 @@ if TYPE_CHECKING:
from openai.types.chat import ChatCompletionMessageParam
-class GoogleLLMWorkerError(Exception):
+class GoogleLLMWorkerError(PipelexError):
"""Base exception for Google LLM Worker errors."""
diff --git a/pipelex/system/configuration/config_check.py b/pipelex/system/configuration/config_check.py
index e6657c215..52ea59b80 100644
--- a/pipelex/system/configuration/config_check.py
+++ b/pipelex/system/configuration/config_check.py
@@ -1,8 +1,8 @@
from rich.panel import Panel
from pipelex.cli.commands.init.config_files import init_config
-from pipelex.config import ConfigPaths
from pipelex.hub import get_console
+from pipelex.system.configuration.configs import ConfigPaths
from pipelex.tools.misc.file_utils import path_exists
from pipelex.urls import URLs
diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.py
index 0b524df54..2d55dc22f 100644
--- a/pipelex/system/configuration/config_loader.py
+++ b/pipelex/system/configuration/config_loader.py
@@ -1,6 +1,7 @@
import os
from typing import Any
+from pipelex.base_exceptions import PipelexError
from pipelex.system.configuration.config_root import (
CONFIG_BASE_OVERRIDES_AFTER_ENV,
CONFIG_BASE_OVERRIDES_BEFORE_ENV,
@@ -13,7 +14,7 @@ CONFIG_DIR_NAME = ".pipelex"
CONFIG_NAME = "pipelex.toml"
-class ConfigError(Exception):
+class ConfigError(PipelexError):
pass
@@ -96,20 +97,19 @@ class ConfigLoader:
deep_update(pipelex_config, local_config)
#################### 3. Load overrides for the current project ####################
- list_of_overrides = [
+ list_of_overrides: list[str] = [
*CONFIG_BASE_OVERRIDES_BEFORE_ENV,
runtime_manager.environment,
runtime_manager.run_mode,
*CONFIG_BASE_OVERRIDES_AFTER_ENV,
]
for override in list_of_overrides:
- if override:
- if override == runtime_manager.run_mode.UNIT_TEST:
- override_path = os.path.join(os.getcwd(), "tests", f"pipelex_{override}.toml")
- else:
- override_path = os.path.join(os.getcwd(), "pipelex" if self.is_in_pipelex_config else "", f"pipelex_{override}.toml")
- if override_dict := load_toml_from_path_if_exists(override_path):
- deep_update(pipelex_config, override_dict)
+ if override == runtime_manager.run_mode and runtime_manager.is_unit_testing:
+ override_path = os.path.join(os.getcwd(), "tests", f"pipelex_{override}.toml")
+ else:
+ override_path = os.path.join(os.getcwd(), "pipelex" if self.is_in_pipelex_config else "", f"pipelex_{override}.toml")
+ if override_dict := load_toml_from_path_if_exists(override_path):
+ deep_update(pipelex_config, override_dict)
return pipelex_config
diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.py
new file mode 100644
index 000000000..b38fec174
--- /dev/null
+++ b/pipelex/system/configuration/configs.py
@@ -0,0 +1,171 @@
+from typing import cast
+
+import shortuuid
+from pydantic import Field, field_validator
+
+from pipelex.base_exceptions import PipelexConfigError
+from pipelex.cogt.config_cogt import Cogt
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
+from pipelex.cogt.templating.templating_style import TemplatingStyle
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
+from pipelex.language.plx_config import PlxConfig
+from pipelex.pipeline.track.tracker_config import TrackerConfig
+from pipelex.system.configuration.config_model import ConfigModel
+from pipelex.system.configuration.config_root import ConfigRoot
+from pipelex.tools.aws.aws_config import AwsConfig
+from pipelex.tools.log.log_config import LogConfig
+from pipelex.types import StrEnum
+
+
+class ConfigPaths:
+ DEFAULT_CONFIG_DIR_PATH = "./.pipelex"
+ INFERENCE_DIR_NAME = "inference"
+ INFERENCE_DIR_PATH = f"{DEFAULT_CONFIG_DIR_PATH}/{INFERENCE_DIR_NAME}"
+ BACKENDS_FILE_NAME = "backends.toml"
+ BACKENDS_FILE_PATH = f"{INFERENCE_DIR_PATH}/{BACKENDS_FILE_NAME}"
+ BACKENDS_DIR_NAME = "backends"
+ BACKENDS_DIR_PATH = f"{INFERENCE_DIR_PATH}/{BACKENDS_DIR_NAME}"
+ ROUTING_PROFILES_FILE_NAME = "routing_profiles.toml"
+ ROUTING_PROFILES_FILE_PATH = f"{INFERENCE_DIR_PATH}/{ROUTING_PROFILES_FILE_NAME}"
+ MODEL_DECKS_DIR_NAME = "deck"
+ MODEL_DECKS_DIR_PATH = f"{INFERENCE_DIR_PATH}/{MODEL_DECKS_DIR_NAME}"
+ BASE_DECK_FILE_NAME = "base_deck.toml"
+ BASE_DECK_FILE_PATH = f"{MODEL_DECKS_DIR_PATH}/{BASE_DECK_FILE_NAME}"
+ OVERRIDES_DECK_FILE_NAME = "overrides.toml"
+ OVERRIDES_DECK_FILE_PATH = f"{MODEL_DECKS_DIR_PATH}/{OVERRIDES_DECK_FILE_NAME}"
+
+
+class ValidationErrorReaction(StrEnum):
+ RAISE = "raise"
+ LOG = "log"
+ IGNORE = "ignore"
+
+
+class ValidationErrorConfig(ConfigModel):
+ default_reaction: ValidationErrorReaction = Field(strict=False)
+ reactions: dict[PipeValidationErrorType, ValidationErrorReaction]
+
+ @field_validator("reactions", mode="before")
+ @classmethod
+ def validate_reactions(cls, value: dict[str, str]) -> dict[PipeValidationErrorType, ValidationErrorReaction]:
+ return cast(
+ "dict[PipeValidationErrorType, ValidationErrorReaction]",
+ ConfigModel.transform_dict_str_to_enum(
+ input_dict=value,
+ key_enum_cls=PipeValidationErrorType,
+ value_enum_cls=ValidationErrorReaction,
+ ),
+ )
+
+
+class PipeRunConfig(ConfigModel):
+ pipe_stack_limit: int
+
+
+class DryRunConfig(ConfigModel):
+ apply_to_jinja2_rendering: bool
+ text_gen_truncate_length: int
+ nb_list_items: int
+ nb_extract_pages: int
+ image_urls: list[str]
+ allowed_to_fail_pipes: list[str] = Field(default_factory=list)
+
+ @field_validator("image_urls", mode="before")
+ @classmethod
+ def validate_image_urls(cls, value: list[str]) -> list[str]:
+ if not value:
+ msg = "dry_run_config.image_urls must be a non-empty list"
+ raise PipelexConfigError(msg)
+ return value
+
+
+class StructureConfig(ConfigModel):
+ is_default_text_then_structure: bool
+
+
+class PromptingConfig(ConfigModel):
+ default_prompting_style: TemplatingStyle
+ prompting_styles: dict[str, TemplatingStyle]
+
+ def get_prompting_style(self, prompting_target: PromptingTarget | None = None) -> TemplatingStyle | None:
+ if prompting_target:
+ return self.prompting_styles.get(prompting_target, self.default_prompting_style)
+ return None
+
+
+class FeatureConfig(ConfigModel):
+ is_pipeline_tracking_enabled: bool
+ is_reporting_enabled: bool
+
+
+class ReportingConfig(ConfigModel):
+ is_log_costs_to_console: bool
+ is_generate_cost_report_file_enabled: bool
+ cost_report_dir_path: str
+ cost_report_base_name: str
+ cost_report_extension: str
+ cost_report_unit_scale: float
+
+
+class ObserverConfig(ConfigModel):
+ observer_dir: str
+
+
+class ScanConfig(ConfigModel):
+ excluded_dirs: frozenset[str]
+
+ @field_validator("excluded_dirs", mode="before")
+ @classmethod
+ def validate_excluded_dirs(cls, value: list[str] | frozenset[str]) -> frozenset[str]:
+ if isinstance(value, frozenset):
+ return value
+ return frozenset(value)
+
+
+class BuilderConfig(ConfigModel):
+ fix_loop_max_attempts: int
+ default_output_dir: str
+ default_bundle_file_name: str
+ default_directory_base_name: str
+
+
+class Pipelex(ConfigModel):
+ feature_config: FeatureConfig
+ log_config: LogConfig
+ aws_config: AwsConfig
+
+ validation_error_config: ValidationErrorConfig
+ tracker_config: TrackerConfig
+ structure_config: StructureConfig
+ prompting_config: PromptingConfig
+ plx_config: PlxConfig
+
+ dry_run_config: DryRunConfig
+ pipe_run_config: PipeRunConfig
+ reporting_config: ReportingConfig
+ observer_config: ObserverConfig
+ scan_config: ScanConfig
+ builder_config: BuilderConfig
+
+
+class MigrationConfig(ConfigModel):
+ migration_maps: dict[str, dict[str, str]]
+
+ def text_in_renaming_keys(self, category: str, text: str) -> list[tuple[str, str]]:
+ renaming_map = self.migration_maps.get(category)
+ if not renaming_map:
+ return []
+ return [(key, value) for key, value in renaming_map.items() if text in key]
+
+ def text_in_renaming_values(self, category: str, text: str) -> list[tuple[str, str]]:
+ renaming_map = self.migration_maps.get(category)
+ if not renaming_map:
+ return []
+ return [(key, value) for key, value in renaming_map.items() if text in value]
+
+
+class PipelexConfig(ConfigRoot):
+ session_id: str = shortuuid.uuid()
+ cogt: Cogt
+ pipelex: Pipelex
+ migration: MigrationConfig
diff --git a/pipelex/system/environment.py b/pipelex/system/environment.py
index 926daa4fe..aebc2abef 100644
--- a/pipelex/system/environment.py
+++ b/pipelex/system/environment.py
@@ -2,13 +2,13 @@ import os
from dotenv import load_dotenv
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.tools.misc.placeholder import value_is_placeholder
load_dotenv(dotenv_path=".env", override=True)
-class EnvVarNotFoundError(ToolException):
+class EnvVarNotFoundError(ToolError):
pass
@@ -42,3 +42,9 @@ def any_env_var_is_placeholder(keys: list[str]) -> bool:
def set_env(key: str, value: str) -> None:
os.environ[key] = value
+
+
+def is_env_var_truthy(key: str) -> bool:
+ """Return True if the env var is set and not a falsy sentinel ("false" or "0")."""
+ value = get_optional_env(key)
+ return (value is not None) and (value.lower() not in ["false", "0"])
diff --git a/pipelex/system/exceptions.py b/pipelex/system/exceptions.py
index c91550b6f..6d2f9938d 100644
--- a/pipelex/system/exceptions.py
+++ b/pipelex/system/exceptions.py
@@ -1,19 +1,19 @@
import logging
from typing import ClassVar
-from pipelex.base_exceptions import RootException
+from pipelex.base_exceptions import PipelexError
from pipelex.types import StrEnum
-class ToolException(RootException):
+class ToolError(PipelexError):
pass
-class NestedKeyConflictError(ToolException):
+class NestedKeyConflictError(ToolError):
"""Raised when attempting to create nested keys under a non-dict value."""
-class CredentialsError(RootException):
+class CredentialsError(PipelexError):
pass
@@ -22,7 +22,7 @@ class TracebackMessageErrorMode(StrEnum):
EXCEPTION = "exception"
-class TracebackMessageError(RootException):
+class TracebackMessageError(PipelexError):
error_mode: ClassVar[TracebackMessageErrorMode] = TracebackMessageErrorMode.EXCEPTION
def __init__(self, message: str):
diff --git a/pipelex/system/registries/func_registry.py b/pipelex/system/registries/func_registry.py
index ccbf0e14b..c361675dd 100644
--- a/pipelex/system/registries/func_registry.py
+++ b/pipelex/system/registries/func_registry.py
@@ -5,7 +5,7 @@ from typing import Any, TypeVar, cast, get_type_hints
from pydantic import Field, PrivateAttr, RootModel
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.urls import URLs
FUNC_REGISTRY_LOGGER_CHANNEL_NAME = "func_registry"
@@ -18,7 +18,7 @@ FuncRegistryDict = dict[str, Callable[..., Any]]
PIPE_FUNC_MARKER = "_is_pipe_func"
-class FuncRegistryError(ToolException):
+class FuncRegistryError(ToolError):
pass
diff --git a/pipelex/system/registries/registry_base.py b/pipelex/system/registries/registry_base.py
index 76d087a0a..b46873fb1 100644
--- a/pipelex/system/registries/registry_base.py
+++ b/pipelex/system/registries/registry_base.py
@@ -1,6 +1,3 @@
-from collections.abc import Callable
-from typing import Any
-
from pydantic import BaseModel
ModelType = type[BaseModel]
@@ -15,14 +12,3 @@ class RegistryModels:
all_models.update(model_list)
return list(all_models)
-
-
-class RegistryFuncs:
- @classmethod
- def get_all_functions(cls) -> list[Callable[..., Any]]:
- functions: list[Callable] = [] # pyright: ignore[reportMissingTypeArgument, reportUnknownVariableType]
- for attr in dir(cls):
- attr_value = getattr(cls, attr)
- if callable(attr_value):
- functions.append(attr_value) # pyright: ignore[reportUnknownMemberType]
- return functions # pyright: ignore[reportUnknownVariableType]
diff --git a/pipelex/system/runtime.py b/pipelex/system/runtime.py
index b2b4a1465..b6ca033a6 100644
--- a/pipelex/system/runtime.py
+++ b/pipelex/system/runtime.py
@@ -1,9 +1,10 @@
from pydantic import BaseModel
-from pipelex.system.environment import get_optional_env
+from pipelex.system.environment import get_optional_env, is_env_var_truthy
from pipelex.types import StrEnum
RUN_MODE_ENV_VAR_KEY = "RUN_MODE"
+CODEX_CLOUD_ENV_VAR_KEY = "CODEX_CLOUD"
class IntegrationMode(StrEnum):
@@ -37,12 +38,17 @@ class RunMode(StrEnum):
NORMAL = "normal"
UNIT_TEST = "unit_test"
CI_TEST = "ci_test"
+ CODEX_CLOUD = "codex_cloud"
+ CODEX_CLOUD_TEST = "codex_cloud_test"
@classmethod
def get_from_env_var(cls) -> "RunMode":
if mode_str := get_optional_env(RUN_MODE_ENV_VAR_KEY):
return RunMode(mode_str)
- return RunMode.NORMAL
+ elif is_env_var_truthy(key=CODEX_CLOUD_ENV_VAR_KEY):
+ return RunMode.CODEX_CLOUD
+ else:
+ return RunMode.NORMAL
class WorkerMode(StrEnum):
@@ -109,24 +115,28 @@ class RuntimeManager(BaseModel):
match self.run_mode:
case RunMode.NORMAL:
return False
+ case RunMode.CODEX_CLOUD:
+ return False
case RunMode.UNIT_TEST:
return True
case RunMode.CI_TEST:
return True
+ case RunMode.CODEX_CLOUD_TEST:
+ return True
@property
def is_ci_testing(self) -> bool:
match self.run_mode:
case RunMode.NORMAL:
return False
+ case RunMode.CODEX_CLOUD:
+ return False
case RunMode.UNIT_TEST:
return False
case RunMode.CI_TEST:
return True
-
- @property
- def should_check_intermediate_configs(self) -> bool:
- return self.is_unit_testing
+ case RunMode.CODEX_CLOUD_TEST:
+ return True
runtime_manager = RuntimeManager()
diff --git a/pipelex/system/telemetry/telemetry_manager.py b/pipelex/system/telemetry/telemetry_manager.py
index 63ac2ad35..9067670f9 100644
--- a/pipelex/system/telemetry/telemetry_manager.py
+++ b/pipelex/system/telemetry/telemetry_manager.py
@@ -6,7 +6,7 @@ from posthog import Posthog, new_context, tag # type: ignore[attr-defined]
from posthog.args import ExceptionArg, OptionalCaptureArgs
from typing_extensions import Unpack, override
-from pipelex.system.exceptions import RootException
+from pipelex.system.exceptions import PipelexError
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventName, EventProperty, Setting
from pipelex.system.telemetry.telemetry_config import TelemetryConfig, TelemetryMode
@@ -58,8 +58,8 @@ class TelemetryManager(TelemetryManagerAbstract):
exception: ExceptionArg | None = None,
**kwargs: Unpack[OptionalCaptureArgs],
) -> Any:
- """Capture exception with message sanitization for RootException subclasses."""
- if exception and isinstance(exception, RootException):
+ """Capture exception with message sanitization for PipelexError subclasses."""
+ if exception and isinstance(exception, PipelexError):
# Create a new exception with sanitized message while preserving the class type
# Use __new__ to create an instance without calling __init__, which may require extra args
# This creates a "shell" instance with NO custom attributes (e.g., no tested_concept, wanted_concept, etc.)
@@ -79,7 +79,7 @@ class TelemetryManager(TelemetryManagerAbstract):
return self._original_capture_exception(sanitized_exception, **kwargs)
else:
- # For non-RootException, capture as-is (or auto-detect current exception)
+ # For non-PipelexError, capture as-is (or auto-detect current exception)
return self._original_capture_exception(exception, **kwargs)
# Replace the method
diff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.py
index 8adef1d3d..92d328ee2 100644
--- a/pipelex/test_extras/shared_pytest_plugins.py
+++ b/pipelex/test_extras/shared_pytest_plugins.py
@@ -5,8 +5,8 @@ from pytest import FixtureRequest, Parser
from pipelex.hub import get_console
from pipelex.pipe_run.pipe_run_params import PipeRunMode
-from pipelex.system.environment import is_env_var_set, set_env
-from pipelex.system.runtime import RunMode, runtime_manager
+from pipelex.system.environment import is_env_var_set, is_env_var_truthy, set_env
+from pipelex.system.runtime import CODEX_CLOUD_ENV_VAR_KEY, RunMode, runtime_manager
from pipelex.tools.misc.placeholder import make_placeholder_value, value_is_placeholder
# List of environment variables that may need placeholders in CI
@@ -41,6 +41,9 @@ ENV_VAR_KEYS_WHICH_MAY_NEED_PLACEHOLDERS_IN_CI = [
def set_run_mode():
if is_env_var_set(key="GITHUB_ACTIONS") or is_env_var_set(key="CI"):
runtime_manager.set_run_mode(run_mode=RunMode.CI_TEST)
+ elif is_env_var_truthy(key=CODEX_CLOUD_ENV_VAR_KEY):
+ # we're in codex cloud and this fixture is called by pytest, so we are testing in codex cloud
+ runtime_manager.set_run_mode(run_mode=RunMode.CODEX_CLOUD_TEST)
else:
runtime_manager.set_run_mode(run_mode=RunMode.UNIT_TEST)
diff --git a/pipelex/tools/jinja2/jinja2_errors.py b/pipelex/tools/jinja2/jinja2_errors.py
index c24ad62c9..794788aa4 100644
--- a/pipelex/tools/jinja2/jinja2_errors.py
+++ b/pipelex/tools/jinja2/jinja2_errors.py
@@ -1,21 +1,21 @@
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
-class Jinja2TemplateSyntaxError(ToolException):
+class Jinja2TemplateSyntaxError(ToolError):
pass
-class Jinja2TemplateRenderError(ToolException):
+class Jinja2TemplateRenderError(ToolError):
pass
-class Jinja2StuffError(ToolException):
+class Jinja2StuffError(ToolError):
pass
-class Jinja2ContextError(ToolException):
+class Jinja2ContextError(ToolError):
pass
-class Jinja2DetectVariablesError(ToolException):
+class Jinja2DetectVariablesError(ToolError):
pass
diff --git a/pipelex/tools/log/log_config.py b/pipelex/tools/log/log_config.py
index 1595fb93a..572c020cd 100644
--- a/pipelex/tools/log/log_config.py
+++ b/pipelex/tools/log/log_config.py
@@ -1,5 +1,4 @@
import sys
-from enum import Enum
from typing import cast
from pydantic import Field, field_validator
@@ -27,7 +26,7 @@ class ProblemIds(StrEnum):
AZURE_OPENAI_NO_STREAM_OPTIONS = "Azure OpenAI no stream_options"
-class CallerInfoTemplate(Enum):
+class CallerInfoTemplate(StrEnum):
FILE_LINE = "file_line"
FILE_LINE_FUNC = "file_line_func"
FUNC = "func"
@@ -38,16 +37,21 @@ class CallerInfoTemplate(Enum):
@classmethod
def for_template_key(cls, key: "CallerInfoTemplate") -> str:
- templates: dict[CallerInfoTemplate, str] = {
- cls.FILE_LINE: "{file}:{line}",
- cls.FILE_LINE_FUNC: "{file}:{line} {func}",
- cls.FUNC: "{func}",
- cls.FILE_FUNC: "{file} {func}",
- cls.FUNC_LINE: "{func} {line}",
- cls.FUNC_MODULE: "{func} {module}",
- cls.FUNC_MODULE_LINE: "{func} {module} {line}",
- }
- return templates.get(key, "")
+ match key:
+ case cls.FILE_LINE:
+ return "{file}:{line}"
+ case cls.FILE_LINE_FUNC:
+ return "{file}:{line} {func}"
+ case cls.FUNC:
+ return "{func}"
+ case cls.FILE_FUNC:
+ return "{file} {func}"
+ case cls.FUNC_LINE:
+ return "{func} {line}"
+ case cls.FUNC_MODULE:
+ return "{func} {module}"
+ case cls.FUNC_MODULE_LINE:
+ return "{func} {module} {line}"
class RichLogConfig(ConfigModel):
diff --git a/pipelex/tools/misc/context_provider_abstract.py b/pipelex/tools/misc/context_provider_abstract.py
index ca8ff8f37..1e93e0738 100644
--- a/pipelex/tools/misc/context_provider_abstract.py
+++ b/pipelex/tools/misc/context_provider_abstract.py
@@ -1,10 +1,10 @@
from abc import ABC, abstractmethod
from typing import Any
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
-class ContextProviderException(ToolException):
+class ContextProviderError(ToolError):
def __init__(self, message: str, variable_name: str):
super().__init__(message=message)
self.variable_name = variable_name
diff --git a/pipelex/tools/misc/dict_utils.py b/pipelex/tools/misc/dict_utils.py
index 27e7ff739..20542b5cf 100644
--- a/pipelex/tools/misc/dict_utils.py
+++ b/pipelex/tools/misc/dict_utils.py
@@ -212,8 +212,8 @@ def extract_vars_from_strings_recursive(data: Any) -> set[str]:
"""Recursively traverse the data structure."""
if isinstance(value, dict):
dict_value: dict[Any, Any] = cast("dict[Any, Any]", value)
- for v in dict_value.values():
- traverse(v)
+ for the_value in dict_value.values():
+ traverse(the_value)
elif isinstance(value, list):
list_value: list[Any] = cast("list[Any]", value)
for item in list_value:
diff --git a/pipelex/tools/misc/file_utils.py b/pipelex/tools/misc/file_utils.py
index d5fe8e29a..21f30aae4 100644
--- a/pipelex/tools/misc/file_utils.py
+++ b/pipelex/tools/misc/file_utils.py
@@ -25,8 +25,8 @@ def save_bytes_to_binary_file(file_path: str, byte_data: bytes, create_directory
if create_directory:
ensure_directory_exists(os.path.dirname(file_path))
- with open(file_path, "wb") as f:
- f.write(byte_data)
+ with open(file_path, "wb") as file:
+ file.write(byte_data)
return file_path
diff --git a/pipelex/tools/misc/filetype_utils.py b/pipelex/tools/misc/filetype_utils.py
index 7597bc459..717cd7d83 100644
--- a/pipelex/tools/misc/filetype_utils.py
+++ b/pipelex/tools/misc/filetype_utils.py
@@ -6,10 +6,10 @@ import filetype
from pydantic import BaseModel
from pipelex import log
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
-class FileTypeException(ToolException):
+class FileTypeError(ToolError):
pass
@@ -34,7 +34,7 @@ def detect_file_type_from_path(path: str | Path) -> FileType:
kind = filetype.guess(path) # pyright: ignore[reportUnknownMemberType]
if kind is None:
msg = f"Could not identify file type of '{path!s}'"
- raise FileTypeException(msg)
+ raise FileTypeError(msg)
extension = f"{kind.extension}"
mime = f"{kind.mime}"
return FileType(extension=extension, mime=mime)
@@ -56,7 +56,7 @@ def detect_file_type_from_bytes(buf: bytes) -> FileType:
kind = filetype.guess(buf) # pyright: ignore[reportUnknownMemberType]
if kind is None:
msg = f"Could not identify file type of given bytes: {buf[:300]!r}"
- raise FileTypeException(msg)
+ raise FileTypeError(msg)
extension = f"{kind.extension}"
mime = f"{kind.mime}"
return FileType(extension=extension, mime=mime)
@@ -90,6 +90,6 @@ def detect_file_type_from_base64(b64: str | bytes) -> FileType:
raw = base64.b64decode(b64_bytes, validate=True)
except binascii.Error as exc: # malformed Base-64
msg = f"Could not identify file type of given bytes because input is not valid Base-64: {exc}\n{b64_bytes[:100]!r}"
- raise FileTypeException(msg) from exc
+ raise FileTypeError(msg) from exc
return detect_file_type_from_bytes(buf=raw)
diff --git a/pipelex/tools/misc/json_utils.py b/pipelex/tools/misc/json_utils.py
index 6b83b5461..40842e3fd 100644
--- a/pipelex/tools/misc/json_utils.py
+++ b/pipelex/tools/misc/json_utils.py
@@ -5,18 +5,18 @@ from typing import Any, Union, cast
from kajson import kajson
from pydantic import BaseModel
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.tools.misc.file_utils import save_text_to_path
from pipelex.tools.typing.pydantic_utils import CustomBaseModel
JsonContent = Union[dict[str, Any], list[Any]]
-class ArgumentTypeError(ToolException):
+class ArgumentTypeError(ToolError):
pass
-class JsonTypeError(ToolException):
+class JsonTypeError(ToolError):
pass
diff --git a/pipelex/tools/misc/pretty.py b/pipelex/tools/misc/pretty.py
index ef6cf3383..33dfd279e 100644
--- a/pipelex/tools/misc/pretty.py
+++ b/pipelex/tools/misc/pretty.py
@@ -4,10 +4,10 @@ from io import StringIO
from typing import Any, ClassVar
from kajson import kajson
-from rich import print as rich_print
from rich.console import Console, Group
from rich.json import JSON
from rich.markdown import Markdown
+from rich.measure import Measurement
from rich.panel import Panel
from rich.pretty import Pretty
from rich.style import StyleType
@@ -73,8 +73,17 @@ def pretty_print(
inner_title: str | None = None,
border_style: StyleType | None = None,
width: int | None = None,
+ console_width: int | None = None,
):
- PrettyPrinter.pretty_print(content=content, title=title, subtitle=subtitle, inner_title=inner_title, border_style=border_style, width=width)
+ PrettyPrinter.pretty_print(
+ content=content,
+ title=title,
+ subtitle=subtitle,
+ inner_title=inner_title,
+ border_style=border_style,
+ width=width,
+ console_width=console_width,
+ )
def pretty_print_md(
@@ -84,10 +93,19 @@ def pretty_print_md(
inner_title: str | None = None,
border_style: StyleType | None = None,
width: int | None = None,
+ console_width: int | None = None,
):
width = width or PrettyPrinter.pretty_width()
md_content = Markdown(content)
- PrettyPrinter.pretty_print(content=md_content, title=title, subtitle=subtitle, inner_title=inner_title, border_style=border_style, width=width)
+ PrettyPrinter.pretty_print(
+ content=md_content,
+ title=title,
+ subtitle=subtitle,
+ inner_title=inner_title,
+ border_style=border_style,
+ width=width,
+ console_width=console_width,
+ )
class PrettyPrinter:
@@ -102,6 +120,7 @@ class PrettyPrinter:
inner_title: str | None = None,
border_style: StyleType | None = None,
width: int | None = None,
+ console_width: int | None = None,
):
match cls.mode:
case PrettyPrintMode.RICH:
@@ -112,9 +131,10 @@ class PrettyPrinter:
inner_title=inner_title,
border_style=border_style,
width=width,
+ console_width=console_width,
)
case PrettyPrintMode.POOR:
- cls.pretty_print_without_rich(content=content, title=title, subtitle=subtitle, inner_title=inner_title)
+ cls.pretty_print_without_rich(content=content, title=title, subtitle=subtitle, inner_title=inner_title, console_width=console_width)
@classmethod
def pretty_print_using_rich(
@@ -125,12 +145,19 @@ class PrettyPrinter:
inner_title: str | None = None,
border_style: StyleType | None = None,
width: int | None = None,
+ console_width: int | None = None,
):
panel = cls.make_pretty_panel(
- content=content, title=title, subtitle=subtitle, inner_title=inner_title, border_style=border_style, width=width
+ content=content,
+ title=title,
+ subtitle=subtitle,
+ inner_title=inner_title,
+ border_style=border_style,
+ width=width,
+ console_width=console_width,
)
- rich_print("", panel, "", sep="\n")
+ Console(width=console_width).print("", panel, "", sep="\n")
@classmethod
def pretty_width(cls, width: int | None = None, depth: int | None = None) -> int:
@@ -153,8 +180,19 @@ class PrettyPrinter:
inner_title: str | None = None,
border_style: StyleType | None = None,
width: int | None = None,
+ console_width: int | None = None,
) -> Panel:
pretty = cls.make_pretty(content, inner_title=inner_title, depth=0)
+ # When width is not specified, measure the content to determine optimal console width
+ if width is None:
+ # Create a console to measure the panel
+ measure_console = Console(width=console_width)
+ measurement = Measurement.get(measure_console, measure_console.options, pretty)
+ # Use the maximum width that fits the content, with some buffer for panel border rendering
+ width = measurement.maximum + 4
+ # print(f"width: {width}")
+ if console_width is not None:
+ width = min(width, console_width)
return cls.wrap_in_panel(pretty=pretty, title=title, subtitle=subtitle, border_style=border_style, width=width)
@classmethod
@@ -256,6 +294,8 @@ class PrettyPrinter:
title: TextType | None = None,
subtitle: TextType | None = None,
inner_title: str | None = None,
+ width: int | None = None,
+ console_width: int | None = None,
):
if isinstance(content, str) and content.startswith(("http://", "https://")):
cls.pretty_print_url_without_rich(content=content, title=title, subtitle=subtitle)
@@ -265,7 +305,7 @@ class PrettyPrinter:
title_str += f"\n{subtitle!s}"
if inner_title:
title_str += f"\n{inner_title}"
- terminal_width = shutil.get_terminal_size().columns
+ terminal_width = console_width or shutil.get_terminal_size().columns
content_str = f"{content}"
# Split title into lines if it contains newlines
@@ -274,6 +314,8 @@ class PrettyPrinter:
# Calculate max content width based on longest title line
max_title_len = max(len(line) for line in title_lines) if title_lines else 0
max_content_width = terminal_width - max_title_len - 8 # Accounting for frame and padding
+ if width:
+ max_content_width = min(max_content_width, width)
wrapped_lines: list[str] = []
for line in content_str.splitlines():
while len(line) > max_content_width:
diff --git a/pipelex/tools/misc/toml_utils.py b/pipelex/tools/misc/toml_utils.py
index 1ef69cd36..0f43a9cc9 100644
--- a/pipelex/tools/misc/toml_utils.py
+++ b/pipelex/tools/misc/toml_utils.py
@@ -5,11 +5,11 @@ from typing import Any
import tomli
import tomlkit
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.tools.misc.file_utils import path_exists
-class TomlError(ToolException):
+class TomlError(ToolError):
def __init__(self, message: str, doc: str, pos: int, lineno: int, colno: int):
super().__init__(message)
self.doc = doc
@@ -44,8 +44,8 @@ def load_toml_from_path(path: str) -> dict[str, Any]:
"""
try:
- with open(path, "rb") as f:
- return tomli.load(f)
+ with open(path, "rb") as file:
+ return tomli.load(file)
except tomli.TOMLDecodeError as exc:
msg = f"TOML parsing error in file '{path}': {exc.msg}"
raise TomlError(message=msg, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from exc
@@ -68,8 +68,8 @@ def load_toml_with_tomlkit(path: str) -> tomlkit.TOMLDocument:
TOMLDocument that preserves formatting and comments
"""
- with open(path, encoding="utf-8") as f:
- return tomlkit.load(f)
+ with open(path, encoding="utf-8") as file:
+ return tomlkit.load(file)
def save_toml_to_path(data: dict[str, Any] | tomlkit.TOMLDocument, path: str) -> None:
@@ -80,5 +80,5 @@ def save_toml_to_path(data: dict[str, Any] | tomlkit.TOMLDocument, path: str) ->
path: Path where the TOML file should be saved
"""
- with open(path, "w", encoding="utf-8") as f:
- tomlkit.dump(data, f) # type: ignore[arg-type]
+ with open(path, "w", encoding="utf-8") as file:
+ tomlkit.dump(data, file) # type: ignore[arg-type]
diff --git a/pipelex/tools/pdf/pypdfium2_renderer.py b/pipelex/tools/pdf/pypdfium2_renderer.py
index 97fdb9f34..cf57d9353 100644
--- a/pipelex/tools/pdf/pypdfium2_renderer.py
+++ b/pipelex/tools/pdf/pypdfium2_renderer.py
@@ -7,7 +7,7 @@ from typing import TYPE_CHECKING
import pypdfium2 as pdfium
from pypdfium2.raw import FPDFBitmap_BGRA
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx_async
from pipelex.tools.misc.path_utils import clarify_path_or_url
@@ -16,7 +16,7 @@ if TYPE_CHECKING:
PDFIUM2_REFERENCE_DPI = 72
-class PyPdfium2RendererError(ToolException):
+class PyPdfium2RendererError(ToolError):
pass
diff --git a/pipelex/tools/secrets/secrets_errors.py b/pipelex/tools/secrets/secrets_errors.py
index 9ad716962..089b30924 100644
--- a/pipelex/tools/secrets/secrets_errors.py
+++ b/pipelex/tools/secrets/secrets_errors.py
@@ -1,5 +1,5 @@
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
-class SecretNotFoundError(ToolException):
+class SecretNotFoundError(ToolError):
pass
diff --git a/pipelex/tools/secrets/secrets_utils.py b/pipelex/tools/secrets/secrets_utils.py
index 9d5d4972a..0a6d3acd0 100644
--- a/pipelex/tools/secrets/secrets_utils.py
+++ b/pipelex/tools/secrets/secrets_utils.py
@@ -1,23 +1,23 @@
import re
from pipelex.system.environment import EnvVarNotFoundError, get_optional_env, get_required_env
-from pipelex.system.exceptions import ToolException
+from pipelex.system.exceptions import ToolError
from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
from pipelex.types import StrEnum
-class VarNotFoundError(ToolException):
+class VarNotFoundError(ToolError):
def __init__(self, var_name: str, message: str):
self.var_name = var_name
super().__init__(message)
-class VarFallbackPatternError(ToolException):
+class VarFallbackPatternError(ToolError):
pass
-class UnknownVarPrefixError(ToolException):
+class UnknownVarPrefixError(ToolError):
"""Raised when an unknown variable prefix is used in variable substitution."""
def __init__(self, var_name: str, message: str):
diff --git a/pipelex/tools/typing/class_utils.py b/pipelex/tools/typing/class_utils.py
index c61024901..0d2dd7f28 100644
--- a/pipelex/tools/typing/class_utils.py
+++ b/pipelex/tools/typing/class_utils.py
@@ -53,16 +53,16 @@ def has_compatible_field(class_1: type[Any], class_2: type[Any]) -> bool:
fields: dict[str, FieldInfo] = class_1.model_fields # type: ignore[attr-defined]
- def _is_compatible(t: Any) -> bool:
+ def _is_compatible(type_param: Any) -> bool:
# Unwrap Annotated[T, ...]
- if get_origin(t) is Annotated:
- t = get_args(t)[0]
+ if get_origin(type_param) is Annotated:
+ type_param = get_args(type_param)[0]
- origin = get_origin(t)
+ origin = get_origin(type_param)
# Handle unions, including PEP 604 (T | None)
if origin in (Union, _UnionType):
- for arg in get_args(t):
+ for arg in get_args(type_param):
if arg is _NoneType:
continue
if _is_compatible(arg):
@@ -71,7 +71,7 @@ def has_compatible_field(class_1: type[Any], class_2: type[Any]) -> bool:
# Base case: direct match / subclass
try:
- return t is class_2 or (isinstance(t, type) and issubclass(t, class_2))
+ return type_param is class_2 or (isinstance(type_param, type) and issubclass(type_param, class_2))
except TypeError:
# Not a class type (e.g., typing constructs you don't care about)
return False
diff --git a/pipelex/tools/typing/module_inspector.py b/pipelex/tools/typing/module_inspector.py
index dd5fce6bb..4c1818d78 100644
--- a/pipelex/tools/typing/module_inspector.py
+++ b/pipelex/tools/typing/module_inspector.py
@@ -6,8 +6,10 @@ import sys
from pathlib import Path
from typing import Any
+from pipelex.base_exceptions import PipelexError
-class ModuleFileError(Exception):
+
+class ModuleFileError(PipelexError):
"""Exception raised for errors related to module file operations."""
@@ -132,12 +134,12 @@ def find_class_names_in_file(file_path: str, base_class_names: list[str] | None
try:
# Read and parse the file
- with open(file_path, encoding="utf-8") as f:
- source = f.read()
+ with open(file_path, encoding="utf-8") as file:
+ source = file.read()
tree = ast.parse(source, filename=file_path)
- except Exception as e:
- msg = f"Failed to parse {file_path}: {e}"
- raise ModuleFileError(msg) from e
+ except Exception as exc:
+ msg = f"Failed to parse {file_path}: {exc}"
+ raise ModuleFileError(msg) from exc
class_names: list[str] = []
@@ -196,12 +198,12 @@ def find_decorated_function_names_in_file(
try:
# Read and parse the file
- with open(file_path, encoding="utf-8") as f:
- source = f.read()
+ with open(file_path, encoding="utf-8") as file:
+ source = file.read()
tree = ast.parse(source, filename=file_path)
- except Exception as e:
- msg = f"Failed to parse {file_path}: {e}"
- raise ModuleFileError(msg) from e
+ except Exception as exc:
+ msg = f"Failed to parse {file_path}: {exc}"
+ raise ModuleFileError(msg) from exc
function_names: list[str] = []
diff --git a/pyproject.toml b/pyproject.toml
index 68a5e6533..6ac2b70a2 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -86,7 +86,7 @@ dev = [
[project.scripts]
pipelex = "pipelex.cli._cli:app"
-pipelex-dev = "pipelex.dev_cli._dev_cli:app"
+pipelex-dev = "pipelex.cli.dev_cli._dev_cli:app"
[build-system]
requires = ["hatchling"]
@@ -205,9 +205,9 @@ typeCheckingMode = "strict"
minversion = "9.0"
addopts = [
"--import-mode=importlib",
- "-ra", # Show all test outcomes (including skips)
+ "-ra", # Show all test outcomes (including skips)
"-m",
- "not (inference or llm or img_gen or extract or needs_output or pipelex_api)",
+ "not (inference or llm or img_gen or extract or pipelex_api)",
]
asyncio_default_fixture_loop_scope = "session"
xfail_strict = true
@@ -220,7 +220,6 @@ filterwarnings = [
"ignore:Use.*Types instead",
]
markers = [
- "needs_output: tests that need output to be displayed",
"inference: slow and costly due to inference calls",
"llm: slow and costly due to llm inference calls",
"img_gen: slow and costly due to image generation inference calls",
@@ -350,7 +349,6 @@ ignore = [
"UP036", # Version block is outdated for minimum Python version
"SIM102", # Use a single `if` statement instead of nested `if` statements
"S701", # Using jinja2 templates with `autoescape=False` is dangerous and can lead to XSS. Ensure `autoescape=True` or use the `select_autoescape` function.
- "N818", # Exception name `RootException` should be named with an Error suffix
"TRY301", # Abstract `raise` to an inner function
"PERF401", # Use a list comprehension to create a transformed list
"PLW2901", # `for` loop variable `line` overwritten by assignment target
@@ -372,14 +370,14 @@ required-version = ">=0.7.2"
[tool.hatch.build.targets.wheel]
packages = ["pipelex"]
-exclude = ["pipelex/dev_cli"]
+exclude = ["pipelex/cli/dev_cli"]
[tool.hatch.build.targets.wheel.force-include]
"pyproject.toml" = "pipelex/pyproject.toml"
[tool.hatch.build.targets.sdist]
packages = ["pipelex"]
-exclude = ["pipelex/dev_cli"]
+exclude = ["pipelex/cli/dev_cli"]
[tool.hatch.build.targets.sdist.force-include]
"pyproject.toml" = "pipelex/pyproject.toml"
@@ -395,5 +393,21 @@ reports = false
disable = ["all"]
enable = [
"W0101", # Unreachable code: Used when there is some code behind a "return" or "raise" statement, which will never be accessed.
+ "C0103", # invalid-name (naming convention)
]
ignore = [".venv", "__pycache__", "build", "dist", ".git"]
+
+[tool.pylint.basic]
+# Variables / attributes / arguments: snake_case, not length 1
+variable-rgx = "^(?!.$)[a-z_][a-z0-9_]*$"
+argument-rgx = "^(?!.$)[a-z_][a-z0-9_]*$"
+attr-rgx = "^(?!.$)[a-z_][a-z0-9_]*$"
+
+# Functions / methods: snake_case, not length 1
+function-rgx = "^(?!.$)[a-z_][a-z0-9_]*$"
+method-rgx = "^(?!.$)[a-z_][a-z0-9_]*$"
+
+# Classes: CapWords, not length 1
+class-rgx = "^(?!.$)[A-Z_][a-zA-Z0-9_]*$"
+typevar-rgx = "^[A-Za-z_][A-Za-z0-9_]*$"
+good-names = ["_"]
diff --git a/tests/conftest.py b/tests/conftest.py
index 2edc9a91c..b2d5805ee 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,11 +1,10 @@
+from pathlib import Path
+
import pytest
-from rich.traceback import Traceback
-import pipelex.config
-import pipelex.pipelex
from pipelex import log
-from pipelex.config import get_config
-from pipelex.hub import get_console, get_report_delegate
+from pipelex.hub import get_library_manager, set_current_library
+from pipelex.pipelex import Pipelex
from pipelex.system.runtime import IntegrationMode
pytest_plugins = [
@@ -27,20 +26,51 @@ def routing_profile_setup(request: pytest.FixtureRequest): # noqa: ARG001 # py
@pytest.fixture(scope="module", autouse=True)
def reset_pipelex_config_fixture(routing_profile_setup: str | None): # noqa: ARG001
- # Code to run before each test
- # The routing_profile_setup dependency ensures any routing overrides happen first
- console = get_console()
- console.print("[magenta]pipelex setup[/magenta]")
- try:
- pipelex_instance = pipelex.pipelex.Pipelex.make(integration_mode=IntegrationMode.PYTEST)
- config = get_config()
- log.verbose(config, title="Test config")
- assert isinstance(config, pipelex.config.PipelexConfig)
- except Exception as exc:
- console.print(Traceback())
- pytest.exit(f"Critical Pipelex setup error: {exc}")
+ pipelex_instance = Pipelex.make(integration_mode=IntegrationMode.PYTEST)
yield
- # Code to run after each test
- get_report_delegate().generate_report()
- console.print("[dim magenta]pipelex teardown[/dim magenta]")
pipelex_instance.teardown()
+
+
[email protected](scope="class")
+def load_test_library():
+ library_id = None
+
+ def _load(library_dirs: list[Path]) -> None:
+ nonlocal library_id
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+
+ library_manager.load_libraries(
+ library_id=library_id,
+ library_dirs=library_dirs,
+ )
+
+ log.verbose(f"Loaded libraries: {[str(p) for p in library_dirs]}")
+
+ yield _load
+
+ if library_id is not None:
+ library_manager = get_library_manager()
+ library_manager.teardown(library_id=library_id)
+ log.verbose(f"Torn down library: {library_id}")
+
+
[email protected](scope="class")
+def load_empty_library():
+ library_id = None
+
+ def _load() -> None:
+ nonlocal library_id
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+
+ log.verbose(f"Opened empty library: {library_id}")
+
+ yield _load
+
+ if library_id is not None:
+ library_manager = get_library_manager()
+ library_manager.teardown(library_id=library_id)
+ log.verbose(f"Torn down library: {library_id}")
diff --git a/tests/e2e/pipelex/cli/test_validate_cmd.py b/tests/e2e/pipelex/cli/test_validate_cmd.py
new file mode 100644
index 000000000..bd9ec0455
--- /dev/null
+++ b/tests/e2e/pipelex/cli/test_validate_cmd.py
@@ -0,0 +1,6 @@
+from pipelex.cli.commands.validate_cmd import do_validate_all_libraries_and_dry_run
+
+
+class TestValidateCommand:
+ def test_validate_all(self):
+ do_validate_all_libraries_and_dry_run()
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_1.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_1.py
index 0e33a3e4e..576ae0f07 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_1.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_1.py
@@ -1,3 +1,6 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pytest import FixtureRequest
@@ -10,7 +13,7 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition import CategoryInput
+from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition import CategoryInput
@pytest.mark.llm
@@ -27,8 +30,10 @@ class TestPipeConditionExpressionTemplate:
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
category: str,
+ load_test_library: Callable[[list[Path]], None],
):
"""Test that PipeCondition routes to the correct pipe based on expression."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create input data
category_input = CategoryInput(category=category)
stuff = StuffFactory.make_stuff(
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_2.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_2.py
index 97069a906..c614ac0e1 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_2.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_2.py
@@ -1,3 +1,6 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pytest import FixtureRequest
@@ -10,7 +13,7 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition import CategoryInput
+from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition import CategoryInput
@pytest.mark.llm
@@ -27,7 +30,9 @@ class TestPipeConditionExpression:
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
category: str,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition/")])
"""Test that PipeCondition routes to the correct pipe based on expression."""
# Create input data
category_input = CategoryInput(category=category)
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py
index 92e2dab1e..7779c4569 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/test_pipe_condition_simple.py
@@ -1,29 +1,33 @@
"""Test simple pipe condition functionality with dry run validation."""
+from pathlib import Path
+from typing import Callable
+
import pytest
from pipelex import log, pretty_print
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.exceptions import PipeRunInputsError
-from pipelex.core.pipes.input_requirements import TypedNamedInputRequirement
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
+from pipelex.core.pipes.inputs.input_requirements import TypedNamedInputRequirement
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory
+from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.pipeline.exceptions import DryRunMissingInputsError
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition import CategoryInput
+from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition import CategoryInput
@pytest.mark.dry_runnable
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
class TestPipeConditionSimple:
- async def test_direct_pipe_condition_should_fail(self):
+ async def test_direct_pipe_condition_should_fail(self, load_test_library: Callable[[list[Path]], None]):
"""Test a PipeCondition created directly in code that should FAIL dry run."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create a PipeCondition directly in Python that requires an input
pipe_condition_blueprint = PipeConditionBlueprint(
description="Test condition that should fail",
@@ -56,8 +60,9 @@ class TestPipeConditionSimple:
assert "user_category" in error.missing_inputs
assert "Missing required inputs" in str(error)
- async def test_direct_pipe_condition_should_succeed(self):
+ async def test_direct_pipe_condition_should_succeed(self, load_test_library: Callable[[list[Path]], None]):
"""Test a PipeCondition created directly in code that should SUCCEED dry run."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create a PipeCondition directly in Python
pipe_condition_blueprint = PipeConditionBlueprint(
description="Test condition that should succeed",
@@ -103,7 +108,7 @@ class TestPipeConditionSimple:
log.info("✅ Direct PipeCondition SUCCEEDED completely!")
pretty_print(pipe_output)
- except DryRunMissingInputsError as exc:
+ except PipeRunError as exc:
msg_exc = str(exc)
# If it fails, it should NOT be due to missing inputs
assert "missing required inputs" not in str(msg_exc)
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_1.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_1.py
index 6d45035da..4994996bb 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_1.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_1.py
@@ -1,4 +1,5 @@
-from typing import cast
+from pathlib import Path
+from typing import Callable, cast
import pytest
from pytest import FixtureRequest
@@ -9,14 +10,15 @@ from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_pipe_router, get_required_pipe
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
-from pipelex.pipe_run.pipe_run_params import PipeRunMode
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_parallel.pipe_parallel import ContentAnalysis, DocumentInput, LengthAnalysis
+from tests.integration.pipelex.pipes.controller.pipe_parallel.pipe_parallel import ContentAnalysis, DocumentInput, LengthAnalysis
@pytest.mark.llm
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
class TestPipeParallelDocumentAnalysis:
@pytest.mark.parametrize(
@@ -32,8 +34,10 @@ class TestPipeParallelDocumentAnalysis:
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
document_text: str,
+ load_test_library: Callable[[list[Path]], None],
):
"""Test that PipeParallel processes document analysis in parallel."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_parallel")])
# Create input data
document_input = DocumentInput(text=document_text)
stuff = StuffFactory.make_stuff(
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/discord_newsletter.plx b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/discord_newsletter.plx
new file mode 100644
index 000000000..e5585be3c
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/discord_newsletter.plx
@@ -0,0 +1,130 @@
+domain = "discord_newsletter_e2e"
+description = "Create newsletters from Discord channel content by summarizing messages and organizing them according to newsletter format"
+
+[concept]
+DiscordChannelUpdateE2E = "A Discord channel with its messages for newsletter generation"
+ChannelSummaryE2E = "A summarized Discord channel for newsletter inclusion"
+HtmlNewsletterE2E = "The final newsletter content in html format with organized channel summaries"
+
+[pipe.write_discord_newsletter_e2e]
+type = "PipeSequence"
+description = "Create a newsletter from Discord articles by summarizing channels and organizing content"
+inputs = { discord_channel_updates = "DiscordChannelUpdateE2E[]" }
+output = "HtmlNewsletterE2E"
+steps = [
+ { pipe = "summarize_discord_channel_update_e2e", batch_over = "discord_channel_updates", batch_as = "discord_channel_update", result = "channel_summaries" },
+ { pipe = "write_weekly_summary_e2e", result = "weekly_summary" },
+ { pipe = "format_html_newsletter_e2e", result = "html_newsletter" },
+]
+
+
+[pipe.summarize_discord_channel_update_e2e]
+type = "PipeCondition"
+description = "Select the appropriate summary pipe based on the channel name"
+inputs = { discord_channel_update = "DiscordChannelUpdateE2E" }
+output = "ChannelSummaryE2E"
+expression = "discord_channel_update.name"
+outcomes = { "Introduce-Yourself" = "summarize_discord_channel_update_for_new_members_e2e" }
+default_outcome = "summarize_discord_channel_update_general_e2e"
+
+[pipe.summarize_discord_channel_update_for_new_members_e2e]
+type = "PipeLLM"
+description = "Summarize the new member announcements"
+inputs = { discord_channel_update = "DiscordChannelUpdateE2E" }
+output = "ChannelSummaryE2E"
+system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."
+prompt = """
+Analyze this Discord channel update and create a newsletter-friendly summary.
+
+Channel Information:
+@discord_channel_update
+
+Summarize with one bullet point for each new member
+"""
+
+[pipe.summarize_discord_channel_update_general_e2e]
+type = "PipeLLM"
+description = "Summarize a Discord channel's messages into newsletter-friendly content"
+inputs = { discord_channel_update = "DiscordChannelUpdateE2E" }
+output = "ChannelSummaryE2E"
+system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."
+prompt = """
+Analyze this Discord channel update and create a newsletter-friendly summary.
+
+Channel Information:
+@discord_channel_update
+
+The summary should be informative and engaging for newsletter readers who want to understand what happened in this channel during the week.
+Generate one or more summary items: each one can correspond to a single message or to a bunch of messages that were part of the same conversation.
+Each summary item should be in plain text, no bullet points. You can make some parts **bold** to highlight important information.
+
+Make sure to preserve the channel name and position from the input for proper ordering in the newsletter.
+"""
+
+[pipe.write_weekly_summary_e2e]
+type = "PipeLLM"
+description = "Combine channel summaries into a short summary of the week's Share channel content (200 characters)"
+inputs = { channel_summaries = "ChannelSummaryE2E" }
+output = "Text"
+prompt = """
+Write a single overall summary of the week's content based on the following Share channel summaries:
+
+{% for channel in channel_summaries.content.items if channel.category == "Share" %}
+{{ channel }}
+{% endfor %}
+
+Keep it short: 200 characters.
+"""
+
+[pipe.format_html_newsletter_e2e]
+type = "PipeCompose"
+description = "Combine weekly and channel summaries into a complete newsletter following specific formatting requirements"
+inputs = { weekly_summary = "Text", channel_summaries = "ChannelSummaryE2E" }
+output = "HtmlNewsletterE2E"
+
+[pipe.format_html_newsletter_e2e.template]
+category = "html"
+template = """
+<!-- Weekly Summary -->
+<h2>☀️ Weekly Summary</h2>
+<p>
+$weekly_summary
+</p>
+
+<!-- New Members Section -->
+{% set introduce_channels = channel_summaries.content.items | selectattr('category', 'equalto', 'Introduce Yourself') | list %}
+{% if introduce_channels %}
+ <h2>🙌 New members</h2>
+ <ul>
+ {% for channel in introduce_channels %}
+ {% for item in channel.summary_items_as_html %}
+ <li>{{ item }}</li>
+ {% endfor %}
+ {% endfor %}
+ </ul>
+{% endif %}
+
+<!-- Share Channel Section -->
+{% set regular_channels = channel_summaries.content.items | selectattr('category', 'equalto', 'Share') | list | sort(attribute='position') %}
+{% if regular_channels %}
+ {% for channel in regular_channels %}
+ <h2>{{ channel.channel_name }}</h2>
+ {% for item in channel.summary_items_as_html %}
+ <p>{{ item }}</p>
+ {% endfor %}
+ {% endfor %}
+{% endif %}
+
+<!-- Geographic Hubs Section -->
+{% set geo_hubs = channel_summaries.content.items | selectattr('category', 'equalto', 'Geographic Hubs') | list | sort(attribute='position') %}
+{% if geo_hubs %}
+ <h2>🌎 Geographic hubs</h2>
+ {% for channel in geo_hubs %}
+ <h3>{{ channel.channel_name }}</h3>
+ {% for item in channel.summary_items_as_html %}
+ <p>{{ item }}</p>
+ {% endfor %}
+ {% endfor %}
+{% endif %}
+"""
+
diff --git a/tests/test_pipelines/discord_newsletter.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/discord_newsletter.py
similarity index 100%
rename from tests/test_pipelines/discord_newsletter.py
rename to tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/discord_newsletter.py
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py
index 3c8a6d066..756c7ab61 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_batching.py
@@ -1,26 +1,28 @@
"""Test pipe sequence functionality with batching operations."""
-from typing import cast
+from pathlib import Path
+from typing import Callable, cast
import pytest
from pipelex import log
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.input_requirements import TypedNamedInputRequirement
+from pipelex.core.pipes.inputs.input_requirements import TypedNamedInputRequirement
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_required_pipe
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.execute import execute_pipeline
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_sequence.pipe_sequence import Document, ProductRating
+from tests.integration.pipelex.pipes.controller.pipe_sequence.pipe_sequence import Document, ProductRating
@pytest.mark.dry_runnable
@pytest.mark.inference
@pytest.mark.asyncio
-async def test_review_analysis_sequence_with_batching(pipe_run_mode: PipeRunMode):
+async def test_review_analysis_sequence_with_batching(pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]):
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence")])
"""Test customer review analysis sequence with batching."""
# Create test input - a document with reviews
if pipe_run_mode == PipeRunMode.DRY:
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
index fc17917c1..14c36d7ab 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
@@ -33,6 +33,7 @@ class TestPipeSequenceMultiplicity:
# Execute the pipeline
pipe_output = await execute_pipeline(
+ library_path="tests/integration/pipelex/pipes/controller/pipe_sequence/",
pipe_code="creative_ideation_sequence",
inputs=working_memory,
pipe_run_mode=pipe_run_mode,
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_simple.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_simple.py
index 1c477f1c1..66671ed17 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_simple.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_simple.py
@@ -1,5 +1,8 @@
"""Test simple pipe sequence functionality without batching."""
+from pathlib import Path
+from typing import Callable
+
import pytest
from pipelex import pretty_print
@@ -16,21 +19,22 @@ from pipelex.pipeline.job_metadata import JobMetadata
@pytest.mark.dry_runnable
@pytest.mark.inference
@pytest.mark.asyncio
-async def test_simple_text_sequence(pipe_run_mode: PipeRunMode):
- """Test simple text processing sequence without batching."""
- # Create test input
- raw_text_stuff = StuffFactory.make_stuff(
- name="raw_text",
- concept=ConceptFactory.make(
- concept_code="RawText",
- domain="simple_text_processing",
- description="simple_text_processing.RawText",
- structure_class_name="TextContent",
- ),
- content=TextContent(text="This is some messy text with bad spacing."),
- )
-
- if pipe_run_mode == PipeRunMode.DRY:
+class TestPipeSequenceSimple:
+ async def test_simple_text_sequence(self, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]):
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence/")])
+ """Test simple text processing sequence without batching."""
+ # Create test input
+ raw_text_stuff = StuffFactory.make_stuff(
+ name="raw_text",
+ concept=ConceptFactory.make(
+ concept_code="RawText",
+ domain="simple_text_processing",
+ description="simple_text_processing.RawText",
+ structure_class_name="TextContent",
+ ),
+ content=TextContent(text="This is some messy text with bad spacing."),
+ )
+
# Create working memory with the required input for dry run
working_memory = WorkingMemoryFactory.make_from_multiple_stuffs([raw_text_stuff])
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
index 2ddc2c22c..390215b0f 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
@@ -3,8 +3,9 @@
import pytest
from pipelex import pretty_print
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipeline.execute import execute_pipeline
-from tests.test_pipelines.test_tweet import OptimizedTweet
+from tests.e2e.pipelex.pipes.pipe_controller.pipe_sequence.test_tweet import OptimizedTweet
SAMPLE_DRAFT_TWEET = """
Local high school basketball star Maria Rodriguez was the talk of Division I scouts - 6'2",
@@ -83,16 +84,18 @@ What $20 decision are you avoiding right now? 👇
"""
@pytest.mark.llm
@pytest.mark.inference
@pytest.mark.asyncio
class TestPipeSequenceWithInputMemory:
"""Test pipe sequence functionality with input memory."""
- async def test_optimize_tweet_sequence_with_input_memory(self):
+ async def test_optimize_tweet_sequence_with_input_memory(self, pipe_run_mode: PipeRunMode):
"""Test the optimize_tweet_sequence pipeline using inputs parameter."""
# Execute the pipeline using inputs
pipe_output = await execute_pipeline(
+ library_path="tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/",
pipe_code="optimize_tweet_sequence",
inputs={
"draft_tweet": {
@@ -104,7 +107,9 @@ class TestPipeSequenceWithInputMemory:
"content": SAMPLE_WRITING_STYLE,
},
},
+ pipe_run_mode=pipe_run_mode,
)
+ pretty_print(pipe_output, title="Pipe output for optimize_tweet_sequence")
# Get the optimized tweet
optimized_tweet = pipe_output.main_stuff_as(content_type=OptimizedTweet)
@@ -113,15 +118,11 @@ class TestPipeSequenceWithInputMemory:
assert pipe_output is not None
assert pipe_output.working_memory is not None
assert pipe_output.main_stuff is not None
- assert optimized_tweet is not None
+ assert optimized_tweet
assert isinstance(optimized_tweet, OptimizedTweet)
- assert len(optimized_tweet.text) > 0
-
- # Log output and generate report
- pretty_print(pipe_output, title="Pipe output for optimize_tweet_sequence")
-
- # Verify the optimized tweet is different from the draft
- assert optimized_tweet.text != SAMPLE_DRAFT_TWEET
- # Verify it's not empty
- assert len(optimized_tweet.text.strip()) > 0
- assert "Maria Rodriguez" in optimized_tweet.text
+ assert len(optimized_tweet.lead_tweet.strip()) > 0
+ match pipe_run_mode:
+ case PipeRunMode.DRY:
+ pass
+ case PipeRunMode.LIVE:
+ assert "Maria Rodriguez" in optimized_tweet.lead_tweet
diff --git a/tests/test_pipelines/test_tweet.plx b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_tweet.plx
similarity index 100%
rename from tests/test_pipelines/test_tweet.plx
rename to tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_tweet.plx
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_tweet.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_tweet.py
new file mode 100644
index 000000000..50db267e1
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_tweet.py
@@ -0,0 +1,10 @@
+from pydantic import Field
+
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+
+class OptimizedTweet(StructuredContent):
+ """A tweet optimized for Twitter/X engagement following best practices."""
+
+ lead_tweet: str
+ follow_up_tweets: list[str] | None = Field(default_factory=list[str])
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.plx b/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.plx
new file mode 100644
index 000000000..66458a666
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.plx
@@ -0,0 +1,58 @@
+domain = "pipe_llm_vision_e2e"
+description = "Test PipeLLM with vision capabilities"
+
+[concept]
+VisionAnalysisE2E = "Some analysis based on the image"
+BasicDescriptionE2E = "Basic description of the image"
+
+[concept.PhotoE2E]
+description = "A photo"
+refines = "Image"
+
+[pipe.describe_image_e2e]
+type = "PipeLLM"
+description = "Describe what is in the image"
+inputs = { image = "Image" }
+output = "Text"
+model = "llm_for_testing_vision"
+prompt = """
+Describe what you see in this image in 1-2 sentences, be concise.
+$image
+"""
+
+[pipe.describe_image_number_1_only_e2e]
+type = "PipeLLM"
+description = "Describe what is in the image"
+inputs = { image_a = "Image", image_b = "Image" }
+output = "BasicDescriptionE2E"
+model = "llm_for_diagram_to_text"
+prompt = """
+Describe what you see in $image_a only.
+Completely ignore $image_b.
+"""
+
+[pipe.describe_image_number_2_only_e2e]
+type = "PipeLLM"
+description = "Describe what is in the image"
+inputs = { image_a = "Image", image_b = "Image" }
+output = "BasicDescriptionE2E"
+model = "llm_for_diagram_to_text"
+prompt = """
+Describe what you see in $image_b only.
+Completely ignore $image_a.
+"""
+
+
+[pipe.vision_analysis_e2e]
+type = "PipeLLM"
+description = "Provide detailed analysis of the image"
+inputs = { image = "PhotoE2E" }
+output = "VisionAnalysisE2E"
+model = "llm_for_diagram_to_text"
+system_prompt = "You are an expert image analyst. Provide detailed, accurate descriptions."
+prompt = """
+Analyze this image and describe what's the main topic etc.
+$image
+--------------------------------
+"""
+
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.py b/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.py
new file mode 100644
index 000000000..eb28e0e9e
--- /dev/null
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_llm_vision.py
@@ -0,0 +1,18 @@
+from pydantic import Field
+
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+
+class BasicDescription(StructuredContent):
+ """Basic description of the image."""
+
+ description: str = Field(description="Concise description of the image")
+
+
+class VisionAnalysisE2E(StructuredContent):
+ """Structured analysis of visual content from an image."""
+
+ main_subject: str = Field(description="The primary subject or focus of the image")
+ category: str = Field(description="The category of the image")
+ description: str = Field(description="Concise description of the image")
+ time_scale: str = Field(description="The time scale of gant chart: 'days', 'weeks', 'months', 'years' or something else")
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py b/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
index ccf05052b..79c174a0d 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
+++ b/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
@@ -1,27 +1,32 @@
"""E2E test for PipeLLM with vision capabilities."""
+from pathlib import Path
+
import pytest
-from pipelex import pretty_print
+from pipelex import pretty_print, pretty_print_md
from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipeline.execute import execute_pipeline
+from tests.e2e.pipelex.pipes.pipe_operators.pipe_llm_vision import VisionAnalysisE2E
from tests.integration.pipelex.cogt.test_data import LLMVisionTestCases
+from tests.integration.pipelex.test_data import PipeTestCases
@pytest.mark.llm
@pytest.mark.inference
@pytest.mark.asyncio
class TestPipeLLMVision:
- """Test PipeLLM with vision capabilities."""
-
- async def test_describe_image(self):
- """Test the describe_image pipeline with a simple image."""
+ async def test_describe_image_single(self, pipe_run_mode: PipeRunMode):
# Execute the pipeline with an image
pipe_output = await execute_pipeline(
- pipe_code="describe_image",
+ pipe_code="describe_image_e2e",
+ library_path=str(Path("tests/e2e/pipelex/pipes/pipe_operators")),
inputs={
- "image": ImageContent(url=LLMVisionTestCases.PATH_IMG_GANTT_1),
+ "image": ImageContent(url=LLMVisionTestCases.URL_CLOUDFRONT_ALAN_TURING),
},
+ pipe_run_mode=pipe_run_mode,
)
# Get the result as text
@@ -43,37 +48,39 @@ class TestPipeLLMVision:
@pytest.mark.parametrize(
"pipe_code",
[
- "describe_image_number_1_only",
- "describe_image_number_2_only",
+ "describe_image_number_1_only_e2e",
+ "describe_image_number_2_only_e2e",
],
)
- async def test_describe_images_discriminate(self, pipe_code: str):
+ async def test_describe_images_multiple(self, pipe_run_mode: PipeRunMode, pipe_code: str):
"""Test the describe_image pipeline with multiple images to discriminate."""
# Execute the pipeline with an image
pipe_output = await execute_pipeline(
pipe_code=pipe_code,
+ library_path=str(Path("tests/e2e/pipelex/pipes/pipe_operators")),
inputs={
- "image_a": ImageContent(url=LLMVisionTestCases.PATH_IMG_GANTT_1),
- "image_b": ImageContent(url=LLMVisionTestCases.PATH_IMG_JPEG_3),
+ "image_a": ImageContent(url=LLMVisionTestCases.URL_CLOUDFRONT_ALAN_TURING),
+ "image_b": ImageContent(url=PipeTestCases.URL_IMG_FASHION_PHOTO_1),
},
+ pipe_run_mode=pipe_run_mode,
)
- result = pipe_output.main_stuff
- pretty_print(result, title=f"Image Description ({pipe_code})")
+ description = pipe_output.main_stuff_as_str
+ pretty_print_md(description, title=f"Image Description ({pipe_code})")
- async def test_structured_analysis_of_image_with_gantt_chart(self):
+ async def test_structured_analysis_of_image_with_gantt_chart(self, pipe_run_mode: PipeRunMode):
"""Test vision with a more complex image (Gantt chart)."""
# Execute the pipeline with a complex image
pipe_output = await execute_pipeline(
- pipe_code="vision_analysis",
+ pipe_code="vision_analysis_e2e",
inputs={
- "image": ImageContent(url=LLMVisionTestCases.PATH_IMG_GANTT_1),
+ "image": ImageContent(url=PipeTestCases.URL_IMG_GANTT_1),
},
+ pipe_run_mode=pipe_run_mode,
)
# Get the result as text
- # result = pipe_output.main_stuff_as(content_type=VisionAnalysis)
- result = pipe_output.main_stuff
+ result = pipe_output.main_stuff_as(content_type=VisionAnalysisE2E)
# Log output
pretty_print(result, title="Gantt Chart Description")
diff --git a/tests/integration/pipelex/client/test_client.py b/tests/integration/pipelex/client/test_client.py
index 77f3a6807..fb66cb352 100644
--- a/tests/integration/pipelex/client/test_client.py
+++ b/tests/integration/pipelex/client/test_client.py
@@ -5,7 +5,7 @@ from pipelex import pretty_print
from pipelex.client.client import PipelexClient
from pipelex.client.protocol import PipelineState
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_factory import StuffFactory
diff --git a/tests/integration/pipelex/cogt/test_data.py b/tests/integration/pipelex/cogt/test_data.py
index bd13b9b6d..bebe14c97 100644
--- a/tests/integration/pipelex/cogt/test_data.py
+++ b/tests/integration/pipelex/cogt/test_data.py
@@ -90,6 +90,7 @@ class LLMVisionTestCases:
class LLMTestConstants:
USER_TEXT_SHORT = "In one sentence, who is Bill Gates?"
+ USER_TEXT_TO_EXTRACT_PERSON = "It's Robert, the nice plumber, he turns 57 next week."
# USER_TEXT_SHORT = "What's the biggest football match tonight in Europe?"
PROMPT_TEMPLATE_TEXT = "Can you give one example of flower which is {color} in color ?"
PROMPT_COLOR_EXAMPLES: ClassVar[list[str]] = [
@@ -125,14 +126,14 @@ How tall do you think he was when he was 12? and at 15?
[
("name: John, age: 30", Person(name="John", age=30)),
# ("Betty Draper, 51", Person(name="Betty Draper", age=51)),
- ("Whiskers, the cat", Pet(species=PetSpecies.CAT, name="Whiskers")),
- ("Whiskers, the dog", Pet(species=PetSpecies.DOG, name="Whiskers")),
+ ("Whiskers, a very nice cat", Pet(species=PetSpecies.CAT, name="Whiskers")),
+ ("Whiskers, a cute little dog", Pet(species=PetSpecies.DOG, name="Whiskers")),
],
[
# ("name: Alice, age: 25", Person(name="Alice", age=25)),
("My sister's plumber, Bob Smith, is 42", Employee(name="Bob Smith", age=42, job="plumber")),
- ("Fluffy, the hamster", Pet(species=PetSpecies.HAMSTER, name="Fluffy")),
- ("Rex, the dog", Pet(species=PetSpecies.DOG, name="Rex")),
+ ("Fluffy is a funny hamster", Pet(species=PetSpecies.HAMSTER, name="Fluffy")),
+ ("Rex is a big black dog", Pet(species=PetSpecies.DOG, name="Rex")),
],
]
diff --git a/tests/integration/pipelex/cogt/test_extract.py b/tests/integration/pipelex/cogt/test_extract.py
index b277a69ca..3972f840c 100644
--- a/tests/integration/pipelex/cogt/test_extract.py
+++ b/tests/integration/pipelex/cogt/test_extract.py
@@ -14,6 +14,7 @@ from tests.cases import ImageTestCases, PDFTestCases
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
@pytest.mark.usefixtures("routing_profile_override")
[email protected]("ignore:Accessing the 'model_fields' attribute on the instance is deprecated:DeprecationWarning")
class TestExtract:
@pytest.mark.parametrize("file_path", PDFTestCases.DOCUMENT_FILE_PATHS)
async def test_extract_pdf_path(self, extract_handle: str, file_path: str):
diff --git a/tests/integration/pipelex/cogt/test_llm_gen_object.py b/tests/integration/pipelex/cogt/test_llm_gen_object.py
index 768df8a78..c00c72fd8 100644
--- a/tests/integration/pipelex/cogt/test_llm_gen_object.py
+++ b/tests/integration/pipelex/cogt/test_llm_gen_object.py
@@ -63,7 +63,7 @@ class TestLLMGenObject:
pytest.skip(f"'{llm_worker.desc}' does not support object generation")
expected_class = expected_instance.__class__
output = await llm_worker.gen_object(llm_job=llm_job, schema=expected_class)
- pretty_print(output)
+ pretty_print(output, title=f"Output from {llm_preset_id}")
assert isinstance(output, expected_class)
assert output.model_dump(serialize_as_any=True) == expected_instance.model_dump(serialize_as_any=True)
diff --git a/tests/integration/pipelex/cogt/test_llm_inference.py b/tests/integration/pipelex/cogt/test_llm_inference.py
index 99783989d..b8c4ea571 100644
--- a/tests/integration/pipelex/cogt/test_llm_inference.py
+++ b/tests/integration/pipelex/cogt/test_llm_inference.py
@@ -40,7 +40,7 @@ class TestLLMInference:
llm_job = LLMJobFactory.make_llm_job(
llm_prompt=LLMPrompt(
system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
+ user_text=LLMTestConstants.USER_TEXT_TO_EXTRACT_PERSON,
),
llm_job_params=llm_job_params,
)
diff --git a/tests/integration/pipelex/core/memory/test_working_memory_factory.py b/tests/integration/pipelex/core/memory/test_working_memory_factory.py
index 89a73aae1..429ac320c 100644
--- a/tests/integration/pipelex/core/memory/test_working_memory_factory.py
+++ b/tests/integration/pipelex/core/memory/test_working_memory_factory.py
@@ -1,6 +1,6 @@
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Callable
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
@@ -12,7 +12,8 @@ if TYPE_CHECKING:
class TestWorkingMemoryFactory:
- def test_make_from_compact_memory_with_text_content(self):
+ def test_make_from_compact_memory_with_text_content(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
pipeline_inputs: PipelineInputs = {
"text_item": {
"concept": NativeConceptCode.TEXT,
@@ -30,7 +31,8 @@ class TestWorkingMemoryFactory:
assert isinstance(stuff.content, TextContent)
assert stuff.content.text == "Hello, world!"
- def test_make_from_compact_memory_with_complex_nested_content(self):
+ def test_make_from_compact_memory_with_complex_nested_content(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test deserialization of compact memory with complex nested structured content."""
pipeline_inputs: PipelineInputs = {
"complex_page": {
@@ -111,7 +113,8 @@ class TestWorkingMemoryFactory:
assert page_view.url == "mock_url"
assert page_view.caption == "Full page screenshot"
- def test_make_from_compact_memory_empty(self):
+ def test_make_from_compact_memory_empty(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test deserialization of empty compact memory."""
pipeline_inputs: PipelineInputs = {}
@@ -120,7 +123,8 @@ class TestWorkingMemoryFactory:
assert working_memory is not None
assert len(working_memory.root) == 0
- def test_make_from_compact_memory_multiple_items(self):
+ def test_make_from_compact_memory_multiple_items(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test deserialization of compact memory with multiple items."""
pipeline_inputs: PipelineInputs = {
"text1": {
diff --git a/tests/integration/pipelex/fixtures/llm_fixtures.py b/tests/integration/pipelex/fixtures/llm_fixtures.py
index 024229b17..e8f76b086 100644
--- a/tests/integration/pipelex/fixtures/llm_fixtures.py
+++ b/tests/integration/pipelex/fixtures/llm_fixtures.py
@@ -2,6 +2,7 @@
import pytest
+from pipelex import log
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.hub import get_model_deck
from tests.integration.pipelex.fixtures.routing_fixtures import ALL_BACKENDS, check_backend_supports_model
@@ -18,6 +19,25 @@ def is_llm_handle_supported_by_enabled_backends(llm_handle: str) -> bool:
return any(check_backend_supports_model(backend, llm_handle) for backend in ALL_BACKENDS)
+def is_llm_preset_supported(llm_preset_id: str) -> bool:
+ """Check if an LLM preset is supported by at least one enabled backend."""
+ llm_setting = get_model_deck().get_llm_setting(llm_choice=llm_preset_id)
+ model_handle = llm_setting.model
+ model_deck = get_model_deck()
+ inference_model = model_deck.get_optional_inference_model(model_handle=model_handle)
+ if inference_model is None:
+ return False
+ log.debug(f"Inference model found!! {inference_model}")
+ return True
+
+
+def is_llm_preset_supported_by_enabled_backends(llm_preset_id: str) -> bool:
+ """Check if an LLM preset is supported by at least one enabled backend."""
+ llm_setting = get_model_deck().get_llm_setting(llm_choice=llm_preset_id)
+ # return is_llm_handle_supported_by_enabled_backends(llm_setting.model)
+ return any(check_backend_supports_model(backend, llm_setting.model) for backend in ALL_BACKENDS)
+
+
# ================================================================================================
# LLM Handles by Manufacturer
# each can be supported by multiple backends
@@ -43,6 +63,7 @@ ANTHROPIC_MODELS = [
"claude-4.5-haiku",
"claude-4.5-sonnet",
"claude-opus-4",
+ "claude-4.5-opus",
]
# --- DeepSeek Models ----------------------------------------------------------------------------
@@ -58,6 +79,7 @@ GOOGLE_MODELS = [
"gemini-2.5-flash-lite",
"gemini-2.5-pro",
"gemini-flash-1.5-8b",
+ "gemini-3.0-pro",
]
# --- Groq Models --------------------------------------------------------------------------------
@@ -118,6 +140,8 @@ OPENAI_MODELS = [
"gpt-5-chat",
"gpt-5-mini",
"gpt-5-nano",
+ "gpt-5.1",
+ "gpt-5.1-chat",
"o1",
"o1-mini",
"o3",
@@ -159,18 +183,6 @@ ALL_LLM_HANDLES = [
]
- params=[
- # "llm_for_testing_gen_text",
- # "llm_for_testing_gen_object",
- "llm_for_creative_writing",
- ],
-)
-def llm_preset_id(request: pytest.FixtureRequest) -> str:
- assert isinstance(request.param, str)
- return request.param
-
-
@pytest.fixture(
params=[
LLMJobParams(
@@ -198,6 +210,23 @@ def llm_handle(request: pytest.FixtureRequest) -> str:
return llm_handle_param
+ params=[
+ # "llm_for_testing_gen_text",
+ # "llm_for_testing_gen_object",
+ "llm_for_creative_writing",
+ ],
+)
+def llm_preset_id(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ llm_preset_id_param = request.param
+ if not is_llm_preset_supported(llm_preset_id=llm_preset_id_param):
+ pytest.skip(f"LLM preset '{llm_preset_id_param}' not supported by any enabled backend")
+ # if not is_llm_preset_supported_by_enabled_backends(llm_preset_id=llm_preset_id_param):
+ # pytest.skip(f"LLM preset '{llm_preset_id_param}' not supported by any enabled backend")
+ return llm_preset_id_param
+
+
@pytest.fixture(
params=[
# "gpt-4o",
diff --git a/tests/integration/pipelex/fixtures/routing_fixtures.py b/tests/integration/pipelex/fixtures/routing_fixtures.py
index 58a88a598..ddbda05d8 100644
--- a/tests/integration/pipelex/fixtures/routing_fixtures.py
+++ b/tests/integration/pipelex/fixtures/routing_fixtures.py
@@ -7,8 +7,8 @@ from pathlib import Path
import pytest
from pytest import MonkeyPatch
-from pipelex.config import ConfigPaths
from pipelex.hub import get_console
+from pipelex.system.configuration.configs import ConfigPaths
from pipelex.tools.misc.toml_utils import load_toml_from_path, load_toml_with_tomlkit, save_toml_to_path
# ================================================================================================
diff --git a/tests/integration/pipelex/kit/conftest.py b/tests/integration/pipelex/kit/conftest.py
new file mode 100644
index 000000000..09ddd2219
--- /dev/null
+++ b/tests/integration/pipelex/kit/conftest.py
@@ -0,0 +1,7 @@
+import pytest
+
+
[email protected](params=["pipelex_language", "coding_standards"])
+def agent_set(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
diff --git a/tests/integration/pipelex/kit/test_cursor_export.py b/tests/integration/pipelex/kit/test_cursor_export.py
deleted file mode 100644
index 2a74c6243..000000000
--- a/tests/integration/pipelex/kit/test_cursor_export.py
+++ /dev/null
@@ -1,46 +0,0 @@
-from pathlib import Path
-
-from pipelex.kit.cursor_export import export_cursor_rules
-from pipelex.kit.index_loader import load_index
-
-
-class TestCursorExport:
- """Test Cursor rules export functionality."""
-
- def test_export_cursor_rules_dry_run(self, tmp_path: Path):
- """Test Cursor export in dry-run mode."""
- idx = load_index()
- repo_root = tmp_path
-
- # Dry run should not create files
- export_cursor_rules(repo_root, idx, dry_run=True)
- cursor_rules_dir = repo_root / ".cursor" / "rules"
- assert not cursor_rules_dir.exists() or len(list(cursor_rules_dir.iterdir())) == 0
-
- def test_export_cursor_rules_creates_mdc_files(self, tmp_path: Path):
- """Test that Cursor export creates .mdc files."""
- idx = load_index()
- repo_root = tmp_path
-
- export_cursor_rules(repo_root, idx, dry_run=False)
-
- # Verify output directory exists and contains .mdc files
- cursor_rules_dir = repo_root / ".cursor" / "rules"
- assert cursor_rules_dir.exists()
- mdc_files = list(cursor_rules_dir.glob("*.mdc"))
- assert len(mdc_files) > 0, "Expected .mdc files to be created"
-
- def test_export_cursor_rules_have_front_matter(self, tmp_path: Path):
- """Test that exported .mdc files have YAML front-matter."""
- idx = load_index()
- repo_root = tmp_path
-
- export_cursor_rules(repo_root, idx, dry_run=False)
-
- # Check first .mdc file for front-matter
- cursor_rules_dir = repo_root / ".cursor" / "rules"
- mdc_files = list(cursor_rules_dir.glob("*.mdc"))
- if mdc_files:
- content = mdc_files[0].read_text(encoding="utf-8")
- assert content.startswith("---\n"), "Expected YAML front-matter to start with ---"
- assert "---\n" in content[4:], "Expected YAML front-matter to end with ---"
diff --git a/tests/integration/pipelex/kit/test_cursor_rules_update.py b/tests/integration/pipelex/kit/test_cursor_rules_update.py
new file mode 100644
index 000000000..d5a08a82d
--- /dev/null
+++ b/tests/integration/pipelex/kit/test_cursor_rules_update.py
@@ -0,0 +1,65 @@
+from pathlib import Path
+
+import pytest
+
+from pipelex.kit.cursor_rules import update_cursor_rules
+from pipelex.kit.exceptions import KitError
+from pipelex.kit.index_loader import load_index
+
+
+class TestUpdateCursorRules:
+ def test_update_cursor_rules_dry_run(self, tmp_path: Path, agent_set: str):
+ kit_index = load_index()
+ repo_root = tmp_path
+
+ update_cursor_rules(repo_root, kit_index, agent_set=agent_set, dry_run=True)
+ cursor_rules_dir = repo_root / ".cursor" / "rules"
+ assert not cursor_rules_dir.exists() or len(list(cursor_rules_dir.iterdir())) == 0
+
+ def test_update_cursor_rules_creates_mdc_files(self, tmp_path: Path, agent_set: str):
+ kit_index = load_index()
+ repo_root = tmp_path
+
+ update_cursor_rules(repo_root, kit_index, agent_set=agent_set, dry_run=False)
+
+ cursor_rules_dir = repo_root / ".cursor" / "rules"
+ assert cursor_rules_dir.exists()
+ mdc_files = list(cursor_rules_dir.glob("*.mdc"))
+ assert len(mdc_files) > 0, "Expected .mdc files to be created"
+
+ def test_update_cursor_rules_have_front_matter(self, tmp_path: Path, agent_set: str):
+ kit_index = load_index()
+ repo_root = tmp_path
+
+ update_cursor_rules(repo_root, kit_index, agent_set=agent_set, dry_run=False)
+
+ # Check first .mdc file for front-matter
+ cursor_rules_dir = repo_root / ".cursor" / "rules"
+ mdc_files = list(cursor_rules_dir.glob("*.mdc"))
+ if mdc_files:
+ content = mdc_files[0].read_text(encoding="utf-8")
+ assert content.startswith("---\n"), "Expected YAML front-matter to start with ---"
+ assert "---\n" in content[4:], "Expected YAML front-matter to end with ---"
+
+ def test_update_cursor_rules_with_agent_set(self, tmp_path: Path, agent_set: str):
+ kit_index = load_index()
+ repo_root = tmp_path
+
+ update_cursor_rules(repo_root, kit_index, agent_set=agent_set, dry_run=False)
+
+ cursor_rules_dir = repo_root / ".cursor" / "rules"
+ exported_files: set[str] = set()
+ for mdc_file in cursor_rules_dir.glob("*.mdc"):
+ exported_files.add(mdc_file.name)
+
+ expected_files: set[str] = set()
+ for file_name in kit_index.agent_rules.sets[agent_set]:
+ expected_files.add(f"{file_name.removesuffix('.md')}.mdc")
+
+ assert exported_files == expected_files
+
+ def test_update_cursor_rules_invalid_agent_set(self, tmp_path: Path):
+ kit_index = load_index()
+
+ with pytest.raises(KitError, match=r"Agent set 'unknown' not found in index.toml"):
+ update_cursor_rules(tmp_path, kit_index, agent_set="unknown", dry_run=True)
diff --git a/tests/integration/pipelex/kit/test_idempotency.py b/tests/integration/pipelex/kit/test_idempotency.py
new file mode 100644
index 000000000..c2359779c
--- /dev/null
+++ b/tests/integration/pipelex/kit/test_idempotency.py
@@ -0,0 +1,114 @@
+"""Test idempotency of single-file agent rules generation."""
+
+from pathlib import Path
+
+from pipelex.kit.index_loader import load_index
+from pipelex.kit.single_file_agent_rules import update_single_file_agent_rules
+
+
+class TestIdempotency:
+ def test_update_single_file_agent_rules_is_idempotent_empty_file(self, tmp_path: Path, agent_set: str):
+ """Test that running update twice on an empty file produces identical results."""
+ idx = load_index()
+
+ repo_root = tmp_path / "repo"
+ repo_root.mkdir()
+ target_file = repo_root / "test_target.md"
+
+ # Start with empty file
+ target_file.write_text("", encoding="utf-8")
+
+ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
+
+ # First update
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ first_result = target_file.read_text(encoding="utf-8")
+
+ # Second update (should be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ second_result = target_file.read_text(encoding="utf-8")
+
+ # Third update (should also be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ third_result = target_file.read_text(encoding="utf-8")
+
+ assert first_result == second_result, "Second run produced different output than first run"
+ assert second_result == third_result, "Third run produced different output than second run"
+
+ def test_update_single_file_agent_rules_is_idempotent_file_without_h1(self, tmp_path: Path, agent_set: str):
+ """Test that running update twice on a file without H1 produces identical results."""
+ idx = load_index()
+
+ repo_root = tmp_path / "repo"
+ repo_root.mkdir()
+ target_file = repo_root / "test_target.md"
+
+ # Start with file that has content but no H1
+ target_file.write_text("Some existing content without heading.\n", encoding="utf-8")
+
+ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
+
+ # First update
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ first_result = target_file.read_text(encoding="utf-8")
+
+ # Second update (should be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ second_result = target_file.read_text(encoding="utf-8")
+
+ # Third update (should also be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ third_result = target_file.read_text(encoding="utf-8")
+
+ assert first_result == second_result, "Second run produced different output than first run"
+ assert second_result == third_result, "Third run produced different output than second run"
+
+ def test_update_single_file_agent_rules_is_idempotent_file_with_h1(self, tmp_path: Path, agent_set: str):
+ """Test that running update twice on a file with existing H1 produces identical results without duplicate headings."""
+ idx = load_index()
+
+ repo_root = tmp_path / "repo"
+ repo_root.mkdir()
+ target_file = repo_root / "test_target.md"
+
+ # Start with file that has an H1 heading already
+ target_file.write_text("# My Existing Heading\n\nSome existing content.\n", encoding="utf-8")
+
+ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
+
+ # First update
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ first_result = target_file.read_text(encoding="utf-8")
+
+ # Verify no heading_1 was added inside markers (since file already has H1)
+ assert first_result.count("# Pipelex Coding Rules") == 0, "Should not add heading_1 when file already has H1"
+ assert first_result.count("# My Existing Heading") == 1, "Original H1 should be preserved"
+
+ # Second update (should be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ second_result = target_file.read_text(encoding="utf-8")
+
+ # Third update (should also be identical)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
+ third_result = target_file.read_text(encoding="utf-8")
+
+ assert first_result == second_result, "Second run produced different output than first run"
+ assert second_result == third_result, "Third run produced different output than second run"
diff --git a/tests/integration/pipelex/kit/test_merged_rules.py b/tests/integration/pipelex/kit/test_merged_rules.py
index fdad74a6b..0ab6c993b 100644
--- a/tests/integration/pipelex/kit/test_merged_rules.py
+++ b/tests/integration/pipelex/kit/test_merged_rules.py
@@ -3,7 +3,7 @@
import pytest
from pipelex.kit.index_loader import load_index
-from pipelex.kit.targets_update import build_merged_rules
+from pipelex.kit.single_file_agent_rules import build_merged_rules
class TestMergedRules:
diff --git a/tests/integration/pipelex/kit/test_targets_update.py b/tests/integration/pipelex/kit/test_update_single_file_agent_rules.py
similarity index 67%
rename from tests/integration/pipelex/kit/test_targets_update.py
rename to tests/integration/pipelex/kit/test_update_single_file_agent_rules.py
index 36a23917d..e312353db 100644
--- a/tests/integration/pipelex/kit/test_targets_update.py
+++ b/tests/integration/pipelex/kit/test_update_single_file_agent_rules.py
@@ -2,14 +2,11 @@ from pathlib import Path
from pipelex.kit.index_loader import load_index
from pipelex.kit.markers import find_span
-from pipelex.kit.targets_update import build_merged_rules, update_targets
+from pipelex.kit.single_file_agent_rules import update_single_file_agent_rules
-class TestTargetsUpdate:
- """Test target file updating functionality."""
-
- def test_update_targets_dry_run(self, tmp_path: Path):
- """Test updating targets in dry-run mode."""
+class TestUpdateSingleFileAgentRules:
+ def test_update_single_file_agent_rules_dry_run(self, tmp_path: Path, agent_set: str):
idx = load_index()
# Create a temporary repo root with a target file
@@ -18,20 +15,20 @@ class TestTargetsUpdate:
target_file = repo_root / "test_target.md"
target_file.write_text("# Test\n\nOriginal content\n", encoding="utf-8")
- merged_rules = build_merged_rules(idx)
-
# Create a test target
test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
original_content = target_file.read_text(encoding="utf-8")
# Dry run should not modify file
- update_targets(repo_root, merged_rules, test_targets, dry_run=True, diff=False, backup=None)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=True, diff=False, backup=None
+ )
assert target_file.read_text(encoding="utf-8") == original_content
- def test_update_targets_inserts_with_markers(self, tmp_path: Path):
- """Test that update_targets inserts content with markers."""
+ def test_update_single_file_agent_rules_inserts_with_markers(self, tmp_path: Path, agent_set: str):
+ """Test that agent rules update inserts content with markers."""
idx = load_index()
repo_root = tmp_path / "repo"
@@ -39,11 +36,11 @@ class TestTargetsUpdate:
target_file = repo_root / "test_target.md"
target_file.write_text("# Test\n\nOriginal content\n", encoding="utf-8")
- merged_rules = build_merged_rules(idx)
-
test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
- update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=None)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
updated_content = target_file.read_text(encoding="utf-8")
target = test_targets["test"]
@@ -56,8 +53,8 @@ class TestTargetsUpdate:
span = find_span(updated_content, target.marker_begin, target.marker_end)
assert span is not None
- def test_update_targets_replaces_existing_markers(self, tmp_path: Path):
- """Test that update_targets replaces content between existing markers."""
+ def test_update_single_file_agent_rules_replaces_existing_markers(self, tmp_path: Path, agent_set: str):
+ """Test that agent rules update replaces content between existing markers."""
idx = load_index()
repo_root = tmp_path / "repo"
@@ -70,8 +67,6 @@ class TestTargetsUpdate:
initial_content = f"# Test\n\n{marker_begin}\nOld content\n{marker_end}\n"
target_file.write_text(initial_content, encoding="utf-8")
- merged_rules = build_merged_rules(idx)
-
test_targets = {
"test": idx.agent_rules.targets["agents"].model_copy(
update={
@@ -82,7 +77,9 @@ class TestTargetsUpdate:
)
}
- update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=None)
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=None
+ )
updated_content = target_file.read_text(encoding="utf-8")
@@ -93,8 +90,8 @@ class TestTargetsUpdate:
# Verify old content is replaced
assert "Old content" not in updated_content
- def test_update_targets_creates_backup(self, tmp_path: Path):
- """Test that update_targets creates backup files when requested."""
+ def test_update_single_file_agent_rules_creates_backup(self, tmp_path: Path, agent_set: str):
+ """Test that agent rules update creates backup files when requested."""
idx = load_index()
repo_root = tmp_path / "repo"
@@ -103,11 +100,11 @@ class TestTargetsUpdate:
original_content = "# Test\n\nOriginal content\n"
target_file.write_text(original_content, encoding="utf-8")
- merged_rules = build_merged_rules(idx)
-
test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}
- update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=".bak")
+ update_single_file_agent_rules(
+ repo_root=repo_root, kit_index=idx, agent_set=agent_set, targets=test_targets, dry_run=False, diff=False, backup=".bak"
+ )
# Verify backup exists
backup_file = target_file.with_suffix(target_file.suffix + ".bak")
diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py
index 5888f70d3..a896062a2 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py
@@ -1,13 +1,18 @@
from __future__ import annotations
-from typing import cast
+from pathlib import Path
+from typing import TYPE_CHECKING, cast
import pytest
from pytest import FixtureRequest
+if TYPE_CHECKING:
+ from collections.abc import Callable
+
from pipelex import pretty_print
-from pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -27,9 +32,13 @@ from pipelex.pipeline.job_metadata import JobMetadata
class TestPipeBatchSimple:
"""Simple integration test for PipeBatch controller."""
- async def test_simple_batch_processing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
- """Test PipeBatch with a simple batch processing scenario."""
- # Create PipeBatch instance - it will call the uppercase_transformer pipe from the PLX
+ async def test_simple_batch_processing(
+ self,
+ request: FixtureRequest,
+ pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
+ ):
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_batch")])
domain = "test_integration"
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="TestConcept1",
@@ -51,7 +60,6 @@ class TestPipeBatchSimple:
branch_pipe_code="uppercase_transformer", # This exists in the PLX file
inputs={
"text_list": concept_1.concept_string,
- "text_item": concept_2.concept_string,
},
output=concept_2.concept_string,
input_list_name="text_list",
diff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx b/tests/integration/pipelex/pipes/controller/pipe_batch/uppercase_transformer.plx
similarity index 93%
rename from tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx
rename to tests/integration/pipelex/pipes/controller/pipe_batch/uppercase_transformer.plx
index dbbfd8445..d34b6b104 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx
+++ b/tests/integration/pipelex/pipes/controller/pipe_batch/uppercase_transformer.plx
@@ -1,4 +1,4 @@
-domain = "test_integration"
+domain = "test_integration1"
description = "Simple pipes for testing PipeBatch integration"
[concept]
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition.py b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition.py
rename to tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition.py
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.plx b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.plx
rename to tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_1.plx
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.plx b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.plx
rename to tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_2.plx
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.plx b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.plx
rename to tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.plx
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.py b/tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.py
rename to tests/integration/pipelex/pipes/controller/pipe_condition/pipe_condition_complex.py
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py
index e5b2bb99b..168b19211 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py
@@ -1,5 +1,7 @@
"""Complex integration test for PipeCondition controller with multiple inputs and nested conditions."""
+from collections.abc import Callable
+from pathlib import Path
from typing import Literal, cast
import pytest
@@ -8,7 +10,7 @@ from pytest import FixtureRequest
from pipelex import pretty_print
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.exceptions import PipeRunInputsError
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import get_pipe_router, get_required_pipe
@@ -16,7 +18,7 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition_complex import (
+from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition_complex import (
DocumentRequest,
UserProfile,
)
@@ -27,8 +29,11 @@ from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition_complex
@pytest.mark.llm
@pytest.mark.asyncio(loop_scope="class")
class TestPipeConditionComplex:
- async def test_technical_urgent_routing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_technical_urgent_routing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test technical document with urgent priority routing."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create complex input data
doc_request = DocumentRequest(document_type="technical", priority="urgent", language="english", complexity="high")
user_profile = UserProfile(user_level="expert", department="technical")
@@ -77,8 +82,11 @@ class TestPipeConditionComplex:
if pipe_run_mode != PipeRunMode.DRY:
assert "URGENT_TECHNICAL_PROCESSED" in final_result.content.text
- async def test_business_finance_routing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_business_finance_routing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test business document for finance department routing."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(document_type="business", priority="normal", language="english", complexity="medium")
user_profile = UserProfile(user_level="intermediate", department="finance")
@@ -122,8 +130,9 @@ class TestPipeConditionComplex:
assert isinstance(final_result.content, TextContent)
assert "FINANCE_BUSINESS_PROCESSED" in final_result.content.text
- async def test_legal_complex_routing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_legal_complex_routing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]):
"""Test complex legal document routing."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(document_type="legal", priority="normal", language="spanish", complexity="high")
user_profile = UserProfile(user_level="expert", department="legal")
@@ -167,8 +176,11 @@ class TestPipeConditionComplex:
assert isinstance(final_result.content, TextContent)
assert "COMPLEX_LEGAL_PROCESSED" in final_result.content.text
- async def test_technical_expert_high_complexity_routing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_technical_expert_high_complexity_routing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test technical document with expert user and high complexity."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(
document_type="technical",
priority="normal", # Not urgent, so should check user level + complexity
@@ -221,8 +233,9 @@ class TestPipeConditionComplex:
assert "EXPERT_TECHNICAL_PROCESSED" in final_result.content.text
# DRY RUN TESTS
- async def test_complex_pipeline_dry_run_success(self, request: FixtureRequest):
+ async def test_complex_pipeline_dry_run_success(self, request: FixtureRequest, load_test_library: Callable[[list[Path]], None]):
"""Test complex pipeline dry run with valid inputs - should succeed."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(document_type="business", priority="urgent", language="english", complexity="low")
user_profile = UserProfile(user_level="beginner", department="marketing")
@@ -263,8 +276,9 @@ class TestPipeConditionComplex:
assert pipe_output is not None
assert pipe_output.working_memory is not None
- async def test_complex_pipeline_dry_run_missing_inputs(self, request: FixtureRequest):
+ async def test_complex_pipeline_dry_run_missing_inputs(self, request: FixtureRequest, load_test_library: Callable[[list[Path]], None]):
"""Test complex pipeline dry run with missing inputs - should fail with PipeRouterError."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(document_type="technical", priority="urgent", language="english", complexity="high")
doc_stuff = StuffFactory.make_stuff(
@@ -322,7 +336,9 @@ class TestPipeConditionComplex:
expected_output_contains: str,
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
doc_request = DocumentRequest(
document_type=doc_type,
priority=priority,
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
index 801f2504c..f8e75b03e 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_simple.py
@@ -1,16 +1,17 @@
"""Simple integration test for PipeCondition controller."""
-from typing import cast
+from pathlib import Path
+from typing import Callable, cast
import pytest
from pytest import FixtureRequest
from pipelex import pretty_print
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.exceptions import PipeRunInputsError
+from pipelex.core.pipes.inputs.exceptions import PipeRunInputsError
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import get_pipe_router, get_required_pipe
@@ -21,15 +22,18 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.test_pipelines.pipe_controllers.pipe_condition.pipe_condition import CategoryInput
+from tests.integration.pipelex.pipes.controller.pipe_condition.pipe_condition import CategoryInput
@pytest.mark.dry_runnable
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
class TestPipeConditionSimple:
- async def test_condition_long_text_processing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_condition_long_text_processing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test PipeCondition with long text that should trigger capitalize_long_text pipe."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
pipe_condition_blueprint = PipeConditionBlueprint(
description="Text length condition for testing",
inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},
@@ -97,8 +101,11 @@ class TestPipeConditionSimple:
assert final_result_in_memory.content.text == "LONG: HELLO WORLD"
assert f"{final_result_in_memory.concept.domain}.{final_result_in_memory.concept.code}" == f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"
- async def test_condition_short_text_processing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_condition_short_text_processing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test PipeCondition with short text that should trigger add_prefix_short_text pipe."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create PipeCondition instance - pipes are loaded from PLX files
pipe_condition_blueprint = PipeConditionBlueprint(
description="Text length condition for short text testing",
@@ -167,8 +174,9 @@ class TestPipeConditionSimple:
assert final_result_in_memory.content.text == "SHORT: hi"
assert f"{final_result_in_memory.concept.domain}.{final_result_in_memory.concept.code}" == f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"
- async def test_condition_dry_run_success(self, request: FixtureRequest):
+ async def test_condition_dry_run_success(self, request: FixtureRequest, load_test_library: Callable[[list[Path]], None]):
"""Test PipeCondition dry run with valid inputs using real pipe - should succeed."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create test data using CategoryInput for the real pipe basic_condition_by_category_2
category_input = CategoryInput(category="small")
input_data_stuff = StuffFactory.make_stuff(
@@ -213,8 +221,9 @@ class TestPipeConditionSimple:
assert isinstance(original_input.content, CategoryInput)
assert original_input.content.category == "small"
- async def test_condition_dry_run_missing_inputs_failure(self, request: FixtureRequest):
+ async def test_condition_dry_run_missing_inputs_failure(self, request: FixtureRequest, load_test_library: Callable[[list[Path]], None]):
"""Test PipeCondition dry run with missing inputs using real pipe - should fail with PipeRouterError."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create empty working memory - missing required input
empty_working_memory = WorkingMemoryFactory.make_empty()
@@ -236,8 +245,9 @@ class TestPipeConditionSimple:
assert "input_data" in error.missing_inputs
assert "Missing required inputs" in str(error)
- async def test_condition_dry_run_medium_category_validation(self, request: FixtureRequest):
+ async def test_condition_dry_run_medium_category_validation(self, request: FixtureRequest, load_test_library: Callable[[list[Path]], None]):
"""Test PipeCondition dry run with medium category - should validate the 'medium' branch."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Create test data using CategoryInput for the medium category
category_input = CategoryInput(category="medium")
input_data_stuff = StuffFactory.make_stuff(
diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
index ec5f9a9fb..6618e394e 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_validation.py
@@ -1,3 +1,6 @@
+from pathlib import Path
+from typing import Callable
+
from pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactory
from pipelex.hub import get_concept_library
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
@@ -8,8 +11,9 @@ from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
class TestPipeConditionValidation:
"""Tests for PipeCondition validate_inputs method"""
- def test_pipe_condition_creation(self):
+ def test_pipe_condition_creation(self, load_test_library: Callable[[list[Path]], None]):
"""Test basic PipeCondition creation"""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
domain = "test_domain"
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="TestConcept",
@@ -44,13 +48,14 @@ class TestPipeConditionValidation:
assert pipe_condition.code == "test_condition"
assert pipe_condition.domain == domain
assert len(pipe_condition.outcome_map) == 2
- assert pipe_condition.expression == "input_var"
+ assert pipe_condition.expression == "{{ input_var }}"
assert pipe_condition.default_outcome == "default_pipe"
concept_library.teardown()
- def test_pipe_condition_expression_template_vs_expression(self):
+ def test_pipe_condition_expression_template_vs_expression(self, load_test_library: Callable[[list[Path]], None]):
"""Test that both expression_template and expression formats work"""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])
# Test with expression_template
domain = "test_domain"
concept_library = get_concept_library()
@@ -100,6 +105,6 @@ class TestPipeConditionValidation:
)
# Both should have the same applied expression template format
- assert pipe_condition_template.applied_expression_template == "{{ var }}"
- assert pipe_condition_expr.applied_expression_template == "{{ var }}"
+ assert pipe_condition_template.expression == "{{ var }}"
+ assert pipe_condition_expr.expression == "{{ var }}"
concept_library.teardown()
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.plx b/tests/integration/pipelex/pipes/controller/pipe_condition/text_length_condition.plx
similarity index 95%
rename from tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.plx
rename to tests/integration/pipelex/pipes/controller/pipe_condition/text_length_condition.plx
index 160053049..96cde5d9d 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.plx
+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/text_length_condition.plx
@@ -1,4 +1,4 @@
-domain = "test_integration"
+domain = "test_integration2"
description = "Test pipes for PipeCondition based on text length"
[pipe]
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.plx b/tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_text_analysis.plx
similarity index 96%
rename from tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.plx
rename to tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_text_analysis.plx
index 1c3ad3e6e..550ac8d12 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.plx
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_text_analysis.plx
@@ -1,4 +1,4 @@
-domain = "test_integration"
+domain = "test_integration3"
description = "Test pipes for PipeParallel text analysis"
[pipe]
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel.py
rename to tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel.py
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.plx b/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.plx
rename to tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.plx
diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py
index 407af4bb5..aa313c6c2 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py
@@ -1,10 +1,11 @@
-from typing import cast
+from pathlib import Path
+from typing import Callable, cast
import pytest
from pytest import FixtureRequest
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff_factory import StuffFactory
@@ -21,8 +22,9 @@ from pipelex.pipeline.job_metadata import JobMetadata
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
class TestPipeParallelSimple:
- async def test_parallel_text_analysis(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_parallel_text_analysis(self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]):
"""Test PipeParallel running three text analysis pipes in parallel."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_parallel")])
# Create PipeParallel instance - pipes are loaded from PLX files
pipe_parallel_blueprint = PipeParallelBlueprint(
description="Parallel text analysis pipeline",
@@ -137,8 +139,11 @@ class TestPipeParallelSimple:
assert isinstance(final_result.content, TextContent)
assert final_result.content.text == "The weather is beautiful today. I love sunny days and outdoor activities."
- async def test_parallel_short_text_analysis(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_parallel_short_text_analysis(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test PipeParallel with shorter text to verify consistent behavior."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_parallel")])
# Create PipeParallel instance
pipe_parallel_blueprint = PipeParallelBlueprint(
description="Parallel text analysis pipeline for short text",
diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py
index ab098d8ee..7a3fdec43 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py
@@ -1,6 +1,8 @@
+from typing import Callable
+
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.pipes.input_requirements import InputRequirements
+from pipelex.core.pipes.inputs.input_requirements import InputRequirements
from pipelex.hub import get_concept_library, get_pipe_library
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallelFactory
@@ -12,7 +14,8 @@ from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory
class TestPipeParallelValidation:
"""Tests for PipeParallel creation and basic structure"""
- def test_pipe_parallel_with_real_pipe_structure(self):
+ def test_pipe_parallel_with_real_pipe_structure(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeParallel structure with a real pipe"""
# Create a real PipeLLM that will infer inputs from the prompt template
domain = "test_domain"
@@ -93,7 +96,8 @@ class TestPipeParallelValidation:
concept_library.teardown()
- def test_pipe_parallel_creation(self):
+ def test_pipe_parallel_creation(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test basic PipeParallel creation and structure"""
# Create a simple PipeParallel with proper inputs
domain = "test_domain"
@@ -146,7 +150,8 @@ class TestPipeParallelValidation:
concept_library.teardown()
- def test_pipe_parallel_needed_inputs_structure(self):
+ def test_pipe_parallel_needed_inputs_structure(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that PipeParallel needed_inputs method can be called and returns expected structure"""
domain = "test_domain"
concept_library = get_concept_library()
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.plx b/tests/integration/pipelex/pipes/controller/pipe_sequence/capitalize_text.plx
similarity index 95%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.plx
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/capitalize_text.plx
index f5e5fdd6b..af2b8845b 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.plx
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/capitalize_text.plx
@@ -1,4 +1,4 @@
-domain = "test_integration"
+domain = "test_integration4"
description = "Test pipe for capitalizing text to uppercase"
[pipe]
diff --git a/tests/test_pipelines/discord_newsletter.plx b/tests/integration/pipelex/pipes/controller/pipe_sequence/discord_newsletter.plx
similarity index 100%
rename from tests/test_pipelines/discord_newsletter.plx
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/discord_newsletter.plx
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/discord_newsletter.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/discord_newsletter.py
new file mode 100644
index 000000000..5bd123297
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/discord_newsletter.py
@@ -0,0 +1,55 @@
+from pydantic import Field
+
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+
+class Attachment(StructuredContent):
+ """Represents a Discord message attachment"""
+
+ name: str = Field(..., description="Name of the attachment file")
+ url: str = Field(..., description="URL of the attachment")
+
+
+class Embed(StructuredContent):
+ """Represents a Discord message embed"""
+
+ title: str = Field(..., description="Title of the embed")
+ description: str = Field(..., description="Description of the embed content")
+ type: str = Field(..., description="Type of the embed (e.g., article, video)")
+
+
+class DiscordMessage(StructuredContent):
+ """Represents a Discord message within a channel"""
+
+ author: str = Field(..., description="Author of the message")
+ content: str = Field(..., description="Content of the message")
+ attachments: list[Attachment] = Field(default_factory=empty_list_factory_of(Attachment), description="List of message attachments")
+ embeds: list[Embed] = Field(default_factory=empty_list_factory_of(Embed), description="List of message embeds")
+ link: str = Field(..., description="Link to the message")
+
+
+class DiscordChannelUpdate(StructuredContent):
+ """Represents a Discord channel with its messages"""
+
+ name: str = Field(..., description="Name of the Discord channel")
+ position: int = Field(..., description="Position of the channel")
+ messages: list[DiscordMessage] = Field(default_factory=empty_list_factory_of(DiscordMessage), description="List of messages in the channel")
+
+
+class ChannelSummary(StructuredContent):
+ """Represents a summarized Discord channel for newsletter inclusion"""
+
+ channel_name: str = Field(..., description="Name of the Discord channel")
+ summary_items: list[str] = Field(..., description="Well-written summaries of the channel's activity")
+
+
+class Newsletter(StructuredContent):
+ """Represents the final newsletter content"""
+
+ weekly_summary: str = Field(..., description="200 character summary of weekly Share channel content")
+ new_members: list[str] = Field(default_factory=empty_list_factory_of(str), description="New member introductions in bullet points")
+ channel_sections: list[ChannelSummary] = Field(default_factory=empty_list_factory_of(ChannelSummary), description="Ordered channel summaries")
+ geographic_hubs: list[ChannelSummary] = Field(
+ default_factory=empty_list_factory_of(ChannelSummary), description="Geographic hub channels grouped at end"
+ )
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence.py
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence.py
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.plx b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_1.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.plx
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_1.plx
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plx b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.plx
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.plx
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plx b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.plx
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.plx
rename to tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.plx
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_dry_run.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_dry_run.py
index 4f4ce427d..d5cfab502 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_dry_run.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_dry_run.py
@@ -1,4 +1,5 @@
-from typing import TYPE_CHECKING
+from pathlib import Path
+from typing import TYPE_CHECKING, Callable
import pytest
from pytest import FixtureRequest
@@ -14,7 +15,7 @@ from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.misc.json_utils import load_json_list_from_path
-from tests.test_pipelines.discord_newsletter import ChannelSummary, DiscordChannelUpdate
+from tests.integration.pipelex.pipes.controller.pipe_sequence.discord_newsletter import ChannelSummary, DiscordChannelUpdate
if TYPE_CHECKING:
from pipelex.core.memory.working_memory import WorkingMemory
@@ -28,8 +29,10 @@ class TestPipeSequenceDryRun:
self,
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
) -> None:
"""Test that the Discord newsletter pipeline creates correct working memory with ListContent for batched inputs."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence")])
# Load the discord channel updates data from JSON
discord_channel_updates_data = load_json_list_from_path(path="tests/data/discord_newsletter/discord_sample.json")
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py
index 17cb30c39..1e11f8071 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_simple.py
@@ -1,6 +1,7 @@
"""Simple integration test for PipeSequence controller."""
-from typing import cast
+from pathlib import Path
+from typing import Callable, cast
import pytest
from pytest import FixtureRequest
@@ -8,7 +9,7 @@ from pytest import FixtureRequest
from pipelex import pretty_print
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
@@ -27,8 +28,11 @@ from pipelex.pipeline.job_metadata import JobMetadata
class TestPipeSequenceSimple:
"""Simple integration test for PipeSequence controller."""
- async def test_simple_sequence_processing(self, request: FixtureRequest, pipe_run_mode: PipeRunMode):
+ async def test_simple_sequence_processing(
+ self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]
+ ):
"""Test PipeSequence with a simple 2-step text transformation scenario."""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence")])
domain = "test_integration"
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="TestConcept1",
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py
index 4314130f0..30856bfb9 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_validation.py
@@ -1,3 +1,6 @@
+from pathlib import Path
+from typing import Callable
+
from pipelex.core.concepts.concept_factory import ConceptBlueprint, ConceptFactory
from pipelex.hub import get_concept_library
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
@@ -8,8 +11,9 @@ from pipelex.pipe_controllers.sub_pipe_factory import SubPipeBlueprint
class TestPipeSequenceValidation:
"""Tests for PipeSequence validate_inputs method"""
- def test_pipe_sequence_creation(self):
+ def test_pipe_sequence_creation(self, load_test_library: Callable[[list[Path]], None]):
"""Test basic PipeSequence creation"""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence")])
domain = "test_domain"
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="TestConcept",
@@ -47,8 +51,9 @@ class TestPipeSequenceValidation:
concept_library.teardown()
- def test_pipe_sequence_multiple_sub_pipes(self):
+ def test_pipe_sequence_multiple_sub_pipes(self, load_test_library: Callable[[list[Path]], None]):
"""Test PipeSequence with multiple sequential sub-pipes"""
+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_sequence")])
domain = "test_domain"
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="TestConcept",
diff --git a/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py b/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py
index b0ae7ea8e..df21cc580 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_extract/test_pipe_extract.py
@@ -1,11 +1,16 @@
+from typing import Callable
+
import pytest
from pipelex import pretty_print
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
+from pipelex.core.stuffs.pdf_content import PDFContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.hub import get_concept_library, get_pipe_router
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
from pipelex.pipe_operators.extract.pipe_extract_factory import PipeExtractFactory
@@ -20,8 +25,12 @@ from tests.integration.pipelex.test_data import PipeExtractTestCases
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
class TestPipeExtract:
- @pytest.fixture(scope="class", autouse=True)
- def setup(self):
+ @pytest.fixture(
+ scope="class",
+ autouse=True,
+ )
+ def setup(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
concept_library = get_concept_library()
concept_1 = ConceptFactory.make_from_blueprint(
concept_code="PageScan",
@@ -46,7 +55,7 @@ class TestPipeExtract:
pipe_extract_blueprint = PipeExtractBlueprint(
description="OCR test for image processing",
inputs={"page_scan": NativeConceptCode.IMAGE},
- output=NativeConceptCode.TEXT_AND_IMAGES,
+ output="Page[]",
page_images=True,
page_image_captions=False,
page_views=True,
@@ -61,9 +70,17 @@ class TestPipeExtract:
blueprint=pipe_extract_blueprint,
),
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
- working_memory=WorkingMemoryFactory.make_from_image(
- image_url=image_url,
- name="page_scan",
+ working_memory=WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=ConceptFactory.make(
+ concept_code="PageScan",
+ domain="extract",
+ description="Lorem Ipsum",
+ structure_class_name="PageScan",
+ ),
+ content=ImageContent(url=image_url),
+ name="page_scan",
+ ),
),
)
pipe_extract_output = await get_pipe_router().run(
@@ -86,7 +103,7 @@ class TestPipeExtract:
blueprint = PipeExtractBlueprint(
description="OCR test for PDF processing",
inputs={input_name: NativeConceptCode.PDF},
- output=NativeConceptCode.TEXT_AND_IMAGES,
+ output="Page[]",
model=extract_choice_for_pdf,
page_images=True,
page_image_captions=page_image_captions,
@@ -101,9 +118,14 @@ class TestPipeExtract:
blueprint=blueprint,
),
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
- working_memory=WorkingMemoryFactory.make_from_pdf(
- pdf_url=pdf_url,
- name=input_name,
+ working_memory=WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(
+ native_concept_code=NativeConceptCode.PDF,
+ ),
+ content=PDFContent(url=pdf_url),
+ name=input_name,
+ ),
),
)
pipe_extract_output = await get_pipe_router().run(
diff --git a/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py b/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py
index 3400299f6..ab5881e8c 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py
@@ -1,8 +1,10 @@
+from typing import Callable
+
import pytest
from pipelex import log, pretty_print
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
@@ -33,7 +35,9 @@ class TestPipeFunc:
async def test_wrap_lines_pipe_func(
self,
pipe_run_mode: PipeRunMode,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
# Sample source code to test with
sample_code = """def hello_world():
print("Hello, World!")
diff --git a/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen_run.py b/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen_run.py
index 1829b4f3e..8e3b56b9d 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen_run.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_img_gen/test_pipe_img_gen_run.py
@@ -1,9 +1,9 @@
-from typing import TYPE_CHECKING, cast
+from typing import TYPE_CHECKING, Callable, cast
import pytest
from pipelex import pretty_print
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.hub import get_pipe_router
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
from pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactory
@@ -28,7 +28,9 @@ class TestPipeImgGenRun:
img_gen_handle: str,
topic: str,
image_desc: str,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
pipe_img_gen_blueprint = PipeImgGenBlueprint(
description="Image generation test",
img_gen_prompt=image_desc,
diff --git a/tests/test_pipelines/pipe_operators/pipe_llm_vision.plx b/tests/integration/pipelex/pipes/operator/pipe_llm/pipe_llm_vision.plx
similarity index 100%
rename from tests/test_pipelines/pipe_operators/pipe_llm_vision.plx
rename to tests/integration/pipelex/pipes/operator/pipe_llm/pipe_llm_vision.plx
diff --git a/tests/test_pipelines/pipe_operators/pipe_llm_vision.py b/tests/integration/pipelex/pipes/operator/pipe_llm/pipe_llm_vision.py
similarity index 100%
rename from tests/test_pipelines/pipe_operators/pipe_llm_vision.py
rename to tests/integration/pipelex/pipes/operator/pipe_llm/pipe_llm_vision.py
diff --git a/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py b/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py
index 2c7c40ef2..7522405f3 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_llm/test_pipe_llm.py
@@ -1,10 +1,15 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pipelex import log, pretty_print
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.stuff import Stuff
-from pipelex.hub import get_class_registry, get_pipe_library, get_pipe_router
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry, get_native_concept, get_pipe_library, get_pipe_router
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint, StructuringMethod
from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
@@ -17,11 +22,13 @@ from tests.integration.pipelex.test_data import BasicStructuredDataTestCases, Pi
@pytest.mark.llm
@pytest.mark.inference
@pytest.mark.asyncio(loop_scope="class")
-class TestPipeLLM:
+class TestPipeLLMBasic:
async def test_pipe_llm_simple(
self,
pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/operator/pipe_llm")])
pipe_llm_blueprint = PipeLLMBlueprint(
description="LLM test for basic text generation",
output=NativeConceptCode.TEXT,
@@ -77,10 +84,18 @@ class TestPipeLLM:
pipe_run_mode: PipeRunMode,
llm: str,
llm_to_structure: str,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/operator/pipe_llm")])
# TODO: Add assertion on generated objects vs expected results
pretty_print(data, title="data")
- working_memory = WorkingMemoryFactory.make_from_text(text=data, name="data")
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=get_native_concept(NativeConceptCode.TEXT),
+ content=TextContent(text=data),
+ name="data",
+ ),
+ )
# Create pipe blueprint
pipe_llm_blueprint = PipeLLMBlueprint(
@@ -136,7 +151,9 @@ class TestPipeLLM:
stuff: Stuff,
attribute_paths: list[str],
pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/operator/pipe_llm")])
for attribute_path in attribute_paths:
stuff_name = attribute_path
if not stuff_name:
diff --git a/tests/test_pipelines/concepts/test_structures_basic.plx b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_basic.plx
similarity index 100%
rename from tests/test_pipelines/concepts/test_structures_basic.plx
rename to tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_basic.plx
diff --git a/tests/test_pipelines/concepts/test_structures_basic.py b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_basic.py
similarity index 100%
rename from tests/test_pipelines/concepts/test_structures_basic.py
rename to tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_basic.py
diff --git a/tests/test_pipelines/concepts/test_structures_complex.plx b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.plx
similarity index 88%
rename from tests/test_pipelines/concepts/test_structures_complex.plx
rename to tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.plx
index f35ba8018..8484412ea 100644
--- a/tests/test_pipelines/concepts/test_structures_complex.plx
+++ b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.plx
@@ -1,4 +1,4 @@
-domain = "test_structured_generations"
+domain = "test_structured_generations2"
description = "Concepts to test complex structures (dicts, unions, etc.)"
[concept]
diff --git a/tests/test_pipelines/concepts/test_structures_complex.py b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.py
similarity index 94%
rename from tests/test_pipelines/concepts/test_structures_complex.py
rename to tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.py
index fc5d60d68..3935a51ad 100644
--- a/tests/test_pipelines/concepts/test_structures_complex.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_llm/test_structures_complex.py
@@ -4,7 +4,7 @@ from pydantic import Field
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
-from tests.test_pipelines.concepts.test_structures_basic import ConceptWithOptionals, ConceptWithSimpleStructure
+from tests.integration.pipelex.pipes.operator.pipe_llm.test_structures_basic import ConceptWithOptionals, ConceptWithSimpleStructure
class ConceptWithDicts(StructuredContent):
diff --git a/tests/test_pipelines/failing_pipelines.plx b/tests/integration/pipelex/pipes/pipelines/failing_pipelines.plx
similarity index 100%
rename from tests/test_pipelines/failing_pipelines.plx
rename to tests/integration/pipelex/pipes/pipelines/failing_pipelines.plx
diff --git a/tests/test_pipelines/misc_tests/flows.plx b/tests/integration/pipelex/pipes/pipelines/flows.plx
similarity index 100%
rename from tests/test_pipelines/misc_tests/flows.plx
rename to tests/integration/pipelex/pipes/pipelines/flows.plx
diff --git a/tests/test_pipelines/misc_tests/multiplicity.plx b/tests/integration/pipelex/pipes/pipelines/multiplicity.plx
similarity index 100%
rename from tests/test_pipelines/misc_tests/multiplicity.plx
rename to tests/integration/pipelex/pipes/pipelines/multiplicity.plx
diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.plx b/tests/integration/pipelex/pipes/pipelines/test_image_inputs.plx
similarity index 100%
rename from tests/test_pipelines/misc_tests/test_image_inputs.plx
rename to tests/integration/pipelex/pipes/pipelines/test_image_inputs.plx
diff --git a/tests/test_pipelines/misc_tests/test_structures.py b/tests/integration/pipelex/pipes/pipelines/test_structures.py
similarity index 100%
rename from tests/test_pipelines/misc_tests/test_structures.py
rename to tests/integration/pipelex/pipes/pipelines/test_structures.py
diff --git a/tests/test_pipelines/misc_tests/tests.plx b/tests/integration/pipelex/pipes/pipelines/tests.plx
similarity index 100%
rename from tests/test_pipelines/misc_tests/tests.plx
rename to tests/integration/pipelex/pipes/pipelines/tests.plx
diff --git a/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
index 62a42a27f..28ee6e484 100644
--- a/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
+++ b/tests/integration/pipelex/pipes/test_bracket_notation_controllers.py
@@ -1,5 +1,7 @@
"""Integration tests for bracket notation in controller pipe factories."""
+from typing import Callable
+
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.hub import get_concept_library
@@ -15,7 +17,8 @@ from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallel
class TestBracketNotationInControllers:
"""Test that controller factories correctly handle bracket notation in inputs and outputs."""
- def test_pipe_parallel_with_bracket_notation(self):
+ def test_pipe_parallel_with_bracket_notation(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeParallel factory with bracket notation."""
domain = "test"
concept_library = get_concept_library()
@@ -47,7 +50,8 @@ class TestBracketNotationInControllers:
concept_library.teardown()
- def test_pipe_condition_with_bracket_notation(self):
+ def test_pipe_condition_with_bracket_notation(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeCondition factory with bracket notation."""
domain = "test"
concept_library = get_concept_library()
@@ -86,7 +90,8 @@ class TestBracketNotationInControllers:
concept_library.teardown()
- def test_pipe_batch_with_bracket_notation(self):
+ def test_pipe_batch_with_bracket_notation(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeBatch factory with bracket notation."""
domain = "test"
concept_library = get_concept_library()
diff --git a/tests/integration/pipelex/pipes/test_bracket_notation_operators.py b/tests/integration/pipelex/pipes/test_bracket_notation_operators/test_bracket_notation_operators.py
similarity index 84%
rename from tests/integration/pipelex/pipes/test_bracket_notation_operators.py
rename to tests/integration/pipelex/pipes/test_bracket_notation_operators/test_bracket_notation_operators.py
index de667649b..f47f54262 100644
--- a/tests/integration/pipelex/pipes/test_bracket_notation_operators.py
+++ b/tests/integration/pipelex/pipes/test_bracket_notation_operators/test_bracket_notation_operators.py
@@ -1,6 +1,9 @@
"""Integration tests for bracket notation in operator pipe factories."""
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pathlib import Path
+from typing import Callable
+
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.text_content import TextContent
@@ -31,7 +34,8 @@ async def process_function(working_memory: WorkingMemory) -> ListContent[TextCon
class TestBracketNotationInOperators:
"""Test that operator factories correctly handle bracket notation in inputs and outputs."""
- def test_pipe_llm_with_bracket_output_variable_list(self):
+ def test_pipe_llm_with_bracket_output_variable_list(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeLLM factory with variable list output (Text[])."""
blueprint = PipeLLMBlueprint(
description="Generate multiple items",
@@ -49,7 +53,8 @@ class TestBracketNotationInOperators:
assert pipe.output.code == "Text"
assert pipe.output_multiplicity is True
- def test_pipe_llm_with_bracket_output_fixed_count(self):
+ def test_pipe_llm_with_bracket_output_fixed_count(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeLLM factory with fixed count output (Text[5])."""
blueprint = PipeLLMBlueprint(
description="Generate exactly 5 items",
@@ -67,7 +72,8 @@ class TestBracketNotationInOperators:
assert pipe.output.code == "Text"
assert pipe.output_multiplicity == 5
- def test_pipe_llm_with_bracket_inputs(self):
+ def test_pipe_llm_with_bracket_inputs(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeLLM factory with bracket notation in inputs."""
blueprint = PipeLLMBlueprint(
description="Process multiple documents",
@@ -87,7 +93,8 @@ class TestBracketNotationInOperators:
assert pipe.inputs.root["documents"].multiplicity is True
assert pipe.inputs.root["query"].multiplicity is None
- def test_pipe_img_gen_with_bracket_output(self):
+ def test_pipe_img_gen_with_bracket_output(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeImgGen factory with fixed count output (Image[3])."""
blueprint = PipeImgGenBlueprint(
description="Generate 3 images",
@@ -104,8 +111,9 @@ class TestBracketNotationInOperators:
assert pipe.output.code == "Image"
assert pipe.output_multiplicity == 3
- def test_pipe_func_with_bracket_input_and_output(self):
- """Test PipeFunc factory with bracket notation."""
+ def test_pipe_func_with_bracket_input_and_output(self, load_test_library: Callable[[list[Path]], None]):
+ load_test_library([Path(Path(__file__).parent)])
+
blueprint = PipeFuncBlueprint(
description="Process items",
inputs={"two_texts": f"{NativeConceptCode.TEXT}[2]"},
@@ -122,7 +130,8 @@ class TestBracketNotationInOperators:
assert pipe.inputs.root["two_texts"].multiplicity == 2
assert pipe.output.code == "Text"
- def test_pipe_compose_with_bracket_notation(self):
+ def test_pipe_compose_with_bracket_notation(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeCompose factory with bracket notation."""
blueprint = PipeComposeBlueprint(
description="Compose multiple items",
@@ -140,7 +149,8 @@ class TestBracketNotationInOperators:
assert pipe.inputs.root["items"].multiplicity is True
assert pipe.output.code == "Text"
- def test_pipe_extract_with_bracket_output(self):
+ def test_pipe_extract_with_bracket_output(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test PipeExtract factory with bracket notation in output."""
blueprint = PipeExtractBlueprint(
description="Extract pages",
diff --git a/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.plx b/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.plx
new file mode 100644
index 000000000..89e5060aa
--- /dev/null
+++ b/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.plx
@@ -0,0 +1,130 @@
+domain = "discord_newsletter_flow"
+description = "Create newsletters from Discord channel content by summarizing messages and organizing them according to newsletter format"
+
+[concept]
+DiscordChannelUpdateFlow = "A Discord channel with its messages for newsletter generation"
+ChannelSummaryFlow = "A summarized Discord channel for newsletter inclusion"
+HtmlNewsletterFlow = "The final newsletter content in html format with organized channel summaries"
+
+[pipe.write_discord_newsletter_flow]
+type = "PipeSequence"
+description = "Create a newsletter from Discord articles by summarizing channels and organizing content"
+inputs = { discord_channel_updates = "DiscordChannelUpdateFlow[]" }
+output = "HtmlNewsletterFlow"
+steps = [
+ { pipe = "summarize_discord_channel_update_flow", batch_over = "discord_channel_updates", batch_as = "discord_channel_update", result = "channel_summaries" },
+ { pipe = "write_weekly_summary_flow", result = "weekly_summary" },
+ { pipe = "format_html_newsletter_flow", result = "html_newsletter" },
+]
+
+
+[pipe.summarize_discord_channel_update_flow]
+type = "PipeCondition"
+description = "Select the appropriate summary pipe based on the channel name"
+inputs = { discord_channel_update = "DiscordChannelUpdateFlow" }
+output = "ChannelSummaryFlow"
+expression = "discord_channel_update.name"
+outcomes = { "Introduce-Yourself" = "summarize_discord_channel_update_for_new_members_flow" }
+default_outcome = "summarize_discord_channel_update_general_flow"
+
+[pipe.summarize_discord_channel_update_for_new_members_flow]
+type = "PipeLLM"
+description = "Summarize the new member announcements"
+inputs = { discord_channel_update = "DiscordChannelUpdateFlow" }
+output = "ChannelSummaryFlow"
+system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."
+prompt = """
+Analyze this Discord channel update and create a newsletter-friendly summary.
+
+Channel Information:
+@discord_channel_update
+
+Summarize with one bullet point for each new member
+"""
+
+[pipe.summarize_discord_channel_update_general_flow]
+type = "PipeLLM"
+description = "Summarize a Discord channel's messages into newsletter-friendly content"
+inputs = { discord_channel_update = "DiscordChannelUpdateFlow" }
+output = "ChannelSummaryFlow"
+system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."
+prompt = """
+Analyze this Discord channel update and create a newsletter-friendly summary.
+
+Channel Information:
+@discord_channel_update
+
+The summary should be informative and engaging for newsletter readers who want to understand what happened in this channel during the week.
+Generate one or more summary items: each one can correspond to a single message or to a bunch of messages that were part of the same conversation.
+Each summary item should be in plain text, no bullet points. You can make some parts **bold** to highlight important information.
+
+Make sure to preserve the channel name and position from the input for proper ordering in the newsletter.
+"""
+
+[pipe.write_weekly_summary_flow]
+type = "PipeLLM"
+description = "Combine channel summaries into a short summary of the week's Share channel content (200 characters)"
+inputs = { channel_summaries = "ChannelSummaryFlow" }
+output = "Text"
+prompt = """
+Write a single overall summary of the week's content based on the following Share channel summaries:
+
+{% for channel in channel_summaries.content.items if channel.category == "Share" %}
+{{ channel }}
+{% endfor %}
+
+Keep it short: 200 characters.
+"""
+
+[pipe.format_html_newsletter_flow]
+type = "PipeCompose"
+description = "Combine weekly and channel summaries into a complete newsletter following specific formatting requirements"
+inputs = { weekly_summary = "Text", channel_summaries = "ChannelSummaryFlow" }
+output = "HtmlNewsletterFlow"
+
+[pipe.format_html_newsletter_flow.template]
+category = "html"
+template = """
+<!-- Weekly Summary -->
+<h2>☀️ Weekly Summary</h2>
+<p>
+$weekly_summary
+</p>
+
+<!-- New Members Section -->
+{% set introduce_channels = channel_summaries.content.items | selectattr('category', 'equalto', 'Introduce Yourself') | list %}
+{% if introduce_channels %}
+ <h2>🙌 New members</h2>
+ <ul>
+ {% for channel in introduce_channels %}
+ {% for item in channel.summary_items_as_html %}
+ <li>{{ item }}</li>
+ {% endfor %}
+ {% endfor %}
+ </ul>
+{% endif %}
+
+<!-- Share Channel Section -->
+{% set regular_channels = channel_summaries.content.items | selectattr('category', 'equalto', 'Share') | list | sort(attribute='position') %}
+{% if regular_channels %}
+ {% for channel in regular_channels %}
+ <h2>{{ channel.channel_name }}</h2>
+ {% for item in channel.summary_items_as_html %}
+ <p>{{ item }}</p>
+ {% endfor %}
+ {% endfor %}
+{% endif %}
+
+<!-- Geographic Hubs Section -->
+{% set geo_hubs = channel_summaries.content.items | selectattr('category', 'equalto', 'Geographic Hubs') | list | sort(attribute='position') %}
+{% if geo_hubs %}
+ <h2>🌎 Geographic hubs</h2>
+ {% for channel in geo_hubs %}
+ <h3>{{ channel.channel_name }}</h3>
+ {% for item in channel.summary_items_as_html %}
+ <p>{{ item }}</p>
+ {% endfor %}
+ {% endfor %}
+{% endif %}
+"""
+
diff --git a/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.py b/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.py
new file mode 100644
index 000000000..5bd123297
--- /dev/null
+++ b/tests/integration/pipelex/pipes/test_flow_factory/discord_newsletter.py
@@ -0,0 +1,55 @@
+from pydantic import Field
+
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+
+class Attachment(StructuredContent):
+ """Represents a Discord message attachment"""
+
+ name: str = Field(..., description="Name of the attachment file")
+ url: str = Field(..., description="URL of the attachment")
+
+
+class Embed(StructuredContent):
+ """Represents a Discord message embed"""
+
+ title: str = Field(..., description="Title of the embed")
+ description: str = Field(..., description="Description of the embed content")
+ type: str = Field(..., description="Type of the embed (e.g., article, video)")
+
+
+class DiscordMessage(StructuredContent):
+ """Represents a Discord message within a channel"""
+
+ author: str = Field(..., description="Author of the message")
+ content: str = Field(..., description="Content of the message")
+ attachments: list[Attachment] = Field(default_factory=empty_list_factory_of(Attachment), description="List of message attachments")
+ embeds: list[Embed] = Field(default_factory=empty_list_factory_of(Embed), description="List of message embeds")
+ link: str = Field(..., description="Link to the message")
+
+
+class DiscordChannelUpdate(StructuredContent):
+ """Represents a Discord channel with its messages"""
+
+ name: str = Field(..., description="Name of the Discord channel")
+ position: int = Field(..., description="Position of the channel")
+ messages: list[DiscordMessage] = Field(default_factory=empty_list_factory_of(DiscordMessage), description="List of messages in the channel")
+
+
+class ChannelSummary(StructuredContent):
+ """Represents a summarized Discord channel for newsletter inclusion"""
+
+ channel_name: str = Field(..., description="Name of the Discord channel")
+ summary_items: list[str] = Field(..., description="Well-written summaries of the channel's activity")
+
+
+class Newsletter(StructuredContent):
+ """Represents the final newsletter content"""
+
+ weekly_summary: str = Field(..., description="200 character summary of weekly Share channel content")
+ new_members: list[str] = Field(default_factory=empty_list_factory_of(str), description="New member introductions in bullet points")
+ channel_sections: list[ChannelSummary] = Field(default_factory=empty_list_factory_of(ChannelSummary), description="Ordered channel summaries")
+ geographic_hubs: list[ChannelSummary] = Field(
+ default_factory=empty_list_factory_of(ChannelSummary), description="Geographic hub channels grouped at end"
+ )
diff --git a/tests/integration/pipelex/pipes/test_flow_factory.py b/tests/integration/pipelex/pipes/test_flow_factory/test_flow_factory.py
similarity index 90%
rename from tests/integration/pipelex/pipes/test_flow_factory.py
rename to tests/integration/pipelex/pipes/test_flow_factory/test_flow_factory.py
index 81e6b5449..16156aaec 100644
--- a/tests/integration/pipelex/pipes/test_flow_factory.py
+++ b/tests/integration/pipelex/pipes/test_flow_factory/test_flow_factory.py
@@ -6,7 +6,7 @@ import pytest
from pipelex import log, pretty_print
from pipelex.builder.flow_factory import FlowFactory
from pipelex.builder.pipe.pipe_signature import PipeSignature
-from pipelex.core.pipes.pipe_blueprint import AllowedPipeTypes
+from pipelex.core.pipes.pipe_blueprint import PipeType
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
from pipelex.tools.misc.file_utils import get_incremental_directory_path, remove_folder
from pipelex.tools.misc.json_utils import save_as_json_to_path
@@ -21,7 +21,6 @@ class TestFlowFactory:
("plx_name", "plx_file"),
[
("discord_newsletter", "discord_newsletter.plx"),
- ("tricky_questions", "tricky_questions.plx"),
],
)
async def test_load_plx_and_convert_to_flow(
@@ -48,7 +47,7 @@ class TestFlowFactory:
try:
# Load PLX file from test pipelines directory
- test_pipelines_dir = Path(__file__).parent.parent.parent.parent / "test_pipelines"
+ test_pipelines_dir = Path(__file__).parent
plx_file_path = test_pipelines_dir / plx_file
log.info(f"Loading PLX file: {plx_file_path}")
@@ -83,13 +82,13 @@ class TestFlowFactory:
async def test_flow_preserves_sequence_structure(self):
"""Verify that PipeSequence controllers preserve their step structure."""
- test_pipelines_dir = Path(__file__).parent.parent.parent.parent / "test_pipelines"
+ test_pipelines_dir = Path(__file__).parent
plx_file_path = test_pipelines_dir / "discord_newsletter.plx"
flow = FlowFactory.make_from_plx_file(plx_file_path)
# Find the sequence pipe
- sequence_element = flow.flow_elements.get("write_discord_newsletter")
+ sequence_element = flow.flow_elements.get("write_discord_newsletter_flow")
assert sequence_element is not None
assert sequence_element.controller_blueprint is not None
assert sequence_element.controller_blueprint.type == "PipeSequence"
@@ -106,17 +105,17 @@ class TestFlowFactory:
async def test_flow_converts_operators_to_signatures(self):
"""Verify that pipe operators are converted to signatures (simplified view)."""
- test_pipelines_dir = Path(__file__).parent.parent.parent.parent / "test_pipelines"
+ test_pipelines_dir = Path(__file__).parent
plx_file_path = test_pipelines_dir / "discord_newsletter.plx"
flow = FlowFactory.make_from_plx_file(plx_file_path)
# Find an operator pipe (LLM pipe) - converted to signature
- operator_element = flow.flow_elements.get("summarize_discord_channel_update_for_new_members")
+ operator_element = flow.flow_elements.get("summarize_discord_channel_update_for_new_members_flow")
assert operator_element is not None
assert operator_element.operator_signature is not None
assert isinstance(operator_element.operator_signature, PipeSignature)
- assert operator_element.operator_signature.type == AllowedPipeTypes.PIPE_LLM
+ assert operator_element.operator_signature.type == PipeType.PIPE_LLM
# Verify it has the signature properties
assert hasattr(operator_element.operator_signature, "code")
diff --git a/tests/integration/pipelex/pipes/test_image_inputs.py b/tests/integration/pipelex/pipes/test_image_inputs.py
index 657f0b9a1..333d24af5 100644
--- a/tests/integration/pipelex/pipes/test_image_inputs.py
+++ b/tests/integration/pipelex/pipes/test_image_inputs.py
@@ -1,15 +1,18 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pytest import FixtureRequest
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
-from pipelex.hub import get_pipe_router, get_required_pipe
+from pipelex.hub import get_native_concept, get_pipe_router, get_required_pipe
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
@@ -17,7 +20,7 @@ from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
from pipelex.pipeline.job_metadata import JobMetadata
from tests.cases import ImageTestCases
-from tests.test_pipelines.misc_tests.test_structures import Article
+from tests.integration.pipelex.pipes.pipelines.test_structures import Article
@pytest.mark.dry_runnable
@@ -31,9 +34,16 @@ class TestImageInputs:
self,
request: FixtureRequest,
pipe_run_mode: PipeRunMode,
+ load_test_library: Callable[[list[Path]], None],
) -> None:
- """Test that an image is indeed given to the LLM, and that it can extract extact whats on the image."""
- working_memory = WorkingMemoryFactory.make_from_image(name="image", image_url=ImageTestCases.IMAGE_FILE_PATH_PNG)
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=get_native_concept(NativeConceptCode.IMAGE),
+ content=ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG),
+ name="image",
+ ),
+ )
pipe_output = await get_pipe_router().run(
pipe_job=PipeJobFactory.make_pipe_job(
@@ -56,10 +66,8 @@ class TestImageInputs:
assert pipe_output.working_memory is not None
assert pipe_output.main_stuff is not None
- async def test_describe_page(self, request: FixtureRequest, pipe_run_mode: PipeRunMode) -> None:
- """Test that a pipe can accept a PageContent input, give to the LLM the image via subattributes,
- But also accepts basic objects
- """
+ async def test_describe_page(self, request: FixtureRequest, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]) -> None:
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
# Create the page content
# image_content = ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG)
image_content = ImageContent(url=f"file://{ImageTestCases.IMAGE_FILE_PATH_PNG}")
@@ -101,10 +109,8 @@ class TestImageInputs:
assert pipe_output.main_stuff is not None
@pytest.mark.usefixtures("request")
- async def test_image_input_within_concept_with_text(self) -> None:
- """Test that a pipe can accept a PageContent input, give to the LLM the image via subattributes,
- But also accepts basic objects
- """
+ async def test_image_input_within_concept_with_text(self, load_test_library: Callable[[list[Path]], None]) -> None:
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
pipe_llm_blueprint = PipeLLMBlueprint(
description="Test that a pipe can accept a PageContent input, give to the LLM the image via subattributes",
inputs={"page": "Page"},
diff --git a/tests/integration/pipelex/pipes/test_pipe_compose.py b/tests/integration/pipelex/pipes/test_pipe_compose.py
index efe306684..735e7bd61 100644
--- a/tests/integration/pipelex/pipes/test_pipe_compose.py
+++ b/tests/integration/pipelex/pipes/test_pipe_compose.py
@@ -1,4 +1,4 @@
-from typing import TYPE_CHECKING, cast
+from typing import TYPE_CHECKING, Callable, cast
import pytest
@@ -6,9 +6,11 @@ from pipelex import pretty_print
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormat
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.hub import get_pipe_router
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_native_concept, get_pipe_router
from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
from pipelex.pipe_operators.compose.pipe_compose_factory import PipeComposeFactory
from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
@@ -28,7 +30,9 @@ class TestPipeCompose:
self,
pipe_run_mode: PipeRunMode,
template_source: str,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
pipe_compose_blueprint = PipeComposeBlueprint(
description="Jinja2 test for any context",
template=TemplateBlueprint(
@@ -57,8 +61,16 @@ class TestPipeCompose:
self,
pipe_run_mode: PipeRunMode,
template_source: str,
+ load_empty_library: Callable[[], None],
):
- working_memory = WorkingMemoryFactory.make_from_text(text="[some text from test_pipe_compose_for_stuff]", name="place_holder")
+ load_empty_library()
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=get_native_concept(NativeConceptCode.TEXT),
+ content=TextContent(text="[some text from test_pipe_compose_for_stuff]"),
+ name="place_holder",
+ ),
+ )
pipe_compose_blueprint = PipeComposeBlueprint(
description="Jinja2 test for stuff context",
diff --git a/tests/integration/pipelex/pipes/test_pipe_running_variants.py b/tests/integration/pipelex/pipes/test_pipe_running_variants.py
index ce9f95ca8..4ee7c5546 100644
--- a/tests/integration/pipelex/pipes/test_pipe_running_variants.py
+++ b/tests/integration/pipelex/pipes/test_pipe_running_variants.py
@@ -1,3 +1,6 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pytest import FixtureRequest
@@ -28,7 +31,9 @@ class TestPipeRunningVariants:
topic: str,
stuff: Stuff,
pipe_code: str,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
log.verbose(stuff, title=f"{topic}: start from '{stuff.stuff_name}', run pipe '{pipe_code}'")
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
_ = await get_pipe_router().run(
@@ -47,7 +52,9 @@ class TestPipeRunningVariants:
request: FixtureRequest,
topic: str,
pipe_code: str,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
log.verbose(f"{topic}: just run pipe '{pipe_code}'")
pipe_output = await get_pipe_router().run(
pipe_job=PipeJobFactory.make_pipe_job(
@@ -71,7 +78,9 @@ class TestPipeRunningVariants:
topic: str,
pipe_code: str,
output_multiplicity: VariableMultiplicity | None,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
log.verbose(f"{topic}: just run pipe '{pipe_code}'")
pipe_output = await get_pipe_router().run(
pipe_job=PipeJobFactory.make_pipe_job(
@@ -98,15 +107,9 @@ class TestPipeRunningVariants:
pipe_code: str,
exception: type[Exception],
expected_error_message: str,
+ load_test_library: Callable[[list[Path]], None],
):
- # failing_pipelines_file_paths = get_config().pipelex.library_config.failing_pipelines_file_paths
- # library_manager = get_library_manager()
- # Reset library to avoid pipe name collisions from previous test runs
- # library_manager.reset()
- # library_manager.load_libraries(
- # library_file_paths=[Path(failing_pipeline_file_path) for failing_pipeline_file_path in failing_pipelines_file_paths],
- # )
-
+ load_test_library([Path("tests/integration/pipelex/pipes/pipelines")])
log.verbose(f"This pipe '{pipe_code}' is supposed to cause an error of type: {exception.__name__}")
with pytest.raises(PipeStackOverflowError) as exc:
await get_pipe_router().run(
diff --git a/tests/integration/pipelex/plugins/test_external_plugin.py b/tests/integration/pipelex/plugins/test_external_plugin.py
index 68cad8395..36c758cca 100644
--- a/tests/integration/pipelex/plugins/test_external_plugin.py
+++ b/tests/integration/pipelex/plugins/test_external_plugin.py
@@ -1,3 +1,5 @@
+from typing import Callable
+
import pytest
from polyfactory.factories.pydantic_factory import ModelFactory
from typing_extensions import override
@@ -10,7 +12,7 @@ from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.hub import get_inference_manager, get_pipe_router, get_report_delegate
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory
@@ -65,7 +67,8 @@ class MockExternalLLMWorker(LLMWorkerAbstract):
@pytest.mark.asyncio(loop_scope="class")
class TestExternalPlugin:
- async def test_external_llm_worker(self):
+ async def test_external_llm_worker(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
llm_worker = MockExternalLLMWorker(reporting_delegate=get_report_delegate())
llm_job = LLMJobFactory.make_llm_job(
llm_prompt=LLMPrompt(
@@ -85,7 +88,8 @@ class TestExternalPlugin:
assert generated_object
pretty_print(generated_object)
- async def test_pipe_llm_with_external_llm_handle(self):
+ async def test_pipe_llm_with_external_llm_handle(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
llm_handle = EXTERNAL_PLUGIN_NAME
get_inference_manager().set_llm_worker_from_external_plugin(
llm_handle=llm_handle,
diff --git a/tests/unit/pipelex/tools/config/test_load_config.py b/tests/integration/pipelex/system/test_load_config.py
similarity index 85%
rename from tests/unit/pipelex/tools/config/test_load_config.py
rename to tests/integration/pipelex/system/test_load_config.py
index 68a7d79d5..703fc808c 100644
--- a/tests/unit/pipelex/tools/config/test_load_config.py
+++ b/tests/integration/pipelex/system/test_load_config.py
@@ -1,12 +1,12 @@
-from pipelex.config import PipelexConfig
from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.configuration.configs import PipelexConfig
from pipelex.tools.misc.json_utils import deep_update
from pipelex.tools.misc.toml_utils import load_toml_from_path
class TestLoadConfig:
def test_load_kit_config(self):
- # The base condif must validate
+ # The base config must validate
config = config_manager.load_config()
PipelexConfig.model_validate(config)
diff --git a/tests/integration/pipelex/test_data.py b/tests/integration/pipelex/test_data.py
index 455ba18f5..8018cd127 100644
--- a/tests/integration/pipelex/test_data.py
+++ b/tests/integration/pipelex/test_data.py
@@ -1,7 +1,7 @@
from typing import ClassVar
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
@@ -110,13 +110,6 @@ class PipeTestCases:
concept_string="answer.Question",
content=USER_TEXT_TRICKY_2,
)
- BLUEPRINT_AND_PIPE: ClassVar[list[tuple[str, StuffBlueprint, str]]] = [ # topic, blueprint, pipe
- (
- "Tricky question conclude",
- TRICKY_QUESTION_BLUEPRINT,
- "conclude_tricky_question_by_steps",
- ),
- ]
NO_INPUT: ClassVar[list[tuple[str, str]]] = [ # topic, pipe
(
"Test with no input",
@@ -161,15 +154,6 @@ class PipeTestCases:
]
-class LibraryTestCases:
- TEST_PIPELINES_DIR_PATH = "tests/test_pipelines"
- KNOWN_CONCEPTS_AND_PIPES: ClassVar[list[tuple[str, str]]] = [ # concept, pipe
- ("cars.CarDescription", "generate_car_description"),
- ("animals.AnimalDescription", "generate_animal_description"),
- ("flowers.FlowerDescription", "generate_flower_description"),
- ]
-
-
class PipeExtractTestCases:
PIPE_OCR_IMAGE_TEST_CASES: ClassVar[list[str]] = [
ImageTestCases.IMAGE_FILE_PATH_PNG,
diff --git a/tests/integration/pipelex/test_fundamentals.py b/tests/integration/pipelex/test_fundamentals.py
index fc9f5f1fc..022f8d247 100644
--- a/tests/integration/pipelex/test_fundamentals.py
+++ b/tests/integration/pipelex/test_fundamentals.py
@@ -1,10 +1,5 @@
import pytest
-from pipelex.config import get_config
-from pipelex.hub import get_pipes
-from pipelex.pipe_run.dry_run import dry_run_pipes
-from pipelex.pipelex import Pipelex
-
# We use gha_disabled here because those tests are called directly and explicitly by the tests-check.yml file before the rest of the tests.
@pytest.mark.gha_disabled
@@ -13,21 +8,3 @@ class TestFundamentals:
# This test does nothing but the conftest runs Pipelex.make()
# Therefore this test will fail if Pipelex.make() fails.
pass
-
- def test_validate_libraries(self):
- Pipelex.get_instance().validate_libraries()
-
- @pytest.mark.asyncio(loop_scope="class")
- async def test_dry_run_all_pipes(self):
- results = await dry_run_pipes(pipes=get_pipes(), raise_on_failure=False)
-
- # Check if there were any failures
- allowed_to_fail_pipes = get_config().pipelex.dry_run_config.allowed_to_fail_pipes
-
- failed_pipes = {
- pipe_code: output for pipe_code, output in results.items() if output.status.is_failure and pipe_code not in allowed_to_fail_pipes
- }
-
- if failed_pipes:
- failure_details = "\n".join([f" - {pipe_code}: {output.error_message}" for pipe_code, output in failed_pipes.items()])
- pytest.fail(f"Dry run failed for {len(failed_pipes)} pipes:\n{failure_details}")
diff --git a/tests/integration/pipelex/test_inline_structure_concepts.py b/tests/integration/pipelex/test_inline_structure_concepts.py
index e93811494..24c733ba2 100644
--- a/tests/integration/pipelex/test_inline_structure_concepts.py
+++ b/tests/integration/pipelex/test_inline_structure_concepts.py
@@ -8,7 +8,7 @@ from pipelex.core.concepts.concept_blueprint import (
)
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.exceptions import StructureClassError
+from pipelex.core.concepts.exceptions import ConceptFactoryError
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.hub import get_class_registry
@@ -180,7 +180,7 @@ class TestInlineStructureConcepts:
def test_invalid_string_reference_raises_error(self):
"""Test that invalid string references raise appropriate errors."""
with pytest.raises(
- StructureClassError,
+ ConceptFactoryError,
match="Structure class 'NonExistentClass' set for concept 'TestInvalidRef' in domain 'test_domain' is \
not a registered subclass of StuffContent",
):
diff --git a/tests/integration/pipelex/test_libraries.py b/tests/integration/pipelex/test_libraries.py
deleted file mode 100644
index 58598bb17..000000000
--- a/tests/integration/pipelex/test_libraries.py
+++ /dev/null
@@ -1,101 +0,0 @@
-from pathlib import Path
-
-import pytest
-from rich import box
-from rich.table import Table
-
-from pipelex import pretty_print
-from pipelex.core.concepts.concept_library import ConceptLibrary
-from pipelex.core.pipes.pipe_library import PipeLibrary
-from pipelex.hub import get_console
-from pipelex.libraries.library_manager_factory import LibraryManagerFactory
-from tests.integration.pipelex.test_data import LibraryTestCases
-
-
-def pretty_print_all_pipes(
- pipe_library: PipeLibrary,
- title: str | None = None,
-):
- console = get_console()
- table = Table(
- title=title,
- show_header=True,
- show_lines=True,
- header_style="bold cyan",
- box=box.SQUARE_DOUBLE_HEAD,
- )
- table.add_column("Domain")
- table.add_column("Code")
- table.add_column("Definition")
- table.add_column("Class")
- table.add_column("Input")
- table.add_column("Output")
-
- ordered_items = sorted(pipe_library.root.values(), key=lambda x: (x.domain, x.code))
- for pipe in ordered_items:
- table.add_row(
- pipe.domain,
- pipe.code,
- pipe.description,
- pipe.__class__.__name__,
- ", ".join([f"{name}: {concept_code}" for name, concept_code in pipe.inputs.items]),
- pipe.output.code,
- )
-
- console.print("\n")
- console.print(table)
-
-
-def pretty_print_all_concepts(
- concept_library: ConceptLibrary,
- title: str | None = None,
-):
- console = get_console()
- table = Table(
- title=title,
- show_header=True,
- show_lines=True,
- header_style="bold cyan",
- box=box.SQUARE_DOUBLE_HEAD,
- )
- table.add_column("Domain")
- table.add_column("Code")
- table.add_column("Definition")
- table.add_column("Class")
- table.add_column("Inherits From")
- # make a list ordered by domain then code
- ordered_concepts = sorted(concept_library.root.values(), key=lambda x: (x.domain, x.code))
- for concept in ordered_concepts:
- table.add_row(
- concept.domain,
- concept.code,
- concept.description,
- concept.structure_class_name,
- concept.refines,
- )
-
- console.print("\n")
- console.print(table)
-
-
-class TestLibraries:
- @pytest.mark.parametrize(("known_concept", "known_pipe"), LibraryTestCases.KNOWN_CONCEPTS_AND_PIPES)
- def test_load_combo_libraries(
- self,
- known_concept: str,
- known_pipe: str,
- ):
- library_manager = LibraryManagerFactory.make_empty()
- test_pipelines_dir = [Path(LibraryTestCases.TEST_PIPELINES_DIR_PATH)]
- library_manager.load_libraries(library_dirs=test_pipelines_dir)
- # Verify that libraries were loaded
- assert len(library_manager.concept_library.root) > 0, "No concepts were loaded"
- assert len(library_manager.pipe_library.root) > 0, "No pipes were loaded"
-
- # Test individual concepts and pipes
- assert library_manager.concept_library.get_required_concept(concept_string=known_concept) is not None
- pretty_print(
- f"Concept: {known_concept} is correctly loaded as {library_manager.concept_library.get_required_concept(concept_string=known_concept)}",
- )
- assert library_manager.pipe_library.get_optional_pipe(known_pipe) is not None
- pretty_print(f"Pipe: {known_pipe} is correctly loaded as {library_manager.pipe_library.get_optional_pipe(known_pipe)}")
diff --git a/tests/pipelex_unit_test.toml b/tests/pipelex_unit_test.toml
index 54e968154..ef0ab2f6e 100644
--- a/tests/pipelex_unit_test.toml
+++ b/tests/pipelex_unit_test.toml
@@ -1,4 +1,7 @@
+[cogt.model_deck_config]
+# Allow tests to run with restrictive routing profiles that don't support all presets
+missing_presets_reaction = "none"
+
[pipelex.feature_config]
is_reporting_enabled = true
-
diff --git a/tests/test_pipelines/concepts/datetime.plx b/tests/test_pipelines/concepts/datetime.plx
deleted file mode 100644
index 297b38e25..000000000
--- a/tests/test_pipelines/concepts/datetime.plx
+++ /dev/null
@@ -1,5 +0,0 @@
-domain = "event"
-
-[concept]
-DateTimeEvent = "A date time event with start and end time"
-
diff --git a/tests/test_pipelines/concepts/datetime.py b/tests/test_pipelines/concepts/datetime.py
deleted file mode 100644
index 0358fb0ba..000000000
--- a/tests/test_pipelines/concepts/datetime.py
+++ /dev/null
@@ -1,12 +0,0 @@
-from datetime import datetime
-
-from pipelex.core.stuffs.structured_content import StructuredContent
-
-
-class DateTimeEvent(StructuredContent):
- """Test model for datetime content."""
-
- event_name: str
- start_time: datetime
- end_time: datetime
- created_at: datetime
diff --git a/tests/test_pipelines/concepts/is_compatible_concept_code.plx b/tests/test_pipelines/concepts/is_compatible_concept_code.plx
deleted file mode 100644
index 0a26845cc..000000000
--- a/tests/test_pipelines/concepts/is_compatible_concept_code.plx
+++ /dev/null
@@ -1,42 +0,0 @@
-domain = "concept_library_tests"
-
-[concept]
-# Simple concept with no structure - should default to Text
-SimpleTextConcept = "A simple concept that should default to Text"
-
-# Concept with explicit structure class
-FundamentalsDoc = "A comprehensive overview of the fundamental concepts and principles of software engineering."
-
-# Concept that explicitly refines Text
-[concept.ExplicitTextConcept]
-description = "A concept that explicitly refines Text"
-refines = "Text"
-
-# Concept that refines Image
-[concept.ImageBasedConcept]
-description = "A concept based on images"
-refines = "Image"
-
-# Concept that refines FundamentalsDoc
-[concept.DocumentationConcept]
-description = "A specialized documentation concept"
-
-# Concept that refines both Text and Image (multiple inheritance)
-[concept.MultiMediaConcept]
-description = "A concept that combines text and images"
-
-# Concept with custom structure that doesn't refine anything
-[concept.IndependentConcept]
-description = "An independent concept with custom structure"
-structure = "IndependentConcept"
-
-# Concept that refines a native concept (changed from non-native)
-[concept.SpecializedDoc]
-description = "A specialized document that builds on Text"
-refines = "Text"
-
-# Concept that refines a native concept (changed from non-native chain)
-[concept.DerivedTextConcept]
-description = "A concept derived from Text"
-refines = "Text"
-
diff --git a/tests/test_pipelines/concepts/is_compatible_concept_code.py b/tests/test_pipelines/concepts/is_compatible_concept_code.py
deleted file mode 100644
index 8db193a47..000000000
--- a/tests/test_pipelines/concepts/is_compatible_concept_code.py
+++ /dev/null
@@ -1,43 +0,0 @@
-from pydantic import Field
-
-from pipelex.core.stuffs.structured_content import StructuredContent
-
-
-class FundamentalsDoc(StructuredContent):
- project_overview: str | None = Field(
- None,
- description="Mission, key features, architecture diagram, demo links",
- )
- core_concepts: dict[str, str] | None = Field(
- None,
- description=(
- "Names and definitions for project-specific terms, acronyms, data model names, background knowledge, business rules, domain entities"
- ),
- )
- repository_map: str | None = Field(
- None,
- description="Directory layout explanation and purpose of each folder",
- )
-
-
-class DocumentationConcept(StructuredContent):
- """A specialized documentation concept that extends FundamentalsDoc."""
-
- title: str = Field(..., description="Title of the documentation")
- sections: list[str] = Field(default_factory=list, description="List of section names")
- last_updated: str | None = Field(None, description="Last update timestamp")
-
-
-class MultiMediaConcept(StructuredContent):
- """A concept that combines text and images."""
-
- text_content: str = Field(..., description="The text content")
- image_urls: list[str] = Field(default_factory=list, description="List of image URLs")
- caption: str | None = Field(None, description="Optional caption for the content")
-
-
-class IndependentConcept(StructuredContent):
- """An independent concept with its own structure."""
-
- unique_field: str = Field(..., description="A unique field for this concept")
- metadata: dict[str, str] = Field(default_factory=dict, description="Metadata dictionary")
diff --git a/tests/test_pipelines/misc_tests/subfolder_1/__init__.py b/tests/test_pipelines/misc_tests/subfolder_1/__init__.py
deleted file mode 100644
index 3596aa421..000000000
--- a/tests/test_pipelines/misc_tests/subfolder_1/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# This file makes the directory a Python package
diff --git a/tests/test_pipelines/misc_tests/subfolder_1/cars.plx b/tests/test_pipelines/misc_tests/subfolder_1/cars.plx
deleted file mode 100644
index d8ce8851c..000000000
--- a/tests/test_pipelines/misc_tests/subfolder_1/cars.plx
+++ /dev/null
@@ -1,22 +0,0 @@
-
-
-domain = "cars"
-description = "The test domain for handling car descriptions"
-system_prompt = "You are an expert at describing cars based on their brand, model, and year."
-
-[concept]
-Car = "A car reference, including its brand, model, and year"
-CarDescription = "A detailed description of a car model"
-
-[pipe]
-[pipe.generate_car_description]
-type = "PipeLLM"
-description = "Generate a description of a car"
-inputs = { car = "Car" }
-output = "CarDescription"
-prompt = """
-Given the reference to a car, generate a description of the car.
-
-@car
-"""
-
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/__init__.py b/tests/test_pipelines/misc_tests/subfolder_2/__init__.py
deleted file mode 100644
index 3596aa421..000000000
--- a/tests/test_pipelines/misc_tests/subfolder_2/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# This file makes the directory a Python package
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/animals.plx b/tests/test_pipelines/misc_tests/subfolder_2/animals.plx
deleted file mode 100644
index adfbf094c..000000000
--- a/tests/test_pipelines/misc_tests/subfolder_2/animals.plx
+++ /dev/null
@@ -1,22 +0,0 @@
-
-
-domain = "animals"
-description = "The test domain for handling animal descriptions"
-system_prompt = "You are an expert at describing animals based on their species, breed, and age."
-
-[concept]
-Animal = "An animal reference, including its species, breed, and age"
-AnimalDescription = "A detailed description of an animal"
-
-[pipe]
-[pipe.generate_animal_description]
-type = "PipeLLM"
-description = "Generate a description of an animal"
-inputs = { animal = "Animal" }
-output = "AnimalDescription"
-prompt = """
-Given the reference to an animal, generate a description of the animal.
-
-@animal
-"""
-
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.plx b/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.plx
deleted file mode 100644
index 77e85bc8f..000000000
--- a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.plx
+++ /dev/null
@@ -1,20 +0,0 @@
-domain = "flowers"
-description = "A test domain related to flowers"
-system_prompt = "You are an expert at describing flowers based on their species and color."
-
-[concept]
-Flower = "A flower reference, including its species and color"
-FlowerDescription = "A detailed description of a flower"
-
-[pipe]
-[pipe.generate_flower_description]
-type = "PipeLLM"
-description = "Generate a description of a flower"
-inputs = { flower = "Flower" }
-output = "FlowerDescription"
-prompt = """
-Given the reference to a flower, generate a description of the flower.
-
-@flower
-"""
-
diff --git a/tests/test_pipelines/misc_tests/test_compose.plx b/tests/test_pipelines/misc_tests/test_compose.plx
deleted file mode 100644
index d54f33add..000000000
--- a/tests/test_pipelines/misc_tests/test_compose.plx
+++ /dev/null
@@ -1,17 +0,0 @@
-domain = "test_jinja2"
-description = "This library is intended for testing Jinja2"
-
-[pipe.jinja2_test_1]
-type = "PipeCompose"
-description = "Jinja2 test 1"
-inputs = { text = "Text" }
-output = "Text"
-
-[pipe.jinja2_test_1.template]
-template = """
-This is a simple test prompt:
-@text
-"""
-templating_style = { tag_style = "square_brackets", text_format = "markdown" }
-category = "llm_prompt"
-
diff --git a/tests/test_pipelines/pipe_operators/__init__.py b/tests/test_pipelines/pipe_operators/__init__.py
deleted file mode 100644
index 254dcf49c..000000000
--- a/tests/test_pipelines/pipe_operators/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-"""Test pipelines for pipe operators."""
diff --git a/tests/test_pipelines/test_tweet.py b/tests/test_pipelines/test_tweet.py
deleted file mode 100644
index 51df37c10..000000000
--- a/tests/test_pipelines/test_tweet.py
+++ /dev/null
@@ -1,7 +0,0 @@
-from pipelex.core.stuffs.text_content import TextContent
-
-
-class OptimizedTweet(TextContent):
- """A tweet optimized for Twitter/X engagement following best practices."""
-
- # TBD
diff --git a/tests/test_pipelines/tricky_questions.plx b/tests/test_pipelines/tricky_questions.plx
deleted file mode 100644
index 96a117249..000000000
--- a/tests/test_pipelines/tricky_questions.plx
+++ /dev/null
@@ -1,67 +0,0 @@
-domain = "test_tricky_questions"
-description = "Domain for testing tricky questions"
-
-[concept]
-AnswerToAQuestion = "Answer to a question"
-Question = "A question"
-QuestionAnalysis = "Analysis of a question"
-ThoughtfulAnswerConclusion = "Conclusion of a thoughtful answer"
-
-[concept.ThoughtfulAnswer]
-description = "A thoughtful answer to a question"
-
-[pipe]
-[pipe.analyse_question_tricky]
-type = "PipeLLM"
-description = "Analyze a question to determine whether it's straightforward or tricky"
-inputs = { question = "Question" }
-output = "QuestionAnalysis"
-model = "llm_to_engineer"
-prompt = """
-Here is a question for an LLM:
-@question
-Do you think it's tricky, or maybe even a deceptive trap?
-Does it assume things that are not necessarily true?
-Does it suggest patterns that aren't applicable?
-Please explain what you think and then give a rating between 0 to 100 of trickiness and another rating between 0 to 100 of deceptiveness.
-If there's an obvious trap, state it without getting into details.
-"""
-
-[pipe.answer_after_analysis]
-type = "PipeLLM"
-description = "Answer knowingly after analyzing a question"
-inputs = { question = "Question", question_analysis = "QuestionAnalysis" }
-output = "ThoughtfulAnswer"
-model = "llm_to_engineer"
-prompt = """
-A question was asked:
-@question
-A thoughtful analysis was given:
-@question_analysis
-If the question was tricky or deceptive, don't get fooled!
-Answer in 4 parts:
-1- the_trap: Explain the trap in a 1 sentence
-2- the_counter: Counter by stating the right way to think about the question and avoid the trap
-3- the_lesson: Did we learn anything?
-4- the_answer: Then give a good answer expressed without mentioning the trap
-"""
-
-[pipe.conclude_thoughtful_answer]
-type = "PipeCompose"
-description = "Conclude a thoughtful answer"
-inputs = { thoughtful_answer = "ThoughtfulAnswer" }
-output = "ThoughtfulAnswerConclusion"
-template = "After analyzing the question, here is my answer: $thoughtful_answer.the_answer"
-
-
-[pipe.conclude_tricky_question_by_steps]
-type = "PipeSequence"
-description = "Answer a tricky question by first analyzing its trickiness and then concluding"
-inputs = { question = "Question" }
-output = "ThoughtfulAnswerConclusion"
-steps = [
- { pipe = "analyse_question_tricky", result = "question_analysis" },
- { pipe = "answer_after_analysis", result = "thoughtful_answer" },
- { pipe = "conclude_thoughtful_answer", result = "thoughtful_answer_conclusion" },
-]
-
diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py
index 579c7cdf4..65ac95d34 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py
@@ -11,7 +11,7 @@ class PipeConditionTestCases:
pipe_code="template_condition",
description="Conditional with template",
inputs={"item": "Item"},
- output="ProcessedItem",
+ output="native.Anything",
jinja2_expression_template="{{ item.category }}",
outcomes={
"A": "process_a",
@@ -23,7 +23,7 @@ class PipeConditionTestCases:
PipeConditionBlueprint(
description="Conditional with template",
inputs={"item": "Item"},
- output="ProcessedItem",
+ output="native.Anything",
type="PipeCondition",
pipe_category="PipeController",
expression=None,
diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
index 583b8cd6c..cb9e558d3 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py
@@ -18,7 +18,7 @@ class PipeExtractTestCases:
source=None,
description="Extract text from image",
inputs={"image": "Image"},
- output="Page",
+ output="Page[]",
type="PipeExtract",
pipe_category="PipeOperator",
model=ExtractSkill.EXTRACT_TEXT_FROM_VISUALS,
@@ -31,7 +31,7 @@ class PipeExtractTestCases:
pipe_code="advanced_extract",
description="Extract with page options",
inputs={"document": "PDF"},
- output="Page",
+ output="Page[]",
extract_skill=ExtractSkill.EXTRACT_TEXT_FROM_PDF,
page_images=True,
page_image_captions=True,
@@ -41,7 +41,7 @@ class PipeExtractTestCases:
source=None,
description="Extract with page options",
inputs={"document": "PDF"},
- output="Page",
+ output="Page[]",
type="PipeExtract",
pipe_category="PipeOperator",
model=ExtractSkill.EXTRACT_TEXT_FROM_PDF,
diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_extract_skill_presets.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_extract_skill_presets.py
index b69dca574..c448301a8 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_extract_skill_presets.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_extract_skill_presets.py
@@ -5,7 +5,7 @@ from pipelex.cogt.exceptions import ModelChoiceNotFoundError
from pipelex.cogt.models.model_deck_check import check_extract_choice_with_deck
-class TestExtractSkill:
+class TestSkillPresetsExtract:
"""Test that all ExtractSkill enum values are valid ExtractModelChoice values in the model deck."""
def test_all_extract_skills_are_valid_extract_choices(self):
diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py
index 33b023e51..82c4d216d 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py
@@ -10,18 +10,17 @@ class PipeImgGenTestCases:
PipeImgGenSpec(
pipe_code="img_generator",
description="Generate an image",
- inputs={},
- output="GeneratedImage",
+ inputs={"my_prompt": "Text"},
+ output="native.Image",
img_gen_skill=ImgGenSkill.GEN_IMAGE_BASIC,
),
PipeImgGenBlueprint(
description="Generate an image",
- inputs=None,
- output="GeneratedImage",
+ inputs={"my_prompt": "Text"},
+ output="native.Image",
type="PipeImgGen",
pipe_category="PipeOperator",
- img_gen_prompt=None,
- img_gen_prompt_var_name=None,
+ img_gen_prompt_var_name="my_prompt",
model=ImgGenSkill.GEN_IMAGE_BASIC,
aspect_ratio=None,
background=None,
@@ -53,7 +52,7 @@ class PipeImgGenTestCases:
output_format=None,
is_raw=None,
seed=None,
- img_gen_prompt_var_name=None,
+ img_gen_prompt_var_name="description",
),
)
diff --git a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py
index 4f7b9b9e0..14fff9054 100644
--- a/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py
+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py
@@ -1,3 +1,5 @@
+from typing import Callable
+
import pytest
from pipelex import pretty_print
@@ -12,7 +14,10 @@ class TestPipeLLMBlueprintConversion:
("test_name", "pipe_spec", "expected_blueprint"),
PipeLLMTestCases.TEST_CASES,
)
- def test_pipe_llm_spec_to_blueprint(self, test_name: str, pipe_spec: PipeLLMSpec, expected_blueprint: PipeLLMBlueprint):
+ def test_pipe_llm_spec_to_blueprint(
+ self, test_name: str, pipe_spec: PipeLLMSpec, expected_blueprint: PipeLLMBlueprint, load_empty_library: Callable[[], None]
+ ):
+ load_empty_library()
blueprint = pipe_spec.to_blueprint()
assert blueprint == expected_blueprint
diff --git a/tests/unit/pipelex/builder/test_runner_generator.py b/tests/unit/pipelex/builder/test_runner_generator.py
index 129e8c863..fb97efb0f 100644
--- a/tests/unit/pipelex/builder/test_runner_generator.py
+++ b/tests/unit/pipelex/builder/test_runner_generator.py
@@ -4,7 +4,7 @@ from __future__ import annotations
from pipelex.builder.runner_code import generate_compact_memory_entry, value_to_python_code
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
class TestValueToPythonCode:
diff --git a/tests/unit/pipelex/client/test_api_serialization.py b/tests/unit/pipelex/client/test_api_serialization.py
index f81df1f4a..6bc0f3f18 100644
--- a/tests/unit/pipelex/client/test_api_serialization.py
+++ b/tests/unit/pipelex/client/test_api_serialization.py
@@ -9,13 +9,22 @@ from pydantic import BaseModel
from pipelex.client.api_serializer import ApiSerializer
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.number_content import NumberContent
+from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
-from tests.test_pipelines.concepts.datetime import DateTimeEvent
+
+
+class DateTimeEvent(StructuredContent):
+ """Test model for datetime content."""
+
+ event_name: str
+ start_time: datetime
+ end_time: datetime
+ created_at: datetime
# Test models for complex scenarios
diff --git a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
index bb40da0c9..b6c48bddd 100644
--- a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
+++ b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py
@@ -3,7 +3,7 @@
from typing import ClassVar
from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
@@ -367,7 +367,7 @@ class PipeSorterTestCases:
("multiple_chains", MULTIPLE_CHAINS_PIPES, MULTIPLE_CHAINS_EXPECTED, None),
("pipe_batch", PIPE_BATCH_PIPES, PIPE_BATCH_EXPECTED, None),
("pipe_condition", PIPE_CONDITION_PIPES, PIPE_CONDITION_EXPECTED, None),
- ("circular_dependency", CIRCULAR_PIPES, None, PipeDefinitionError),
+ ("circular_dependency", CIRCULAR_PIPES, None, PipeValidationError),
("missing_dependency", MISSING_DEPENDENCY_PIPES, MISSING_DEPENDENCY_EXPECTED, None),
("image_inversion", IMAGE_INVERSION_PIPES, IMAGE_INVERSION_EXPECTED, None),
("complex_nested", COMPLEX_NESTED_PIPES, COMPLEX_NESTED_EXPECTED, None),
diff --git a/tests/unit/pipelex/core/concepts/concept_factory/data.py b/tests/unit/pipelex/core/concepts/concept_factory/data.py
index b70fbf5b3..7b3bcbd7c 100644
--- a/tests/unit/pipelex/core/concepts/concept_factory/data.py
+++ b/tests/unit/pipelex/core/concepts/concept_factory/data.py
@@ -6,8 +6,8 @@ from pipelex.core.concepts.concept_blueprint import (
ConceptStructureBlueprint,
)
from pipelex.core.concepts.concept_factory import DomainAndConceptCode
-from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprintFieldType
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
diff --git a/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py b/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py
index 53313f834..59cf33f9d 100644
--- a/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py
+++ b/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py
@@ -7,7 +7,7 @@ from pipelex.core.concepts.concept_blueprint import (
)
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.exceptions import StructureClassError
+from pipelex.core.concepts.exceptions import ConceptFactoryError
from tests.unit.pipelex.core.concepts.concept_factory.data import TestCases
@@ -126,7 +126,7 @@ class TestConceptFactory:
"""Test that make_from_blueprint raises StructureClassError for invalid structure class."""
blueprint = ConceptBlueprint(description="A concept with invalid structure", structure="NonExistentStructureClass")
- with pytest.raises(StructureClassError, match="is not a registered subclass of StuffContent"):
+ with pytest.raises(ConceptFactoryError, match="is not a registered subclass of StuffContent"):
ConceptFactory.make_from_blueprint(
domain="my_domain",
concept_code="TestConcept",
diff --git a/tests/unit/pipelex/core/concepts/concept_library/test_concept_library.py b/tests/unit/pipelex/core/concepts/concept_library/test_concept_library.py
index 011b12c87..a9d3ef239 100644
--- a/tests/unit/pipelex/core/concepts/concept_library/test_concept_library.py
+++ b/tests/unit/pipelex/core/concepts/concept_library/test_concept_library.py
@@ -1,12 +1,15 @@
+from typing import Callable
+
+from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
-from pipelex.hub import get_concept_library
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.hub import get_native_concept
class TestConceptLibrary:
- def test_is_image_concept(self):
- concept_library = get_concept_library()
+ def test_is_image_concept(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
native_image_concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE)
concept_1 = ConceptFactory.make_from_blueprint(
@@ -62,10 +65,33 @@ class TestConceptLibrary:
),
)
- assert concept_library.is_image_concept(concept=native_image_concept) is True
- assert concept_library.is_image_concept(concept=concept_1) is True
- assert concept_library.is_image_concept(concept=concept_2) is True
- assert concept_library.is_image_concept(concept=concept_3) is True
- assert concept_library.is_image_concept(concept=concept_4) is True
- assert concept_library.is_image_concept(concept=concept_5) is False
- assert concept_library.is_image_concept(concept=concept_6) is False
+ assert (
+ Concept.are_concept_compatible(
+ concept_1=native_image_concept, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True
+ )
+ is True
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_1, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is True
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_2, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is True
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_3, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is True
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_4, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is True
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_5, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is False
+ )
+ assert (
+ Concept.are_concept_compatible(concept_1=concept_6, concept_2=get_native_concept(native_concept=NativeConceptCode.IMAGE), strict=True)
+ is False
+ )
diff --git a/tests/unit/pipelex/core/concepts/test_concept.py b/tests/unit/pipelex/core/concepts/test_concept.py
index 91f1d371f..c4e70c197 100644
--- a/tests/unit/pipelex/core/concepts/test_concept.py
+++ b/tests/unit/pipelex/core/concepts/test_concept.py
@@ -3,8 +3,9 @@ import pytest
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.concepts.exceptions import ConceptStringError
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.exceptions import NativeConceptDefinitionError
from pipelex.core.concepts.validation import validate_concept_string
from pipelex.core.domains.domain import SpecialDomain
@@ -48,15 +49,24 @@ class TestConcept:
NativeConceptCode.get_validated_native_concept_string(f"{SpecialDomain.NATIVE}.{NativeConceptCode.NUMBER}")
== f"{SpecialDomain.NATIVE}.{NativeConceptCode.NUMBER}"
)
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.TEXT}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.IMAGE}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.PDF}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.TEXT_AND_IMAGES}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.NUMBER}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.ANYTHING}") is None
- assert NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.DYNAMIC}") is None
- assert NativeConceptCode.get_validated_native_concept_string("RandomConcept") is None
- assert NativeConceptCode.get_validated_native_concept_string("text") is None
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.TEXT}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.IMAGE}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.PDF}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.TEXT_AND_IMAGES}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.NUMBER}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.ANYTHING}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string(f"not_native.{NativeConceptCode.DYNAMIC}")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string("RandomConcept")
+ with pytest.raises(NativeConceptDefinitionError):
+ NativeConceptCode.get_validated_native_concept_string("text")
def test_is_native_concept(self):
"""Test is_native_concept method."""
@@ -211,10 +221,6 @@ class TestConcept:
with pytest.raises(ConceptStringError):
validate_concept_string(f"{valid_domain}.text-name")
- # Invalid native concept
- with pytest.raises(ConceptStringError):
- validate_concept_string(f"{SpecialDomain.NATIVE}.InvalidNativeConcept")
-
def test_are_concept_compatible(self):
concept1 = ConceptFactory.make_from_blueprint(
concept_code="Code1",
diff --git a/tests/unit/pipelex/core/concepts/test_concept_compact_memory.py b/tests/unit/pipelex/core/concepts/test_concept_compact_memory.py
index c947e7b1b..ec1f903a2 100644
--- a/tests/unit/pipelex/core/concepts/test_concept_compact_memory.py
+++ b/tests/unit/pipelex/core/concepts/test_concept_compact_memory.py
@@ -3,7 +3,7 @@
from __future__ import annotations
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
class TestConceptCompactMemory:
diff --git a/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py b/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py
index 16822aaff..959332300 100644
--- a/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py
+++ b/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py
@@ -1,9 +1,10 @@
-from typing import TYPE_CHECKING
+from pathlib import Path
+from typing import TYPE_CHECKING, Callable
import pytest
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.hub import get_concept_library, get_native_concept, get_required_concept
from pipelex.system.registries.class_registry_utils import ClassRegistryUtils
@@ -14,8 +15,8 @@ if TYPE_CHECKING:
from pipelex.core.concepts.concept import Concept
[email protected](scope="module", autouse=True)
-def register_test_concepts():
[email protected](scope="class", autouse=True)
+def register_test_concepts(load_test_library: Callable[[list[Path]], None]):
"""Register test concepts for the module.
This fixture:
@@ -26,6 +27,7 @@ def register_test_concepts():
The cleanup ensures test isolation between modules.
"""
+ load_test_library([Path(__file__).parent])
concept_library = get_concept_library()
# Register the test structure classes
diff --git a/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py b/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py
index 214ba66d5..bc8629558 100644
--- a/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py
+++ b/tests/unit/pipelex/core/concepts/test_structure_generator_escaping.py
@@ -8,7 +8,7 @@ generated Python code.
import ast
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.structure_generator import StructureGenerator
+from pipelex.core.concepts.structure_generation.generator import StructureGenerator
from pipelex.core.stuffs.structured_content import StructuredContent
diff --git a/tests/unit/pipelex/core/concepts/test_structured_output_generator.py b/tests/unit/pipelex/core/concepts/test_structured_output_generator.py
index d25b97b06..8cb417264 100644
--- a/tests/unit/pipelex/core/concepts/test_structured_output_generator.py
+++ b/tests/unit/pipelex/core/concepts/test_structured_output_generator.py
@@ -3,7 +3,8 @@ import pytest
from pipelex import pretty_print
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.structure_generator import ConceptStructureValidationError, StructureGenerator
+from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureValidationError
+from pipelex.core.concepts.structure_generation.generator import StructureGenerator
from pipelex.core.stuffs.structured_content import StructuredContent
diff --git a/tests/unit/pipelex/core/interpreter/test_interpreter.py b/tests/unit/pipelex/core/interpreter/test_interpreter.py
index 32073c02a..a351aa42d 100644
--- a/tests/unit/pipelex/core/interpreter/test_interpreter.py
+++ b/tests/unit/pipelex/core/interpreter/test_interpreter.py
@@ -1,30 +1,17 @@
-from pathlib import Path
-
import pytest
from pipelex import log, pretty_print
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.interpreter import PipelexInterpreter
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
from tests.unit.pipelex.core.test_data import InterpreterTestCases
class TestPipelexInterpreter:
- def test_init_with_both_file_path_and_content(self, tmp_path: Path):
- """Test initialization with both file_path and file_content."""
- test_file = tmp_path / "test.plx"
- test_file.write_text("domain = 'test'\n[concept]\n")
- content = "domain = 'other'\n[concept]\n"
-
- interpreter = PipelexInterpreter(file_path=test_file, file_content=content)
- assert interpreter.file_path == test_file
- assert interpreter.file_content == content
-
@pytest.mark.parametrize(("test_name", "plx_content", "expected_blueprint"), InterpreterTestCases.VALID_TEST_CASES)
def test_make_pipelex_bundle_blueprint(self, test_name: str, plx_content: str, expected_blueprint: PipelexBundleBlueprint):
"""Test making blueprint from various valid PLX content."""
- interpreter = PipelexInterpreter(file_content=plx_content)
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(plx_content=plx_content)
- blueprint = interpreter.make_pipelex_bundle_blueprint()
pretty_print(blueprint, title=f"Blueprint {test_name}")
pretty_print(expected_blueprint, title=f"Expected blueprint {test_name}")
assert blueprint == expected_blueprint
@@ -32,8 +19,6 @@ class TestPipelexInterpreter:
@pytest.mark.parametrize(("test_name", "invalid_plx_content", "expected_exception"), InterpreterTestCases.ERROR_TEST_CASES)
def test_invalid_plx_should_raise_exception(self, test_name: str, invalid_plx_content: str, expected_exception: type[Exception]):
"""Test that invalid PLX content raises appropriate exceptions."""
- interpreter = PipelexInterpreter(file_content=invalid_plx_content)
log.verbose(f"Testing invalid PLX content: {test_name}")
-
with pytest.raises(expected_exception):
- interpreter.make_pipelex_bundle_blueprint()
+ PipelexInterpreter.make_pipelex_bundle_blueprint(plx_content=invalid_plx_content)
diff --git a/tests/unit/pipelex/core/memory/conftest.py b/tests/unit/pipelex/core/memory/conftest.py
index faf9c248a..920ca8b66 100644
--- a/tests/unit/pipelex/core/memory/conftest.py
+++ b/tests/unit/pipelex/core/memory/conftest.py
@@ -5,7 +5,7 @@ from typing import ClassVar
import pytest
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
@@ -13,6 +13,7 @@ from pipelex.core.stuffs.html_content import HtmlContent
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.number_content import NumberContent
+from pipelex.core.stuffs.pdf_content import PDFContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
@@ -67,28 +68,36 @@ class TestWorkingMemoryData:
@pytest.fixture
def single_text_memory() -> WorkingMemory:
"""Create WorkingMemory with single text content."""
- return WorkingMemoryFactory.make_from_text(
- text=TestWorkingMemoryData.SAMPLE_TEXT, concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.TEXT, name="sample_text"
+ return WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ name="sample_text",
+ content=TextContent(text=TestWorkingMemoryData.SAMPLE_TEXT),
+ ),
)
@pytest.fixture
def single_image_memory() -> WorkingMemory:
"""Create WorkingMemory with single image content."""
- return WorkingMemoryFactory.make_from_image(
- image_url=TestWorkingMemoryData.SAMPLE_IMAGE_URL,
- concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.IMAGE,
- name="sample_image",
+ return WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ name="sample_image",
+ content=ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL),
+ ),
)
@pytest.fixture
def single_pdf_memory() -> WorkingMemory:
"""Create WorkingMemory with single PDF content."""
- return WorkingMemoryFactory.make_from_pdf(
- pdf_url=TestWorkingMemoryData.SAMPLE_PDF_URL,
- concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.PDF,
- name="pdf_document",
+ return WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.PDF),
+ name="pdf_document",
+ content=PDFContent(url=TestWorkingMemoryData.SAMPLE_PDF_URL),
+ ),
)
diff --git a/tests/unit/pipelex/core/memory/test_dry_working_memory.py b/tests/unit/pipelex/core/memory/test_dry_working_memory.py
index 9553a47dd..833868012 100644
--- a/tests/unit/pipelex/core/memory/test_dry_working_memory.py
+++ b/tests/unit/pipelex/core/memory/test_dry_working_memory.py
@@ -2,12 +2,12 @@ import pytest
from pipelex import log
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
-from pipelex.core.pipes.input_requirements import TypedNamedInputRequirement
+from pipelex.core.pipes.inputs.input_requirements import TypedNamedInputRequirement
from pipelex.core.stuffs.page_content import PageContent
from pipelex.core.stuffs.text_content import TextContent
-from tests.test_pipelines.tricky_questions import ThoughtfulAnswer
+from tests.unit.pipelex.core.memory.tricky_questions import ThoughtfulAnswer
@pytest.mark.dry_runnable
diff --git a/tests/unit/pipelex/core/memory/test_working_memory_basic.py b/tests/unit/pipelex/core/memory/test_working_memory_basic.py
index de907a377..b369ed661 100644
--- a/tests/unit/pipelex/core/memory/test_working_memory_basic.py
+++ b/tests/unit/pipelex/core/memory/test_working_memory_basic.py
@@ -1,4 +1,4 @@
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.stuffs.text_content import TextContent
diff --git a/tests/test_pipelines/tricky_questions.py b/tests/unit/pipelex/core/memory/tricky_questions.py
similarity index 100%
rename from tests/test_pipelines/tricky_questions.py
rename to tests/unit/pipelex/core/memory/tricky_questions.py
diff --git a/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py b/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py
new file mode 100644
index 000000000..818dbb3f1
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_concept_multiplicity_formatting.py
@@ -0,0 +1,43 @@
+"""Unit tests for concept code formatting with multiplicity notation."""
+
+import pytest
+
+from pipelex.core.pipes.variable_multiplicity import format_concept_with_multiplicity
+
+
+class TestFormatConceptWithMultiplicity:
+ """Test the format_concept_with_multiplicity helper function."""
+
+ @pytest.mark.parametrize(
+ ("concept_code", "multiplicity", "expected"),
+ [
+ # Simple concepts - no multiplicity
+ ("Text", None, "Text"),
+ ("ConceptName", None, "ConceptName"),
+ # Simple concepts - variable length
+ ("Text", True, "Text[]"),
+ ("Invoice", True, "Invoice[]"),
+ # Simple concepts - fixed length
+ ("Text", 3, "Text[3]"),
+ ("Image", 1, "Image[1]"),
+ ("Document", 10, "Document[10]"),
+ # Domain-qualified concepts
+ ("domain.Text", None, "domain.Text"),
+ ("domain.Text", True, "domain.Text[]"),
+ ("domain.Text", 5, "domain.Text[5]"),
+ ("expense.Invoice", None, "expense.Invoice"),
+ ("expense.Invoice", True, "expense.Invoice[]"),
+ ("expense.Invoice", 10, "expense.Invoice[10]"),
+ # Edge cases - underscores, numbers
+ ("my_concept", None, "my_concept"),
+ ("my_concept", True, "my_concept[]"),
+ ("my_domain.MyConcept", 5, "my_domain.MyConcept[5]"),
+ ("Concept123", 2, "Concept123[2]"),
+ # Edge cases - zero and large numbers
+ ("Text", 0, "Text[0]"),
+ ("Text", 999, "Text[999]"),
+ ],
+ )
+ def test_format_concept_with_multiplicity(self, concept_code: str, multiplicity: int | bool | None, expected: str):
+ """Test formatting concept codes with all multiplicity variations."""
+ assert format_concept_with_multiplicity(concept_code, multiplicity) == expected
diff --git a/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
index 9608b96eb..63a2fce48 100644
--- a/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
+++ b/tests/unit/pipelex/core/pipes/test_input_requirements_factory.py
@@ -1,8 +1,11 @@
+from typing import Callable
+
import pytest
-from pipelex.core.concepts.exceptions import ConceptLibraryConceptNotFoundError, ConceptStringError
-from pipelex.core.pipes.input_requirements import InputRequirement
-from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory, InputRequirementsFactoryError
+from pipelex.core.concepts.exceptions import ConceptStringError
+from pipelex.core.pipes.inputs.input_requirements import InputRequirement
+from pipelex.core.pipes.inputs.input_requirements_factory import InputRequirementsFactory, InputRequirementsFactoryError
+from pipelex.libraries.concept.exceptions import ConceptLibraryError
from tests.unit.pipelex.core.pipes.data import (
CONCEPT_CODE_RESOLUTION_TEST_CASES,
DIFFERENT_CONCEPT_CODES_TEST_CASES,
@@ -22,9 +25,15 @@ class TestMakeInputRequirementsFromString:
SINGLE_ITEM_NO_BRACKETS_TEST_CASES,
)
def test_single_item_default_no_brackets(
- self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int | bool | None
+ self,
+ domain: str,
+ requirement_str: str,
+ expected_concept_string: str,
+ expected_multiplicity: int | bool | None,
+ load_empty_library: Callable[[], None],
):
"""Test parsing a concept string without brackets (single item, default)."""
+ load_empty_library()
result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)
assert isinstance(result, InputRequirement)
@@ -35,7 +44,10 @@ class TestMakeInputRequirementsFromString:
("domain", "requirement_str", "expected_concept_string"),
MULTIPLE_ITEMS_EMPTY_BRACKETS_TEST_CASES,
)
- def test_multiple_items_with_empty_brackets(self, domain: str, requirement_str: str, expected_concept_string: str):
+ def test_multiple_items_with_empty_brackets(
+ self, domain: str, requirement_str: str, expected_concept_string: str, load_empty_library: Callable[[], None]
+ ):
+ load_empty_library()
"""Test parsing a concept string with empty brackets (multiple items)."""
result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)
@@ -47,8 +59,11 @@ class TestMakeInputRequirementsFromString:
("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),
FIXED_COUNT_TEST_CASES,
)
- def test_fixed_count_with_number_in_brackets(self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int):
+ def test_fixed_count_with_number_in_brackets(
+ self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int, load_empty_library: Callable[[], None]
+ ):
"""Test parsing a concept string with a number in brackets (fixed count)."""
+ load_empty_library()
result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)
assert isinstance(result, InputRequirement)
@@ -59,7 +74,10 @@ class TestMakeInputRequirementsFromString:
("domain", "requirement_str", "expected_concept_string", "expected_multiplicity"),
VARIOUS_FIXED_COUNTS_TEST_CASES,
)
- def test_various_fixed_counts(self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int):
+ def test_various_fixed_counts(
+ self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int, load_empty_library: Callable[[], None]
+ ):
+ load_empty_library()
"""Test parsing concept strings with various numbers in brackets."""
result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)
assert result.multiplicity == expected_multiplicity, f"Failed for {requirement_str}"
@@ -69,13 +87,15 @@ class TestMakeInputRequirementsFromString:
("domain", "requirement_str", "expected_concept_string"),
DIFFERENT_CONCEPT_CODES_TEST_CASES,
)
- def test_different_concept_codes(self, domain: str, requirement_str: str, expected_concept_string: str):
+ def test_different_concept_codes(self, domain: str, requirement_str: str, expected_concept_string: str, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test parsing various concept codes without multiplicity."""
result = InputRequirementsFactory.make_from_string(domain=domain, requirement_str=requirement_str)
assert result.concept.concept_string == expected_concept_string
assert result.multiplicity is None
- def test_custom_domain_concepts(self):
+ def test_custom_domain_concepts(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test parsing concept codes from custom domains."""
# Note: This test will only work if these concepts exist in the system
# For now, we'll test with native concepts, but the pattern should work for any domain
@@ -83,24 +103,28 @@ class TestMakeInputRequirementsFromString:
assert result.concept.concept_string == "native.Text"
assert result.multiplicity == 3
- def test_concept_not_found_raises_error(self):
+ def test_concept_not_found_raises_error(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that an invalid concept code raises ConceptLibraryConceptNotFoundError."""
- with pytest.raises(ConceptLibraryConceptNotFoundError):
+ with pytest.raises(ConceptLibraryError):
InputRequirementsFactory.make_from_string(domain="nonexistent", requirement_str="nonexistent.InvalidConcept")
- def test_concept_not_found_with_multiplicity_raises_error(self):
+ def test_concept_not_found_with_multiplicity_raises_error(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that an invalid concept code with multiplicity raises ConceptLibraryConceptNotFoundError."""
- with pytest.raises(ConceptLibraryConceptNotFoundError):
+ with pytest.raises(ConceptLibraryError):
InputRequirementsFactory.make_from_string(domain="nonexistent", requirement_str="nonexistent.InvalidConcept[5]")
- def test_empty_string_raises_value_error(self):
+ def test_empty_string_raises_value_error(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that an empty string raises InputRequirementsFactorySyntaxError."""
with pytest.raises(InputRequirementsFactoryError, match="Invalid input requirement string") as exc_info:
InputRequirementsFactory.make_from_string(domain="native", requirement_str="")
# This error is raised directly without a cause
assert exc_info.value.__cause__ is None
- def test_malformed_brackets_with_non_digit(self):
+ def test_malformed_brackets_with_non_digit(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that brackets with non-digit content are treated as part of concept string."""
# The regex will match "native.Text[abc]" as concept="native.Text[abc]", multiplicity=None
# This will then fail during concept validation with ConceptCodeError
@@ -109,7 +133,8 @@ class TestMakeInputRequirementsFromString:
assert exc_info.value.__cause__ is not None
assert isinstance(exc_info.value.__cause__, ConceptStringError)
- def test_multiplicity_zero_in_brackets(self):
+ def test_multiplicity_zero_in_brackets(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test parsing a concept string with 0 in brackets."""
result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[0]")
@@ -117,12 +142,14 @@ class TestMakeInputRequirementsFromString:
assert result.concept.concept_string == "native.Text"
assert result.multiplicity == 0
- def test_return_type(self):
+ def test_return_type(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that the method returns an InputRequirement instance."""
result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text")
assert isinstance(result, InputRequirement)
- def test_concept_attribute_access(self):
+ def test_concept_attribute_access(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that the returned InputRequirement has proper concept attributes."""
result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[5]")
@@ -131,14 +158,16 @@ class TestMakeInputRequirementsFromString:
assert result.concept.concept_string == "native.Text"
assert result.concept.code == "Text"
- def test_edge_case_very_long_number(self):
+ def test_edge_case_very_long_number(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test parsing with a very long number in brackets."""
result = InputRequirementsFactory.make_from_string(domain="native", requirement_str="native.Text[999999]")
assert result.multiplicity == 999999
assert result.concept.concept_string == "native.Text"
- def test_whitespace_not_trimmed(self):
+ def test_whitespace_not_trimmed(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that whitespace is not automatically trimmed."""
# Whitespace should cause domain validation to fail
with pytest.raises(InputRequirementsFactoryError) as exc_info:
@@ -152,7 +181,8 @@ class TestMakeInputRequirementsFromString:
assert exc_info.value.__cause__ is not None
assert isinstance(exc_info.value.__cause__, ConceptStringError)
- def test_multiple_brackets_treated_as_concept_name(self):
+ def test_multiple_brackets_treated_as_concept_name(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that multiple brackets are treated as part of the concept name."""
# "native.Text[5][10]" should match as concept="native.Text[5]", multiplicity=10
# This will fail during concept code validation
@@ -161,7 +191,8 @@ class TestMakeInputRequirementsFromString:
assert exc_info.value.__cause__ is not None
assert isinstance(exc_info.value.__cause__, ConceptStringError)
- def test_brackets_at_start_treated_as_concept_name(self):
+ def test_brackets_at_start_treated_as_concept_name(self, load_empty_library: Callable[[], None]):
+ load_empty_library()
"""Test that brackets at the start are part of the concept name."""
# This will fail during domain validation
with pytest.raises(InputRequirementsFactoryError) as exc_info:
@@ -181,6 +212,7 @@ class TestMakeInputRequirementsFromString:
expected_concept_string: str,
expected_multiplicity: int | bool | None,
description: str,
+ load_empty_library: Callable[[], None],
):
"""Test that concept codes are resolved correctly with domain and concept_codes_from_same_domain.
@@ -189,6 +221,7 @@ class TestMakeInputRequirementsFromString:
2. Unknown concepts (not native, not in the domain) become implicit concepts
3. concept_codes_from_same_domain helps resolve ambiguous concept codes
"""
+ load_empty_library()
result = InputRequirementsFactory.make_from_string(
domain=domain,
requirement_str=requirement_str,
@@ -204,9 +237,15 @@ class TestMakeInputRequirementsFromString:
EXPLICIT_DOMAIN_IN_STRING_TEST_CASES,
)
def test_explicit_domain_in_string(
- self, domain: str, requirement_str: str, expected_concept_string: str, expected_multiplicity: int | bool | None
+ self,
+ domain: str,
+ requirement_str: str,
+ expected_concept_string: str,
+ expected_multiplicity: int | bool | None,
+ load_empty_library: Callable[[], None],
):
"""Test that explicitly specifying a domain in the requirement string works correctly."""
+ load_empty_library()
result = InputRequirementsFactory.make_from_string(
domain=domain,
requirement_str=requirement_str,
diff --git a/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py b/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py
new file mode 100644
index 000000000..84a766090
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_pipe_blueprint.py
@@ -0,0 +1,144 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.pipes.pipe_blueprint import PipeBlueprint, PipeCategory, PipeType
+
+
+class ConcretePipeBlueprint(PipeBlueprint):
+ pass
+
+
+class TestPipeBlueprintValidation:
+ def test_validate_pipe_type_correct(self):
+ for pipe_type_enum in PipeType:
+ pipe_type = pipe_type_enum.value
+ match pipe_type:
+ case "PipeFunc" | "PipeImgGen" | "PipeCompose" | "PipeLLM" | "PipeExtract":
+ category = "PipeOperator"
+ case "PipeBatch" | "PipeCondition" | "PipeParallel" | "PipeSequence":
+ category = "PipeController"
+ blueprint = ConcretePipeBlueprint(
+ type=pipe_type,
+ pipe_category=category,
+ output="Text",
+ )
+ assert blueprint.type == pipe_type
+
+ def test_validate_pipe_type_incorrect(self):
+ invalid_types = ["InvalidType", "PipeTest", "RandomPipe", "NotAPipe", ""]
+ for invalid_type in invalid_types:
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type=invalid_type,
+ pipe_category="PipeOperator",
+ output="Text",
+ )
+ assert "Invalid pipe type" in str(exc_info.value)
+
+ def test_validate_pipe_category_correct(self):
+ for category_enum in PipeCategory:
+ category = category_enum.value
+ match category:
+ case "PipeOperator":
+ pipe_type = "PipeLLM"
+ case "PipeController":
+ pipe_type = "PipeSequence"
+ blueprint = ConcretePipeBlueprint(
+ type=pipe_type,
+ pipe_category=category,
+ output="Text",
+ )
+ assert blueprint.pipe_category == category
+
+ def test_validate_pipe_category_incorrect(self):
+ invalid_categories = ["InvalidCategory", "Operator", "Controller", "PipeOp", "RandomCategory", ""]
+ for invalid_category in invalid_categories:
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category=invalid_category,
+ output="Text",
+ )
+ assert "Invalid pipe category" in str(exc_info.value)
+
+ def test_validate_inputs_blueprint_correct(self):
+ blueprint = ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs={"text": "Text", "prompt": "Text"},
+ output="Text",
+ )
+ assert blueprint.inputs == {"text": "Text", "prompt": "Text"}
+
+ blueprint = ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs=None,
+ output="Text",
+ )
+ assert blueprint.inputs is None
+
+ blueprint = ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs={"items": "Text[]", "count": "Number[2]"},
+ output="Text",
+ )
+ assert blueprint.inputs == {"items": "Text[]", "count": "Number[2]"}
+
+ def test_validate_inputs_blueprint_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs={"text": "InvalidFormat!"},
+ output="Text",
+ )
+ assert "Invalid input syntax" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs={"text": "invalid_concept"},
+ output="Text",
+ )
+ assert "Invalid concept string or code" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ inputs={"text": "Text"},
+ output="InvalidConcept!",
+ )
+ assert "Invalid concept specification syntax" in str(exc_info.value)
+
+ def test_extra_fields_raises_error(self):
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ output="Text",
+ extra_field="should not be allowed", # type: ignore[call-arg]
+ )
+ assert "extra fields" in str(exc_info.value).lower() or "Extra inputs are not permitted" in str(exc_info.value)
+
+ def test_missing_mandatory_fields_raises_error(self):
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(pipe_category="PipeOperator", output="Text") # type: ignore[call-arg]
+ assert "type" in str(exc_info.value).lower()
+
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ output="Text",
+ ) # type: ignore[call-arg]
+ assert "pipe_category" in str(exc_info.value).lower() or "pipe_category" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ ConcretePipeBlueprint(
+ type="PipeLLM",
+ pipe_category="PipeOperator",
+ ) # type: ignore[call-arg]
+ assert "output" in str(exc_info.value).lower()
diff --git a/tests/unit/pipelex/core/pipes/test_validation.py b/tests/unit/pipelex/core/pipes/test_validation.py
index 1e20169ea..c9142da35 100644
--- a/tests/unit/pipelex/core/pipes/test_validation.py
+++ b/tests/unit/pipelex/core/pipes/test_validation.py
@@ -1,6 +1,5 @@
import pytest
-from pipelex.core.pipes.exceptions import PipeBlueprintValueError
from pipelex.core.pipes.validation import is_valid_input_name, validate_input_name
@@ -90,13 +89,13 @@ class TestInputNameValidation:
],
)
def test_validate_input_name_invalid(self, invalid_input_name: str):
- """Test validate_input_name raises PipeBlueprintValueError for invalid input names."""
- with pytest.raises(PipeBlueprintValueError, match="Invalid input name syntax"):
+ """Test validate_input_name raises ValueError for invalid input names."""
+ with pytest.raises(ValueError, match="Invalid input name syntax"):
validate_input_name(invalid_input_name)
def test_validate_input_name_error_message(self):
"""Test that the error message is helpful and descriptive."""
- with pytest.raises(PipeBlueprintValueError) as exc_info:
+ with pytest.raises(ValueError, match="Invalid input name syntax") as exc_info:
validate_input_name("myInput.fieldName")
error_message = str(exc_info.value)
diff --git a/tests/unit/pipelex/core/stuffs/data.py b/tests/unit/pipelex/core/stuffs/data.py
index cde07771e..d9179c911 100644
--- a/tests/unit/pipelex/core/stuffs/data.py
+++ b/tests/unit/pipelex/core/stuffs/data.py
@@ -2,7 +2,7 @@ from pydantic import Field
from pipelex.client.protocol import StuffContentOrData
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.stuff import DictStuff, Stuff
diff --git a/tests/test_pipelines/test_content_factory.py b/tests/unit/pipelex/core/stuffs/test_content_factory.py
similarity index 100%
rename from tests/test_pipelines/test_content_factory.py
rename to tests/unit/pipelex/core/stuffs/test_content_factory.py
diff --git a/tests/unit/pipelex/core/stuffs/test_stuff_content_factory.py b/tests/unit/pipelex/core/stuffs/test_stuff_content_factory.py
index 40395c9f4..5b56cdee5 100644
--- a/tests/unit/pipelex/core/stuffs/test_stuff_content_factory.py
+++ b/tests/unit/pipelex/core/stuffs/test_stuff_content_factory.py
@@ -1,10 +1,11 @@
-from typing import Any, ClassVar
+from pathlib import Path
+from typing import Any, Callable, ClassVar
import pytest
from pipelex import log
from pipelex.core.concepts.concept_factory import ConceptFactory
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.stuff_factory import StuffContentFactory
@@ -143,8 +144,9 @@ class TestStuffContentFactory:
("test_name", "blueprint"),
TestCases.TEST_BLUEPRINTS,
)
- def test_blueprint_scenarios(self, test_name: str, blueprint: dict[str, Any]):
+ def test_blueprint_scenarios(self, test_name: str, blueprint: dict[str, Any], load_test_library: Callable[[list[Path]], None]):
"""Test various blueprint scenarios with parametrized test cases."""
+ load_test_library([Path("tests/unit/pipelex/core/stuffs")])
content = blueprint["content"]
if test_name.startswith("text_"):
diff --git a/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py b/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
index 2d680a651..41ec5e290 100644
--- a/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
+++ b/tests/unit/pipelex/core/stuffs/test_stuff_factory_implicit_memory.py
@@ -1,5 +1,6 @@
import os
-from typing import Any
+from pathlib import Path
+from typing import Any, Callable
import pytest
@@ -15,7 +16,8 @@ from tests.unit.pipelex.core.stuffs.data import ERROR_TEST_CASES, SEARCH_DOMAIN_
@pytest.fixture(scope="class")
-def setup_test_concept():
+def setup_test_concept(load_test_library: Callable[[list[Path]], None]):
+ load_test_library([Path(__file__).parent])
# Register the class in the class registry
ClassRegistryUtils.register_classes_in_file(
file_path=os.path.join(os.path.dirname(__file__), "data.py"),
diff --git a/tests/unit/pipelex/core/test_data/errors/invalid_plx.py b/tests/unit/pipelex/core/test_data/errors/invalid_plx.py
index ffbd0f01f..a1571ec44 100644
--- a/tests/unit/pipelex/core/test_data/errors/invalid_plx.py
+++ b/tests/unit/pipelex/core/test_data/errors/invalid_plx.py
@@ -1,6 +1,4 @@
-from pydantic import ValidationError
-
-from pipelex.core.interpreter import PLXDecodeError
+from pipelex.core.interpreter.interpreter import PipelexInterpreterError, PLXDecodeError
INVALID_PLX_SYNTAX = (
"invalid_plx_syntax",
@@ -63,7 +61,7 @@ description = "Domain without domain field"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
INVALID_DOMAIN_NAME = (
@@ -74,7 +72,7 @@ description = "Domain with invalid characters"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
EMPTY_DOMAIN = (
@@ -85,7 +83,7 @@ description = "Domain with empty string"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
INVALID_ROOT_KEY = (
@@ -97,7 +95,7 @@ invalid_root_key = "This key should not be allowed at root level"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
MULTIPLE_INVALID_ROOT_KEYS = (
@@ -111,7 +109,7 @@ unknown_field = "Another unknown field"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
WRONG_TYPE_FOR_DOMAIN = (
@@ -133,7 +131,7 @@ description = 456
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
WRONG_TYPE_FOR_CONCEPT_SECTION = (
@@ -142,7 +140,7 @@ WRONG_TYPE_FOR_CONCEPT_SECTION = (
description = "Domain with wrong type for concept"
concept = "should_be_dict_not_string"
""",
- ValidationError,
+ PipelexInterpreterError,
)
WRONG_TYPE_FOR_PIPE_SECTION = (
@@ -151,7 +149,7 @@ WRONG_TYPE_FOR_PIPE_SECTION = (
description = "Domain with wrong type for pipe"
pipe = "should_be_dict_not_string"
""",
- ValidationError,
+ PipelexInterpreterError,
)
INVALID_NESTED_SECTION = (
@@ -165,7 +163,7 @@ some_key = "This section is not allowed"
[concept]
TestConcept = "A test concept"
""",
- ValidationError,
+ PipelexInterpreterError,
)
INVALID_TABLE_SYNTAX = (
diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py b/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py
index e7df0c42d..4230dae80 100644
--- a/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py
+++ b/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py
@@ -1,7 +1,7 @@
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
PIPE_COMPOSE_WITHOUT_CATEGORY = (
@@ -12,6 +12,7 @@ description = "Domain with template processing pipe"
[pipe.process_template]
type = "PipeCompose"
description = "Process a Jinja2 template"
+inputs = { name = "Text" }
output = "Text"
template = "Hello {{ name }}!"
""",
@@ -22,6 +23,9 @@ template = "Hello {{ name }}!"
"process_template": PipeComposeBlueprint(
type="PipeCompose",
description="Process a Jinja2 template",
+ inputs={
+ "name": NativeConceptCode.TEXT,
+ },
output=NativeConceptCode.TEXT,
template="Hello {{ name }}!",
),
@@ -37,6 +41,7 @@ description = "Domain with template processing pipe"
[pipe.compose_output]
type = "PipeCompose"
description = "Process a Jinja2 template"
+inputs = { name = "Text" }
output = "Text"
[pipe.compose_output.template]
@@ -50,6 +55,9 @@ category = "markdown"
"compose_output": PipeComposeBlueprint(
type="PipeCompose",
description="Process a Jinja2 template",
+ inputs={
+ "name": NativeConceptCode.TEXT,
+ },
output=NativeConceptCode.TEXT,
template=TemplateBlueprint(
template="Hello {{ name }}!",
diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py b/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py
index 5d86d0d11..95309e12e 100644
--- a/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py
+++ b/tests/unit/pipelex/core/test_data/pipes/operators/extract/pipe_extract.py
@@ -1,5 +1,5 @@
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
PIPE_EXTRACT = (
@@ -10,6 +10,7 @@ description = "Domain with extract pipe"
[pipe.extract_text]
type = "PipeExtract"
description = "Extract text from document"
+inputs = { document = "PDF" }
output = "Page[]"
model = "extract_text_from_pdf"
""",
@@ -20,6 +21,9 @@ model = "extract_text_from_pdf"
"extract_text": PipeExtractBlueprint(
type="PipeExtract",
description="Extract text from document",
+ inputs={
+ "document": NativeConceptCode.PDF,
+ },
output=NativeConceptCode.PAGE.as_output_multiple_indeterminate,
model="extract_text_from_pdf",
),
diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/img_gen/pipe_img_gen.py b/tests/unit/pipelex/core/test_data/pipes/operators/img_gen/pipe_img_gen.py
index 67db2e092..49486e1e4 100644
--- a/tests/unit/pipelex/core/test_data/pipes/operators/img_gen/pipe_img_gen.py
+++ b/tests/unit/pipelex/core/test_data/pipes/operators/img_gen/pipe_img_gen.py
@@ -1,5 +1,5 @@
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
PIPE_IMG_GEN = (
diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py b/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py
index 5479a7b39..da348241f 100644
--- a/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py
+++ b/tests/unit/pipelex/core/test_data/pipes/operators/llm/pipe_llm.py
@@ -1,6 +1,6 @@
# ruff: noqa: E501
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
# Basic PipeLLM with prompt_template
@@ -98,6 +98,7 @@ description = "Domain with pipe definitions"
[pipe.expert_analysis]
type = "PipeLLM"
description = "Expert analysis with system prompt"
+inputs = { extracted_text = "Text" }
output = "Text"
prompt = """
Extract all articles/items from this invoice text: $extracted_text. For each item find: item name, quantity, unit price, total price, description, and product code if
@@ -111,6 +112,9 @@ Extract all articles/items from this invoice text: $extracted_text. For each ite
"expert_analysis": PipeLLMBlueprint(
type="PipeLLM",
description="Expert analysis with system prompt",
+ inputs={
+ "extracted_text": NativeConceptCode.TEXT,
+ },
output=NativeConceptCode.TEXT,
prompt="""Extract all articles/items from this invoice text: $extracted_text. For each item find: item name, quantity, unit price, total price, description, and product code if
available. Return each article as separate structured data.
diff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_blueprint.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_blueprint.py
new file mode 100644
index 000000000..4e5e446c7
--- /dev/null
+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_blueprint.py
@@ -0,0 +1,22 @@
+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
+
+
+class TestPipeBatchBlueprint:
+ def test_pipe_dependencies_correct(self):
+ blueprint = PipeBatchBlueprint(
+ branch_pipe_code="process_item",
+ input_list_name="items",
+ input_item_name="item",
+ inputs={"items": "Text[]"},
+ output="Text[]",
+ )
+ assert blueprint.pipe_dependencies == {"process_item"}
+
+ blueprint = PipeBatchBlueprint(
+ branch_pipe_code="transform_data",
+ input_list_name="data_list",
+ input_item_name="data",
+ inputs={"data_list": "Number[]"},
+ output="Number[]",
+ )
+ assert blueprint.pipe_dependencies == {"transform_data"}
diff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py
index 653cfe705..c3a650f03 100644
--- a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py
+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py
@@ -1,3 +1,5 @@
+from typing import Callable
+
import pytest
from pipelex import log
@@ -15,7 +17,9 @@ class TestPipeBatchValidateInputs:
self,
test_id: str,
blueprint: PipeBatchBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
# Validation happens automatically during instantiation via model_validator
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/__init__.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py
similarity index 100%
rename from tests/test_pipelines/pipe_controllers/pipe_sequence/__init__.py
rename to tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py
diff --git a/tests/unit/pipelex/pipe_controllers/condition/data.py b/tests/unit/pipelex/pipe_controllers/condition/data.py
index 11ff05efa..7d3f0ddc0 100644
--- a/tests/unit/pipelex/pipe_controllers/condition/data.py
+++ b/tests/unit/pipelex/pipe_controllers/condition/data.py
@@ -94,36 +94,6 @@ class PipeConditionInputTestCases:
VALID_SINGLE_OUTCOME,
]
- # Error test cases: (test_id, blueprint_dict, expected_error_message_fragment)
- # Using dicts instead of blueprints to avoid validation errors during import
- ERROR_BOTH_EXPRESSION_AND_TEMPLATE: ClassVar[tuple[str, dict[str, Any], str]] = (
- "both_expression_and_template",
- {
- "description": "Test case: both_expression_and_template",
- "inputs": {"data": "native.Text"},
- "output": "native.Text",
- "expression": "data",
- "expression_template": "{{ data }}",
- "outcomes": {"A": "pipe_a"},
- "default_outcome": "pipe_default",
- },
- "exactly one of expression_template or expression",
- )
-
- ERROR_NEITHER_EXPRESSION_NOR_TEMPLATE: ClassVar[tuple[str, dict[str, Any], str]] = (
- "neither_expression_nor_template",
- {
- "description": "Test case: neither_expression_nor_template",
- "inputs": {"data": "native.Text"},
- "output": "native.Text",
- "expression": None,
- "expression_template": None,
- "outcomes": {"A": "pipe_a"},
- "default_outcome": "pipe_default",
- },
- "exactly one of expression_template or expression",
- )
-
ERROR_EMPTY_OUTCOMES: ClassVar[tuple[str, dict[str, Any], str]] = (
"empty_outcomes",
{
@@ -138,7 +108,5 @@ class PipeConditionInputTestCases:
)
ERROR_CASES: ClassVar[list[tuple[str, dict[str, Any], str]]] = [
- ERROR_BOTH_EXPRESSION_AND_TEMPLATE,
- ERROR_NEITHER_EXPRESSION_NOR_TEMPLATE,
ERROR_EMPTY_OUTCOMES,
]
diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_blueprint.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_blueprint.py
new file mode 100644
index 000000000..fb4a0c10e
--- /dev/null
+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_blueprint.py
@@ -0,0 +1,65 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
+
+
+class TestPipeConditionBlueprint:
+ def test_pipe_dependencies_correct(self):
+ blueprint = PipeConditionBlueprint(
+ expression="status",
+ outcomes={"active": "process_active", "inactive": "process_inactive"},
+ default_outcome="process_default",
+ inputs={"status": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"process_active", "process_inactive", "process_default"}
+
+ blueprint = PipeConditionBlueprint(
+ expression="type",
+ outcomes={"A": "handle_a"},
+ default_outcome=SpecialOutcome.FAIL,
+ inputs={"type": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"handle_a"}
+
+ blueprint = PipeConditionBlueprint(
+ expression="flag",
+ outcomes={"yes": "process_yes", "no": "process_no"},
+ default_outcome=SpecialOutcome.CONTINUE,
+ inputs={"flag": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"process_yes", "process_no"}
+
+ def test_validate_outcome_map_correct(self):
+ blueprint = PipeConditionBlueprint(
+ expression="value",
+ outcomes={"high": "process_high"},
+ default_outcome="process_default",
+ inputs={"value": "Text"},
+ output="Text",
+ )
+ assert blueprint.outcomes == {"high": "process_high"}
+
+ blueprint = PipeConditionBlueprint(
+ expression="status",
+ outcomes={"active": "process_active", "inactive": "process_inactive"},
+ default_outcome="process_default",
+ inputs={"status": "Text"},
+ output="Text",
+ )
+ assert len(blueprint.outcomes) == 2
+
+ def test_validate_outcome_map_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeConditionBlueprint(
+ expression="value",
+ outcomes={},
+ default_outcome="process_default",
+ inputs={"value": "Text"},
+ output="Text",
+ )
+ assert "PipeConditionBlueprint must have at least one mapping in outcomes" in str(exc_info.value)
diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
index f06f8d936..f46102909 100644
--- a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py
@@ -1,10 +1,9 @@
-from typing import Any
+from typing import Any, Callable
import pytest
from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
from pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory
from tests.unit.pipelex.pipe_controllers.condition.data import PipeConditionInputTestCases
@@ -19,7 +18,9 @@ class TestPipeConditionValidateInputs:
self,
test_id: str,
blueprint: PipeConditionBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
# Validation happens automatically during instantiation via model_validator
@@ -42,17 +43,19 @@ class TestPipeConditionValidateInputs:
test_id: str,
blueprint_dict: dict[str, Any],
expected_error_message_fragment: str,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing error case: {test_id}")
- with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012
+ with pytest.raises((PipeValidationError, ValueError)) as exc_info: # noqa: PT012
# Construct blueprint from dict at test time to trigger validation
blueprint = PipeConditionBlueprint.model_validate(blueprint_dict)
PipeConditionFactory.make_from_blueprint(
domain="test_domain",
pipe_code=f"test_pipe_{test_id}",
blueprint=blueprint,
- )
+ ).validate_with_libraries()
error_str = str(exc_info.value)
assert expected_error_message_fragment in error_str, (
diff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py
new file mode 100644
index 000000000..3ee38ce2e
--- /dev/null
+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py
@@ -0,0 +1,96 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+
+
+class TestPipeParallelBlueprint:
+ def test_pipe_dependencies_correct(self):
+ blueprint = PipeParallelBlueprint(
+ parallels=[
+ SubPipeBlueprint(pipe="process_a", result="result_a"),
+ SubPipeBlueprint(pipe="process_b", result="result_b"),
+ ],
+ add_each_output=True,
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"process_a", "process_b"}
+
+ blueprint = PipeParallelBlueprint(
+ parallels=[
+ SubPipeBlueprint(pipe="step1", result="result1"),
+ SubPipeBlueprint(pipe="step2", result="result2"),
+ SubPipeBlueprint(pipe="step3", result="result3"),
+ ],
+ add_each_output=True,
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"step1", "step2", "step3"}
+
+ def test_validate_combined_output_correct(self):
+ blueprint = PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ combined_output="Text",
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.combined_output == "Text"
+
+ blueprint = PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ combined_output="Number",
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.combined_output == "Number"
+
+ def test_validate_combined_output_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ combined_output="InvalidConcept!",
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert "Combined output 'InvalidConcept!' is not a valid concept string or code" in str(exc_info.value)
+
+ def test_validate_output_options_correct(self):
+ blueprint = PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ add_each_output=True,
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.add_each_output is True
+
+ blueprint = PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ combined_output="Text",
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.combined_output == "Text"
+
+ blueprint = PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ add_each_output=True,
+ combined_output="Text",
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.add_each_output is True
+ assert blueprint.combined_output == "Text"
+
+ def test_validate_output_options_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeParallelBlueprint(
+ parallels=[SubPipeBlueprint(pipe="process", result="result")],
+ add_each_output=False,
+ combined_output=None,
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert "PipeParallel requires either add_each_output to be True or combined_output to be set" in str(exc_info.value)
diff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
index 250ad6501..4f4dd8bf1 100644
--- a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py
@@ -1,10 +1,9 @@
-from typing import Any
+from typing import Any, Callable
import pytest
from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallelFactory
from tests.unit.pipelex.pipe_controllers.parallel.data import PipeParallelInputTestCases
@@ -19,7 +18,9 @@ class TestPipeParallelValidateInputs:
self,
test_id: str,
blueprint: PipeParallelBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
# Validation happens automatically during instantiation via model_validator
@@ -42,10 +43,12 @@ class TestPipeParallelValidateInputs:
test_id: str,
blueprint_dict: dict[str, Any],
expected_error_message_fragment: str,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing error case: {test_id}")
- with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012
+ with pytest.raises((PipeValidationError, ValueError)) as exc_info: # noqa: PT012
# Construct blueprint from dict at test time to trigger validation
blueprint = PipeParallelBlueprint.model_validate(blueprint_dict)
PipeParallelFactory.make_from_blueprint(
diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_blueprint.py b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_blueprint.py
new file mode 100644
index 000000000..c67287855
--- /dev/null
+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_blueprint.py
@@ -0,0 +1,78 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+
+
+class TestPipeSequenceBlueprint:
+ def test_pipe_dependencies_correct(self):
+ blueprint = PipeSequenceBlueprint(
+ steps=[
+ SubPipeBlueprint(pipe="step1", result="result1"),
+ SubPipeBlueprint(pipe="step2", result="result2"),
+ ],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"step1", "step2"}
+
+ blueprint = PipeSequenceBlueprint(
+ steps=[
+ SubPipeBlueprint(pipe="process_a", result="result_a"),
+ SubPipeBlueprint(pipe="process_b", result="result_b"),
+ SubPipeBlueprint(pipe="process_c", result="result_c"),
+ ],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.pipe_dependencies == {"process_a", "process_b", "process_c"}
+
+ def test_ordered_pipe_dependencies_correct(self):
+ blueprint = PipeSequenceBlueprint(
+ steps=[
+ SubPipeBlueprint(pipe="step1", result="result1"),
+ SubPipeBlueprint(pipe="step2", result="result2"),
+ SubPipeBlueprint(pipe="step3", result="result3"),
+ ],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.ordered_pipe_dependencies == ["step1", "step2", "step3"]
+
+ blueprint = PipeSequenceBlueprint(
+ steps=[
+ SubPipeBlueprint(pipe="first", result="first_result"),
+ SubPipeBlueprint(pipe="second", result="second_result"),
+ ],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert blueprint.ordered_pipe_dependencies == ["first", "second"]
+
+ def test_validate_steps_correct(self):
+ blueprint = PipeSequenceBlueprint(
+ steps=[SubPipeBlueprint(pipe="step1", result="result1")],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert len(blueprint.steps) == 1
+
+ blueprint = PipeSequenceBlueprint(
+ steps=[
+ SubPipeBlueprint(pipe="step1", result="result1"),
+ SubPipeBlueprint(pipe="step2", result="result2"),
+ ],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert len(blueprint.steps) == 2
+
+ def test_validate_steps_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeSequenceBlueprint(
+ steps=[],
+ inputs={"data": "Text"},
+ output="Text",
+ )
+ assert "PipeSequence must have at least 1 step" in str(exc_info.value)
diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
index 3f73c4efe..77d523fc1 100644
--- a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py
@@ -1,10 +1,9 @@
-from typing import Any
+from typing import Any, Callable
import pytest
from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
from pipelex.pipe_controllers.sequence.pipe_sequence_factory import PipeSequenceFactory
from tests.unit.pipelex.pipe_controllers.sequence.data import PipeSequenceInputTestCases
@@ -19,7 +18,9 @@ class TestPipeSequenceValidateInputs:
self,
test_id: str,
blueprint: PipeSequenceBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
# Validation happens automatically during instantiation via model_validator
@@ -42,17 +43,19 @@ class TestPipeSequenceValidateInputs:
test_id: str,
blueprint_dict: dict[str, Any],
expected_error_message_fragment: str,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing error case: {test_id}")
- with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012
+ with pytest.raises((PipeValidationError, ValueError)) as exc_info: # noqa: PT012
# Construct blueprint from dict at test time to trigger validation
blueprint = PipeSequenceBlueprint.model_validate(blueprint_dict)
PipeSequenceFactory.make_from_blueprint(
domain="test_domain",
pipe_code=f"test_pipe_{test_id}",
blueprint=blueprint,
- )
+ ).validate_with_libraries()
error_str = str(exc_info.value)
assert expected_error_message_fragment in error_str, (
diff --git a/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py
new file mode 100644
index 000000000..01d3a3ffe
--- /dev/null
+++ b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py
@@ -0,0 +1,57 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
+
+
+class TestSubPipeBlueprint:
+ def test_validate_multiple_output_correct(self):
+ blueprint = SubPipeBlueprint(pipe="process")
+ assert blueprint.pipe == "process"
+ assert blueprint.nb_output is None
+ assert blueprint.multiple_output is None
+
+ blueprint = SubPipeBlueprint(pipe="process", nb_output=3)
+ assert blueprint.nb_output == 3
+ assert blueprint.multiple_output is None
+
+ blueprint = SubPipeBlueprint(pipe="process", multiple_output=True)
+ assert blueprint.nb_output is None
+ assert blueprint.multiple_output is True
+
+ def test_validate_multiple_output_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ SubPipeBlueprint(
+ pipe="process",
+ nb_output=3,
+ multiple_output=True,
+ )
+ assert "PipeStepBlueprint should have no more than '1' of nb_output or multiple_output" in str(exc_info.value)
+
+ def test_validate_batch_params_correct(self):
+ blueprint = SubPipeBlueprint(pipe="process")
+ assert blueprint.batch_over is None
+ assert blueprint.batch_as is None
+
+ blueprint = SubPipeBlueprint(
+ pipe="process",
+ batch_over="items",
+ batch_as="item",
+ )
+ assert blueprint.batch_over == "items"
+ assert blueprint.batch_as == "item"
+
+ def test_validate_batch_params_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ SubPipeBlueprint(
+ pipe="process",
+ batch_over="items",
+ )
+ assert "When 'batch_over' is specified, 'batch_as' must also be provided" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ SubPipeBlueprint(
+ pipe="process",
+ batch_as="item",
+ )
+ assert "When 'batch_as' is specified, 'batch_over' must also be provided" in str(exc_info.value)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
index b0066b52b..56ef984c0 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py
@@ -3,7 +3,6 @@ from typing import ClassVar
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint
-from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError
class PipeComposeInputTestCases:
@@ -95,46 +94,3 @@ class PipeComposeInputTestCases:
VALID_WITH_HTML_TEMPLATE,
VALID_COMPLEX_JINJA2,
]
-
- # Error test cases: (test_id, blueprint, expected_error_type, expected_error_message_fragment)
- ERROR_MISSING_INPUT_IN_TEMPLATE: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (
- "missing_input_in_template",
- PipeComposeBlueprint(
- description="Test case: missing_input_in_template",
- inputs={},
- output="native.Text",
- template="Hello {{ name }}!",
- ),
- Exception, # Will be PipeDefinitionError
- "Required variable 'name' is not in the inputs",
- )
-
- ERROR_TWO_INPUTS_THREE_VARIABLES: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (
- "two_inputs_three_variables",
- PipeComposeBlueprint(
- description="Test case: two_inputs_three_variables",
- inputs={"first_name": "native.Text", "last_name": "native.Text"},
- output="native.Text",
- template="Hello {{ first_name }} {{ last_name }} from {{ city }}!",
- ),
- Exception, # Will be PipeDefinitionError
- "Required variable 'city' is not in the inputs",
- )
-
- ERROR_INVALID_JINJA2_SYNTAX: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (
- "invalid_jinja2_syntax",
- PipeComposeBlueprint(
- description="Test case: invalid_jinja2_syntax",
- inputs={"name": "native.Text"},
- output="native.Text",
- template="Hello {{ name }!",
- ),
- Jinja2TemplateSyntaxError,
- "Could not parse Jinja2 template",
- )
-
- ERROR_CASES: ClassVar[list[tuple[str, PipeComposeBlueprint, type[Exception], str]]] = [
- ERROR_MISSING_INPUT_IN_TEMPLATE,
- ERROR_TWO_INPUTS_THREE_VARIABLES,
- ERROR_INVALID_JINJA2_SYNTAX,
- ]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_blueprint.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py
index be758053a..02c5500f8 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py
@@ -1,3 +1,5 @@
+from typing import Callable
+
import pytest
from pipelex import log
@@ -15,7 +17,9 @@ class TestPipeComposeValidateInputs:
self,
test_id: str,
blueprint: PipeComposeBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
# Validation happens automatically during instantiation via model_validator
@@ -28,28 +32,3 @@ class TestPipeComposeValidateInputs:
# Assert that the pipe was created successfully
assert pipe_compose is not None
assert pipe_compose.code == f"test_pipe_{test_id}"
-
- @pytest.mark.parametrize(
- ("test_id", "blueprint", "expected_error_type", "expected_error_message_fragment"),
- PipeComposeInputTestCases.ERROR_CASES,
- )
- def test_validate_inputs_error_cases(
- self,
- test_id: str,
- blueprint: PipeComposeBlueprint,
- expected_error_type: type[Exception],
- expected_error_message_fragment: str,
- ):
- log.verbose(f"Testing error case: {test_id}")
-
- with pytest.raises(expected_error_type) as exc_info:
- PipeComposeFactory.make_from_blueprint(
- domain="test_domain",
- pipe_code=f"test_pipe_{test_id}",
- blueprint=blueprint,
- )
-
- error_str = str(exc_info.value)
- assert expected_error_message_fragment in error_str, (
- f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"
- )
diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/data.py b/tests/unit/pipelex/pipe_operators/pipe_extract/data.py
index 230517642..2cf760a0b 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_extract/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/data.py
@@ -12,7 +12,7 @@ class PipeExtractInputTestCases:
PipeExtractBlueprint(
description="Test case: valid_image_input",
inputs={"document_image": "native.Image"},
- output="native.Page",
+ output="native.Page[]",
),
)
@@ -21,7 +21,7 @@ class PipeExtractInputTestCases:
PipeExtractBlueprint(
description="Test case: valid_pdf_input",
inputs={"document": "native.PDF"},
- output="native.Page",
+ output="native.Page[]",
),
)
@@ -30,7 +30,7 @@ class PipeExtractInputTestCases:
PipeExtractBlueprint(
description="Test case: valid_image_with_page_images",
inputs={"invoice_image": "native.Image"},
- output="native.Page",
+ output="native.Page[]",
page_images=True,
),
)
@@ -40,7 +40,7 @@ class PipeExtractInputTestCases:
PipeExtractBlueprint(
description="Test case: valid_pdf_with_page_views",
inputs={"contract": "native.PDF"},
- output="native.Page",
+ output="native.Page[]",
page_views=True,
page_views_dpi=150,
),
@@ -51,7 +51,7 @@ class PipeExtractInputTestCases:
PipeExtractBlueprint(
description="Test case: valid_image_with_captions",
inputs={"report_image": "native.Image"},
- output="native.Page",
+ output="native.Page[]",
page_images=True,
page_image_captions=True,
),
@@ -64,40 +64,3 @@ class PipeExtractInputTestCases:
VALID_PDF_WITH_PAGE_VIEWS,
VALID_IMAGE_WITH_CAPTIONS,
]
-
- # Error test cases: (test_id, blueprint, expected_error_message_fragment)
- ERROR_NO_INPUT: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (
- "no_input",
- PipeExtractBlueprint(
- description="Test case: no_input",
- inputs={},
- output="native.Page",
- ),
- "missing_input_variable",
- )
-
- ERROR_TOO_MANY_INPUTS: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (
- "too_many_inputs",
- PipeExtractBlueprint(
- description="Test case: too_many_inputs",
- inputs={"image1": "native.Image", "image2": "native.Image"},
- output="native.Page",
- ),
- "too_many_candidate_inputs",
- )
-
- ERROR_WRONG_INPUT_TYPE: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (
- "wrong_input_type",
- PipeExtractBlueprint(
- description="Test case: wrong_input_type",
- inputs={"text_doc": "native.Text"},
- output="native.Page",
- ),
- "inadequate_input_concept",
- )
-
- ERROR_CASES: ClassVar[list[tuple[str, PipeExtractBlueprint, str]]] = [
- ERROR_NO_INPUT,
- ERROR_TOO_MANY_INPUTS,
- ERROR_WRONG_INPUT_TYPE,
- ]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py
new file mode 100644
index 000000000..3b6d8e0c7
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py
@@ -0,0 +1,63 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
+
+
+class TestPipeExtractBlueprint:
+ def test_force_output_correct(self):
+ blueprint = PipeExtractBlueprint(
+ inputs={"document": "PDF"},
+ output="Page[]",
+ )
+ assert blueprint.output == "Page[]"
+
+ blueprint = PipeExtractBlueprint(
+ inputs={"image": "Image"},
+ output="Page[]",
+ )
+ assert blueprint.output == "Page[]"
+
+ def test_validate_inputs_correct(self):
+ blueprint = PipeExtractBlueprint(
+ inputs={"document": "PDF"},
+ output="Page[]",
+ )
+ assert blueprint.nb_inputs == 1
+ assert blueprint.input_names == ["document"]
+
+ blueprint = PipeExtractBlueprint(
+ inputs={"image": "Image"},
+ output="Page[]",
+ )
+ assert blueprint.nb_inputs == 1
+ assert blueprint.input_names == ["image"]
+
+ def test_validate_inputs_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeExtractBlueprint(
+ inputs={},
+ output="Page[]",
+ )
+ assert "Missing input provided for PipeExtract" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ PipeExtractBlueprint(
+ inputs={"doc1": "PDF", "doc2": "PDF"},
+ output="Page[]",
+ )
+ assert "Too many inputs provided for PipeExtract" in str(exc_info.value)
+
+ def test_validate_output_correct(self):
+ blueprint = PipeExtractBlueprint(
+ inputs={"document": "PDF"},
+ output="Page[]",
+ )
+ assert blueprint.output == "Page[]"
+
+ def test_validate_output_incorrect(self):
+ blueprint = PipeExtractBlueprint(
+ inputs={"document": "PDF"},
+ output="Page[]",
+ )
+ assert blueprint.output == "Page[]"
diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py
deleted file mode 100644
index 65bb53a5f..000000000
--- a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py
+++ /dev/null
@@ -1,55 +0,0 @@
-import pytest
-
-from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
-from pipelex.pipe_operators.extract.pipe_extract_factory import PipeExtractFactory
-from tests.unit.pipelex.pipe_operators.pipe_extract.data import PipeExtractInputTestCases
-
-
-class TestPipeExtractValidateInputs:
- @pytest.mark.parametrize(
- ("test_id", "blueprint"),
- PipeExtractInputTestCases.VALID_CASES,
- )
- def test_validate_inputs_valid_cases(
- self,
- test_id: str,
- blueprint: PipeExtractBlueprint,
- ):
- log.verbose(f"Testing valid case: {test_id}")
-
- # Validation happens automatically during instantiation via model_validator
- pipe_extract = PipeExtractFactory.make_from_blueprint(
- domain="test_domain",
- pipe_code=f"test_pipe_{test_id}",
- blueprint=blueprint,
- )
-
- # Assert that the pipe was created successfully
- assert pipe_extract is not None
- assert pipe_extract.code == f"test_pipe_{test_id}"
-
- @pytest.mark.parametrize(
- ("test_id", "blueprint", "expected_error_message_fragment"),
- PipeExtractInputTestCases.ERROR_CASES,
- )
- def test_validate_inputs_error_cases(
- self,
- test_id: str,
- blueprint: PipeExtractBlueprint,
- expected_error_message_fragment: str,
- ):
- log.verbose(f"Testing error case: {test_id}")
-
- with pytest.raises(StaticValidationError) as exc_info:
- PipeExtractFactory.make_from_blueprint(
- domain="test_domain",
- pipe_code=f"test_pipe_{test_id}",
- blueprint=blueprint,
- )
-
- error_str = str(exc_info.value)
- assert expected_error_message_fragment in error_str, (
- f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"
- )
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/data.py b/tests/unit/pipelex/pipe_operators/pipe_func/data.py
index 8aaef4cda..650398bd0 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/data.py
@@ -44,25 +44,6 @@ async def process_all(working_memory: WorkingMemory) -> TextContent:
return TextContent(text=f"processed all: {text}, {image.url}, {number}")
-# Error test functions - these have validation issues
-# NOT registered automatically - must be registered in test fixtures
-async def no_return_type_func(working_memory: WorkingMemory): # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
- """Test function with no return type annotation."""
- return TextContent(text="no return type")
-
-
-async def wrong_return_type_func(working_memory: WorkingMemory) -> str: # noqa: ARG001 # pyright: ignore[reportUnusedParameter]
- """Test function with wrong return type (not StuffContent)."""
- return "wrong type"
-
-
-# Map of error function names to actual functions (for test fixtures)
-ERROR_TEST_FUNCTIONS = {
- "no_return_type": no_return_type_func,
- "wrong_return_type": wrong_return_type_func,
-}
-
-
class PipeFuncInputTestCases:
"""Test cases for PipeFunc input validation."""
@@ -124,43 +105,3 @@ class PipeFuncInputTestCases:
VALID_IMAGE_INPUT,
VALID_MIXED_INPUTS,
]
-
- # Error test cases: (test_id, blueprint, expected_error_message_substring)
- ERROR_FUNCTION_NOT_FOUND: ClassVar[tuple[str, PipeFuncBlueprint, str]] = (
- "function_not_found",
- PipeFuncBlueprint(
- description="Test case: function_not_found",
- inputs={},
- output="native.Text",
- function_name="nonexistent_function",
- ),
- "not found in registry",
- )
-
- ERROR_NO_RETURN_TYPE: ClassVar[tuple[str, PipeFuncBlueprint, str]] = (
- "no_return_type",
- PipeFuncBlueprint(
- description="Test case: no_return_type",
- inputs={},
- output="native.Text",
- function_name="no_return_type",
- ),
- "has no return type annotation",
- )
-
- ERROR_WRONG_RETURN_TYPE: ClassVar[tuple[str, PipeFuncBlueprint, str]] = (
- "wrong_return_type",
- PipeFuncBlueprint(
- description="Test case: wrong_return_type",
- inputs={},
- output="native.Text",
- function_name="wrong_return_type",
- ),
- "is not a subclass of StuffContent",
- )
-
- ERROR_CASES: ClassVar[list[tuple[str, PipeFuncBlueprint, str]]] = [
- ERROR_FUNCTION_NOT_FOUND,
- ERROR_NO_RETURN_TYPE,
- ERROR_WRONG_RETURN_TYPE,
- ]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_blueprint.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py
index 199d0aa68..0c9ad0066 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py
@@ -1,56 +1,33 @@
+from pathlib import Path
+from typing import Callable
+
import pytest
from pipelex import log
-from pipelex.core.pipe_errors import PipeDefinitionError
from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
from pipelex.pipe_operators.func.pipe_func_factory import PipeFuncFactory
-from pipelex.system.registries.func_registry import func_registry
-from tests.unit.pipelex.pipe_operators.pipe_func.data import ERROR_TEST_FUNCTIONS, PipeFuncInputTestCases
+from tests.unit.pipelex.pipe_operators.pipe_func.data import PipeFuncInputTestCases
[email protected](scope="class")
-def register_error_test_functions():
- """Register error test functions that have validation issues.
+class TestPipeFuncValidation:
+ """Test PipeFunc with valid, registered, and functioning functions.
- These functions have intentional validation issues (no return type, wrong return type)
- and need to be registered bypassing normal eligibility checks to test validation logic.
- They are properly cleaned up after the test class completes.
+ Error cases (function not found, invalid return types, etc.) are tested
+ at the registry level in test_func_registry_utils.py.
"""
- registered_names: list[str] = []
-
- # Register each error test function
- # Note: We use register_function first (which will skip ineligible functions),
- # then manually add if needed for testing validation
- for func_name, func in ERROR_TEST_FUNCTIONS.items():
- # Try normal registration first
- func_registry.register_function(func, name=func_name)
-
- # If not registered (due to eligibility check), manually add for testing
- if not func_registry.has_function(func_name):
- # Directly add to registry for testing validation (intentionally bypassing checks)
- func_registry.root[func_name] = func
-
- registered_names.append(func_name)
- yield registered_names
-
- # Cleanup: use proper accessor to unregister all error test functions
- for func_name in registered_names:
- if func_registry.has_function(func_name):
- func_registry.unregister_function_by_name(func_name)
-
-
[email protected]("register_error_test_functions")
-class TestPipeFuncValidation:
@pytest.mark.parametrize(
("test_id", "blueprint"),
PipeFuncInputTestCases.VALID_CASES,
)
- def test_pipe_func_validate_valid_cases(
+ def test_pipe_func_with_valid_functions(
self,
test_id: str,
blueprint: PipeFuncBlueprint,
+ load_test_library: Callable[[list[Path]], None],
):
+ load_test_library([Path(Path(__file__).parent)])
+ """Test that PipeFunc works correctly with valid, registered functions."""
log.verbose(f"Testing valid case: {test_id}")
pipe_func = PipeFuncFactory.make_from_blueprint(
@@ -63,27 +40,3 @@ class TestPipeFuncValidation:
assert pipe_func is not None
assert pipe_func.code == f"test_pipe_{test_id}"
assert pipe_func.function_name == blueprint.function_name
-
- @pytest.mark.parametrize(
- ("test_id", "blueprint", "expected_error_substring"),
- PipeFuncInputTestCases.ERROR_CASES,
- )
- def test_pipe_func_validate_error_cases(
- self,
- test_id: str,
- blueprint: PipeFuncBlueprint,
- expected_error_substring: str,
- ):
- log.verbose(f"Testing error case: {test_id}")
-
- # Assert that creating the pipe raises PipeDefinitionError
- with pytest.raises(PipeDefinitionError) as exc_info:
- PipeFuncFactory.make_from_blueprint(
- domain="test_domain",
- pipe_code=f"test_pipe_{test_id}",
- blueprint=blueprint,
- )
-
- # Assert that the error message contains the expected substring
- error_message = str(exc_info.value)
- assert expected_error_substring in error_message, f"Expected error message to contain '{expected_error_substring}', but got: {error_message}"
diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py
index 8e6de6897..6b558bb2c 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py
@@ -1,6 +1,7 @@
from typing import ClassVar
from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, OutputFormat
+from pipelex.core.pipes.exceptions import PipeValidationErrorType
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
@@ -11,7 +12,7 @@ class PipeImgGenInputTestCases:
VALID_TEXT_INPUT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_text_input",
PipeImgGenBlueprint(
- description="Test case: valid_text_input",
+ description="VALID_TEXT_INPUT: Test case: valid_text_input",
inputs={"prompt": "native.Text"},
output="native.Image",
),
@@ -20,7 +21,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_INLINE_PROMPT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_inline_prompt",
PipeImgGenBlueprint(
- description="Test case: valid_with_inline_prompt",
+ description="VALID_WITH_INLINE_PROMPT: Test case: valid_with_inline_prompt",
inputs={},
output="native.Image",
img_gen_prompt="A beautiful sunset over the ocean",
@@ -30,7 +31,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_ASPECT_RATIO: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_aspect_ratio",
PipeImgGenBlueprint(
- description="Test case: valid_with_aspect_ratio",
+ description="VALID_WITH_ASPECT_RATIO: Test case: valid_with_aspect_ratio",
inputs={"prompt": "native.Text"},
output="native.Image",
aspect_ratio=AspectRatio.LANDSCAPE_16_9,
@@ -40,7 +41,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_NB_OUTPUT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_nb_output",
PipeImgGenBlueprint(
- description="Test case: valid_with_nb_output",
+ description="VALID_WITH_NB_OUTPUT: Test case: valid_with_nb_output",
inputs={"prompt": "native.Text"},
output="native.Image[3]",
),
@@ -49,7 +50,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_SEED: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_seed",
PipeImgGenBlueprint(
- description="Test case: valid_with_seed",
+ description="VALID_WITH_SEED: Test case: valid_with_seed",
inputs={"prompt": "native.Text"},
output="native.Image",
seed=42,
@@ -59,7 +60,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_SEED_AUTO: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_seed_auto",
PipeImgGenBlueprint(
- description="Test case: valid_with_seed_auto",
+ description="VALID_WITH_SEED_AUTO: Test case: valid_with_seed_auto",
inputs={"prompt": "native.Text"},
output="native.Image",
seed="auto",
@@ -69,7 +70,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_BACKGROUND: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_background",
PipeImgGenBlueprint(
- description="Test case: valid_with_background",
+ description="VALID_WITH_BACKGROUND: Test case: valid_with_background",
inputs={"prompt": "native.Text"},
output="native.Image",
background=Background.TRANSPARENT,
@@ -79,7 +80,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_OUTPUT_FORMAT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_output_format",
PipeImgGenBlueprint(
- description="Test case: valid_with_output_format",
+ description="VALID_WITH_OUTPUT_FORMAT: Test case: valid_with_output_format",
inputs={"prompt": "native.Text"},
output="native.Image",
output_format=OutputFormat.PNG,
@@ -89,7 +90,7 @@ class PipeImgGenInputTestCases:
VALID_WITH_IS_RAW: ClassVar[tuple[str, PipeImgGenBlueprint]] = (
"valid_with_is_raw",
PipeImgGenBlueprint(
- description="Test case: valid_with_is_raw",
+ description="VALID_WITH_IS_RAW: Test case: valid_with_is_raw",
inputs={"prompt": "native.Text"},
output="native.Image",
is_raw=True,
@@ -108,51 +109,17 @@ class PipeImgGenInputTestCases:
VALID_WITH_IS_RAW,
]
- # Error test cases: (test_id, blueprint, expected_error_message_fragment)
- ERROR_NO_INPUT_NO_PROMPT: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (
- "no_input_no_prompt",
- PipeImgGenBlueprint(
- description="Test case: no_input_no_prompt",
- inputs={},
- output="native.Image",
- ),
- "missing_input_variable",
- )
-
- ERROR_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (
- "multiple_inputs",
- PipeImgGenBlueprint(
- description="Test case: multiple_inputs",
- inputs={"prompt1": "native.Text", "prompt2": "native.Text"},
- output="native.Image",
- ),
- "too_many_candidate_inputs",
- )
-
- ERROR_WRONG_INPUT_TYPE: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (
+ # Error test cases: (test_id, blueprint, expected_error_type)
+ ERROR_WRONG_INPUT_TYPE: ClassVar[tuple[str, PipeImgGenBlueprint, PipeValidationErrorType]] = (
"wrong_input_type",
PipeImgGenBlueprint(
- description="Test case: wrong_input_type",
+ description="ERROR_WRONG_INPUT_TYPE: Test case: wrong_input_type",
inputs={"image": "native.Image"},
output="native.Image",
),
- "inadequate_input_concept",
- )
-
- ERROR_BOTH_PROMPT_AND_INPUT: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (
- "both_prompt_and_input",
- PipeImgGenBlueprint(
- description="Test case: both_prompt_and_input",
- inputs={"prompt": "native.Text"},
- output="native.Image",
- img_gen_prompt="A beautiful sunset",
- ),
- "There must be no inputs if img_gen_prompt is provided",
+ PipeValidationErrorType.IMG_GEN_INPUT_NOT_TEXT_COMPATIBLE,
)
- ERROR_CASES: ClassVar[list[tuple[str, PipeImgGenBlueprint, str]]] = [
- ERROR_NO_INPUT_NO_PROMPT,
- ERROR_MULTIPLE_INPUTS,
+ ERROR_CASES: ClassVar[list[tuple[str, PipeImgGenBlueprint, PipeValidationErrorType]]] = [
ERROR_WRONG_INPUT_TYPE,
- ERROR_BOTH_PROMPT_AND_INPUT,
]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_blueprint.py
new file mode 100644
index 000000000..1ed75375f
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_blueprint.py
@@ -0,0 +1,45 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
+
+
+class TestPipeImgGenBlueprint:
+ def test_validate_inputs_correct(self):
+ blueprint = PipeImgGenBlueprint(
+ inputs=None,
+ img_gen_prompt="A beautiful sunset",
+ output="Image",
+ )
+ assert blueprint.inputs is None
+ assert blueprint.img_gen_prompt == "A beautiful sunset"
+
+ blueprint = PipeImgGenBlueprint(
+ inputs={"prompt": "ImgGenPrompt"},
+ output="Image",
+ )
+ assert blueprint.nb_inputs == 1
+ assert blueprint.input_names == ["prompt"]
+
+ def test_validate_inputs_incorrect(self):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeImgGenBlueprint(
+ inputs={},
+ output="Image",
+ )
+ assert "If no inputs are provided, you must provide an 'img_gen_prompt' as attribute" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ PipeImgGenBlueprint(
+ inputs={"prompt": "ImgGenPrompt"},
+ img_gen_prompt="A beautiful sunset",
+ output="Image",
+ )
+ assert "You must provide either an 'img_gen_prompt' as attribute or as a single text input, but not both" in str(exc_info.value)
+
+ with pytest.raises(ValidationError) as exc_info:
+ PipeImgGenBlueprint(
+ inputs={"prompt1": "ImgGenPrompt", "prompt2": "ImgGenPrompt"},
+ output="Image",
+ )
+ assert "Too many inputs provided for PipeImgGen" in str(exc_info.value)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
index b032057c6..fda6d1ac2 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py
@@ -1,8 +1,9 @@
+from typing import Callable
+
import pytest
from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
-from pipelex.core.pipe_errors import PipeDefinitionError
+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
from pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactory
from tests.unit.pipelex.pipe_operators.pipe_img_gen.data import PipeImgGenInputTestCases
@@ -17,7 +18,9 @@ class TestPipeImgGenValidateInputs:
self,
test_id: str,
blueprint: PipeImgGenBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
pipe_img_gen = PipeImgGenFactory.make_from_blueprint(
@@ -31,25 +34,25 @@ class TestPipeImgGenValidateInputs:
assert pipe_img_gen.code == f"test_pipe_{test_id}"
@pytest.mark.parametrize(
- ("test_id", "blueprint", "expected_error_message_fragment"),
+ ("test_id", "blueprint", "expected_error_type"),
PipeImgGenInputTestCases.ERROR_CASES,
)
def test_validate_inputs_error_cases(
self,
test_id: str,
blueprint: PipeImgGenBlueprint,
- expected_error_message_fragment: str,
+ expected_error_type: PipeValidationErrorType,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing error case: {test_id}")
- with pytest.raises((StaticValidationError, PipeDefinitionError)) as exc_info:
+ with pytest.raises(PipeValidationError) as exc_info:
PipeImgGenFactory.make_from_blueprint(
domain="test_domain",
pipe_code=f"test_pipe_{test_id}",
blueprint=blueprint,
- )
+ ).validate_with_libraries()
- error_str = str(exc_info.value)
- assert expected_error_message_fragment in error_str, (
- f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"
- )
+ # Verify the error type matches
+ assert exc_info.value.error_type == expected_error_type, f"Expected error type {expected_error_type}, but got {exc_info.value.error_type}"
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/data.py b/tests/unit/pipelex/pipe_operators/pipe_llm/data.py
index 1b7e5d706..abd738367 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/data.py
@@ -75,43 +75,3 @@ class PipeLLMInputTestCases:
VALID_IMAGE_MENTIONED_IN_PROMPT,
VALID_TEXT_AND_IMAGE_BOTH_IN_PROMPT,
]
-
- # Error test cases: (test_id, blueprint, expected_error_message_fragment)
- ERROR_MISSING_INPUT_IN_PROMPT: ClassVar[tuple[str, PipeLLMBlueprint, str]] = (
- "missing_input_in_prompt",
- PipeLLMBlueprint(
- description="Test case: missing_input_in_prompt",
- inputs={},
- output="native.Text",
- prompt="Process this text: @user_text",
- ),
- "missing_input_variable",
- )
-
- ERROR_EXTRANEOUS_INPUT: ClassVar[tuple[str, PipeLLMBlueprint, str]] = (
- "extraneous_input",
- PipeLLMBlueprint(
- description="Test case: extraneous_input",
- inputs={"user_text": "native.Text", "extra_input": "native.Text"},
- output="native.Text",
- prompt="Process this: @user_text",
- ),
- "extraneous_input",
- )
-
- ERROR_TWO_INPUTS_THREE_VARIABLES: ClassVar[tuple[str, PipeLLMBlueprint, str]] = (
- "two_inputs_three_variables",
- PipeLLMBlueprint(
- description="Test case: two_inputs_three_variables",
- inputs={"input_one": "native.Text", "input_two": "native.Text"},
- output="native.Text",
- prompt="Process @input_one, @input_two, and @input_three",
- ),
- "missing_input_variable",
- )
-
- ERROR_CASES: ClassVar[list[tuple[str, PipeLLMBlueprint, str]]] = [
- ERROR_MISSING_INPUT_IN_PROMPT,
- ERROR_EXTRANEOUS_INPUT,
- ERROR_TWO_INPUTS_THREE_VARIABLES,
- ]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
new file mode 100644
index 000000000..fee37780f
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
@@ -0,0 +1,127 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+
+
+class TestPipeLLMBlueprint:
+ def test_validate_inputs_correct_with_prompt_variables(self):
+ """Test that prompt variables matching inputs are valid."""
+ blueprint = PipeLLMBlueprint(
+ inputs={"input_one": "native.Text", "input_two": "native.Text"},
+ output="native.Text",
+ prompt="Process @input_one and @input_two",
+ )
+ assert blueprint.nb_inputs == 2
+ assert set(blueprint.input_names) == {"input_one", "input_two"}
+
+ def test_validate_inputs_correct_with_system_prompt_variables(self):
+ """Test that system_prompt variables matching inputs are valid."""
+ blueprint = PipeLLMBlueprint(
+ inputs={"context": "native.Text"},
+ output="native.Text",
+ system_prompt="Use this context: @context",
+ prompt="Generate a response",
+ )
+ assert blueprint.nb_inputs == 1
+ assert blueprint.input_names == ["context"]
+
+ def test_validate_inputs_correct_with_both_prompts(self):
+ """Test that variables in both prompt and system_prompt are validated."""
+ blueprint = PipeLLMBlueprint(
+ inputs={"context": "native.Text", "query": "native.Text"},
+ output="native.Text",
+ system_prompt="Context: @context",
+ prompt="Answer this query: @query",
+ )
+ assert blueprint.nb_inputs == 2
+ assert set(blueprint.input_names) == {"context", "query"}
+
+ def test_validate_inputs_correct_no_variables(self):
+ """Test that prompts without variables work fine."""
+ blueprint = PipeLLMBlueprint(
+ inputs=None,
+ output="native.Text",
+ prompt="Generate a random story",
+ )
+ assert blueprint.inputs is None
+
+ def test_validate_inputs_correct_with_inline_variables(self):
+ """Test that inline variables ($variable) are also validated."""
+ blueprint = PipeLLMBlueprint(
+ inputs={"topic": "native.Text", "style": "native.Text"},
+ output="native.Text",
+ prompt="Write about $topic in $style style",
+ )
+ assert blueprint.nb_inputs == 2
+ assert set(blueprint.input_names) == {"topic", "style"}
+
+ def test_validate_inputs_incorrect_missing_variable(self):
+ """Test that missing input variable raises ValueError."""
+ with pytest.raises(ValueError, match="Missing input variable") as exc_info:
+ PipeLLMBlueprint(
+ inputs={"input_one": "native.Text", "input_two": "native.Text"},
+ output="native.Text",
+ prompt="Process @input_one, @input_two, and @input_three",
+ )
+ error_msg = str(exc_info.value)
+ assert "Missing input variable(s) in prompt template" in error_msg
+ assert "input_three" in error_msg
+ assert "not declared in inputs" in error_msg
+
+ def test_validate_inputs_incorrect_missing_multiple_variables(self):
+ """Test that multiple missing variables are all reported."""
+ with pytest.raises(ValueError, match="Missing input variable") as exc_info:
+ PipeLLMBlueprint(
+ inputs={"input_one": "native.Text"},
+ output="native.Text",
+ prompt="Process @input_one, @input_two, and @input_three",
+ )
+ error_msg = str(exc_info.value)
+ assert "Missing input variable(s) in prompt template" in error_msg
+ assert "input_two" in error_msg
+ assert "input_three" in error_msg
+
+ def test_validate_inputs_incorrect_missing_in_system_prompt(self):
+ """Test that missing variables in system_prompt are caught."""
+ with pytest.raises(ValueError, match="Missing input variable") as exc_info:
+ PipeLLMBlueprint(
+ inputs={"query": "native.Text"},
+ output="native.Text",
+ system_prompt="Use context: @context",
+ prompt="Answer: @query",
+ )
+ error_msg = str(exc_info.value)
+ assert "Missing input variable(s) in prompt template" in error_msg
+ assert "context" in error_msg
+
+ def test_validate_inputs_incorrect_empty_inputs_with_variables(self):
+ """Test that variables in prompt require inputs to be declared."""
+ with pytest.raises((ValidationError, ValueError)) as exc_info:
+ PipeLLMBlueprint(
+ inputs={},
+ output="native.Text",
+ prompt="Process @data",
+ )
+ error_msg = str(exc_info.value)
+ assert "Missing input variable(s) in prompt template" in error_msg or "data" in error_msg
+
+ def test_validate_inputs_ignores_internal_variables(self):
+ """Test that internal variables (starting with _) are ignored."""
+ blueprint = PipeLLMBlueprint(
+ inputs=None,
+ output="native.Text",
+ prompt="Use internal variable @_internal_var",
+ )
+ # Should not raise error even though _internal_var is not in inputs
+ assert blueprint.inputs is None
+
+ def test_validate_inputs_ignores_special_variables(self):
+ """Test that special variables like preliminary_text are ignored."""
+ blueprint = PipeLLMBlueprint(
+ inputs=None,
+ output="native.Text",
+ prompt="Use @preliminary_text and @place_holder",
+ )
+ # Should not raise error even though these special variables are not in inputs
+ assert blueprint.inputs is None
diff --git a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py
index 5215c7723..58372a64a 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py
@@ -1,7 +1,8 @@
+from typing import Callable
+
import pytest
from pipelex import log
-from pipelex.core.exceptions import StaticValidationError
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory
from tests.unit.pipelex.pipe_operators.pipe_llm.data import PipeLLMInputTestCases
@@ -16,7 +17,9 @@ class TestPipeLLMValidateInputs:
self,
test_id: str,
blueprint: PipeLLMBlueprint,
+ load_empty_library: Callable[[], None],
):
+ load_empty_library()
log.verbose(f"Testing valid case: {test_id}")
pipe_llm = PipeLLMFactory.make_from_blueprint(
@@ -25,28 +28,5 @@ class TestPipeLLMValidateInputs:
blueprint=blueprint,
)
- pipe_llm.validate_inputs()
-
- @pytest.mark.parametrize(
- ("test_id", "blueprint", "expected_error_message_fragment"),
- PipeLLMInputTestCases.ERROR_CASES,
- )
- def test_validate_inputs_error_cases(
- self,
- test_id: str,
- blueprint: PipeLLMBlueprint,
- expected_error_message_fragment: str,
- ):
- log.verbose(f"Testing error case: {test_id}")
-
- with pytest.raises(StaticValidationError) as exc_info:
- PipeLLMFactory.make_from_blueprint(
- domain="test_domain",
- pipe_code=f"test_pipe_{test_id}",
- blueprint=blueprint,
- )
-
- error_str = str(exc_info.value)
- assert expected_error_message_fragment in error_str, (
- f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"
- )
+ pipe_llm.validate_inputs_static()
+ pipe_llm.validate_inputs_with_library()
diff --git a/tests/unit/test_pipelex_check_initialization.py b/tests/unit/pipelex/system/test_pipelex_check_initialization.py
similarity index 99%
rename from tests/unit/test_pipelex_check_initialization.py
rename to tests/unit/pipelex/system/test_pipelex_check_initialization.py
index 890e89c0d..81f60501d 100644
--- a/tests/unit/test_pipelex_check_initialization.py
+++ b/tests/unit/pipelex/system/test_pipelex_check_initialization.py
@@ -7,8 +7,8 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
-from pipelex.config import ConfigPaths
from pipelex.system.configuration.config_check import check_is_initialized
+from pipelex.system.configuration.configs import ConfigPaths
class TestPipelexCheckInitialization:
diff --git a/tests/unit/pipelex/tools/misc/conftest.py b/tests/unit/pipelex/tools/misc/conftest.py
new file mode 100644
index 000000000..e691d0d68
--- /dev/null
+++ b/tests/unit/pipelex/tools/misc/conftest.py
@@ -0,0 +1,19 @@
+import pytest
+
+from tests.unit.pipelex.tools.misc.test_data import Address, ComplexUser, UserPreferences
+
+
+def complex_user() -> ComplexUser:
+ """Create a complex nested object for testing."""
+ return ComplexUser(
+ name="John Doe",
+ age=30,
+ addresses=[
+ Address(street="123 Main St", city="Springfield", country="USA", postal_code="12345"),
+ Address(street="456 Side St", city="Brooklyn", country="USA"),
+ ],
+ preferences=UserPreferences(theme="light", notifications=False, tags=["python", "coding"]),
+ metadata={"last_login": "2024-03-20", "login_count": 42, "is_active": True},
+ )
diff --git a/tests/unit/pipelex/tools/misc/test_data.py b/tests/unit/pipelex/tools/misc/test_data.py
new file mode 100644
index 000000000..7ca651d3b
--- /dev/null
+++ b/tests/unit/pipelex/tools/misc/test_data.py
@@ -0,0 +1,23 @@
+from pydantic import BaseModel, Field
+
+
+class Address(BaseModel):
+ street: str
+ city: str
+ country: str
+ postal_code: str | None = None
+
+
+class UserPreferences(BaseModel):
+ theme: str = "dark"
+ notifications: bool = True
+ tags: list[str] = Field(default_factory=list)
+
+
+class ComplexUser(BaseModel):
+ name: str
+ age: int
+ email: str | None
+ addresses: list[Address]
+ preferences: UserPreferences
+ metadata: dict[str, str | int | bool] = Field(default_factory=dict)
diff --git a/tests/unit/pipelex/tools/misc/test_dict_utils.py b/tests/unit/pipelex/tools/misc/test_dict_utils.py
index fe34164ed..e1fa5aab3 100644
--- a/tests/unit/pipelex/tools/misc/test_dict_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_dict_utils.py
@@ -2,7 +2,7 @@ from typing import Any
import pytest
-from pipelex.core.concepts.concept_native import NativeConceptCode
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.system.exceptions import NestedKeyConflictError
from pipelex.tools.misc.dict_utils import apply_to_strings_in_list, apply_to_strings_recursive, insert_before, substitute_nested_in_context
diff --git a/tests/unit/pipelex/tools/misc/test_filetype_utils.py b/tests/unit/pipelex/tools/misc/test_filetype_utils.py
index 7b619285f..95efde43e 100644
--- a/tests/unit/pipelex/tools/misc/test_filetype_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_filetype_utils.py
@@ -7,7 +7,7 @@ from pytest_mock import MockerFixture
from pipelex.tools.misc.filetype_utils import (
FileType,
- FileTypeException,
+ FileTypeError,
detect_file_type_from_base64,
detect_file_type_from_bytes,
detect_file_type_from_path,
@@ -30,7 +30,7 @@ class TestFileType:
class TestFileTypeException:
def test_file_type_exception_inheritance(self):
- error = FileTypeException("test message")
+ error = FileTypeError("test message")
assert isinstance(error, Exception)
assert str(error) == "test message"
@@ -69,7 +69,7 @@ class TestDetectFileTypeFromPath:
# Mock filetype.guess to return None (unrecognized file type)
mock_filetype_guess = mocker.patch("filetype.guess", return_value=None)
- with pytest.raises(FileTypeException, match=r"Could not identify file type of '/unknown/file\.xyz'"):
+ with pytest.raises(FileTypeError, match=r"Could not identify file type of '/unknown/file\.xyz'"):
detect_file_type_from_path("/unknown/file.xyz")
mock_filetype_guess.assert_called_once_with("/unknown/file.xyz")
@@ -79,7 +79,7 @@ class TestDetectFileTypeFromPath:
mock_filetype_guess = mocker.patch("filetype.guess", return_value=None)
path = Path("/unknown/file.xyz")
- with pytest.raises(FileTypeException, match=r"Could not identify file type of '/unknown/file\.xyz'"):
+ with pytest.raises(FileTypeError, match=r"Could not identify file type of '/unknown/file\.xyz'"):
detect_file_type_from_path(path)
mock_filetype_guess.assert_called_once_with(path)
@@ -106,7 +106,7 @@ class TestDetectFileTypeFromBytes:
mock_filetype_guess = mocker.patch("filetype.guess", return_value=None)
test_bytes = b"unknown file content"
- with pytest.raises(FileTypeException, match="Could not identify file type of given bytes: b'unknown file content'"):
+ with pytest.raises(FileTypeError, match="Could not identify file type of given bytes: b'unknown file content'"):
detect_file_type_from_bytes(test_bytes)
mock_filetype_guess.assert_called_once_with(test_bytes)
@@ -117,7 +117,7 @@ class TestDetectFileTypeFromBytes:
# Create bytes longer than 300 characters to test truncation
test_bytes = b"a" * 350
- with pytest.raises(FileTypeException) as exc_info:
+ with pytest.raises(FileTypeError) as exc_info:
detect_file_type_from_bytes(test_bytes)
# Check that the error message contains truncated bytes (first 300 chars)
@@ -206,14 +206,14 @@ class TestDetectFileTypeFromBase64:
# Test with invalid base64 string
invalid_b64 = "invalid!base64!string!"
- with pytest.raises(FileTypeException, match="Could not identify file type of given bytes because input is not valid Base-64"):
+ with pytest.raises(FileTypeError, match="Could not identify file type of given bytes because input is not valid Base-64"):
detect_file_type_from_base64(invalid_b64)
def test_detect_file_type_from_base64_invalid_base64_bytes(self, mocker: MockerFixture): # noqa: ARG002
# Test with invalid base64 bytes
invalid_b64_bytes = b"invalid!base64!bytes!"
- with pytest.raises(FileTypeException, match="Could not identify file type of given bytes because input is not valid Base-64"):
+ with pytest.raises(FileTypeError, match="Could not identify file type of given bytes because input is not valid Base-64"):
detect_file_type_from_base64(invalid_b64_bytes)
def test_detect_file_type_from_base64_data_url_no_comma(self, mocker: MockerFixture):
@@ -238,7 +238,7 @@ class TestDetectFileTypeFromBase64:
# Test that binascii.Error is properly chained when base64 decode fails
mocker.patch("base64.b64decode", side_effect=binascii.Error("Invalid base64"))
- with pytest.raises(FileTypeException) as exc_info:
+ with pytest.raises(FileTypeError) as exc_info:
detect_file_type_from_base64("invalid")
# Check that the original exception is chained
diff --git a/tests/unit/pipelex/tools/misc/test_pretty.py b/tests/unit/pipelex/tools/misc/test_pretty.py
index 63c5afcfb..025075c9e 100644
--- a/tests/unit/pipelex/tools/misc/test_pretty.py
+++ b/tests/unit/pipelex/tools/misc/test_pretty.py
@@ -2,32 +2,10 @@ import re
import textwrap
import pytest
-from pydantic import BaseModel, Field
from pytest import CaptureFixture
-from pipelex.tools.misc.pretty import PrettyPrinter, pretty_print
-
-
-class Address(BaseModel):
- street: str
- city: str
- country: str
- postal_code: str | None = None
-
-
-class UserPreferences(BaseModel):
- theme: str = "dark"
- notifications: bool = True
- tags: list[str] = Field(default_factory=list)
-
-
-class ComplexUser(BaseModel):
- name: str
- age: int
- email: str | None
- addresses: list[Address]
- preferences: UserPreferences
- metadata: dict[str, str | int | bool] = Field(default_factory=dict)
+from pipelex.tools.misc.pretty import pretty_print
+from tests.unit.pipelex.tools.misc.conftest import ComplexUser
def remove_ansi_escape_codes(text: str) -> str:
@@ -35,9 +13,16 @@ def remove_ansi_escape_codes(text: str) -> str:
return ansi_escape.sub("", text)
+def widths() -> tuple[int, int]:
+ """Return tuple of (width, console_width) for pretty print tests."""
+ return (180, 200)
+
+
class TestPrettyPrint:
- def test_pretty_print_with_brackets_optional_edge_case(self, capsys: CaptureFixture[str]):
- pretty_print(content="Optional[float]", title="title")
+ def test_pretty_print_with_brackets_optional_edge_case(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="Optional[float]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -53,8 +38,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_brackets_basic(self, capsys: CaptureFixture[str]):
- pretty_print(content="SomethingBeforeBracketsAnd[SomethingBetweenBrackets]", title="title")
+ def test_pretty_print_with_brackets_basic(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="SomethingBeforeBracketsAnd[SomethingBetweenBrackets]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -71,8 +57,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_nested_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="List[Optional[int]]", title="title")
+ def test_pretty_print_with_nested_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="List[Optional[int]]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -89,8 +76,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_multiple_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="Union[str, List[int], Dict[str, Any]]", title="title")
+ def test_pretty_print_with_multiple_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="Union[str, List[int], Dict[str, Any]]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -107,8 +95,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_special_chars_and_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="@decorator[*args, **kwargs]", title="title")
+ def test_pretty_print_with_special_chars_and_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="@decorator[*args, **kwargs]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -125,8 +114,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_empty_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="List[]", title="title")
+ def test_pretty_print_with_empty_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="List[]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -143,8 +133,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_unmatched_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="Unmatched[bracket", title="title")
+ def test_pretty_print_with_unmatched_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="Unmatched[bracket", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -161,8 +152,9 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- def test_pretty_print_with_unicode_and_brackets(self, capsys: CaptureFixture[str]):
- pretty_print(content="🐍Python[版本3.11]", title="title")
+ def test_pretty_print_with_unicode_and_brackets(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ pretty_print(content="🐍Python[版本3.11]", title="title", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
@@ -179,34 +171,17 @@ class TestPrettyPrint:
assert output == expected_output, f"Output did not match expected:\n{output}"
- # This test needs output to be displayed, so we disabled it by default in pyproject.toml
- # So to run it you must bypass the marker restrictions defined in the pyproject.toml pytest section
- # and make sure you turn on outputs with -s like this:
- # pytest -m "" -k test_pretty_print_pydantic_object -s
- @pytest.mark.gha_disabled
- @pytest.mark.codex_disabled
- @pytest.mark.needs_output
- def test_pretty_print_pydantic_object(self, capsys: CaptureFixture[str]):
- # Create a complex nested object
- user = ComplexUser(
- name="John Doe",
- age=30,
- addresses=[
- Address(street="123 Main St", city="Springfield", country="USA", postal_code="12345"),
- Address(street="456 Side St", city="Brooklyn", country="USA"),
- ],
- preferences=UserPreferences(theme="light", notifications=False, tags=["python", "coding"]),
- metadata={"last_login": "2024-03-20", "login_count": 42, "is_active": True},
- )
-
- pretty_print(content=user, title="Complex User")
+ def test_pretty_print_pydantic_object(self, complex_user: ComplexUser, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+
+ pretty_print(content=complex_user, title="Complex User", width=width, console_width=console_width)
captured = capsys.readouterr()
output = remove_ansi_escape_codes(captured.out)
- expected_output = textwrap.dedent("""\
+ expected_output = textwrap.dedent("""
╭─ Complex User ──────────────────────────────────────────────────────────────────────────────────╮
+ │ │
│ ComplexUser( │
│ name='John Doe', │
│ age=30, │
@@ -222,39 +197,9 @@ class TestPrettyPrint:
│ ), │
│ metadata={'last_login': '2024-03-20', 'login_count': 42, 'is_active': True} │
│ ) │
+ │ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
- """)
-
- assert output == expected_output, f"Make sure you enable pytest '-s' option. Output did not match expected:\n{output}"
-
-class TestPrettyPrintInSandbox:
- def test_empty_content(self, capsys: CaptureFixture[str]):
- PrettyPrinter.pretty_print_without_rich(content="")
-
- captured = capsys.readouterr()
- output = remove_ansi_escape_codes(captured.err)
-
- expected_output = textwrap.dedent("""\
- ╭────╮
- │ │
- ╰────╯
- """)
-
- assert output == expected_output, f"Output did not match expected:\n{output}"
-
- def test_none_content(self, capsys: CaptureFixture[str]):
- PrettyPrinter.pretty_print_without_rich(content=None, title="title", subtitle="subtitle")
-
- captured = capsys.readouterr()
- output = remove_ansi_escape_codes(captured.err)
-
- expected_output = textwrap.dedent("""\
- ╭────────────╮
- │ title: │
- │ subtitle: │
- │ None │
- ╰────────────╯
""")
- assert output == expected_output, f"Output did not match expected:\n{output}"
+ assert output == expected_output, f"Make sure you enable pytest '-s' option. Output did not match expected:\n{output}"
diff --git a/tests/unit/pipelex/tools/misc/test_pretty_without_rich.py b/tests/unit/pipelex/tools/misc/test_pretty_without_rich.py
new file mode 100644
index 000000000..46e5cf1aa
--- /dev/null
+++ b/tests/unit/pipelex/tools/misc/test_pretty_without_rich.py
@@ -0,0 +1,72 @@
+import re
+import textwrap
+
+import pytest
+from pytest import CaptureFixture
+
+from pipelex.tools.misc.pretty import PrettyPrinter
+from tests.unit.pipelex.tools.misc.conftest import ComplexUser
+
+
+def remove_ansi_escape_codes(text: str) -> str:
+ ansi_escape = re.compile(r"\x1B\[[0-?]*[ -/]*[@-~]")
+ return ansi_escape.sub("", text)
+
+
+def widths() -> tuple[int, int]:
+ """Return tuple of (width, console_width) for pretty print tests."""
+ return (100, 150)
+
+
+class TestPrettyPrintWithoutRich:
+ def test_pretty_without_rich_empty_content(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ PrettyPrinter.pretty_print_without_rich(content="", width=width, console_width=console_width)
+
+ captured = capsys.readouterr()
+ output = remove_ansi_escape_codes(captured.err)
+
+ expected_output = textwrap.dedent("""\
+ ╭────╮
+ │ │
+ ╰────╯
+ """)
+
+ assert output == expected_output, f"Output did not match expected:\n{output}"
+
+ def test_pretty_without_rich_none_content(self, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ PrettyPrinter.pretty_print_without_rich(content=None, title="title", subtitle="subtitle", width=width, console_width=console_width)
+
+ captured = capsys.readouterr()
+ output = remove_ansi_escape_codes(captured.err)
+
+ expected_output = textwrap.dedent("""\
+ ╭────────────╮
+ │ title: │
+ │ subtitle: │
+ │ None │
+ ╰────────────╯
+ """)
+
+ assert output == expected_output, f"Output did not match expected:\n{output}"
+
+ def test_pretty_without_rich_pydantic_object(self, complex_user: ComplexUser, widths: tuple[int, int], capsys: CaptureFixture[str]):
+ width, console_width = widths
+ PrettyPrinter.pretty_print_without_rich(content=complex_user, title="Complex User", width=width, console_width=console_width)
+
+ captured = capsys.readouterr()
+ output = remove_ansi_escape_codes(captured.err)
+
+ expected_output = textwrap.dedent("""\
+ ╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+ │ Complex User: │
+ │ name='John Doe' age=30 email='[email protected]' addresses=[Address(street='123 Main St', city='Sprin │
+ │ gfield', country='USA', postal_code='12345'), Address(street='456 Side St', city='Brooklyn', country │
+ │ ='USA', postal_code=None)] preferences=UserPreferences(theme='light', notifications=False, tags=['py │
+ │ thon', 'coding']) metadata={'last_login': '2024-03-20', 'login_count': 42, 'is_active': True} │
+ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ """)
+
+ assert output == expected_output, f"Output did not match expected:\n{output}"
diff --git a/tests/test_pipelines/test.plx b/tests/unit/pipelex/tools/test.plx
similarity index 100%
rename from tests/test_pipelines/test.plx
rename to tests/unit/pipelex/tools/test.plx
diff --git a/tests/test_pipelines/test_file_func_registry.py b/tests/unit/pipelex/tools/test_file_func_registry.py
similarity index 96%
rename from tests/test_pipelines/test_file_func_registry.py
rename to tests/unit/pipelex/tools/test_file_func_registry.py
index 84cdb5a8b..7813a9e69 100644
--- a/tests/test_pipelines/test_file_func_registry.py
+++ b/tests/unit/pipelex/tools/test_file_func_registry.py
@@ -38,9 +38,9 @@ def read_file_content(working_memory: WorkingMemory) -> ListContent[CodebaseFile
with open(file_path.path, encoding="utf-8") as file:
content = file.read()
codebase_files.append(CodebaseFileContent(file_path=file_path.path, file_content=content))
- except Exception as e:
+ except Exception as exc:
codebase_files.append(
- CodebaseFileContent(file_path=file_path.path, file_content=f"# File not found or unreadable: {file_path.path}\n# Error: {e!s}"),
+ CodebaseFileContent(file_path=file_path.path, file_content=f"# File not found or unreadable: {file_path.path}\n# Error: {exc!s}"),
)
return ListContent[CodebaseFileContent](items=codebase_files)
diff --git a/tests/unit/pipelex/tools/test_func_registry_utils.py b/tests/unit/pipelex/tools/test_func_registry_utils.py
index fa6a22721..b69c696a1 100644
--- a/tests/unit/pipelex/tools/test_func_registry_utils.py
+++ b/tests/unit/pipelex/tools/test_func_registry_utils.py
@@ -71,6 +71,16 @@ def invalid_function_wrong_return_type(working_memory: WorkingMemory) -> str:
'''This should not be registered - wrong return type.'''
return "test"
+# No return type annotation - should be rejected
+async def invalid_function_no_return_type(working_memory: WorkingMemory):
+ '''This should not be registered - no return type annotation.'''
+ return TextContent(text="no return type")
+
+# Wrong return type (not StuffContent) - should be rejected
+async def invalid_function_returns_str(working_memory: WorkingMemory) -> str:
+ '''This should not be registered - returns str instead of StuffContent.'''
+ return "wrong type"
+
# Too many parameters - should be rejected
def invalid_function_too_many_params(working_memory: WorkingMemory, extra_param: str) -> StructuredContent:
'''This should not be registered - too many parameters.'''
@@ -98,6 +108,8 @@ def invalid_function_no_type_hints(working_memory):
"invalid_function_wrong_param_name",
"invalid_function_wrong_param_type",
"invalid_function_wrong_return_type",
+ "invalid_function_no_return_type",
+ "invalid_function_returns_str",
"invalid_function_too_many_params",
"invalid_function_no_params",
"invalid_function_no_type_hints",
diff --git a/tests/unit/pipelex/tools/test_log_config.py b/tests/unit/pipelex/tools/test_log_config.py
new file mode 100644
index 000000000..381ca9681
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_config.py
@@ -0,0 +1,31 @@
+import pytest
+
+from pipelex.tools.log.log_config import CallerInfoTemplate, LogConfig
+from pipelex.tools.log.log_levels import LogLevel
+
+
+class TestLogConfigUtilities:
+ @pytest.mark.parametrize(
+ ("template_key", "expected_template"),
+ [
+ (CallerInfoTemplate.FILE_LINE, "{file}:{line}"),
+ (CallerInfoTemplate.FILE_LINE_FUNC, "{file}:{line} {func}"),
+ (CallerInfoTemplate.FUNC, "{func}"),
+ (CallerInfoTemplate.FILE_FUNC, "{file} {func}"),
+ (CallerInfoTemplate.FUNC_LINE, "{func} {line}"),
+ (CallerInfoTemplate.FUNC_MODULE, "{func} {module}"),
+ (CallerInfoTemplate.FUNC_MODULE_LINE, "{func} {module} {line}"),
+ ],
+ )
+ def test_caller_info_template_mapping(self, template_key: CallerInfoTemplate, expected_template: str) -> None:
+ assert CallerInfoTemplate.for_template_key(template_key) == expected_template
+
+ def test_validate_package_log_levels_converts_strings_to_enum(self) -> None:
+ package_levels = {"pipelex": "DEBUG", "pipelex.tools": "INFO"}
+
+ validated_levels = LogConfig.validate_package_log_levels(package_levels)
+
+ assert validated_levels == {
+ "pipelex": LogLevel.DEBUG,
+ "pipelex.tools": LogLevel.INFO,
+ }
diff --git a/tests/unit/pipelex/tools/test_log_levels.py b/tests/unit/pipelex/tools/test_log_levels.py
new file mode 100644
index 000000000..548093dad
--- /dev/null
+++ b/tests/unit/pipelex/tools/test_log_levels.py
@@ -0,0 +1,42 @@
+import logging
+
+import pytest
+
+from pipelex.tools.log.log_levels import (
+ LOGGING_LEVEL_DEV,
+ LOGGING_LEVEL_OFF,
+ LOGGING_LEVEL_VERBOSE,
+ LogLevel,
+)
+
+
+class TestLogLevels:
+ @pytest.mark.parametrize(
+ ("log_level", "expected_int_level"),
+ [
+ (LogLevel.VERBOSE, LOGGING_LEVEL_VERBOSE),
+ (LogLevel.DEBUG, logging.DEBUG),
+ (LogLevel.DEV, LOGGING_LEVEL_DEV),
+ (LogLevel.INFO, logging.INFO),
+ (LogLevel.WARNING, logging.WARNING),
+ (LogLevel.ERROR, logging.ERROR),
+ (LogLevel.CRITICAL, logging.CRITICAL),
+ (LogLevel.OFF, LOGGING_LEVEL_OFF),
+ ],
+ )
+ def test_int_logging_level_returns_correct_value(self, log_level: LogLevel, expected_int_level: int) -> None:
+ assert log_level.int_logging_level == expected_int_level
+
+ @pytest.mark.parametrize(
+ ("raw_level", "expected_enum"),
+ [
+ (LOGGING_LEVEL_VERBOSE, LogLevel.VERBOSE),
+ (LOGGING_LEVEL_DEV, LogLevel.DEV),
+ (LOGGING_LEVEL_OFF, LogLevel.OFF),
+ (logging.CRITICAL + 1, LogLevel.OFF),
+ (logging.DEBUG, LogLevel.DEBUG),
+ (logging.INFO, LogLevel.INFO),
+ ],
+ )
+ def test_from_int_converts_to_enum(self, raw_level: int, expected_enum: LogLevel) -> None:
+ assert LogLevel.from_int(raw_level) == expected_enum
diff --git a/uv.lock b/uv.lock
index 699db5841..83877a579 100644
--- a/uv.lock
+++ b/uv.lock
@@ -180,7 +180,7 @@ wheels = [
[[package]]
name = "anthropic"
-version = "0.72.0"
+version = "0.74.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@@ -192,9 +192,9 @@ dependencies = [
{ name = "sniffio" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/49/07/61f3ca8e69c5dcdaec31b36b79a53ea21c5b4ca5e93c7df58c71f43bf8d8/anthropic-0.72.0.tar.gz", hash = "sha256:8971fe76dcffc644f74ac3883069beb1527641115ae0d6eb8fa21c1ce4082f7a", size = 493721, upload-time = "2025-10-28T19:13:01.755Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/7b/609eea5c54ae69b1a4a94169d4b0c86dc5c41b43509989913f6cdc61b81d/anthropic-0.74.1.tar.gz", hash = "sha256:04c087b2751385c524f6d332d066a913870e4de8b3e335fb0a0c595f1f88dc6e", size = 428981, upload-time = "2025-11-19T22:17:31.533Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/7b/b7/160d4fb30080395b4143f1d1a4f6c646ba9105561108d2a434b606c03579/anthropic-0.72.0-py3-none-any.whl", hash = "sha256:0e9f5a7582f038cab8efbb4c959e49ef654a56bfc7ba2da51b5a7b8a84de2e4d", size = 357464, upload-time = "2025-10-28T19:13:00.215Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/45/6b18d0692302b8cbc01a10c35b43953d3c4172fbd4f83337b8ed21a8eaa4/anthropic-0.74.1-py3-none-any.whl", hash = "sha256:b07b998d1cee7f41d9f02530597d7411672b362cc2417760a40c0167b81c6e65", size = 371473, upload-time = "2025-11-19T22:17:29.998Z" },
]
[[package]]
@@ -214,14 +214,14 @@ wheels = [
[[package]]
name = "astroid"
-version = "4.0.1"
+version = "4.0.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions", marker = "python_full_version < '3.11'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/a7/d1/6eee8726a863f28ff50d26c5eacb1a590f96ccbb273ce0a8c047ffb10f5a/astroid-4.0.1.tar.gz", hash = "sha256:0d778ec0def05b935e198412e62f9bcca8b3b5c39fdbe50b0ba074005e477aab", size = 405414, upload-time = "2025-10-11T15:15:42.6Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b7/22/97df040e15d964e592d3a180598ace67e91b7c559d8298bdb3c949dc6e42/astroid-4.0.2.tar.gz", hash = "sha256:ac8fb7ca1c08eb9afec91ccc23edbd8ac73bb22cbdd7da1d488d9fb8d6579070", size = 405714, upload-time = "2025-11-09T21:21:18.373Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/47/f4/034361a9cbd9284ef40c8ad107955ede4efae29cbc17a059f63f6569c06a/astroid-4.0.1-py3-none-any.whl", hash = "sha256:37ab2f107d14dc173412327febf6c78d39590fdafcb44868f03b6c03452e3db0", size = 276268, upload-time = "2025-10-11T15:15:40.585Z" },
+ { url = "https://files.pythonhosted.org/packages/93/ac/a85b4bfb4cf53221513e27f33cc37ad158fce02ac291d18bee6b49ab477d/astroid-4.0.2-py3-none-any.whl", hash = "sha256:d7546c00a12efc32650b19a2bb66a153883185d3179ab0d4868086f807338b9b", size = 276354, upload-time = "2025-11-09T21:21:16.54Z" },
]
[[package]]
@@ -307,16 +307,16 @@ wheels = [
[[package]]
name = "boto3-stubs"
-version = "1.40.69"
+version = "1.41.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "botocore-stubs" },
{ name = "types-s3transfer" },
{ name = "typing-extensions", marker = "python_full_version < '3.12'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/e3/f7/10c3314c273fcd74bbf588c521cd30cba5784c492ba14858014bdd295af3/boto3_stubs-1.40.69.tar.gz", hash = "sha256:244121672c614b58426f1630f5685a821f4a8cff49deed59a287d9d91c7a4d64", size = 99409, upload-time = "2025-11-08T08:35:02.954Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/e9/de/ed64826dfe146e1aa91126e89a876f8c321a2039f6e29eb41ccfacc61bbd/boto3_stubs-1.41.0.tar.gz", hash = "sha256:74d138f2d2f5f48140be81d680722b0194e09bcdf8d2080a914c21d277c2cfe3", size = 99816, upload-time = "2025-11-19T21:09:18.788Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ed/54/1f5d5db7812524809656baadc63669f4456a88bc65f1b30b576f3d2309f9/boto3_stubs-1.40.69-py3-none-any.whl", hash = "sha256:004e9647c310dcaec45145645dfa9aec73fb233f2649900eeaa976894378d57c", size = 68982, upload-time = "2025-11-08T08:34:57.269Z" },
+ { url = "https://files.pythonhosted.org/packages/69/05/71601d7c9e845d34628108170d6e4ec0c91fc6fd44c71ba8d52bc607aeb8/boto3_stubs-1.41.0-py3-none-any.whl", hash = "sha256:9e373ac5f9c721c5e615ef694a992c333480c71905d3b8e940175dd7265eef93", size = 69211, upload-time = "2025-11-19T21:09:10.914Z" },
]
[[package]]
@@ -335,32 +335,41 @@ wheels = [
[[package]]
name = "botocore-stubs"
-version = "1.40.69"
+version = "1.40.76"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "types-awscrt" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/2a/57/da68248f3b5fcd41f47dd4dfad6b6ba5fdcdedfa86e04d2b9822d5788dfd/botocore_stubs-1.40.69.tar.gz", hash = "sha256:31d1bb9a5140df9ba83d6d4d66b68998767279850de50e28caade315244b0a78", size = 42235, upload-time = "2025-11-07T21:18:47.479Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c9/03/02f3ca1bd4baae56dfb148d8f7a18466f0b10b0006c57811dc73d24ccedc/botocore_stubs-1.40.76.tar.gz", hash = "sha256:261d51e71a164a93abf9c6a5c779c5664d307ec4450de917ad9cbe7c29782223", size = 42227, upload-time = "2025-11-18T20:30:28.629Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b1/19/0f822401c3fda8de004be0d9ed9195ea63f76bde2b1d12c81e9138cd94b3/botocore_stubs-1.40.69-py3-none-any.whl", hash = "sha256:79837e22d89c65cb09484aede593aca5b5027ec6a535c34498aebca693c3b856", size = 66542, upload-time = "2025-11-07T21:18:44.688Z" },
+ { url = "https://files.pythonhosted.org/packages/70/2d/3b71d25da85e10700287d90bd5c525821985d80b55895d81f7d0683e4e07/botocore_stubs-1.40.76-py3-none-any.whl", hash = "sha256:84c9a24c2a38a59dd173f2655afacf7b6484c05f3d61cf198b59bb54a6efebc5", size = 66542, upload-time = "2025-11-18T20:30:26.596Z" },
]
[[package]]
name = "cachetools"
-version = "6.2.1"
+version = "6.2.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/cc/7e/b975b5814bd36faf009faebe22c1072a1fa1168db34d285ef0ba071ad78c/cachetools-6.2.1.tar.gz", hash = "sha256:3f391e4bd8f8bf0931169baf7456cc822705f4e2a31f840d218f445b9a854201", size = 31325, upload-time = "2025-10-12T14:55:30.139Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/fb/44/ca1675be2a83aeee1886ab745b28cda92093066590233cc501890eb8417a/cachetools-6.2.2.tar.gz", hash = "sha256:8e6d266b25e539df852251cfd6f990b4bc3a141db73b939058d809ebd2590fc6", size = 31571, upload-time = "2025-11-13T17:42:51.465Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/96/c5/1e741d26306c42e2bf6ab740b2202872727e0f606033c9dd713f8b93f5a8/cachetools-6.2.1-py3-none-any.whl", hash = "sha256:09868944b6dde876dfd44e1d47e18484541eaf12f26f29b7af91b26cc892d701", size = 11280, upload-time = "2025-10-12T14:55:28.382Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/46/eb6eca305c77a4489affe1c5d8f4cae82f285d9addd8de4ec084a7184221/cachetools-6.2.2-py3-none-any.whl", hash = "sha256:6c09c98183bf58560c97b2abfcedcbaf6a896a490f534b031b661d3723b45ace", size = 11503, upload-time = "2025-11-13T17:42:50.232Z" },
]
[[package]]
name = "certifi"
-version = "2025.10.5"
+version = "2025.11.12"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/4c/5b/b6ce21586237c77ce67d01dc5507039d444b630dd76611bbca2d8e5dcd91/certifi-2025.10.5.tar.gz", hash = "sha256:47c09d31ccf2acf0be3f701ea53595ee7e0b8fa08801c6624be771df09ae7b43", size = 164519, upload-time = "2025-10-05T04:12:15.808Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/8c/58f469717fa48465e4a50c014a0400602d3c437d7c0c468e17ada824da3a/certifi-2025.11.12.tar.gz", hash = "sha256:d8ab5478f2ecd78af242878415affce761ca6bc54a22a27e026d7c25357c3316", size = 160538, upload-time = "2025-11-12T02:54:51.517Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e4/37/af0d2ef3967ac0d6113837b44a4f0bfe1328c2b9763bd5b1744520e5cfed/certifi-2025.10.5-py3-none-any.whl", hash = "sha256:0f212c2744a9bb6de0c56639a6f68afe01ecd92d91f14ae897c4fe7bbeeef0de", size = 163286, upload-time = "2025-10-05T04:12:14.03Z" },
+ { url = "https://files.pythonhosted.org/packages/70/7d/9bc192684cea499815ff478dfcdc13835ddf401365057044fb721ec6bddb/certifi-2025.11.12-py3-none-any.whl", hash = "sha256:97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b", size = 159438, upload-time = "2025-11-12T02:54:49.735Z" },
+]
+
+[[package]]
+name = "cfgv"
+version = "3.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" },
]
[[package]]
@@ -438,14 +447,14 @@ wheels = [
[[package]]
name = "click"
-version = "8.3.0"
+version = "8.3.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/46/61/de6cd827efad202d7057d93e0fed9294b96952e188f7384832791c7b2254/click-8.3.0.tar.gz", hash = "sha256:e7b8232224eba16f4ebe410c25ced9f7875cb5f3263ffc93cc3e8da705e229c4", size = 276943, upload-time = "2025-09-18T17:32:23.696Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/3d/fa/656b739db8587d7b5dfa22e22ed02566950fbfbcdc20311993483657a5c0/click-8.3.1.tar.gz", hash = "sha256:12ff4785d337a1bb490bb7e9c2b1ee5da3112e94a8622f26a6c77f5d2fc6842a", size = 295065, upload-time = "2025-11-15T20:45:42.706Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/db/d3/9dcc0f5797f070ec8edf30fbadfb200e71d9db6b84d211e3b2085a7589a0/click-8.3.0-py3-none-any.whl", hash = "sha256:9b9f285302c6e3064f4330c05f05b81945b2a39544279343e6e7c5f27a9baddc", size = 107295, upload-time = "2025-09-18T17:32:22.42Z" },
+ { url = "https://files.pythonhosted.org/packages/98/78/01c019cdb5d6498122777c1a43056ebb3ebfeef2076d9d026bfe15583b2b/click-8.3.1-py3-none-any.whl", hash = "sha256:981153a64e25f12d547d3426c367a4857371575ee7ad18df2a6183ab0545b2a6", size = 108274, upload-time = "2025-11-15T20:45:41.139Z" },
]
[[package]]
@@ -459,75 +468,75 @@ wheels = [
[[package]]
name = "coverage"
-version = "7.11.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/32/e6/7c4006cf689ed7a4aa75dcf1f14acbc04e585714c220b5cc6d231096685a/coverage-7.11.2.tar.gz", hash = "sha256:ae43149b7732df15c3ca9879b310c48b71d08cd8a7ba77fda7f9108f78499e93", size = 814849, upload-time = "2025-11-08T20:26:33.011Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/83/5b/d943b719938467d313973fd83af9c810e248fcec33165d5ab0148ab1c602/coverage-7.11.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:004bdc5985b86f565772af627925e368256ee2172623db10a0d78a3b53f20ef1", size = 216802, upload-time = "2025-11-08T20:23:47.186Z" },
- { url = "https://files.pythonhosted.org/packages/8b/f7/d3c096ca6a6212e8a536ae2144406d28b43e7528ff05a0bf6a5336319d0d/coverage-7.11.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:3aa8c62460499e10ceac5ea61cc09c4f7ddcd8a68c6313cf08785ad353dfd311", size = 217317, upload-time = "2025-11-08T20:23:50.255Z" },
- { url = "https://files.pythonhosted.org/packages/10/46/d0dbafbd3604293b73a44ae9c88e339921c13f309138b31ec60b451895b9/coverage-7.11.2-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d73da4893125e0671f762e408dea9957b2bda0036c9589c2fd258a6b870acbdb", size = 244068, upload-time = "2025-11-08T20:23:51.63Z" },
- { url = "https://files.pythonhosted.org/packages/3d/16/ef8aba300f7224167c556d15852bf35d42c7af93b68f3ef82323737515e8/coverage-7.11.2-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:805efa416085999da918f15f81b26636d8e79863e1fbac1495664686d1e6a6e9", size = 245896, upload-time = "2025-11-08T20:23:53.1Z" },
- { url = "https://files.pythonhosted.org/packages/7f/ea/02fa537e61bc61fd111d5d9611184a354dd26bbc31e58ccd922f76404723/coverage-7.11.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c65f4291aec39692a3bfbe1d92ae5bea58c16b5553fdf021de61c655d987233f", size = 247755, upload-time = "2025-11-08T20:23:54.88Z" },
- { url = "https://files.pythonhosted.org/packages/41/3b/6cc19074059c030e489fd5ff934aa49521a75ba6236d27badb3b4270b21c/coverage-7.11.2-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b7658f3d4f728092368c091c18efcfb679be9b612c93bfdf345f33635a325188", size = 244714, upload-time = "2025-11-08T20:23:56.655Z" },
- { url = "https://files.pythonhosted.org/packages/a6/d5/b3480a0fd9c45fad37884c38ee943788ef43b64abf156b3f8e6af096c62e/coverage-7.11.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:9f5f6ee021b3b25e748a9a053f3a8dd61a62b6689efd6425cb47e27360994903", size = 245800, upload-time = "2025-11-08T20:23:58.06Z" },
- { url = "https://files.pythonhosted.org/packages/07/2a/34f1476db9c58c410193f8f0cbecdfd9931912ed07de628fdffe0dae216d/coverage-7.11.2-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:9a95b7a6043b221ec1a0d4d5481e424272b37028353265fbe5fcd3768d652eb7", size = 243808, upload-time = "2025-11-08T20:23:59.756Z" },
- { url = "https://files.pythonhosted.org/packages/73/fd/b43a0a4f6306a486d31cdd4166afd4dc0b08a8f072d7ab2ccc23893b6d19/coverage-7.11.2-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:94ced4a29a6987af99faaa49a513bf8d0458e8af004c54174e05dd7a8a31c7d9", size = 244070, upload-time = "2025-11-08T20:24:01.281Z" },
- { url = "https://files.pythonhosted.org/packages/cc/8c/bcbe2c9cb81ef008d05b04ebc37a3a1c65d61b61c9cf772f0ae473ddc56b/coverage-7.11.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8014a28a37ffabf7da7107f4f154d68c6b89672f27fef835a0574591c5cd140b", size = 244688, upload-time = "2025-11-08T20:24:02.641Z" },
- { url = "https://files.pythonhosted.org/packages/6b/f7/c6c276f6663a1d7e29f8cc4a5a8c76dbf834ecb74017936187146adbce9e/coverage-7.11.2-cp310-cp310-win32.whl", hash = "sha256:43ecf9dca4fcb3baf8a886019dd5ce663c95a5e1c5172719c414f0ebd9eeb785", size = 219382, upload-time = "2025-11-08T20:24:04.476Z" },
- { url = "https://files.pythonhosted.org/packages/4f/aa/0d07b2d567f1d005088b4afad533b4a6af48ec75f3f9071afbe5f7076cab/coverage-7.11.2-cp310-cp310-win_amd64.whl", hash = "sha256:230317450af65a37c1fdbdd3546f7277e0c1c1b65e0d57409248e5dd0fa13493", size = 220319, upload-time = "2025-11-08T20:24:06.464Z" },
- { url = "https://files.pythonhosted.org/packages/89/39/326336c0adc6dc624be0edb5143dec90a9da2626335e83f6d09da120922f/coverage-7.11.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:36c41bf2ee6f6062de8177e249fee17cd5c9662cd373f7a41e6468a34c5b9c0f", size = 216927, upload-time = "2025-11-08T20:24:08.167Z" },
- { url = "https://files.pythonhosted.org/packages/b7/68/cd1d3422fc9525827cddf62b2385f78356b88e745e90e8e512fefcc05f8f/coverage-7.11.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:397778cf6d50df59c890bd3ac10acb5bf413388ff6a013305134f1403d5db648", size = 217429, upload-time = "2025-11-08T20:24:09.939Z" },
- { url = "https://files.pythonhosted.org/packages/36/73/3f384dd79d6bbdf7fbceda3c7e0db33e148559bc18c49022c9c0c5e512c1/coverage-7.11.2-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:c85f44ed4260221e46a4e9e8e8df4b359ab6c0a742c79e85d649779bcf77b534", size = 247832, upload-time = "2025-11-08T20:24:11.897Z" },
- { url = "https://files.pythonhosted.org/packages/45/3c/27839b6f343998e82f3e470397c058566c953dc71fe37e0abb953133a341/coverage-7.11.2-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:cbffd1d5c5bf4c576ca247bf77646cdad4dced82928337eeb0b85e2b3be4d64b", size = 249749, upload-time = "2025-11-08T20:24:13.705Z" },
- { url = "https://files.pythonhosted.org/packages/6e/51/011102c7f6902084e632128ac0f42cd3345acc543a7c9f8ce5e1a94397ef/coverage-7.11.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ea10a57568af7cf082a7a4d98a699f993652c2ffbdd5a6c9d63c9ca10b693b4d", size = 251860, upload-time = "2025-11-08T20:24:15.113Z" },
- { url = "https://files.pythonhosted.org/packages/bb/4c/4622eb7aac98c2552ed8a176a6015ea8cf36a2ec75cbcfb5f2ccf100bbd6/coverage-7.11.2-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4b1bea4c707f4c09f682fe0e646a114dfd068f627880d4a208850d01f8164ad", size = 247942, upload-time = "2025-11-08T20:24:16.637Z" },
- { url = "https://files.pythonhosted.org/packages/95/94/42ba12fc827fb504f8f8ec5313e46cf5582cdb9d4823e76d70ed22e88bdf/coverage-7.11.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1ac3f647ecf25d883051ef42d38d823016e715b9f289f8c1768be5117075d1bd", size = 249553, upload-time = "2025-11-08T20:24:18.153Z" },
- { url = "https://files.pythonhosted.org/packages/a3/47/2cd8014c872a3e469ffe50fbc692d02c7460e20cd701a0d6366fbef759e3/coverage-7.11.2-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:d423991415f73a70c0a5f3e0a226cf4ab374dd0da7409978069b844df3d31582", size = 247627, upload-time = "2025-11-08T20:24:19.644Z" },
- { url = "https://files.pythonhosted.org/packages/a9/31/e722f2c7f0f16954d13e6441a24d841174bcb1ff2421c6504c024c09c7af/coverage-7.11.2-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:0f4a958ff286038ac870f836351e9fb8912f1614d1cdbda200fc899235f7dc9b", size = 247353, upload-time = "2025-11-08T20:24:21.28Z" },
- { url = "https://files.pythonhosted.org/packages/0a/dd/d4fd26be0ce7993f0013df9788e52cd83a1adf5cfb9887bfd1b38722380e/coverage-7.11.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4d1ff4b87ad438148976f2215141a490ae000e878536370d53f8da8c59a175a6", size = 248251, upload-time = "2025-11-08T20:24:22.724Z" },
- { url = "https://files.pythonhosted.org/packages/1c/33/003f7b5f10fae2ad7390e57a1520c46a24bd46e374b197e97050ae47751f/coverage-7.11.2-cp311-cp311-win32.whl", hash = "sha256:e448ceee2fb880427eafc9a3f8e6162b2ac7cc3e9b30b85d6511f25cc8a11820", size = 219410, upload-time = "2025-11-08T20:24:24.15Z" },
- { url = "https://files.pythonhosted.org/packages/22/e8/5db102c57143f33a9229ecdc8d7976ad0c5d103fcd26f2b939db96789990/coverage-7.11.2-cp311-cp311-win_amd64.whl", hash = "sha256:bc65e32fe5bb942f0f5247e1500e355cbbdf326181198f5e27e3bb3ddb81e203", size = 220342, upload-time = "2025-11-08T20:24:25.947Z" },
- { url = "https://files.pythonhosted.org/packages/bf/b2/9908f6b4b979045c01e02a069ae5f73c16dff022c296a5e1fd756c602c6c/coverage-7.11.2-cp311-cp311-win_arm64.whl", hash = "sha256:e8eb6cbd7d3b238335b5da0f3ce281102435afb503be4d7bdd69eea3c700a952", size = 219014, upload-time = "2025-11-08T20:24:27.382Z" },
- { url = "https://files.pythonhosted.org/packages/4f/98/aef630a13bc974333aeb83d69765eb513f790bf4bd5b79b8036ec176de8e/coverage-7.11.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:eaa2a5eeb82fa7a6a9cd65c4f968ee2a53839d451b4e88e060c67d87a0a40732", size = 217103, upload-time = "2025-11-08T20:24:28.938Z" },
- { url = "https://files.pythonhosted.org/packages/f9/1f/41f144dc49c07043230ad79126a9c79236724579c43175e476e0731ddc2a/coverage-7.11.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:07e14a4050525fd98bf3d793f229eb8b3ae81678f4031e38e6a18a068bd59fd4", size = 217467, upload-time = "2025-11-08T20:24:30.758Z" },
- { url = "https://files.pythonhosted.org/packages/a1/fa/6fc4b47c7c8323b0326c57786858b6185668f008edc2ea626bc35fb53e28/coverage-7.11.2-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:03e7e7dc31a7deaebf121c3c3bd3c6442b7fbf50aca72aae2a1d08aa30ca2a20", size = 248947, upload-time = "2025-11-08T20:24:32.559Z" },
- { url = "https://files.pythonhosted.org/packages/22/38/03bb7b3d991259ef8d483b83560f87eb4c6d5e8889ad836d212e010d08b3/coverage-7.11.2-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d752a8e398a19e2fb24781e4c73089bfeb417b6ac55f96c2c42cfe5bdb21cc18", size = 251707, upload-time = "2025-11-08T20:24:34.371Z" },
- { url = "https://files.pythonhosted.org/packages/83/6c/c32c7c76c8373978bf68bcfd87a1d265ace9c973ed9a007cada37f25948a/coverage-7.11.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a02818ec44803e325d66bd022828212df934739b894d1699c9a05b9105d30f2", size = 252793, upload-time = "2025-11-08T20:24:35.921Z" },
- { url = "https://files.pythonhosted.org/packages/60/16/86582ab283bad8e137f76e97c5b75a81f547174bca9bb2eba8b7be33d8b6/coverage-7.11.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d30a717493583c2a83c99f195e934c073be7f4291b32b7352c246d52e43f6893", size = 249331, upload-time = "2025-11-08T20:24:37.462Z" },
- { url = "https://files.pythonhosted.org/packages/9e/8a/24449d3e2a84bd38c1903757265cd45b6c9021ecf013f27e33155dba5ada/coverage-7.11.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:55ae008253df6000bc885a780c1b0e939bd8c932f41e16df1cfe19a00428a98a", size = 250728, upload-time = "2025-11-08T20:24:38.936Z" },
- { url = "https://files.pythonhosted.org/packages/86/bc/fcfe9bdda15f48ef6d78a8524837216752fe82474965d42310e6296c8bde/coverage-7.11.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:17047fb65fcd1ce8a2f97dd2247c2b59cb4bc8848b3911db02dcb05856f91b71", size = 248877, upload-time = "2025-11-08T20:24:40.444Z" },
- { url = "https://files.pythonhosted.org/packages/51/27/58db09afcb155f41739330c521258782eefc12fe18f70d3b8e5dbc61857b/coverage-7.11.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5f72a49504e1f35443b157d97997c9259a017384373eab52fd09b8ade2ae4674", size = 248455, upload-time = "2025-11-08T20:24:42.479Z" },
- { url = "https://files.pythonhosted.org/packages/24/6b/1eba5fa2b01b1aa727aa2a1c480c5f475fccecf32decae95b890cef7ee68/coverage-7.11.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5c31cdbb95ab0f4a60224a04efc43cfb406ce904f0b60fb6b2a72f37718ea5cb", size = 250316, upload-time = "2025-11-08T20:24:44.029Z" },
- { url = "https://files.pythonhosted.org/packages/08/58/46d3dcb99366c74b0478f2a58fd97e82419871a50989937e08578f9a5c5c/coverage-7.11.2-cp312-cp312-win32.whl", hash = "sha256:632904d126ca97e5d4ecf7e51ae8b20f086b6f002c6075adcfd4ff3a28574527", size = 219617, upload-time = "2025-11-08T20:24:46.086Z" },
- { url = "https://files.pythonhosted.org/packages/94/19/ab26b96a5c6fd0b5d644524997b60523b3ccbe7473a069e1385a272be238/coverage-7.11.2-cp312-cp312-win_amd64.whl", hash = "sha256:c7ea5dec77d79dabb7b5fc712c59361aac52e459cd22028480625c3c743323d0", size = 220427, upload-time = "2025-11-08T20:24:47.809Z" },
- { url = "https://files.pythonhosted.org/packages/5b/c5/948b268909f04eb2b0a55e22f1e4b3ffd472a8a398d05ebcf95c36d8b1eb/coverage-7.11.2-cp312-cp312-win_arm64.whl", hash = "sha256:ed6ba9f1777fdd1c8e5650c7d123211fa484a187c61af4d82948dc5ee3c0afcc", size = 219068, upload-time = "2025-11-08T20:24:49.813Z" },
- { url = "https://files.pythonhosted.org/packages/ec/00/57f3f8adaced9e4c74f482932e093176df7e400b4bb95dc1f3cd499511b5/coverage-7.11.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:38a5509fe7fabb6fb3161059b947641753b6529150ef483fc01c4516a546f2ad", size = 217125, upload-time = "2025-11-08T20:24:51.368Z" },
- { url = "https://files.pythonhosted.org/packages/fc/2a/ff1a55673161608c895080950cdfbb6485c95e6fa57a92d2cd1e463717b3/coverage-7.11.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7e01ab8d69b6cffa2463e78a4d760a6b69dfebe5bf21837eabcc273655c7e7b3", size = 217499, upload-time = "2025-11-08T20:24:53.238Z" },
- { url = "https://files.pythonhosted.org/packages/73/e3/eaac01709ffbef291a12ca2526b6247f55ab17724e2297cc70921cd9a81f/coverage-7.11.2-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:b4776c6555a9f378f37fa06408f2e1cc1d06e4c4e06adb3d157a4926b549efbe", size = 248479, upload-time = "2025-11-08T20:24:54.825Z" },
- { url = "https://files.pythonhosted.org/packages/75/25/d846d2d08d182eeb30d1eba839fabdd9a3e6c710a1f187657b9c697bab23/coverage-7.11.2-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6f70fa1ef17cba5dada94e144ea1b6e117d4f174666842d1da3aaf765d6eb477", size = 251074, upload-time = "2025-11-08T20:24:56.442Z" },
- { url = "https://files.pythonhosted.org/packages/2e/7a/34c9402ad12bce609be4be1146a7d22a7fae8e9d752684b6315cce552a65/coverage-7.11.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:811bff1f93566a8556a9aeb078bd82573e37f4d802a185fba4cbe75468615050", size = 252318, upload-time = "2025-11-08T20:24:57.987Z" },
- { url = "https://files.pythonhosted.org/packages/cf/2f/292fe3cea4cc1c4b8fb060fa60e565ab1b3bfc67bda74bedefb24b4a2407/coverage-7.11.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d0e80c9946da61cc0bf55dfd90d65707acc1aa5bdcb551d4285ea8906255bb33", size = 248641, upload-time = "2025-11-08T20:24:59.642Z" },
- { url = "https://files.pythonhosted.org/packages/c5/af/33ccb2aa2f43bbc330a1fccf84a396b90f2e61c00dccb7b72b2993a3c795/coverage-7.11.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:10f10c9acf584ef82bfaaa7296163bd11c7487237f1670e81fc2fa7e972be67b", size = 250457, upload-time = "2025-11-08T20:25:01.358Z" },
- { url = "https://files.pythonhosted.org/packages/bd/91/4b5b58f34e0587fbc5c1a28d644d9c20c13349c1072aea507b6e372c8f20/coverage-7.11.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:fd3f7cc6cb999e3eff91a2998a70c662b0fcd3c123d875766147c530ca0d3248", size = 248421, upload-time = "2025-11-08T20:25:02.895Z" },
- { url = "https://files.pythonhosted.org/packages/d5/d5/5c5ed220b15f490717522d241629c522fa22275549a6ccfbc96a3654b009/coverage-7.11.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e52a028a56889d3ad036c0420e866e4a69417d3203e2fc5f03dcb8841274b64c", size = 248244, upload-time = "2025-11-08T20:25:04.742Z" },
- { url = "https://files.pythonhosted.org/packages/1e/27/504088aba40735132db838711d966e1314931ff9bddcd0e2ea6bc7e345a7/coverage-7.11.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f6f985e175dfa1fb8c0a01f47186720ae25d5e20c181cc5f3b9eba95589b8148", size = 250004, upload-time = "2025-11-08T20:25:06.633Z" },
- { url = "https://files.pythonhosted.org/packages/ea/89/4d61c0ad0d39656bd5e73fe41a93a34b063c90333258e6307aadcfcdbb97/coverage-7.11.2-cp313-cp313-win32.whl", hash = "sha256:e48b95abe2983be98cdf52900e07127eb7fe7067c87a700851f4f1f53d2b00e6", size = 219639, upload-time = "2025-11-08T20:25:08.27Z" },
- { url = "https://files.pythonhosted.org/packages/e0/a7/a298afa025ebe7a2afd6657871a1ac2d9c49666ce00f9a35ee9df61a3bd8/coverage-7.11.2-cp313-cp313-win_amd64.whl", hash = "sha256:ea910cc737ee8553c81ad5c104bc5b135106ebb36f88be506c3493e001b4c733", size = 220445, upload-time = "2025-11-08T20:25:09.906Z" },
- { url = "https://files.pythonhosted.org/packages/7e/a1/1825f5eadc0a0a6ea1c6e678827e1ec8c0494dbd23270016fccfc3358fbf/coverage-7.11.2-cp313-cp313-win_arm64.whl", hash = "sha256:ef2d3081562cd83f97984a96e02e7a294efa28f58d5e7f4e28920f59fd752b41", size = 219077, upload-time = "2025-11-08T20:25:11.777Z" },
- { url = "https://files.pythonhosted.org/packages/c0/61/98336c6f4545690b482e805c3a1a83fb2db4c19076307b187db3d421b5b3/coverage-7.11.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:87d7c7b0b2279e174f36d276e2afb7bf16c9ea04e824d4fa277eea1854f4cfd4", size = 217818, upload-time = "2025-11-08T20:25:13.697Z" },
- { url = "https://files.pythonhosted.org/packages/57/ee/6dca6e5f1a4affba8d3224996d0e9145e6d67817da753cc436e48bb8d0e6/coverage-7.11.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:940d195f4c8ba3ec6e7c302c9f546cdbe63e57289ed535452bc52089b1634f1c", size = 218170, upload-time = "2025-11-08T20:25:15.284Z" },
- { url = "https://files.pythonhosted.org/packages/ec/17/9c9ca3ef09d3576027e77cf580eb599d8d655f9ca2456a26ca50c53e07e3/coverage-7.11.2-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:e3b92e10ca996b5421232dd6629b9933f97eb57ce374bca800ab56681fbeda2b", size = 259466, upload-time = "2025-11-08T20:25:17.373Z" },
- { url = "https://files.pythonhosted.org/packages/53/96/2001a596827a0b91ba5f627f21b5ce998fa1f27d861a8f6d909f5ea663ff/coverage-7.11.2-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:61d6a7cc1e7a7a761ac59dcc88cee54219fd4231face52bd1257cfd3df29ae9f", size = 261530, upload-time = "2025-11-08T20:25:19.085Z" },
- { url = "https://files.pythonhosted.org/packages/4d/bb/fea7007035fdc3c40fcca0ab740da549ff9d38fa50b0d37cd808fbbf9683/coverage-7.11.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bee1911c44c52cad6b51d436aa8c6ff5ca5d414fa089c7444592df9e7b890be9", size = 263963, upload-time = "2025-11-08T20:25:21.168Z" },
- { url = "https://files.pythonhosted.org/packages/d2/b3/7452071353441b632ebea42f6ad328a7ab592e4bc50a31f9921b41667017/coverage-7.11.2-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4c4423ea9c28749080b41e18ec74d658e6c9f148a6b47e719f3d7f56197f8227", size = 258644, upload-time = "2025-11-08T20:25:22.928Z" },
- { url = "https://files.pythonhosted.org/packages/e6/05/6e56b1c2b3308f587508ad4b0a4cb76c8d6179fea2df148e071979b3eb77/coverage-7.11.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:689d3b4dd0d4c912ed8bfd7a1b5ff2c5ecb1fa16571840573174704ff5437862", size = 261539, upload-time = "2025-11-08T20:25:25.277Z" },
- { url = "https://files.pythonhosted.org/packages/91/15/7afeeac2a49f651318e4a83f1d5f4d3d4f4092f1d451ac4aec8069cddbdb/coverage-7.11.2-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:75ef769be19d69ea71b0417d7fbf090032c444792579cdf9b166346a340987d5", size = 259153, upload-time = "2025-11-08T20:25:28.098Z" },
- { url = "https://files.pythonhosted.org/packages/1e/77/08f3b5c7500b2031cee74e8a01f9a5bc407f781ff6a826707563bb9dd5b7/coverage-7.11.2-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:6681164bc697b93676945c8c814b76ac72204c395e11b71ba796a93b33331c24", size = 258043, upload-time = "2025-11-08T20:25:30.087Z" },
- { url = "https://files.pythonhosted.org/packages/ca/49/8e080e7622bd7c82df0f8324bbe0461ed1032a638b80046f1a53a88ea3a8/coverage-7.11.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4aa799c61869318d2b86c0d3c413d6805546aec42069f009cbb27df2eefb2790", size = 260243, upload-time = "2025-11-08T20:25:31.722Z" },
- { url = "https://files.pythonhosted.org/packages/dc/75/da033d8589661527b4a6d30c414005467e48fbccc0f3c10898af183e14e1/coverage-7.11.2-cp313-cp313t-win32.whl", hash = "sha256:9a6468e1a3a40d3d1f9120a9ff221d3eacef4540a6f819fff58868fe0bd44fa9", size = 220309, upload-time = "2025-11-08T20:25:33.9Z" },
- { url = "https://files.pythonhosted.org/packages/29/ef/8a477d41dbcde1f1179c13c43c9f77ee926b793fe3e5f1cf5d868a494679/coverage-7.11.2-cp313-cp313t-win_amd64.whl", hash = "sha256:30c437e8b51ce081fe3903c9e368e85c9a803b093fd062c49215f3bf4fd1df37", size = 221374, upload-time = "2025-11-08T20:25:35.88Z" },
- { url = "https://files.pythonhosted.org/packages/0d/a3/4c3cdd737ed1f630b821430004c2d5f1088b9bc0a7115aa5ad7c40d7d5cb/coverage-7.11.2-cp313-cp313t-win_arm64.whl", hash = "sha256:a35701fe0b5ee9d4b67d31aa76555237af32a36b0cf8dd33f8a74470cf7cd2f5", size = 219648, upload-time = "2025-11-08T20:25:37.572Z" },
- { url = "https://files.pythonhosted.org/packages/05/7a/99766a75c88e576f47c2d9a06416ff5d95be9b42faca5c37e1ab77c4cd1a/coverage-7.11.2-py3-none-any.whl", hash = "sha256:2442afabe9e83b881be083238bb7cf5afd4a10e47f29b6094470338d2336b33c", size = 208891, upload-time = "2025-11-08T20:26:30.739Z" },
+version = "7.12.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/89/26/4a96807b193b011588099c3b5c89fbb05294e5b90e71018e065465f34eb6/coverage-7.12.0.tar.gz", hash = "sha256:fc11e0a4e372cb5f282f16ef90d4a585034050ccda536451901abfb19a57f40c", size = 819341, upload-time = "2025-11-18T13:34:20.766Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/26/4a/0dc3de1c172d35abe512332cfdcc43211b6ebce629e4cc42e6cd25ed8f4d/coverage-7.12.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:32b75c2ba3f324ee37af3ccee5b30458038c50b349ad9b88cee85096132a575b", size = 217409, upload-time = "2025-11-18T13:31:53.122Z" },
+ { url = "https://files.pythonhosted.org/packages/01/c3/086198b98db0109ad4f84241e8e9ea7e5fb2db8c8ffb787162d40c26cc76/coverage-7.12.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:cb2a1b6ab9fe833714a483a915de350abc624a37149649297624c8d57add089c", size = 217927, upload-time = "2025-11-18T13:31:54.458Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/5f/34614dbf5ce0420828fc6c6f915126a0fcb01e25d16cf141bf5361e6aea6/coverage-7.12.0-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:5734b5d913c3755e72f70bf6cc37a0518d4f4745cde760c5d8e12005e62f9832", size = 244678, upload-time = "2025-11-18T13:31:55.805Z" },
+ { url = "https://files.pythonhosted.org/packages/55/7b/6b26fb32e8e4a6989ac1d40c4e132b14556131493b1d06bc0f2be169c357/coverage-7.12.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b527a08cdf15753279b7afb2339a12073620b761d79b81cbe2cdebdb43d90daa", size = 246507, upload-time = "2025-11-18T13:31:57.05Z" },
+ { url = "https://files.pythonhosted.org/packages/06/42/7d70e6603d3260199b90fb48b537ca29ac183d524a65cc31366b2e905fad/coverage-7.12.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9bb44c889fb68004e94cab71f6a021ec83eac9aeabdbb5a5a88821ec46e1da73", size = 248366, upload-time = "2025-11-18T13:31:58.362Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/4a/d86b837923878424c72458c5b25e899a3c5ca73e663082a915f5b3c4d749/coverage-7.12.0-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4b59b501455535e2e5dde5881739897967b272ba25988c89145c12d772810ccb", size = 245366, upload-time = "2025-11-18T13:31:59.572Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/c2/2adec557e0aa9721875f06ced19730fdb7fc58e31b02b5aa56f2ebe4944d/coverage-7.12.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:d8842f17095b9868a05837b7b1b73495293091bed870e099521ada176aa3e00e", size = 246408, upload-time = "2025-11-18T13:32:00.784Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/4b/8bd1f1148260df11c618e535fdccd1e5aaf646e55b50759006a4f41d8a26/coverage-7.12.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:c5a6f20bf48b8866095c6820641e7ffbe23f2ac84a2efc218d91235e404c7777", size = 244416, upload-time = "2025-11-18T13:32:01.963Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/13/3a248dd6a83df90414c54a4e121fd081fb20602ca43955fbe1d60e2312a9/coverage-7.12.0-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:5f3738279524e988d9da2893f307c2093815c623f8d05a8f79e3eff3a7a9e553", size = 244681, upload-time = "2025-11-18T13:32:03.408Z" },
+ { url = "https://files.pythonhosted.org/packages/76/30/aa833827465a5e8c938935f5d91ba055f70516941078a703740aaf1aa41f/coverage-7.12.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:e0d68c1f7eabbc8abe582d11fa393ea483caf4f44b0af86881174769f185c94d", size = 245300, upload-time = "2025-11-18T13:32:04.686Z" },
+ { url = "https://files.pythonhosted.org/packages/38/24/f85b3843af1370fb3739fa7571819b71243daa311289b31214fe3e8c9d68/coverage-7.12.0-cp310-cp310-win32.whl", hash = "sha256:7670d860e18b1e3ee5930b17a7d55ae6287ec6e55d9799982aa103a2cc1fa2ef", size = 220008, upload-time = "2025-11-18T13:32:05.806Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/a2/c7da5b9566f7164db9eefa133d17761ecb2c2fde9385d754e5b5c80f710d/coverage-7.12.0-cp310-cp310-win_amd64.whl", hash = "sha256:f999813dddeb2a56aab5841e687b68169da0d3f6fc78ccf50952fa2463746022", size = 220943, upload-time = "2025-11-18T13:32:07.166Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/0c/0dfe7f0487477d96432e4815537263363fb6dd7289743a796e8e51eabdf2/coverage-7.12.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:aa124a3683d2af98bd9d9c2bfa7a5076ca7e5ab09fdb96b81fa7d89376ae928f", size = 217535, upload-time = "2025-11-18T13:32:08.812Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/f5/f9a4a053a5bbff023d3bec259faac8f11a1e5a6479c2ccf586f910d8dac7/coverage-7.12.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d93fbf446c31c0140208dcd07c5d882029832e8ed7891a39d6d44bd65f2316c3", size = 218044, upload-time = "2025-11-18T13:32:10.329Z" },
+ { url = "https://files.pythonhosted.org/packages/95/c5/84fc3697c1fa10cd8571919bf9693f693b7373278daaf3b73e328d502bc8/coverage-7.12.0-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:52ca620260bd8cd6027317bdd8b8ba929be1d741764ee765b42c4d79a408601e", size = 248440, upload-time = "2025-11-18T13:32:12.536Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/36/2d93fbf6a04670f3874aed397d5a5371948a076e3249244a9e84fb0e02d6/coverage-7.12.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f3433ffd541380f3a0e423cff0f4926d55b0cc8c1d160fdc3be24a4c03aa65f7", size = 250361, upload-time = "2025-11-18T13:32:13.852Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/49/66dc65cc456a6bfc41ea3d0758c4afeaa4068a2b2931bf83be6894cf1058/coverage-7.12.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f7bbb321d4adc9f65e402c677cd1c8e4c2d0105d3ce285b51b4d87f1d5db5245", size = 252472, upload-time = "2025-11-18T13:32:15.068Z" },
+ { url = "https://files.pythonhosted.org/packages/35/1f/ebb8a18dffd406db9fcd4b3ae42254aedcaf612470e8712f12041325930f/coverage-7.12.0-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:22a7aade354a72dff3b59c577bfd18d6945c61f97393bc5fb7bd293a4237024b", size = 248592, upload-time = "2025-11-18T13:32:16.328Z" },
+ { url = "https://files.pythonhosted.org/packages/da/a8/67f213c06e5ea3b3d4980df7dc344d7fea88240b5fe878a5dcbdfe0e2315/coverage-7.12.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3ff651dcd36d2fea66877cd4a82de478004c59b849945446acb5baf9379a1b64", size = 250167, upload-time = "2025-11-18T13:32:17.687Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/00/e52aef68154164ea40cc8389c120c314c747fe63a04b013a5782e989b77f/coverage-7.12.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:31b8b2e38391a56e3cea39d22a23faaa7c3fc911751756ef6d2621d2a9daf742", size = 248238, upload-time = "2025-11-18T13:32:19.2Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/a4/4d88750bcf9d6d66f77865e5a05a20e14db44074c25fd22519777cb69025/coverage-7.12.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:297bc2da28440f5ae51c845a47c8175a4db0553a53827886e4fb25c66633000c", size = 247964, upload-time = "2025-11-18T13:32:21.027Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/6b/b74693158899d5b47b0bf6238d2c6722e20ba749f86b74454fac0696bb00/coverage-7.12.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6ff7651cc01a246908eac162a6a86fc0dbab6de1ad165dfb9a1e2ec660b44984", size = 248862, upload-time = "2025-11-18T13:32:22.304Z" },
+ { url = "https://files.pythonhosted.org/packages/18/de/6af6730227ce0e8ade307b1cc4a08e7f51b419a78d02083a86c04ccceb29/coverage-7.12.0-cp311-cp311-win32.whl", hash = "sha256:313672140638b6ddb2c6455ddeda41c6a0b208298034544cfca138978c6baed6", size = 220033, upload-time = "2025-11-18T13:32:23.714Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/a1/e7f63021a7c4fe20994359fcdeae43cbef4a4d0ca36a5a1639feeea5d9e1/coverage-7.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:a1783ed5bd0d5938d4435014626568dc7f93e3cb99bc59188cc18857c47aa3c4", size = 220966, upload-time = "2025-11-18T13:32:25.599Z" },
+ { url = "https://files.pythonhosted.org/packages/77/e8/deae26453f37c20c3aa0c4433a1e32cdc169bf415cce223a693117aa3ddd/coverage-7.12.0-cp311-cp311-win_arm64.whl", hash = "sha256:4648158fd8dd9381b5847622df1c90ff314efbfc1df4550092ab6013c238a5fc", size = 219637, upload-time = "2025-11-18T13:32:27.265Z" },
+ { url = "https://files.pythonhosted.org/packages/02/bf/638c0427c0f0d47638242e2438127f3c8ee3cfc06c7fdeb16778ed47f836/coverage-7.12.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:29644c928772c78512b48e14156b81255000dcfd4817574ff69def189bcb3647", size = 217704, upload-time = "2025-11-18T13:32:28.906Z" },
+ { url = "https://files.pythonhosted.org/packages/08/e1/706fae6692a66c2d6b871a608bbde0da6281903fa0e9f53a39ed441da36a/coverage-7.12.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8638cbb002eaa5d7c8d04da667813ce1067080b9a91099801a0053086e52b736", size = 218064, upload-time = "2025-11-18T13:32:30.161Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/8b/eb0231d0540f8af3ffda39720ff43cb91926489d01524e68f60e961366e4/coverage-7.12.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:083631eeff5eb9992c923e14b810a179798bb598e6a0dd60586819fc23be6e60", size = 249560, upload-time = "2025-11-18T13:32:31.835Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/a1/67fb52af642e974d159b5b379e4d4c59d0ebe1288677fbd04bbffe665a82/coverage-7.12.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:99d5415c73ca12d558e07776bd957c4222c687b9f1d26fa0e1b57e3598bdcde8", size = 252318, upload-time = "2025-11-18T13:32:33.178Z" },
+ { url = "https://files.pythonhosted.org/packages/41/e5/38228f31b2c7665ebf9bdfdddd7a184d56450755c7e43ac721c11a4b8dab/coverage-7.12.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e949ebf60c717c3df63adb4a1a366c096c8d7fd8472608cd09359e1bd48ef59f", size = 253403, upload-time = "2025-11-18T13:32:34.45Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/4b/df78e4c8188f9960684267c5a4897836f3f0f20a20c51606ee778a1d9749/coverage-7.12.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6d907ddccbca819afa2cd014bc69983b146cca2735a0b1e6259b2a6c10be1e70", size = 249984, upload-time = "2025-11-18T13:32:35.747Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/51/bb163933d195a345c6f63eab9e55743413d064c291b6220df754075c2769/coverage-7.12.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b1518ecbad4e6173f4c6e6c4a46e49555ea5679bf3feda5edb1b935c7c44e8a0", size = 251339, upload-time = "2025-11-18T13:32:37.352Z" },
+ { url = "https://files.pythonhosted.org/packages/15/40/c9b29cdb8412c837cdcbc2cfa054547dd83affe6cbbd4ce4fdb92b6ba7d1/coverage-7.12.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:51777647a749abdf6f6fd8c7cffab12de68ab93aab15efc72fbbb83036c2a068", size = 249489, upload-time = "2025-11-18T13:32:39.212Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/da/b3131e20ba07a0de4437a50ef3b47840dfabf9293675b0cd5c2c7f66dd61/coverage-7.12.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:42435d46d6461a3b305cdfcad7cdd3248787771f53fe18305548cba474e6523b", size = 249070, upload-time = "2025-11-18T13:32:40.598Z" },
+ { url = "https://files.pythonhosted.org/packages/70/81/b653329b5f6302c08d683ceff6785bc60a34be9ae92a5c7b63ee7ee7acec/coverage-7.12.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5bcead88c8423e1855e64b8057d0544e33e4080b95b240c2a355334bb7ced937", size = 250929, upload-time = "2025-11-18T13:32:42.915Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/00/250ac3bca9f252a5fb1338b5ad01331ebb7b40223f72bef5b1b2cb03aa64/coverage-7.12.0-cp312-cp312-win32.whl", hash = "sha256:dcbb630ab034e86d2a0f79aefd2be07e583202f41e037602d438c80044957baa", size = 220241, upload-time = "2025-11-18T13:32:44.665Z" },
+ { url = "https://files.pythonhosted.org/packages/64/1c/77e79e76d37ce83302f6c21980b45e09f8aa4551965213a10e62d71ce0ab/coverage-7.12.0-cp312-cp312-win_amd64.whl", hash = "sha256:2fd8354ed5d69775ac42986a691fbf68b4084278710cee9d7c3eaa0c28fa982a", size = 221051, upload-time = "2025-11-18T13:32:46.008Z" },
+ { url = "https://files.pythonhosted.org/packages/31/f5/641b8a25baae564f9e52cac0e2667b123de961985709a004e287ee7663cc/coverage-7.12.0-cp312-cp312-win_arm64.whl", hash = "sha256:737c3814903be30695b2de20d22bcc5428fdae305c61ba44cdc8b3252984c49c", size = 219692, upload-time = "2025-11-18T13:32:47.372Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/14/771700b4048774e48d2c54ed0c674273702713c9ee7acdfede40c2666747/coverage-7.12.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:47324fffca8d8eae7e185b5bb20c14645f23350f870c1649003618ea91a78941", size = 217725, upload-time = "2025-11-18T13:32:49.22Z" },
+ { url = "https://files.pythonhosted.org/packages/17/a7/3aa4144d3bcb719bf67b22d2d51c2d577bf801498c13cb08f64173e80497/coverage-7.12.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ccf3b2ede91decd2fb53ec73c1f949c3e034129d1e0b07798ff1d02ea0c8fa4a", size = 218098, upload-time = "2025-11-18T13:32:50.78Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/9c/b846bbc774ff81091a12a10203e70562c91ae71badda00c5ae5b613527b1/coverage-7.12.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:b365adc70a6936c6b0582dc38746b33b2454148c02349345412c6e743efb646d", size = 249093, upload-time = "2025-11-18T13:32:52.554Z" },
+ { url = "https://files.pythonhosted.org/packages/76/b6/67d7c0e1f400b32c883e9342de4a8c2ae7c1a0b57c5de87622b7262e2309/coverage-7.12.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bc13baf85cd8a4cfcf4a35c7bc9d795837ad809775f782f697bf630b7e200211", size = 251686, upload-time = "2025-11-18T13:32:54.862Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/75/b095bd4b39d49c3be4bffbb3135fea18a99a431c52dd7513637c0762fecb/coverage-7.12.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:099d11698385d572ceafb3288a5b80fe1fc58bf665b3f9d362389de488361d3d", size = 252930, upload-time = "2025-11-18T13:32:56.417Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/f3/466f63015c7c80550bead3093aacabf5380c1220a2a93c35d374cae8f762/coverage-7.12.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:473dc45d69694069adb7680c405fb1e81f60b2aff42c81e2f2c3feaf544d878c", size = 249296, upload-time = "2025-11-18T13:32:58.074Z" },
+ { url = "https://files.pythonhosted.org/packages/27/86/eba2209bf2b7e28c68698fc13437519a295b2d228ba9e0ec91673e09fa92/coverage-7.12.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:583f9adbefd278e9de33c33d6846aa8f5d164fa49b47144180a0e037f0688bb9", size = 251068, upload-time = "2025-11-18T13:32:59.646Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/55/ca8ae7dbba962a3351f18940b359b94c6bafdd7757945fdc79ec9e452dc7/coverage-7.12.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b2089cc445f2dc0af6f801f0d1355c025b76c24481935303cf1af28f636688f0", size = 249034, upload-time = "2025-11-18T13:33:01.481Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/d7/39136149325cad92d420b023b5fd900dabdd1c3a0d1d5f148ef4a8cedef5/coverage-7.12.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:950411f1eb5d579999c5f66c62a40961f126fc71e5e14419f004471957b51508", size = 248853, upload-time = "2025-11-18T13:33:02.935Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/b6/76e1add8b87ef60e00643b0b7f8f7bb73d4bf5249a3be19ebefc5793dd25/coverage-7.12.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b1aab7302a87bafebfe76b12af681b56ff446dc6f32ed178ff9c092ca776e6bc", size = 250619, upload-time = "2025-11-18T13:33:04.336Z" },
+ { url = "https://files.pythonhosted.org/packages/95/87/924c6dc64f9203f7a3c1832a6a0eee5a8335dbe5f1bdadcc278d6f1b4d74/coverage-7.12.0-cp313-cp313-win32.whl", hash = "sha256:d7e0d0303c13b54db495eb636bc2465b2fb8475d4c8bcec8fe4b5ca454dfbae8", size = 220261, upload-time = "2025-11-18T13:33:06.493Z" },
+ { url = "https://files.pythonhosted.org/packages/91/77/dd4aff9af16ff776bf355a24d87eeb48fc6acde54c907cc1ea89b14a8804/coverage-7.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:ce61969812d6a98a981d147d9ac583a36ac7db7766f2e64a9d4d059c2fe29d07", size = 221072, upload-time = "2025-11-18T13:33:07.926Z" },
+ { url = "https://files.pythonhosted.org/packages/70/49/5c9dc46205fef31b1b226a6e16513193715290584317fd4df91cdaf28b22/coverage-7.12.0-cp313-cp313-win_arm64.whl", hash = "sha256:bcec6f47e4cb8a4c2dc91ce507f6eefc6a1b10f58df32cdc61dff65455031dfc", size = 219702, upload-time = "2025-11-18T13:33:09.631Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/62/f87922641c7198667994dd472a91e1d9b829c95d6c29529ceb52132436ad/coverage-7.12.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:459443346509476170d553035e4a3eed7b860f4fe5242f02de1010501956ce87", size = 218420, upload-time = "2025-11-18T13:33:11.153Z" },
+ { url = "https://files.pythonhosted.org/packages/85/dd/1cc13b2395ef15dbb27d7370a2509b4aee77890a464fb35d72d428f84871/coverage-7.12.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:04a79245ab2b7a61688958f7a855275997134bc84f4a03bc240cf64ff132abf6", size = 218773, upload-time = "2025-11-18T13:33:12.569Z" },
+ { url = "https://files.pythonhosted.org/packages/74/40/35773cc4bb1e9d4658d4fb669eb4195b3151bef3bbd6f866aba5cd5dac82/coverage-7.12.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:09a86acaaa8455f13d6a99221d9654df249b33937b4e212b4e5a822065f12aa7", size = 260078, upload-time = "2025-11-18T13:33:14.037Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/ee/231bb1a6ffc2905e396557585ebc6bdc559e7c66708376d245a1f1d330fc/coverage-7.12.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:907e0df1b71ba77463687a74149c6122c3f6aac56c2510a5d906b2f368208560", size = 262144, upload-time = "2025-11-18T13:33:15.601Z" },
+ { url = "https://files.pythonhosted.org/packages/28/be/32f4aa9f3bf0b56f3971001b56508352c7753915345d45fab4296a986f01/coverage-7.12.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b57e2d0ddd5f0582bae5437c04ee71c46cd908e7bc5d4d0391f9a41e812dd12", size = 264574, upload-time = "2025-11-18T13:33:17.354Z" },
+ { url = "https://files.pythonhosted.org/packages/68/7c/00489fcbc2245d13ab12189b977e0cf06ff3351cb98bc6beba8bd68c5902/coverage-7.12.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:58c1c6aa677f3a1411fe6fb28ec3a942e4f665df036a3608816e0847fad23296", size = 259298, upload-time = "2025-11-18T13:33:18.958Z" },
+ { url = "https://files.pythonhosted.org/packages/96/b4/f0760d65d56c3bea95b449e02570d4abd2549dc784bf39a2d4721a2d8ceb/coverage-7.12.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4c589361263ab2953e3c4cd2a94db94c4ad4a8e572776ecfbad2389c626e4507", size = 262150, upload-time = "2025-11-18T13:33:20.644Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/71/9a9314df00f9326d78c1e5a910f520d599205907432d90d1c1b7a97aa4b1/coverage-7.12.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:91b810a163ccad2e43b1faa11d70d3cf4b6f3d83f9fd5f2df82a32d47b648e0d", size = 259763, upload-time = "2025-11-18T13:33:22.189Z" },
+ { url = "https://files.pythonhosted.org/packages/10/34/01a0aceed13fbdf925876b9a15d50862eb8845454301fe3cdd1df08b2182/coverage-7.12.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:40c867af715f22592e0d0fb533a33a71ec9e0f73a6945f722a0c85c8c1cbe3a2", size = 258653, upload-time = "2025-11-18T13:33:24.239Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/04/81d8fd64928acf1574bbb0181f66901c6c1c6279c8ccf5f84259d2c68ae9/coverage-7.12.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:68b0d0a2d84f333de875666259dadf28cc67858bc8fd8b3f1eae84d3c2bec455", size = 260856, upload-time = "2025-11-18T13:33:26.365Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/76/fa2a37bfaeaf1f766a2d2360a25a5297d4fb567098112f6517475eee120b/coverage-7.12.0-cp313-cp313t-win32.whl", hash = "sha256:73f9e7fbd51a221818fd11b7090eaa835a353ddd59c236c57b2199486b116c6d", size = 220936, upload-time = "2025-11-18T13:33:28.165Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/52/60f64d932d555102611c366afb0eb434b34266b1d9266fc2fe18ab641c47/coverage-7.12.0-cp313-cp313t-win_amd64.whl", hash = "sha256:24cff9d1f5743f67db7ba46ff284018a6e9aeb649b67aa1e70c396aa1b7cb23c", size = 222001, upload-time = "2025-11-18T13:33:29.656Z" },
+ { url = "https://files.pythonhosted.org/packages/77/df/c303164154a5a3aea7472bf323b7c857fed93b26618ed9fc5c2955566bb0/coverage-7.12.0-cp313-cp313t-win_arm64.whl", hash = "sha256:c87395744f5c77c866d0f5a43d97cc39e17c7f1cb0115e54a2fe67ca75c5d14d", size = 220273, upload-time = "2025-11-18T13:33:31.415Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/a3/43b749004e3c09452e39bb56347a008f0a0668aad37324a99b5c8ca91d9e/coverage-7.12.0-py3-none-any.whl", hash = "sha256:159d50c0b12e060b15ed3d39f87ed43d4f7f7ad40b8a534f4dd331adbb51104a", size = 209503, upload-time = "2025-11-18T13:34:18.892Z" },
]
[package.optional-dependencies]
@@ -553,6 +562,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/3f/27/4570e78fc0bf5ea0ca45eb1de3818a23787af9b390c0b0a0033a1b8236f9/diskcache-5.6.3-py3-none-any.whl", hash = "sha256:5e31b2d5fbad117cc363ebaf6b689474db18a1f6438bc82358b024abd4c2ca19", size = 45550, upload-time = "2023-08-31T06:11:58.822Z" },
]
+[[package]]
+name = "distlib"
+version = "0.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/96/8e/709914eb2b5749865801041647dc7f4e6d00b549cfe88b65ca192995f07c/distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d", size = 614605, upload-time = "2025-07-17T16:52:00.465Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16", size = 469047, upload-time = "2025-07-17T16:51:58.613Z" },
+]
+
[[package]]
name = "distro"
version = "1.9.0"
@@ -573,11 +591,11 @@ wheels = [
[[package]]
name = "eval-type-backport"
-version = "0.2.2"
+version = "0.3.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/30/ea/8b0ac4469d4c347c6a385ff09dc3c048c2d021696664e26c7ee6791631b5/eval_type_backport-0.2.2.tar.gz", hash = "sha256:f0576b4cf01ebb5bd358d02314d31846af5e07678387486e2c798af0e7d849c1", size = 9079, upload-time = "2024-12-21T20:09:46.005Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/51/23/079e39571d6dd8d90d7a369ecb55ad766efb6bae4e77389629e14458c280/eval_type_backport-0.3.0.tar.gz", hash = "sha256:1638210401e184ff17f877e9a2fa076b60b5838790f4532a21761cc2be67aea1", size = 9272, upload-time = "2025-11-13T20:56:50.845Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ce/31/55cd413eaccd39125368be33c46de24a1f639f2e12349b0361b4678f3915/eval_type_backport-0.2.2-py3-none-any.whl", hash = "sha256:cb6ad7c393517f476f96d456d0412ea80f0a8cf96f6892834cd9340149111b0a", size = 5830, upload-time = "2024-12-21T20:09:44.175Z" },
+ { url = "https://files.pythonhosted.org/packages/19/d8/2a1c638d9e0aa7e269269a1a1bf423ddd94267f1a01bbe3ad03432b67dd4/eval_type_backport-0.3.0-py3-none-any.whl", hash = "sha256:975a10a0fe333c8b6260d7fdb637698c9a16c3a9e3b6eb943fee6a6f67a37fe8", size = 6061, upload-time = "2025-11-13T20:56:49.499Z" },
]
[[package]]
@@ -594,36 +612,45 @@ wheels = [
[[package]]
name = "execnet"
-version = "2.1.1"
+version = "2.1.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/bb/ff/b4c0dc78fbe20c3e59c0c7334de0c27eb4001a2b2017999af398bf730817/execnet-2.1.1.tar.gz", hash = "sha256:5189b52c6121c24feae288166ab41b32549c7e2348652736540b9e6e7d4e72e3", size = 166524, upload-time = "2024-04-08T09:04:19.245Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/43/09/2aea36ff60d16dd8879bdb2f5b3ee0ba8d08cbbdcdfe870e695ce3784385/execnet-2.1.1-py3-none-any.whl", hash = "sha256:26dee51f1b80cebd6d0ca8e74dd8745419761d3bef34163928cbebbdc4749fdc", size = 40612, upload-time = "2024-04-08T09:04:17.414Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" },
]
[[package]]
name = "faker"
-version = "37.12.0"
+version = "38.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "tzdata" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/3d/84/e95acaa848b855e15c83331d0401ee5f84b2f60889255c2e055cb4fb6bdf/faker-37.12.0.tar.gz", hash = "sha256:7505e59a7e02fa9010f06c3e1e92f8250d4cfbb30632296140c2d6dbef09b0fa", size = 1935741, upload-time = "2025-10-24T15:19:58.764Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/27/022d4dbd4c20567b4c294f79a133cc2f05240ea61e0d515ead18c995c249/faker-38.2.0.tar.gz", hash = "sha256:20672803db9c7cb97f9b56c18c54b915b6f1d8991f63d1d673642dc43f5ce7ab", size = 1941469, upload-time = "2025-11-19T16:37:31.892Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/8e/98/2c050dec90e295a524c9b65c4cb9e7c302386a296b2938710448cbd267d5/faker-37.12.0-py3-none-any.whl", hash = "sha256:afe7ccc038da92f2fbae30d8e16d19d91e92e242f8401ce9caf44de892bab4c4", size = 1975461, upload-time = "2025-10-24T15:19:55.739Z" },
+ { url = "https://files.pythonhosted.org/packages/17/93/00c94d45f55c336434a15f98d906387e87ce28f9918e4444829a8fda432d/faker-38.2.0-py3-none-any.whl", hash = "sha256:35fe4a0a79dee0dc4103a6083ee9224941e7d3594811a50e3969e547b0d2ee65", size = 1980505, upload-time = "2025-11-19T16:37:30.208Z" },
]
[[package]]
name = "fal-client"
-version = "0.9.0"
+version = "0.9.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "httpx" },
{ name = "httpx-sse" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/bb/5e/3d1e10041c20db96abbc21a4d9d4594e05e4643b8ef5d775c28ff28b2826/fal_client-0.9.0.tar.gz", hash = "sha256:c8602e540db23ea19d30f3efc932137119515e3f7f9dfbfd63044332fbd74e86", size = 16483, upload-time = "2025-11-09T14:14:59.855Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b8/a1/98ab1cea4c2424ee612292bc92b07905e1a15a05584f6c263cde38e6a3a2/fal_client-0.9.1.tar.gz", hash = "sha256:c8f7f88f79c4b4c4f069be9f571be924dc7c4a6bf07c252fe0b75f3c46c8d66d", size = 17085, upload-time = "2025-11-13T18:15:09.911Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/60/e1/7f468aeb695950d737971308e223a90c83436606107906ba7ecb43e10406/fal_client-0.9.0-py3-none-any.whl", hash = "sha256:91280ece00211fd36977b1a1b74bc9638c56f8617705bb0c8e9292efa175c293", size = 11000, upload-time = "2025-11-09T14:14:58.649Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/57/775821a71459f2b83bbaa59452a4b1e4772f7c770de88a6f591c9d43c7c8/fal_client-0.9.1-py3-none-any.whl", hash = "sha256:8eba86c947299852c8306f685eee883ce01856543bf4344b87f65abd4b7d7622", size = 11157, upload-time = "2025-11-13T18:15:08.528Z" },
+]
+
+[[package]]
+name = "filelock"
+version = "3.20.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/58/46/0028a82567109b5ef6e4d2a1f04a583fb513e6cf9527fcdd09afd817deeb/filelock-3.20.0.tar.gz", hash = "sha256:711e943b4ec6be42e1d4e6690b48dc175c822967466bb31c0c293f34334c13f4", size = 18922, upload-time = "2025-10-08T18:03:50.056Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/76/91/7216b27286936c16f5b4d0c530087e4a54eead683e6b0b73dd0c64844af6/filelock-3.20.0-py3-none-any.whl", hash = "sha256:339b4732ffda5cd79b13f4e2711a31b0365ce445d95d243bb996273d072546a2", size = 16054, upload-time = "2025-10-08T18:03:48.35Z" },
]
[[package]]
@@ -765,7 +792,7 @@ wheels = [
[[package]]
name = "google-genai"
-version = "1.49.0"
+version = "1.51.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@@ -777,9 +804,9 @@ dependencies = [
{ name = "typing-extensions" },
{ name = "websockets" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/82/49/1a724ee3c3748fa50721d53a52d9fee88c67d0c43bb16eb2b10ee89ab239/google_genai-1.49.0.tar.gz", hash = "sha256:35eb16023b72e298571ae30e919c810694f258f2ba68fc77a2185c7c8829ad5a", size = 253493, upload-time = "2025-11-05T22:41:03.278Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/1c/29245699c7c274ed5709b33b6a5192af2d57da5da3d2f189f222d1895336/google_genai-1.51.0.tar.gz", hash = "sha256:596c1ec964b70fec17a6ccfe6ee4edede31022584e8b1d33371d93037c4001b1", size = 258060, upload-time = "2025-11-18T05:32:47.068Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d5/d3/84a152746dc7bdebb8ba0fd7d6157263044acd1d14b2a53e8df4a307b6b7/google_genai-1.49.0-py3-none-any.whl", hash = "sha256:ad49cd5be5b63397069e7aef9a4fe0a84cbdf25fcd93408e795292308db4ef32", size = 256098, upload-time = "2025-11-05T22:41:01.429Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/28/0185dcda66f1994171067cfdb0e44a166450239d5b11b3a8a281dd2da459/google_genai-1.51.0-py3-none-any.whl", hash = "sha256:bfb7d0c6ba48ba9bda539f0d5e69dad827d8735a8b1e4703bafa0a2945d293e1", size = 260483, upload-time = "2025-11-18T05:32:45.938Z" },
]
[[package]]
@@ -828,6 +855,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" },
]
+[[package]]
+name = "identify"
+version = "2.6.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ff/e7/685de97986c916a6d93b3876139e00eef26ad5bbbd61925d670ae8013449/identify-2.6.15.tar.gz", hash = "sha256:e4f4864b96c6557ef2a1e1c951771838f4edc9df3a72ec7118b338801b11c7bf", size = 99311, upload-time = "2025-10-02T17:43:40.631Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0f/1c/e5fd8f973d4f375adb21565739498e2e9a1e54c858a97b9a8ccfdc81da9b/identify-2.6.15-py2.py3-none-any.whl", hash = "sha256:1181ef7608e00704db228516541eb83a88a9f94433a8c80bb9b5bd54b1d81757", size = 99183, upload-time = "2025-10-02T17:43:39.137Z" },
+]
+
[[package]]
name = "idna"
version = "3.11"
@@ -848,7 +884,7 @@ wheels = [
[[package]]
name = "instructor"
-version = "1.10.0"
+version = "1.13.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiohttp" },
@@ -857,16 +893,18 @@ dependencies = [
{ name = "jinja2" },
{ name = "jiter" },
{ name = "openai" },
+ { name = "pre-commit" },
{ name = "pydantic" },
{ name = "pydantic-core" },
{ name = "requests" },
{ name = "rich" },
{ name = "tenacity" },
+ { name = "ty" },
{ name = "typer" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/a5/67/63c4b4d2cc3c7b4238920ad3388a6f5d67265ab7c09ee34012d6b591130e/instructor-1.10.0.tar.gz", hash = "sha256:887d33e058b913290dbf526b0096b1bb8d7ea1a07d75afecbf716161f959697b", size = 69388981, upload-time = "2025-07-18T15:28:52.386Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/31/f0/7f31609ec2fb84b140ff573abf1cce78cd3a2a3c6479b60aa82b69d40d2a/instructor-1.13.0.tar.gz", hash = "sha256:bf838a5c503fafdd034a9b1f8544c5e1f62462eea9f89932bc75c116ad35ab5a", size = 69898121, upload-time = "2025-11-06T04:19:31.034Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/2c/fb/ffc1ade9779795a8dc8e2379b1bfb522161ee7df8df12722f50d348fb4ea/instructor-1.10.0-py3-none-any.whl", hash = "sha256:9c789f0fce915d5498059afb5314530c8a5b22b0283302679148ddae98f732b0", size = 119455, upload-time = "2025-07-18T15:28:48.785Z" },
+ { url = "https://files.pythonhosted.org/packages/95/64/6542ac826a4c9b937b67c096a785af1aaa26b22fcb7c81223cfe4038205b/instructor-1.13.0-py3-none-any.whl", hash = "sha256:2b735b6ea0d3194548369a18254f1dde83cb5ec0b182de77adbadd8be73caddc", size = 160904, upload-time = "2025-11-06T04:19:24.674Z" },
]
[package.optional-dependencies]
@@ -898,61 +936,74 @@ wheels = [
[[package]]
name = "jiter"
-version = "0.8.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/f8/70/90bc7bd3932e651486861df5c8ffea4ca7c77d28e8532ddefe2abc561a53/jiter-0.8.2.tar.gz", hash = "sha256:cd73d3e740666d0e639f678adb176fad25c1bcbdae88d8d7b857e1783bb4212d", size = 163007, upload-time = "2024-12-09T18:11:08.649Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/f2/f3/8c11e0e87bd5934c414f9b1cfae3cbfd4a938d4669d57cb427e1c4d11a7f/jiter-0.8.2-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:ca8577f6a413abe29b079bc30f907894d7eb07a865c4df69475e868d73e71c7b", size = 303381, upload-time = "2024-12-09T18:09:00.301Z" },
- { url = "https://files.pythonhosted.org/packages/ea/28/4cd3f0bcbf40e946bc6a62a82c951afc386a25673d3d8d5ee461f1559bbe/jiter-0.8.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b25bd626bde7fb51534190c7e3cb97cee89ee76b76d7585580e22f34f5e3f393", size = 311718, upload-time = "2024-12-09T18:09:02.53Z" },
- { url = "https://files.pythonhosted.org/packages/0d/17/57acab00507e60bd954eaec0837d9d7b119b4117ff49b8a62f2b646f32ed/jiter-0.8.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d5c826a221851a8dc028eb6d7d6429ba03184fa3c7e83ae01cd6d3bd1d4bd17d", size = 335465, upload-time = "2024-12-09T18:09:04.044Z" },
- { url = "https://files.pythonhosted.org/packages/74/b9/1a3ddd2bc95ae17c815b021521020f40c60b32137730126bada962ef32b4/jiter-0.8.2-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d35c864c2dff13dfd79fb070fc4fc6235d7b9b359efe340e1261deb21b9fcb66", size = 355570, upload-time = "2024-12-09T18:09:05.445Z" },
- { url = "https://files.pythonhosted.org/packages/78/69/6d29e2296a934199a7d0dde673ecccf98c9c8db44caf0248b3f2b65483cb/jiter-0.8.2-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f557c55bc2b7676e74d39d19bcb8775ca295c7a028246175d6a8b431e70835e5", size = 381383, upload-time = "2024-12-09T18:09:07.499Z" },
- { url = "https://files.pythonhosted.org/packages/22/d7/fbc4c3fb1bf65f9be22a32759b539f88e897aeb13fe84ab0266e4423487a/jiter-0.8.2-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:580ccf358539153db147e40751a0b41688a5ceb275e6f3e93d91c9467f42b2e3", size = 390454, upload-time = "2024-12-09T18:09:09.587Z" },
- { url = "https://files.pythonhosted.org/packages/4d/a0/3993cda2e267fe679b45d0bcc2cef0b4504b0aa810659cdae9737d6bace9/jiter-0.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:af102d3372e917cffce49b521e4c32c497515119dc7bd8a75665e90a718bbf08", size = 345039, upload-time = "2024-12-09T18:09:11.045Z" },
- { url = "https://files.pythonhosted.org/packages/b9/ef/69c18562b4c09ce88fab5df1dcaf643f6b1a8b970b65216e7221169b81c4/jiter-0.8.2-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:cadcc978f82397d515bb2683fc0d50103acff2a180552654bb92d6045dec2c49", size = 376200, upload-time = "2024-12-09T18:09:13.104Z" },
- { url = "https://files.pythonhosted.org/packages/4d/17/0b5a8de46a6ab4d836f70934036278b49b8530c292b29dde3483326d4555/jiter-0.8.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:ba5bdf56969cad2019d4e8ffd3f879b5fdc792624129741d3d83fc832fef8c7d", size = 511158, upload-time = "2024-12-09T18:09:15.222Z" },
- { url = "https://files.pythonhosted.org/packages/6c/b2/c401a0a2554b36c9e6d6e4876b43790d75139cf3936f0222e675cbc23451/jiter-0.8.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:3b94a33a241bee9e34b8481cdcaa3d5c2116f575e0226e421bed3f7a6ea71cff", size = 503956, upload-time = "2024-12-09T18:09:16.595Z" },
- { url = "https://files.pythonhosted.org/packages/d4/02/a0291ed7d72c0ac130f172354ee3cf0b2556b69584de391463a8ee534f40/jiter-0.8.2-cp310-cp310-win32.whl", hash = "sha256:6e5337bf454abddd91bd048ce0dca5134056fc99ca0205258766db35d0a2ea43", size = 202846, upload-time = "2024-12-09T18:09:19.347Z" },
- { url = "https://files.pythonhosted.org/packages/ad/20/8c988831ae4bf437e29f1671e198fc99ba8fe49f2895f23789acad1d1811/jiter-0.8.2-cp310-cp310-win_amd64.whl", hash = "sha256:4a9220497ca0cb1fe94e3f334f65b9b5102a0b8147646118f020d8ce1de70105", size = 204414, upload-time = "2024-12-09T18:09:20.904Z" },
- { url = "https://files.pythonhosted.org/packages/cb/b0/c1a7caa7f9dc5f1f6cfa08722867790fe2d3645d6e7170ca280e6e52d163/jiter-0.8.2-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:2dd61c5afc88a4fda7d8b2cf03ae5947c6ac7516d32b7a15bf4b49569a5c076b", size = 303666, upload-time = "2024-12-09T18:09:23.145Z" },
- { url = "https://files.pythonhosted.org/packages/f5/97/0468bc9eeae43079aaa5feb9267964e496bf13133d469cfdc135498f8dd0/jiter-0.8.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a6c710d657c8d1d2adbbb5c0b0c6bfcec28fd35bd6b5f016395f9ac43e878a15", size = 311934, upload-time = "2024-12-09T18:09:25.098Z" },
- { url = "https://files.pythonhosted.org/packages/e5/69/64058e18263d9a5f1e10f90c436853616d5f047d997c37c7b2df11b085ec/jiter-0.8.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a9584de0cd306072635fe4b89742bf26feae858a0683b399ad0c2509011b9dc0", size = 335506, upload-time = "2024-12-09T18:09:26.407Z" },
- { url = "https://files.pythonhosted.org/packages/9d/14/b747f9a77b8c0542141d77ca1e2a7523e854754af2c339ac89a8b66527d6/jiter-0.8.2-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5a90a923338531b7970abb063cfc087eebae6ef8ec8139762007188f6bc69a9f", size = 355849, upload-time = "2024-12-09T18:09:27.686Z" },
- { url = "https://files.pythonhosted.org/packages/53/e2/98a08161db7cc9d0e39bc385415890928ff09709034982f48eccfca40733/jiter-0.8.2-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d21974d246ed0181558087cd9f76e84e8321091ebfb3a93d4c341479a736f099", size = 381700, upload-time = "2024-12-09T18:09:28.989Z" },
- { url = "https://files.pythonhosted.org/packages/7a/38/1674672954d35bce3b1c9af99d5849f9256ac8f5b672e020ac7821581206/jiter-0.8.2-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:32475a42b2ea7b344069dc1e81445cfc00b9d0e3ca837f0523072432332e9f74", size = 389710, upload-time = "2024-12-09T18:09:30.565Z" },
- { url = "https://files.pythonhosted.org/packages/f8/9b/92f9da9a9e107d019bcf883cd9125fa1690079f323f5a9d5c6986eeec3c0/jiter-0.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8b9931fd36ee513c26b5bf08c940b0ac875de175341cbdd4fa3be109f0492586", size = 345553, upload-time = "2024-12-09T18:09:32.735Z" },
- { url = "https://files.pythonhosted.org/packages/44/a6/6d030003394e9659cd0d7136bbeabd82e869849ceccddc34d40abbbbb269/jiter-0.8.2-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ce0820f4a3a59ddced7fce696d86a096d5cc48d32a4183483a17671a61edfddc", size = 376388, upload-time = "2024-12-09T18:09:34.723Z" },
- { url = "https://files.pythonhosted.org/packages/ad/8d/87b09e648e4aca5f9af89e3ab3cfb93db2d1e633b2f2931ede8dabd9b19a/jiter-0.8.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:8ffc86ae5e3e6a93765d49d1ab47b6075a9c978a2b3b80f0f32628f39caa0c88", size = 511226, upload-time = "2024-12-09T18:09:36.13Z" },
- { url = "https://files.pythonhosted.org/packages/77/95/8008ebe4cdc82eac1c97864a8042ca7e383ed67e0ec17bfd03797045c727/jiter-0.8.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5127dc1abd809431172bc3fbe8168d6b90556a30bb10acd5ded41c3cfd6f43b6", size = 504134, upload-time = "2024-12-09T18:09:37.581Z" },
- { url = "https://files.pythonhosted.org/packages/26/0d/3056a74de13e8b2562e4d526de6dac2f65d91ace63a8234deb9284a1d24d/jiter-0.8.2-cp311-cp311-win32.whl", hash = "sha256:66227a2c7b575720c1871c8800d3a0122bb8ee94edb43a5685aa9aceb2782d44", size = 203103, upload-time = "2024-12-09T18:09:38.881Z" },
- { url = "https://files.pythonhosted.org/packages/4e/1e/7f96b798f356e531ffc0f53dd2f37185fac60fae4d6c612bbbd4639b90aa/jiter-0.8.2-cp311-cp311-win_amd64.whl", hash = "sha256:cde031d8413842a1e7501e9129b8e676e62a657f8ec8166e18a70d94d4682855", size = 206717, upload-time = "2024-12-09T18:09:41.064Z" },
- { url = "https://files.pythonhosted.org/packages/a1/17/c8747af8ea4e045f57d6cfd6fc180752cab9bc3de0e8a0c9ca4e8af333b1/jiter-0.8.2-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:e6ec2be506e7d6f9527dae9ff4b7f54e68ea44a0ef6b098256ddf895218a2f8f", size = 302027, upload-time = "2024-12-09T18:09:43.11Z" },
- { url = "https://files.pythonhosted.org/packages/3c/c1/6da849640cd35a41e91085723b76acc818d4b7d92b0b6e5111736ce1dd10/jiter-0.8.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:76e324da7b5da060287c54f2fabd3db5f76468006c811831f051942bf68c9d44", size = 310326, upload-time = "2024-12-09T18:09:44.426Z" },
- { url = "https://files.pythonhosted.org/packages/06/99/a2bf660d8ccffee9ad7ed46b4f860d2108a148d0ea36043fd16f4dc37e94/jiter-0.8.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:180a8aea058f7535d1c84183c0362c710f4750bef66630c05f40c93c2b152a0f", size = 334242, upload-time = "2024-12-09T18:09:45.915Z" },
- { url = "https://files.pythonhosted.org/packages/a7/5f/cea1c17864828731f11427b9d1ab7f24764dbd9aaf4648a7f851164d2718/jiter-0.8.2-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:025337859077b41548bdcbabe38698bcd93cfe10b06ff66617a48ff92c9aec60", size = 356654, upload-time = "2024-12-09T18:09:47.619Z" },
- { url = "https://files.pythonhosted.org/packages/e9/13/62774b7e5e7f5d5043efe1d0f94ead66e6d0f894ae010adb56b3f788de71/jiter-0.8.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ecff0dc14f409599bbcafa7e470c00b80f17abc14d1405d38ab02e4b42e55b57", size = 379967, upload-time = "2024-12-09T18:09:49.987Z" },
- { url = "https://files.pythonhosted.org/packages/ec/fb/096b34c553bb0bd3f2289d5013dcad6074948b8d55212aa13a10d44c5326/jiter-0.8.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ffd9fee7d0775ebaba131f7ca2e2d83839a62ad65e8e02fe2bd8fc975cedeb9e", size = 389252, upload-time = "2024-12-09T18:09:51.329Z" },
- { url = "https://files.pythonhosted.org/packages/17/61/beea645c0bf398ced8b199e377b61eb999d8e46e053bb285c91c3d3eaab0/jiter-0.8.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:14601dcac4889e0a1c75ccf6a0e4baf70dbc75041e51bcf8d0e9274519df6887", size = 345490, upload-time = "2024-12-09T18:09:52.646Z" },
- { url = "https://files.pythonhosted.org/packages/d5/df/834aa17ad5dcc3cf0118821da0a0cf1589ea7db9832589278553640366bc/jiter-0.8.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:92249669925bc1c54fcd2ec73f70f2c1d6a817928480ee1c65af5f6b81cdf12d", size = 376991, upload-time = "2024-12-09T18:09:53.972Z" },
- { url = "https://files.pythonhosted.org/packages/67/80/87d140399d382fb4ea5b3d56e7ecaa4efdca17cd7411ff904c1517855314/jiter-0.8.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:e725edd0929fa79f8349ab4ec7f81c714df51dc4e991539a578e5018fa4a7152", size = 510822, upload-time = "2024-12-09T18:09:55.439Z" },
- { url = "https://files.pythonhosted.org/packages/5c/37/3394bb47bac1ad2cb0465601f86828a0518d07828a650722e55268cdb7e6/jiter-0.8.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:bf55846c7b7a680eebaf9c3c48d630e1bf51bdf76c68a5f654b8524335b0ad29", size = 503730, upload-time = "2024-12-09T18:09:59.494Z" },
- { url = "https://files.pythonhosted.org/packages/f9/e2/253fc1fa59103bb4e3aa0665d6ceb1818df1cd7bf3eb492c4dad229b1cd4/jiter-0.8.2-cp312-cp312-win32.whl", hash = "sha256:7efe4853ecd3d6110301665a5178b9856be7e2a9485f49d91aa4d737ad2ae49e", size = 203375, upload-time = "2024-12-09T18:10:00.814Z" },
- { url = "https://files.pythonhosted.org/packages/41/69/6d4bbe66b3b3b4507e47aa1dd5d075919ad242b4b1115b3f80eecd443687/jiter-0.8.2-cp312-cp312-win_amd64.whl", hash = "sha256:83c0efd80b29695058d0fd2fa8a556490dbce9804eac3e281f373bbc99045f6c", size = 204740, upload-time = "2024-12-09T18:10:02.146Z" },
- { url = "https://files.pythonhosted.org/packages/6c/b0/bfa1f6f2c956b948802ef5a021281978bf53b7a6ca54bb126fd88a5d014e/jiter-0.8.2-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:ca1f08b8e43dc3bd0594c992fb1fd2f7ce87f7bf0d44358198d6da8034afdf84", size = 301190, upload-time = "2024-12-09T18:10:03.463Z" },
- { url = "https://files.pythonhosted.org/packages/a4/8f/396ddb4e292b5ea57e45ade5dc48229556b9044bad29a3b4b2dddeaedd52/jiter-0.8.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:5672a86d55416ccd214c778efccf3266b84f87b89063b582167d803246354be4", size = 309334, upload-time = "2024-12-09T18:10:05.774Z" },
- { url = "https://files.pythonhosted.org/packages/7f/68/805978f2f446fa6362ba0cc2e4489b945695940656edd844e110a61c98f8/jiter-0.8.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:58dc9bc9767a1101f4e5e22db1b652161a225874d66f0e5cb8e2c7d1c438b587", size = 333918, upload-time = "2024-12-09T18:10:07.158Z" },
- { url = "https://files.pythonhosted.org/packages/b3/99/0f71f7be667c33403fa9706e5b50583ae5106d96fab997fa7e2f38ee8347/jiter-0.8.2-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:37b2998606d6dadbb5ccda959a33d6a5e853252d921fec1792fc902351bb4e2c", size = 356057, upload-time = "2024-12-09T18:10:09.341Z" },
- { url = "https://files.pythonhosted.org/packages/8d/50/a82796e421a22b699ee4d2ce527e5bcb29471a2351cbdc931819d941a167/jiter-0.8.2-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:4ab9a87f3784eb0e098f84a32670cfe4a79cb6512fd8f42ae3d0709f06405d18", size = 379790, upload-time = "2024-12-09T18:10:10.702Z" },
- { url = "https://files.pythonhosted.org/packages/3c/31/10fb012b00f6d83342ca9e2c9618869ab449f1aa78c8f1b2193a6b49647c/jiter-0.8.2-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:79aec8172b9e3c6d05fd4b219d5de1ac616bd8da934107325a6c0d0e866a21b6", size = 388285, upload-time = "2024-12-09T18:10:12.721Z" },
- { url = "https://files.pythonhosted.org/packages/c8/81/f15ebf7de57be488aa22944bf4274962aca8092e4f7817f92ffa50d3ee46/jiter-0.8.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:711e408732d4e9a0208008e5892c2966b485c783cd2d9a681f3eb147cf36c7ef", size = 344764, upload-time = "2024-12-09T18:10:14.075Z" },
- { url = "https://files.pythonhosted.org/packages/b3/e8/0cae550d72b48829ba653eb348cdc25f3f06f8a62363723702ec18e7be9c/jiter-0.8.2-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:653cf462db4e8c41995e33d865965e79641ef45369d8a11f54cd30888b7e6ff1", size = 376620, upload-time = "2024-12-09T18:10:15.487Z" },
- { url = "https://files.pythonhosted.org/packages/b8/50/e5478ff9d82534a944c03b63bc217c5f37019d4a34d288db0f079b13c10b/jiter-0.8.2-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:9c63eaef32b7bebac8ebebf4dabebdbc6769a09c127294db6babee38e9f405b9", size = 510402, upload-time = "2024-12-09T18:10:17.499Z" },
- { url = "https://files.pythonhosted.org/packages/8e/1e/3de48bbebbc8f7025bd454cedc8c62378c0e32dd483dece5f4a814a5cb55/jiter-0.8.2-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:eb21aaa9a200d0a80dacc7a81038d2e476ffe473ffdd9c91eb745d623561de05", size = 503018, upload-time = "2024-12-09T18:10:18.92Z" },
- { url = "https://files.pythonhosted.org/packages/d5/cd/d5a5501d72a11fe3e5fd65c78c884e5164eefe80077680533919be22d3a3/jiter-0.8.2-cp313-cp313-win32.whl", hash = "sha256:789361ed945d8d42850f919342a8665d2dc79e7e44ca1c97cc786966a21f627a", size = 203190, upload-time = "2024-12-09T18:10:20.801Z" },
- { url = "https://files.pythonhosted.org/packages/51/bf/e5ca301245ba951447e3ad677a02a64a8845b185de2603dabd83e1e4b9c6/jiter-0.8.2-cp313-cp313-win_amd64.whl", hash = "sha256:ab7f43235d71e03b941c1630f4b6e3055d46b6cb8728a17663eaac9d8e83a865", size = 203551, upload-time = "2024-12-09T18:10:22.822Z" },
- { url = "https://files.pythonhosted.org/packages/2f/3c/71a491952c37b87d127790dd7a0b1ebea0514c6b6ad30085b16bbe00aee6/jiter-0.8.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b426f72cd77da3fec300ed3bc990895e2dd6b49e3bfe6c438592a3ba660e41ca", size = 308347, upload-time = "2024-12-09T18:10:24.139Z" },
- { url = "https://files.pythonhosted.org/packages/a0/4c/c02408042e6a7605ec063daed138e07b982fdb98467deaaf1c90950cf2c6/jiter-0.8.2-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b2dd880785088ff2ad21ffee205e58a8c1ddabc63612444ae41e5e4b321b39c0", size = 342875, upload-time = "2024-12-09T18:10:25.553Z" },
- { url = "https://files.pythonhosted.org/packages/91/61/c80ef80ed8a0a21158e289ef70dac01e351d929a1c30cb0f49be60772547/jiter-0.8.2-cp313-cp313t-win_amd64.whl", hash = "sha256:3ac9f578c46f22405ff7f8b1f5848fb753cc4b8377fbec8470a7dc3997ca7566", size = 202374, upload-time = "2024-12-09T18:10:26.958Z" },
+version = "0.11.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a3/68/0357982493a7b20925aece061f7fb7a2678e3b232f8d73a6edb7e5304443/jiter-0.11.1.tar.gz", hash = "sha256:849dcfc76481c0ea0099391235b7ca97d7279e0fa4c86005457ac7c88e8b76dc", size = 168385, upload-time = "2025-10-17T11:31:15.186Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/12/10/d099def5716452c8d5ffa527405373a44ddaf8e3c9d4f6de1e1344cffd90/jiter-0.11.1-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:ed58841a491bbbf3f7c55a6b68fff568439ab73b2cce27ace0e169057b5851df", size = 310078, upload-time = "2025-10-17T11:28:36.186Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/56/b81d010b0031ffa96dfb590628562ac5f513ce56aa2ab451d29fb3fedeb9/jiter-0.11.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:499beb9b2d7e51d61095a8de39ebcab1d1778f2a74085f8305a969f6cee9f3e4", size = 317138, upload-time = "2025-10-17T11:28:38.294Z" },
+ { url = "https://files.pythonhosted.org/packages/89/12/31ea12af9d79671cc7bd893bf0ccaf3467624c0fc7146a0cbfe7b549bcfa/jiter-0.11.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b87b2821795e28cc990939b68ce7a038edea680a24910bd68a79d54ff3f03c02", size = 348964, upload-time = "2025-10-17T11:28:40.103Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/d2/95cb6dc5ff962410667a29708c7a6c0691cc3c4866a0bfa79d085b56ebd6/jiter-0.11.1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:83f6fa494d8bba14ab100417c80e70d32d737e805cb85be2052d771c76fcd1f8", size = 363289, upload-time = "2025-10-17T11:28:41.49Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/3e/37006ad5843a0bc3a3ec3a6c44710d7a154113befaf5f26d2fe190668b63/jiter-0.11.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5fbc6aea1daa2ec6f5ed465f0c5e7b0607175062ceebbea5ca70dd5ddab58083", size = 487243, upload-time = "2025-10-17T11:28:43.209Z" },
+ { url = "https://files.pythonhosted.org/packages/80/5c/d38c8c801a322a0c0de47b9618c16fd766366f087ce37c4e55ae8e3c8b03/jiter-0.11.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:302288e2edc43174bb2db838e94688d724f9aad26c5fb9a74f7a5fb427452a6a", size = 376139, upload-time = "2025-10-17T11:28:44.821Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/cd/442ad2389a5570b0ee673f93e14bbe8cdecd3e08a9ba7756081d84065e4c/jiter-0.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:85db563fe3b367bb568af5d29dea4d4066d923b8e01f3417d25ebecd958de815", size = 359279, upload-time = "2025-10-17T11:28:46.152Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/35/8f5810d0e7d00bc395889085dbc1ccc36d454b56f28b2a5359dfd1bab48d/jiter-0.11.1-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f1c1ba2b6b22f775444ef53bc2d5778396d3520abc7b2e1da8eb0c27cb3ffb10", size = 384911, upload-time = "2025-10-17T11:28:48.03Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/bd/8c069ceb0bafcf6b4aa5de0c27f02faf50468df39564a02e1a12389ad6c2/jiter-0.11.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:523be464b14f8fd0cc78da6964b87b5515a056427a2579f9085ce30197a1b54a", size = 517879, upload-time = "2025-10-17T11:28:49.902Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/3c/9163efcf762f79f47433078b4f0a1bddc56096082c02c6cae2f47f07f56f/jiter-0.11.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:25b99b3f04cd2a38fefb22e822e35eb203a2cd37d680dbbc0c0ba966918af336", size = 508739, upload-time = "2025-10-17T11:28:51.785Z" },
+ { url = "https://files.pythonhosted.org/packages/44/07/50690f257935845d3114b95b5dd03749eeaab5e395cbb522f9e957da4551/jiter-0.11.1-cp310-cp310-win32.whl", hash = "sha256:47a79e90545a596bb9104109777894033347b11180d4751a216afef14072dbe7", size = 203948, upload-time = "2025-10-17T11:28:54.368Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/3a/5964a944bf2e98ffd566153fdc2a6a368fcb11b58cc46832ca8c75808dba/jiter-0.11.1-cp310-cp310-win_amd64.whl", hash = "sha256:cace75621ae9bd66878bf69fbd4dfc1a28ef8661e0c2d0eb72d3d6f1268eddf5", size = 207522, upload-time = "2025-10-17T11:28:56.79Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/34/c9e6cfe876f9a24f43ed53fe29f052ce02bd8d5f5a387dbf46ad3764bef0/jiter-0.11.1-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:9b0088ff3c374ce8ce0168523ec8e97122ebb788f950cf7bb8e39c7dc6a876a2", size = 310160, upload-time = "2025-10-17T11:28:59.174Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/9f/b06ec8181d7165858faf2ac5287c54fe52b2287760b7fe1ba9c06890255f/jiter-0.11.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:74433962dd3c3090655e02e461267095d6c84f0741c7827de11022ef8d7ff661", size = 316573, upload-time = "2025-10-17T11:29:00.905Z" },
+ { url = "https://files.pythonhosted.org/packages/66/49/3179d93090f2ed0c6b091a9c210f266d2d020d82c96f753260af536371d0/jiter-0.11.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6d98030e345e6546df2cc2c08309c502466c66c4747b043f1a0d415fada862b8", size = 348998, upload-time = "2025-10-17T11:29:02.321Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/9d/63db2c8eabda7a9cad65a2e808ca34aaa8689d98d498f5a2357d7a2e2cec/jiter-0.11.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1d6db0b2e788db46bec2cf729a88b6dd36959af2abd9fa2312dfba5acdd96dcb", size = 363413, upload-time = "2025-10-17T11:29:03.787Z" },
+ { url = "https://files.pythonhosted.org/packages/25/ff/3e6b3170c5053053c7baddb8d44e2bf11ff44cd71024a280a8438ae6ba32/jiter-0.11.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:55678fbbda261eafe7289165dd2ddd0e922df5f9a1ae46d7c79a5a15242bd7d1", size = 487144, upload-time = "2025-10-17T11:29:05.37Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/50/b63fcadf699893269b997f4c2e88400bc68f085c6db698c6e5e69d63b2c1/jiter-0.11.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6a6b74fae8e40497653b52ce6ca0f1b13457af769af6fb9c1113efc8b5b4d9be", size = 376215, upload-time = "2025-10-17T11:29:07.123Z" },
+ { url = "https://files.pythonhosted.org/packages/39/8c/57a8a89401134167e87e73471b9cca321cf651c1fd78c45f3a0f16932213/jiter-0.11.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0a55a453f8b035eb4f7852a79a065d616b7971a17f5e37a9296b4b38d3b619e4", size = 359163, upload-time = "2025-10-17T11:29:09.047Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/96/30b0cdbffbb6f753e25339d3dbbe26890c9ef119928314578201c758aace/jiter-0.11.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2638148099022e6bdb3f42904289cd2e403609356fb06eb36ddec2d50958bc29", size = 385344, upload-time = "2025-10-17T11:29:10.69Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/d5/31dae27c1cc9410ad52bb514f11bfa4f286f7d6ef9d287b98b8831e156ec/jiter-0.11.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:252490567a5d990986f83b95a5f1ca1bf205ebd27b3e9e93bb7c2592380e29b9", size = 517972, upload-time = "2025-10-17T11:29:12.174Z" },
+ { url = "https://files.pythonhosted.org/packages/61/1e/5905a7a3aceab80de13ab226fd690471a5e1ee7e554dc1015e55f1a6b896/jiter-0.11.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:d431d52b0ca2436eea6195f0f48528202100c7deda354cb7aac0a302167594d5", size = 508408, upload-time = "2025-10-17T11:29:13.597Z" },
+ { url = "https://files.pythonhosted.org/packages/91/12/1c49b97aa49077e136e8591cef7162f0d3e2860ae457a2d35868fd1521ef/jiter-0.11.1-cp311-cp311-win32.whl", hash = "sha256:db6f41e40f8bae20c86cb574b48c4fd9f28ee1c71cb044e9ec12e78ab757ba3a", size = 203937, upload-time = "2025-10-17T11:29:14.894Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/9d/2255f7c17134ee9892c7e013c32d5bcf4bce64eb115402c9fe5e727a67eb/jiter-0.11.1-cp311-cp311-win_amd64.whl", hash = "sha256:0cc407b8e6cdff01b06bb80f61225c8b090c3df108ebade5e0c3c10993735b19", size = 207589, upload-time = "2025-10-17T11:29:16.166Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/28/6307fc8f95afef84cae6caf5429fee58ef16a582c2ff4db317ceb3e352fa/jiter-0.11.1-cp311-cp311-win_arm64.whl", hash = "sha256:fe04ea475392a91896d1936367854d346724a1045a247e5d1c196410473b8869", size = 188391, upload-time = "2025-10-17T11:29:17.488Z" },
+ { url = "https://files.pythonhosted.org/packages/15/8b/318e8af2c904a9d29af91f78c1e18f0592e189bbdb8a462902d31fe20682/jiter-0.11.1-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:c92148eec91052538ce6823dfca9525f5cfc8b622d7f07e9891a280f61b8c96c", size = 305655, upload-time = "2025-10-17T11:29:18.859Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/29/6c7de6b5d6e511d9e736312c0c9bfcee8f9b6bef68182a08b1d78767e627/jiter-0.11.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ecd4da91b5415f183a6be8f7158d127bdd9e6a3174138293c0d48d6ea2f2009d", size = 315645, upload-time = "2025-10-17T11:29:20.889Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/5f/ef9e5675511ee0eb7f98dd8c90509e1f7743dbb7c350071acae87b0145f3/jiter-0.11.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d7e3ac25c00b9275684d47aa42febaa90a9958e19fd1726c4ecf755fbe5e553b", size = 348003, upload-time = "2025-10-17T11:29:22.712Z" },
+ { url = "https://files.pythonhosted.org/packages/56/1b/abe8c4021010b0a320d3c62682769b700fb66f92c6db02d1a1381b3db025/jiter-0.11.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:57d7305c0a841858f866cd459cd9303f73883fb5e097257f3d4a3920722c69d4", size = 365122, upload-time = "2025-10-17T11:29:24.408Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/2d/4a18013939a4f24432f805fbd5a19893e64650b933edb057cd405275a538/jiter-0.11.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e86fa10e117dce22c547f31dd6d2a9a222707d54853d8de4e9a2279d2c97f239", size = 488360, upload-time = "2025-10-17T11:29:25.724Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/77/38124f5d02ac4131f0dfbcfd1a19a0fac305fa2c005bc4f9f0736914a1a4/jiter-0.11.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ae5ef1d48aec7e01ee8420155d901bb1d192998fa811a65ebb82c043ee186711", size = 376884, upload-time = "2025-10-17T11:29:27.056Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/43/59fdc2f6267959b71dd23ce0bd8d4aeaf55566aa435a5d00f53d53c7eb24/jiter-0.11.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eb68e7bf65c990531ad8715e57d50195daf7c8e6f1509e617b4e692af1108939", size = 358827, upload-time = "2025-10-17T11:29:28.698Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/d0/b3cc20ff5340775ea3bbaa0d665518eddecd4266ba7244c9cb480c0c82ec/jiter-0.11.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:43b30c8154ded5845fa454ef954ee67bfccce629b2dea7d01f795b42bc2bda54", size = 385171, upload-time = "2025-10-17T11:29:30.078Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/bc/94dd1f3a61f4dc236f787a097360ec061ceeebebf4ea120b924d91391b10/jiter-0.11.1-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:586cafbd9dd1f3ce6a22b4a085eaa6be578e47ba9b18e198d4333e598a91db2d", size = 518359, upload-time = "2025-10-17T11:29:31.464Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/8c/12ee132bd67e25c75f542c227f5762491b9a316b0dad8e929c95076f773c/jiter-0.11.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:677cc2517d437a83bb30019fd4cf7cad74b465914c56ecac3440d597ac135250", size = 509205, upload-time = "2025-10-17T11:29:32.895Z" },
+ { url = "https://files.pythonhosted.org/packages/39/d5/9de848928ce341d463c7e7273fce90ea6d0ea4343cd761f451860fa16b59/jiter-0.11.1-cp312-cp312-win32.whl", hash = "sha256:fa992af648fcee2b850a3286a35f62bbbaeddbb6dbda19a00d8fbc846a947b6e", size = 205448, upload-time = "2025-10-17T11:29:34.217Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/b0/8002d78637e05009f5e3fb5288f9d57d65715c33b5d6aa20fd57670feef5/jiter-0.11.1-cp312-cp312-win_amd64.whl", hash = "sha256:88b5cae9fa51efeb3d4bd4e52bfd4c85ccc9cac44282e2a9640893a042ba4d87", size = 204285, upload-time = "2025-10-17T11:29:35.446Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/a2/bb24d5587e4dff17ff796716542f663deee337358006a80c8af43ddc11e5/jiter-0.11.1-cp312-cp312-win_arm64.whl", hash = "sha256:9a6cae1ab335551917f882f2c3c1efe7617b71b4c02381e4382a8fc80a02588c", size = 188712, upload-time = "2025-10-17T11:29:37.027Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/4b/e4dd3c76424fad02a601d570f4f2a8438daea47ba081201a721a903d3f4c/jiter-0.11.1-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:71b6a920a5550f057d49d0e8bcc60945a8da998019e83f01adf110e226267663", size = 305272, upload-time = "2025-10-17T11:29:39.249Z" },
+ { url = "https://files.pythonhosted.org/packages/67/83/2cd3ad5364191130f4de80eacc907f693723beaab11a46c7d155b07a092c/jiter-0.11.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0b3de72e925388453a5171be83379549300db01284f04d2a6f244d1d8de36f94", size = 314038, upload-time = "2025-10-17T11:29:40.563Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/3c/8e67d9ba524e97d2f04c8f406f8769a23205026b13b0938d16646d6e2d3e/jiter-0.11.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cc19dd65a2bd3d9c044c5b4ebf657ca1e6003a97c0fc10f555aa4f7fb9821c00", size = 345977, upload-time = "2025-10-17T11:29:42.009Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/a5/489ce64d992c29bccbffabb13961bbb0435e890d7f2d266d1f3df5e917d2/jiter-0.11.1-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d58faaa936743cd1464540562f60b7ce4fd927e695e8bc31b3da5b914baa9abd", size = 364503, upload-time = "2025-10-17T11:29:43.459Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/c0/e321dd83ee231d05c8fe4b1a12caf1f0e8c7a949bf4724d58397104f10f2/jiter-0.11.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:902640c3103625317291cb73773413b4d71847cdf9383ba65528745ff89f1d14", size = 487092, upload-time = "2025-10-17T11:29:44.835Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/5e/8f24ec49c8d37bd37f34ec0112e0b1a3b4b5a7b456c8efff1df5e189ad43/jiter-0.11.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:30405f726e4c2ed487b176c09f8b877a957f535d60c1bf194abb8dadedb5836f", size = 376328, upload-time = "2025-10-17T11:29:46.175Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/70/ded107620e809327cf7050727e17ccfa79d6385a771b7fe38fb31318ef00/jiter-0.11.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3217f61728b0baadd2551844870f65219ac4a1285d5e1a4abddff3d51fdabe96", size = 356632, upload-time = "2025-10-17T11:29:47.454Z" },
+ { url = "https://files.pythonhosted.org/packages/19/53/c26f7251613f6a9079275ee43c89b8a973a95ff27532c421abc2a87afb04/jiter-0.11.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b1364cc90c03a8196f35f396f84029f12abe925415049204446db86598c8b72c", size = 384358, upload-time = "2025-10-17T11:29:49.377Z" },
+ { url = "https://files.pythonhosted.org/packages/84/16/e0f2cc61e9c4d0b62f6c1bd9b9781d878a427656f88293e2a5335fa8ff07/jiter-0.11.1-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:53a54bf8e873820ab186b2dca9f6c3303f00d65ae5e7b7d6bda1b95aa472d646", size = 517279, upload-time = "2025-10-17T11:29:50.968Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5c/4cd095eaee68961bca3081acbe7c89e12ae24a5dae5fd5d2a13e01ed2542/jiter-0.11.1-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:7e29aca023627b0e0c2392d4248f6414d566ff3974fa08ff2ac8dbb96dfee92a", size = 508276, upload-time = "2025-10-17T11:29:52.619Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/25/f459240e69b0e09a7706d96ce203ad615ca36b0fe832308d2b7123abf2d0/jiter-0.11.1-cp313-cp313-win32.whl", hash = "sha256:f153e31d8bca11363751e875c0a70b3d25160ecbaee7b51e457f14498fb39d8b", size = 205593, upload-time = "2025-10-17T11:29:53.938Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/16/461bafe22bae79bab74e217a09c907481a46d520c36b7b9fe71ee8c9e983/jiter-0.11.1-cp313-cp313-win_amd64.whl", hash = "sha256:f773f84080b667c69c4ea0403fc67bb08b07e2b7ce1ef335dea5868451e60fed", size = 203518, upload-time = "2025-10-17T11:29:55.216Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/72/c45de6e320edb4fa165b7b1a414193b3cae302dd82da2169d315dcc78b44/jiter-0.11.1-cp313-cp313-win_arm64.whl", hash = "sha256:635ecd45c04e4c340d2187bcb1cea204c7cc9d32c1364d251564bf42e0e39c2d", size = 188062, upload-time = "2025-10-17T11:29:56.631Z" },
+ { url = "https://files.pythonhosted.org/packages/65/9b/4a57922437ca8753ef823f434c2dec5028b237d84fa320f06a3ba1aec6e8/jiter-0.11.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d892b184da4d94d94ddb4031296931c74ec8b325513a541ebfd6dfb9ae89904b", size = 313814, upload-time = "2025-10-17T11:29:58.509Z" },
+ { url = "https://files.pythonhosted.org/packages/76/50/62a0683dadca25490a4bedc6a88d59de9af2a3406dd5a576009a73a1d392/jiter-0.11.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aa22c223a3041dacb2fcd37c70dfd648b44662b4a48e242592f95bda5ab09d58", size = 344987, upload-time = "2025-10-17T11:30:00.208Z" },
+ { url = "https://files.pythonhosted.org/packages/da/00/2355dbfcbf6cdeaddfdca18287f0f38ae49446bb6378e4a5971e9356fc8a/jiter-0.11.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:330e8e6a11ad4980cd66a0f4a3e0e2e0f646c911ce047014f984841924729789", size = 356399, upload-time = "2025-10-17T11:30:02.084Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/07/c2bd748d578fa933d894a55bff33f983bc27f75fc4e491b354bef7b78012/jiter-0.11.1-cp313-cp313t-win_amd64.whl", hash = "sha256:09e2e386ebf298547ca3a3704b729471f7ec666c2906c5c26c1a915ea24741ec", size = 203289, upload-time = "2025-10-17T11:30:03.656Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/ee/ace64a853a1acbd318eb0ca167bad1cf5ee037207504b83a868a5849747b/jiter-0.11.1-cp313-cp313t-win_arm64.whl", hash = "sha256:fe4a431c291157e11cee7c34627990ea75e8d153894365a3bc84b7a959d23ca8", size = 188284, upload-time = "2025-10-17T11:30:05.046Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/51/bd41562dd284e2a18b6dc0a99d195fd4a3560d52ab192c42e56fe0316643/jiter-0.11.1-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:e642b5270e61dd02265866398707f90e365b5db2eb65a4f30c789d826682e1f6", size = 306871, upload-time = "2025-10-17T11:31:03.616Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/cb/64e7f21dd357e8cd6b3c919c26fac7fc198385bbd1d85bb3b5355600d787/jiter-0.11.1-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:464ba6d000585e4e2fd1e891f31f1231f497273414f5019e27c00a4b8f7a24ad", size = 301454, upload-time = "2025-10-17T11:31:05.338Z" },
+ { url = "https://files.pythonhosted.org/packages/55/b0/54bdc00da4ef39801b1419a01035bd8857983de984fd3776b0be6b94add7/jiter-0.11.1-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:055568693ab35e0bf3a171b03bb40b2dcb10352359e0ab9b5ed0da2bf1eb6f6f", size = 336801, upload-time = "2025-10-17T11:31:06.893Z" },
+ { url = "https://files.pythonhosted.org/packages/de/8f/87176ed071d42e9db415ed8be787ef4ef31a4fa27f52e6a4fbf34387bd28/jiter-0.11.1-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e0c69ea798d08a915ba4478113efa9e694971e410056392f4526d796f136d3fa", size = 343452, upload-time = "2025-10-17T11:31:08.259Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/bc/950dd7f170c6394b6fdd73f989d9e729bd98907bcc4430ef080a72d06b77/jiter-0.11.1-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:0d4d6993edc83cf75e8c6828a8d6ce40a09ee87e38c7bfba6924f39e1337e21d", size = 302626, upload-time = "2025-10-17T11:31:09.645Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/65/43d7971ca82ee100b7b9b520573eeef7eabc0a45d490168ebb9a9b5bb8b2/jiter-0.11.1-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:f78d151c83a87a6cf5461d5ee55bc730dd9ae227377ac6f115b922989b95f838", size = 297034, upload-time = "2025-10-17T11:31:10.975Z" },
+ { url = "https://files.pythonhosted.org/packages/19/4c/000e1e0c0c67e96557a279f8969487ea2732d6c7311698819f977abae837/jiter-0.11.1-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c9022974781155cd5521d5cb10997a03ee5e31e8454c9d999dcdccd253f2353f", size = 337328, upload-time = "2025-10-17T11:31:12.399Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/71/71408b02c6133153336d29fa3ba53000f1e1a3f78bb2fc2d1a1865d2e743/jiter-0.11.1-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:18c77aaa9117510d5bdc6a946baf21b1f0cfa58ef04d31c8d016f206f2118960", size = 343697, upload-time = "2025-10-17T11:31:13.773Z" },
]
[[package]]
@@ -1470,65 +1521,65 @@ wheels = [
[[package]]
name = "numpy"
-version = "2.3.4"
+version = "2.3.5"
source = { registry = "https://pypi.org/simple" }
resolution-markers = [
"python_full_version >= '3.12'",
"python_full_version == '3.11.*'",
]
-sdist = { url = "https://files.pythonhosted.org/packages/b5/f4/098d2270d52b41f1bd7db9fc288aaa0400cb48c2a3e2af6fa365d9720947/numpy-2.3.4.tar.gz", hash = "sha256:a7d018bfedb375a8d979ac758b120ba846a7fe764911a64465fd87b8729f4a6a", size = 20582187, upload-time = "2025-10-15T16:18:11.77Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/60/e7/0e07379944aa8afb49a556a2b54587b828eb41dc9adc56fb7615b678ca53/numpy-2.3.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:e78aecd2800b32e8347ce49316d3eaf04aed849cd5b38e0af39f829a4e59f5eb", size = 21259519, upload-time = "2025-10-15T16:15:19.012Z" },
- { url = "https://files.pythonhosted.org/packages/d0/cb/5a69293561e8819b09e34ed9e873b9a82b5f2ade23dce4c51dc507f6cfe1/numpy-2.3.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7fd09cc5d65bda1e79432859c40978010622112e9194e581e3415a3eccc7f43f", size = 14452796, upload-time = "2025-10-15T16:15:23.094Z" },
- { url = "https://files.pythonhosted.org/packages/e4/04/ff11611200acd602a1e5129e36cfd25bf01ad8e5cf927baf2e90236eb02e/numpy-2.3.4-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:1b219560ae2c1de48ead517d085bc2d05b9433f8e49d0955c82e8cd37bd7bf36", size = 5381639, upload-time = "2025-10-15T16:15:25.572Z" },
- { url = "https://files.pythonhosted.org/packages/ea/77/e95c757a6fe7a48d28a009267408e8aa382630cc1ad1db7451b3bc21dbb4/numpy-2.3.4-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:bafa7d87d4c99752d07815ed7a2c0964f8ab311eb8168f41b910bd01d15b6032", size = 6914296, upload-time = "2025-10-15T16:15:27.079Z" },
- { url = "https://files.pythonhosted.org/packages/a3/d2/137c7b6841c942124eae921279e5c41b1c34bab0e6fc60c7348e69afd165/numpy-2.3.4-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:36dc13af226aeab72b7abad501d370d606326a0029b9f435eacb3b8c94b8a8b7", size = 14591904, upload-time = "2025-10-15T16:15:29.044Z" },
- { url = "https://files.pythonhosted.org/packages/bb/32/67e3b0f07b0aba57a078c4ab777a9e8e6bc62f24fb53a2337f75f9691699/numpy-2.3.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a7b2f9a18b5ff9824a6af80de4f37f4ec3c2aab05ef08f51c77a093f5b89adda", size = 16939602, upload-time = "2025-10-15T16:15:31.106Z" },
- { url = "https://files.pythonhosted.org/packages/95/22/9639c30e32c93c4cee3ccdb4b09c2d0fbff4dcd06d36b357da06146530fb/numpy-2.3.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9984bd645a8db6ca15d850ff996856d8762c51a2239225288f08f9050ca240a0", size = 16372661, upload-time = "2025-10-15T16:15:33.546Z" },
- { url = "https://files.pythonhosted.org/packages/12/e9/a685079529be2b0156ae0c11b13d6be647743095bb51d46589e95be88086/numpy-2.3.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:64c5825affc76942973a70acf438a8ab618dbd692b84cd5ec40a0a0509edc09a", size = 18884682, upload-time = "2025-10-15T16:15:36.105Z" },
- { url = "https://files.pythonhosted.org/packages/cf/85/f6f00d019b0cc741e64b4e00ce865a57b6bed945d1bbeb1ccadbc647959b/numpy-2.3.4-cp311-cp311-win32.whl", hash = "sha256:ed759bf7a70342f7817d88376eb7142fab9fef8320d6019ef87fae05a99874e1", size = 6570076, upload-time = "2025-10-15T16:15:38.225Z" },
- { url = "https://files.pythonhosted.org/packages/7d/10/f8850982021cb90e2ec31990291f9e830ce7d94eef432b15066e7cbe0bec/numpy-2.3.4-cp311-cp311-win_amd64.whl", hash = "sha256:faba246fb30ea2a526c2e9645f61612341de1a83fb1e0c5edf4ddda5a9c10996", size = 13089358, upload-time = "2025-10-15T16:15:40.404Z" },
- { url = "https://files.pythonhosted.org/packages/d1/ad/afdd8351385edf0b3445f9e24210a9c3971ef4de8fd85155462fc4321d79/numpy-2.3.4-cp311-cp311-win_arm64.whl", hash = "sha256:4c01835e718bcebe80394fd0ac66c07cbb90147ebbdad3dcecd3f25de2ae7e2c", size = 10462292, upload-time = "2025-10-15T16:15:42.896Z" },
- { url = "https://files.pythonhosted.org/packages/96/7a/02420400b736f84317e759291b8edaeee9dc921f72b045475a9cbdb26b17/numpy-2.3.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef1b5a3e808bc40827b5fa2c8196151a4c5abe110e1726949d7abddfe5c7ae11", size = 20957727, upload-time = "2025-10-15T16:15:44.9Z" },
- { url = "https://files.pythonhosted.org/packages/18/90/a014805d627aa5750f6f0e878172afb6454552da929144b3c07fcae1bb13/numpy-2.3.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c2f91f496a87235c6aaf6d3f3d89b17dba64996abadccb289f48456cff931ca9", size = 14187262, upload-time = "2025-10-15T16:15:47.761Z" },
- { url = "https://files.pythonhosted.org/packages/c7/e4/0a94b09abe89e500dc748e7515f21a13e30c5c3fe3396e6d4ac108c25fca/numpy-2.3.4-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:f77e5b3d3da652b474cc80a14084927a5e86a5eccf54ca8ca5cbd697bf7f2667", size = 5115992, upload-time = "2025-10-15T16:15:50.144Z" },
- { url = "https://files.pythonhosted.org/packages/88/dd/db77c75b055c6157cbd4f9c92c4458daef0dd9cbe6d8d2fe7f803cb64c37/numpy-2.3.4-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:8ab1c5f5ee40d6e01cbe96de5863e39b215a4d24e7d007cad56c7184fdf4aeef", size = 6648672, upload-time = "2025-10-15T16:15:52.442Z" },
- { url = "https://files.pythonhosted.org/packages/e1/e6/e31b0d713719610e406c0ea3ae0d90760465b086da8783e2fd835ad59027/numpy-2.3.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:77b84453f3adcb994ddbd0d1c5d11db2d6bda1a2b7fd5ac5bd4649d6f5dc682e", size = 14284156, upload-time = "2025-10-15T16:15:54.351Z" },
- { url = "https://files.pythonhosted.org/packages/f9/58/30a85127bfee6f108282107caf8e06a1f0cc997cb6b52cdee699276fcce4/numpy-2.3.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4121c5beb58a7f9e6dfdee612cb24f4df5cd4db6e8261d7f4d7450a997a65d6a", size = 16641271, upload-time = "2025-10-15T16:15:56.67Z" },
- { url = "https://files.pythonhosted.org/packages/06/f2/2e06a0f2adf23e3ae29283ad96959267938d0efd20a2e25353b70065bfec/numpy-2.3.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:65611ecbb00ac9846efe04db15cbe6186f562f6bb7e5e05f077e53a599225d16", size = 16059531, upload-time = "2025-10-15T16:15:59.412Z" },
- { url = "https://files.pythonhosted.org/packages/b0/e7/b106253c7c0d5dc352b9c8fab91afd76a93950998167fa3e5afe4ef3a18f/numpy-2.3.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:dabc42f9c6577bcc13001b8810d300fe814b4cfbe8a92c873f269484594f9786", size = 18578983, upload-time = "2025-10-15T16:16:01.804Z" },
- { url = "https://files.pythonhosted.org/packages/73/e3/04ecc41e71462276ee867ccbef26a4448638eadecf1bc56772c9ed6d0255/numpy-2.3.4-cp312-cp312-win32.whl", hash = "sha256:a49d797192a8d950ca59ee2d0337a4d804f713bb5c3c50e8db26d49666e351dc", size = 6291380, upload-time = "2025-10-15T16:16:03.938Z" },
- { url = "https://files.pythonhosted.org/packages/3d/a8/566578b10d8d0e9955b1b6cd5db4e9d4592dd0026a941ff7994cedda030a/numpy-2.3.4-cp312-cp312-win_amd64.whl", hash = "sha256:985f1e46358f06c2a09921e8921e2c98168ed4ae12ccd6e5e87a4f1857923f32", size = 12787999, upload-time = "2025-10-15T16:16:05.801Z" },
- { url = "https://files.pythonhosted.org/packages/58/22/9c903a957d0a8071b607f5b1bff0761d6e608b9a965945411f867d515db1/numpy-2.3.4-cp312-cp312-win_arm64.whl", hash = "sha256:4635239814149e06e2cb9db3dd584b2fa64316c96f10656983b8026a82e6e4db", size = 10197412, upload-time = "2025-10-15T16:16:07.854Z" },
- { url = "https://files.pythonhosted.org/packages/57/7e/b72610cc91edf138bc588df5150957a4937221ca6058b825b4725c27be62/numpy-2.3.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c090d4860032b857d94144d1a9976b8e36709e40386db289aaf6672de2a81966", size = 20950335, upload-time = "2025-10-15T16:16:10.304Z" },
- { url = "https://files.pythonhosted.org/packages/3e/46/bdd3370dcea2f95ef14af79dbf81e6927102ddf1cc54adc0024d61252fd9/numpy-2.3.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a13fc473b6db0be619e45f11f9e81260f7302f8d180c49a22b6e6120022596b3", size = 14179878, upload-time = "2025-10-15T16:16:12.595Z" },
- { url = "https://files.pythonhosted.org/packages/ac/01/5a67cb785bda60f45415d09c2bc245433f1c68dd82eef9c9002c508b5a65/numpy-2.3.4-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:3634093d0b428e6c32c3a69b78e554f0cd20ee420dcad5a9f3b2a63762ce4197", size = 5108673, upload-time = "2025-10-15T16:16:14.877Z" },
- { url = "https://files.pythonhosted.org/packages/c2/cd/8428e23a9fcebd33988f4cb61208fda832800ca03781f471f3727a820704/numpy-2.3.4-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:043885b4f7e6e232d7df4f51ffdef8c36320ee9d5f227b380ea636722c7ed12e", size = 6641438, upload-time = "2025-10-15T16:16:16.805Z" },
- { url = "https://files.pythonhosted.org/packages/3e/d1/913fe563820f3c6b079f992458f7331278dcd7ba8427e8e745af37ddb44f/numpy-2.3.4-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4ee6a571d1e4f0ea6d5f22d6e5fbd6ed1dc2b18542848e1e7301bd190500c9d7", size = 14281290, upload-time = "2025-10-15T16:16:18.764Z" },
- { url = "https://files.pythonhosted.org/packages/9e/7e/7d306ff7cb143e6d975cfa7eb98a93e73495c4deabb7d1b5ecf09ea0fd69/numpy-2.3.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc8a63918b04b8571789688b2780ab2b4a33ab44bfe8ccea36d3eba51228c953", size = 16636543, upload-time = "2025-10-15T16:16:21.072Z" },
- { url = "https://files.pythonhosted.org/packages/47/6a/8cfc486237e56ccfb0db234945552a557ca266f022d281a2f577b98e955c/numpy-2.3.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:40cc556d5abbc54aabe2b1ae287042d7bdb80c08edede19f0c0afb36ae586f37", size = 16056117, upload-time = "2025-10-15T16:16:23.369Z" },
- { url = "https://files.pythonhosted.org/packages/b1/0e/42cb5e69ea901e06ce24bfcc4b5664a56f950a70efdcf221f30d9615f3f3/numpy-2.3.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ecb63014bb7f4ce653f8be7f1df8cbc6093a5a2811211770f6606cc92b5a78fd", size = 18577788, upload-time = "2025-10-15T16:16:27.496Z" },
- { url = "https://files.pythonhosted.org/packages/86/92/41c3d5157d3177559ef0a35da50f0cda7fa071f4ba2306dd36818591a5bc/numpy-2.3.4-cp313-cp313-win32.whl", hash = "sha256:e8370eb6925bb8c1c4264fec52b0384b44f675f191df91cbe0140ec9f0955646", size = 6282620, upload-time = "2025-10-15T16:16:29.811Z" },
- { url = "https://files.pythonhosted.org/packages/09/97/fd421e8bc50766665ad35536c2bb4ef916533ba1fdd053a62d96cc7c8b95/numpy-2.3.4-cp313-cp313-win_amd64.whl", hash = "sha256:56209416e81a7893036eea03abcb91c130643eb14233b2515c90dcac963fe99d", size = 12784672, upload-time = "2025-10-15T16:16:31.589Z" },
- { url = "https://files.pythonhosted.org/packages/ad/df/5474fb2f74970ca8eb978093969b125a84cc3d30e47f82191f981f13a8a0/numpy-2.3.4-cp313-cp313-win_arm64.whl", hash = "sha256:a700a4031bc0fd6936e78a752eefb79092cecad2599ea9c8039c548bc097f9bc", size = 10196702, upload-time = "2025-10-15T16:16:33.902Z" },
- { url = "https://files.pythonhosted.org/packages/11/83/66ac031464ec1767ea3ed48ce40f615eb441072945e98693bec0bcd056cc/numpy-2.3.4-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:86966db35c4040fdca64f0816a1c1dd8dbd027d90fca5a57e00e1ca4cd41b879", size = 21049003, upload-time = "2025-10-15T16:16:36.101Z" },
- { url = "https://files.pythonhosted.org/packages/5f/99/5b14e0e686e61371659a1d5bebd04596b1d72227ce36eed121bb0aeab798/numpy-2.3.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:838f045478638b26c375ee96ea89464d38428c69170360b23a1a50fa4baa3562", size = 14302980, upload-time = "2025-10-15T16:16:39.124Z" },
- { url = "https://files.pythonhosted.org/packages/2c/44/e9486649cd087d9fc6920e3fc3ac2aba10838d10804b1e179fb7cbc4e634/numpy-2.3.4-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:d7315ed1dab0286adca467377c8381cd748f3dc92235f22a7dfc42745644a96a", size = 5231472, upload-time = "2025-10-15T16:16:41.168Z" },
- { url = "https://files.pythonhosted.org/packages/3e/51/902b24fa8887e5fe2063fd61b1895a476d0bbf46811ab0c7fdf4bd127345/numpy-2.3.4-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:84f01a4d18b2cc4ade1814a08e5f3c907b079c847051d720fad15ce37aa930b6", size = 6739342, upload-time = "2025-10-15T16:16:43.777Z" },
- { url = "https://files.pythonhosted.org/packages/34/f1/4de9586d05b1962acdcdb1dc4af6646361a643f8c864cef7c852bf509740/numpy-2.3.4-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:817e719a868f0dacde4abdfc5c1910b301877970195db9ab6a5e2c4bd5b121f7", size = 14354338, upload-time = "2025-10-15T16:16:46.081Z" },
- { url = "https://files.pythonhosted.org/packages/1f/06/1c16103b425de7969d5a76bdf5ada0804b476fed05d5f9e17b777f1cbefd/numpy-2.3.4-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:85e071da78d92a214212cacea81c6da557cab307f2c34b5f85b628e94803f9c0", size = 16702392, upload-time = "2025-10-15T16:16:48.455Z" },
- { url = "https://files.pythonhosted.org/packages/34/b2/65f4dc1b89b5322093572b6e55161bb42e3e0487067af73627f795cc9d47/numpy-2.3.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:2ec646892819370cf3558f518797f16597b4e4669894a2ba712caccc9da53f1f", size = 16134998, upload-time = "2025-10-15T16:16:51.114Z" },
- { url = "https://files.pythonhosted.org/packages/d4/11/94ec578896cdb973aaf56425d6c7f2aff4186a5c00fac15ff2ec46998b46/numpy-2.3.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:035796aaaddfe2f9664b9a9372f089cfc88bd795a67bd1bfe15e6e770934cf64", size = 18651574, upload-time = "2025-10-15T16:16:53.429Z" },
- { url = "https://files.pythonhosted.org/packages/62/b7/7efa763ab33dbccf56dade36938a77345ce8e8192d6b39e470ca25ff3cd0/numpy-2.3.4-cp313-cp313t-win32.whl", hash = "sha256:fea80f4f4cf83b54c3a051f2f727870ee51e22f0248d3114b8e755d160b38cfb", size = 6413135, upload-time = "2025-10-15T16:16:55.992Z" },
- { url = "https://files.pythonhosted.org/packages/43/70/aba4c38e8400abcc2f345e13d972fb36c26409b3e644366db7649015f291/numpy-2.3.4-cp313-cp313t-win_amd64.whl", hash = "sha256:15eea9f306b98e0be91eb344a94c0e630689ef302e10c2ce5f7e11905c704f9c", size = 12928582, upload-time = "2025-10-15T16:16:57.943Z" },
- { url = "https://files.pythonhosted.org/packages/67/63/871fad5f0073fc00fbbdd7232962ea1ac40eeaae2bba66c76214f7954236/numpy-2.3.4-cp313-cp313t-win_arm64.whl", hash = "sha256:b6c231c9c2fadbae4011ca5e7e83e12dc4a5072f1a1d85a0a7b3ed754d145a40", size = 10266691, upload-time = "2025-10-15T16:17:00.048Z" },
- { url = "https://files.pythonhosted.org/packages/b1/b6/64898f51a86ec88ca1257a59c1d7fd077b60082a119affefcdf1dd0df8ca/numpy-2.3.4-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:6e274603039f924c0fe5cb73438fa9246699c78a6df1bd3decef9ae592ae1c05", size = 21131552, upload-time = "2025-10-15T16:17:55.845Z" },
- { url = "https://files.pythonhosted.org/packages/ce/4c/f135dc6ebe2b6a3c77f4e4838fa63d350f85c99462012306ada1bd4bc460/numpy-2.3.4-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:d149aee5c72176d9ddbc6803aef9c0f6d2ceeea7626574fc68518da5476fa346", size = 14377796, upload-time = "2025-10-15T16:17:58.308Z" },
- { url = "https://files.pythonhosted.org/packages/d0/a4/f33f9c23fcc13dd8412fc8614559b5b797e0aba9d8e01dfa8bae10c84004/numpy-2.3.4-pp311-pypy311_pp73-macosx_14_0_arm64.whl", hash = "sha256:6d34ed9db9e6395bb6cd33286035f73a59b058169733a9db9f85e650b88df37e", size = 5306904, upload-time = "2025-10-15T16:18:00.596Z" },
- { url = "https://files.pythonhosted.org/packages/28/af/c44097f25f834360f9fb960fa082863e0bad14a42f36527b2a121abdec56/numpy-2.3.4-pp311-pypy311_pp73-macosx_14_0_x86_64.whl", hash = "sha256:fdebe771ca06bb8d6abce84e51dca9f7921fe6ad34a0c914541b063e9a68928b", size = 6819682, upload-time = "2025-10-15T16:18:02.32Z" },
- { url = "https://files.pythonhosted.org/packages/c5/8c/cd283b54c3c2b77e188f63e23039844f56b23bba1712318288c13fe86baf/numpy-2.3.4-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:957e92defe6c08211eb77902253b14fe5b480ebc5112bc741fd5e9cd0608f847", size = 14422300, upload-time = "2025-10-15T16:18:04.271Z" },
- { url = "https://files.pythonhosted.org/packages/b0/f0/8404db5098d92446b3e3695cf41c6f0ecb703d701cb0b7566ee2177f2eee/numpy-2.3.4-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:13b9062e4f5c7ee5c7e5be96f29ba71bc5a37fed3d1d77c37390ae00724d296d", size = 16760806, upload-time = "2025-10-15T16:18:06.668Z" },
- { url = "https://files.pythonhosted.org/packages/95/8e/2844c3959ce9a63acc7c8e50881133d86666f0420bcde695e115ced0920f/numpy-2.3.4-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:81b3a59793523e552c4a96109dde028aa4448ae06ccac5a76ff6532a85558a7f", size = 12973130, upload-time = "2025-10-15T16:18:09.397Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/76/65/21b3bc86aac7b8f2862db1e808f1ea22b028e30a225a34a5ede9bf8678f2/numpy-2.3.5.tar.gz", hash = "sha256:784db1dcdab56bf0517743e746dfb0f885fc68d948aba86eeec2cba234bdf1c0", size = 20584950, upload-time = "2025-11-16T22:52:42.067Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/43/77/84dd1d2e34d7e2792a236ba180b5e8fcc1e3e414e761ce0253f63d7f572e/numpy-2.3.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:de5672f4a7b200c15a4127042170a694d4df43c992948f5e1af57f0174beed10", size = 17034641, upload-time = "2025-11-16T22:49:19.336Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/ea/25e26fa5837106cde46ae7d0b667e20f69cbbc0efd64cba8221411ab26ae/numpy-2.3.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:acfd89508504a19ed06ef963ad544ec6664518c863436306153e13e94605c218", size = 12528324, upload-time = "2025-11-16T22:49:22.582Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/1a/e85f0eea4cf03d6a0228f5c0256b53f2df4bc794706e7df019fc622e47f1/numpy-2.3.5-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:ffe22d2b05504f786c867c8395de703937f934272eb67586817b46188b4ded6d", size = 5356872, upload-time = "2025-11-16T22:49:25.408Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/bb/35ef04afd567f4c989c2060cde39211e4ac5357155c1833bcd1166055c61/numpy-2.3.5-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:872a5cf366aec6bb1147336480fef14c9164b154aeb6542327de4970282cd2f5", size = 6893148, upload-time = "2025-11-16T22:49:27.549Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/2b/05bbeb06e2dff5eab512dfc678b1cc5ee94d8ac5956a0885c64b6b26252b/numpy-2.3.5-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3095bdb8dd297e5920b010e96134ed91d852d81d490e787beca7e35ae1d89cf7", size = 14557282, upload-time = "2025-11-16T22:49:30.964Z" },
+ { url = "https://files.pythonhosted.org/packages/65/fb/2b23769462b34398d9326081fad5655198fcf18966fcb1f1e49db44fbf31/numpy-2.3.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8cba086a43d54ca804ce711b2a940b16e452807acebe7852ff327f1ecd49b0d4", size = 16897903, upload-time = "2025-11-16T22:49:34.191Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/14/085f4cf05fc3f1e8aa95e85404e984ffca9b2275a5dc2b1aae18a67538b8/numpy-2.3.5-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6cf9b429b21df6b99f4dee7a1218b8b7ffbbe7df8764dc0bd60ce8a0708fed1e", size = 16341672, upload-time = "2025-11-16T22:49:37.2Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/3b/1f73994904142b2aa290449b3bb99772477b5fd94d787093e4f24f5af763/numpy-2.3.5-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:396084a36abdb603546b119d96528c2f6263921c50df3c8fd7cb28873a237748", size = 18838896, upload-time = "2025-11-16T22:49:39.727Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/b9/cf6649b2124f288309ffc353070792caf42ad69047dcc60da85ee85fea58/numpy-2.3.5-cp311-cp311-win32.whl", hash = "sha256:b0c7088a73aef3d687c4deef8452a3ac7c1be4e29ed8bf3b366c8111128ac60c", size = 6563608, upload-time = "2025-11-16T22:49:42.079Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/44/9fe81ae1dcc29c531843852e2874080dc441338574ccc4306b39e2ff6e59/numpy-2.3.5-cp311-cp311-win_amd64.whl", hash = "sha256:a414504bef8945eae5f2d7cb7be2d4af77c5d1cb5e20b296c2c25b61dff2900c", size = 13078442, upload-time = "2025-11-16T22:49:43.99Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/a7/f99a41553d2da82a20a2f22e93c94f928e4490bb447c9ff3c4ff230581d3/numpy-2.3.5-cp311-cp311-win_arm64.whl", hash = "sha256:0cd00b7b36e35398fa2d16af7b907b65304ef8bb4817a550e06e5012929830fa", size = 10458555, upload-time = "2025-11-16T22:49:47.092Z" },
+ { url = "https://files.pythonhosted.org/packages/44/37/e669fe6cbb2b96c62f6bbedc6a81c0f3b7362f6a59230b23caa673a85721/numpy-2.3.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:74ae7b798248fe62021dbf3c914245ad45d1a6b0cb4a29ecb4b31d0bfbc4cc3e", size = 16733873, upload-time = "2025-11-16T22:49:49.84Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/65/df0db6c097892c9380851ab9e44b52d4f7ba576b833996e0080181c0c439/numpy-2.3.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ee3888d9ff7c14604052b2ca5535a30216aa0a58e948cdd3eeb8d3415f638769", size = 12259838, upload-time = "2025-11-16T22:49:52.863Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/e1/1ee06e70eb2136797abe847d386e7c0e830b67ad1d43f364dd04fa50d338/numpy-2.3.5-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:612a95a17655e213502f60cfb9bf9408efdc9eb1d5f50535cc6eb365d11b42b5", size = 5088378, upload-time = "2025-11-16T22:49:55.055Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/9c/1ca85fb86708724275103b81ec4cf1ac1d08f465368acfc8da7ab545bdae/numpy-2.3.5-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:3101e5177d114a593d79dd79658650fe28b5a0d8abeb8ce6f437c0e6df5be1a4", size = 6628559, upload-time = "2025-11-16T22:49:57.371Z" },
+ { url = "https://files.pythonhosted.org/packages/74/78/fcd41e5a0ce4f3f7b003da85825acddae6d7ecb60cf25194741b036ca7d6/numpy-2.3.5-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8b973c57ff8e184109db042c842423ff4f60446239bd585a5131cc47f06f789d", size = 14250702, upload-time = "2025-11-16T22:49:59.632Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/23/2a1b231b8ff672b4c450dac27164a8b2ca7d9b7144f9c02d2396518352eb/numpy-2.3.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0d8163f43acde9a73c2a33605353a4f1bc4798745a8b1d73183b28e5b435ae28", size = 16606086, upload-time = "2025-11-16T22:50:02.127Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/c5/5ad26fbfbe2012e190cc7d5003e4d874b88bb18861d0829edc140a713021/numpy-2.3.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:51c1e14eb1e154ebd80e860722f9e6ed6ec89714ad2db2d3aa33c31d7c12179b", size = 16025985, upload-time = "2025-11-16T22:50:04.536Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/fa/dd48e225c46c819288148d9d060b047fd2a6fb1eb37eae25112ee4cb4453/numpy-2.3.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b46b4ec24f7293f23adcd2d146960559aaf8020213de8ad1909dba6c013bf89c", size = 18542976, upload-time = "2025-11-16T22:50:07.557Z" },
+ { url = "https://files.pythonhosted.org/packages/05/79/ccbd23a75862d95af03d28b5c6901a1b7da4803181513d52f3b86ed9446e/numpy-2.3.5-cp312-cp312-win32.whl", hash = "sha256:3997b5b3c9a771e157f9aae01dd579ee35ad7109be18db0e85dbdbe1de06e952", size = 6285274, upload-time = "2025-11-16T22:50:10.746Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/57/8aeaf160312f7f489dea47ab61e430b5cb051f59a98ae68b7133ce8fa06a/numpy-2.3.5-cp312-cp312-win_amd64.whl", hash = "sha256:86945f2ee6d10cdfd67bcb4069c1662dd711f7e2a4343db5cecec06b87cf31aa", size = 12782922, upload-time = "2025-11-16T22:50:12.811Z" },
+ { url = "https://files.pythonhosted.org/packages/78/a6/aae5cc2ca78c45e64b9ef22f089141d661516856cf7c8a54ba434576900d/numpy-2.3.5-cp312-cp312-win_arm64.whl", hash = "sha256:f28620fe26bee16243be2b7b874da327312240a7cdc38b769a697578d2100013", size = 10194667, upload-time = "2025-11-16T22:50:16.16Z" },
+ { url = "https://files.pythonhosted.org/packages/db/69/9cde09f36da4b5a505341180a3f2e6fadc352fd4d2b7096ce9778db83f1a/numpy-2.3.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d0f23b44f57077c1ede8c5f26b30f706498b4862d3ff0a7298b8411dd2f043ff", size = 16728251, upload-time = "2025-11-16T22:50:19.013Z" },
+ { url = "https://files.pythonhosted.org/packages/79/fb/f505c95ceddd7027347b067689db71ca80bd5ecc926f913f1a23e65cf09b/numpy-2.3.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:aa5bc7c5d59d831d9773d1170acac7893ce3a5e130540605770ade83280e7188", size = 12254652, upload-time = "2025-11-16T22:50:21.487Z" },
+ { url = "https://files.pythonhosted.org/packages/78/da/8c7738060ca9c31b30e9301ee0cf6c5ffdbf889d9593285a1cead337f9a5/numpy-2.3.5-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:ccc933afd4d20aad3c00bcef049cb40049f7f196e0397f1109dba6fed63267b0", size = 5083172, upload-time = "2025-11-16T22:50:24.562Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/b4/ee5bb2537fb9430fd2ef30a616c3672b991a4129bb1c7dcc42aa0abbe5d7/numpy-2.3.5-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:afaffc4393205524af9dfa400fa250143a6c3bc646c08c9f5e25a9f4b4d6a903", size = 6622990, upload-time = "2025-11-16T22:50:26.47Z" },
+ { url = "https://files.pythonhosted.org/packages/95/03/dc0723a013c7d7c19de5ef29e932c3081df1c14ba582b8b86b5de9db7f0f/numpy-2.3.5-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9c75442b2209b8470d6d5d8b1c25714270686f14c749028d2199c54e29f20b4d", size = 14248902, upload-time = "2025-11-16T22:50:28.861Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/10/ca162f45a102738958dcec8023062dad0cbc17d1ab99d68c4e4a6c45fb2b/numpy-2.3.5-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:11e06aa0af8c0f05104d56450d6093ee639e15f24ecf62d417329d06e522e017", size = 16597430, upload-time = "2025-11-16T22:50:31.56Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/51/c1e29be863588db58175175f057286900b4b3327a1351e706d5e0f8dd679/numpy-2.3.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ed89927b86296067b4f81f108a2271d8926467a8868e554eaf370fc27fa3ccaf", size = 16024551, upload-time = "2025-11-16T22:50:34.242Z" },
+ { url = "https://files.pythonhosted.org/packages/83/68/8236589d4dbb87253d28259d04d9b814ec0ecce7cb1c7fed29729f4c3a78/numpy-2.3.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51c55fe3451421f3a6ef9a9c1439e82101c57a2c9eab9feb196a62b1a10b58ce", size = 18533275, upload-time = "2025-11-16T22:50:37.651Z" },
+ { url = "https://files.pythonhosted.org/packages/40/56/2932d75b6f13465239e3b7b7e511be27f1b8161ca2510854f0b6e521c395/numpy-2.3.5-cp313-cp313-win32.whl", hash = "sha256:1978155dd49972084bd6ef388d66ab70f0c323ddee6f693d539376498720fb7e", size = 6277637, upload-time = "2025-11-16T22:50:40.11Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/88/e2eaa6cffb115b85ed7c7c87775cb8bcf0816816bc98ca8dbfa2ee33fe6e/numpy-2.3.5-cp313-cp313-win_amd64.whl", hash = "sha256:00dc4e846108a382c5869e77c6ed514394bdeb3403461d25a829711041217d5b", size = 12779090, upload-time = "2025-11-16T22:50:42.503Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/88/3f41e13a44ebd4034ee17baa384acac29ba6a4fcc2aca95f6f08ca0447d1/numpy-2.3.5-cp313-cp313-win_arm64.whl", hash = "sha256:0472f11f6ec23a74a906a00b48a4dcf3849209696dff7c189714511268d103ae", size = 10194710, upload-time = "2025-11-16T22:50:44.971Z" },
+ { url = "https://files.pythonhosted.org/packages/13/cb/71744144e13389d577f867f745b7df2d8489463654a918eea2eeb166dfc9/numpy-2.3.5-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:414802f3b97f3c1eef41e530aaba3b3c1620649871d8cb38c6eaff034c2e16bd", size = 16827292, upload-time = "2025-11-16T22:50:47.715Z" },
+ { url = "https://files.pythonhosted.org/packages/71/80/ba9dc6f2a4398e7f42b708a7fdc841bb638d353be255655498edbf9a15a8/numpy-2.3.5-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5ee6609ac3604fa7780e30a03e5e241a7956f8e2fcfe547d51e3afa5247ac47f", size = 12378897, upload-time = "2025-11-16T22:50:51.327Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/6d/db2151b9f64264bcceccd51741aa39b50150de9b602d98ecfe7e0c4bff39/numpy-2.3.5-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:86d835afea1eaa143012a2d7a3f45a3adce2d7adc8b4961f0b362214d800846a", size = 5207391, upload-time = "2025-11-16T22:50:54.542Z" },
+ { url = "https://files.pythonhosted.org/packages/80/ae/429bacace5ccad48a14c4ae5332f6aa8ab9f69524193511d60ccdfdc65fa/numpy-2.3.5-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:30bc11310e8153ca664b14c5f1b73e94bd0503681fcf136a163de856f3a50139", size = 6721275, upload-time = "2025-11-16T22:50:56.794Z" },
+ { url = "https://files.pythonhosted.org/packages/74/5b/1919abf32d8722646a38cd527bc3771eb229a32724ee6ba340ead9b92249/numpy-2.3.5-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1062fde1dcf469571705945b0f221b73928f34a20c904ffb45db101907c3454e", size = 14306855, upload-time = "2025-11-16T22:50:59.208Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/87/6831980559434973bebc30cd9c1f21e541a0f2b0c280d43d3afd909b66d0/numpy-2.3.5-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ce581db493ea1a96c0556360ede6607496e8bf9b3a8efa66e06477267bc831e9", size = 16657359, upload-time = "2025-11-16T22:51:01.991Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/91/c797f544491ee99fd00495f12ebb7802c440c1915811d72ac5b4479a3356/numpy-2.3.5-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:cc8920d2ec5fa99875b670bb86ddeb21e295cb07aa331810d9e486e0b969d946", size = 16093374, upload-time = "2025-11-16T22:51:05.291Z" },
+ { url = "https://files.pythonhosted.org/packages/74/a6/54da03253afcbe7a72785ec4da9c69fb7a17710141ff9ac5fcb2e32dbe64/numpy-2.3.5-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:9ee2197ef8c4f0dfe405d835f3b6a14f5fee7782b5de51ba06fb65fc9b36e9f1", size = 18594587, upload-time = "2025-11-16T22:51:08.585Z" },
+ { url = "https://files.pythonhosted.org/packages/80/e9/aff53abbdd41b0ecca94285f325aff42357c6b5abc482a3fcb4994290b18/numpy-2.3.5-cp313-cp313t-win32.whl", hash = "sha256:70b37199913c1bd300ff6e2693316c6f869c7ee16378faf10e4f5e3275b299c3", size = 6405940, upload-time = "2025-11-16T22:51:11.541Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/81/50613fec9d4de5480de18d4f8ef59ad7e344d497edbef3cfd80f24f98461/numpy-2.3.5-cp313-cp313t-win_amd64.whl", hash = "sha256:b501b5fa195cc9e24fe102f21ec0a44dffc231d2af79950b451e0d99cea02234", size = 12920341, upload-time = "2025-11-16T22:51:14.312Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/ab/08fd63b9a74303947f34f0bd7c5903b9c5532c2d287bead5bdf4c556c486/numpy-2.3.5-cp313-cp313t-win_arm64.whl", hash = "sha256:a80afd79f45f3c4a7d341f13acbe058d1ca8ac017c165d3fa0d3de6bc1a079d7", size = 10262507, upload-time = "2025-11-16T22:51:16.846Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/65/f9dea8e109371ade9c782b4e4756a82edf9d3366bca495d84d79859a0b79/numpy-2.3.5-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:f0963b55cdd70fad460fa4c1341f12f976bb26cb66021a5580329bd498988310", size = 16910689, upload-time = "2025-11-16T22:52:23.247Z" },
+ { url = "https://files.pythonhosted.org/packages/00/4f/edb00032a8fb92ec0a679d3830368355da91a69cab6f3e9c21b64d0bb986/numpy-2.3.5-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:f4255143f5160d0de972d28c8f9665d882b5f61309d8362fdd3e103cf7bf010c", size = 12457053, upload-time = "2025-11-16T22:52:26.367Z" },
+ { url = "https://files.pythonhosted.org/packages/16/a4/e8a53b5abd500a63836a29ebe145fc1ab1f2eefe1cfe59276020373ae0aa/numpy-2.3.5-pp311-pypy311_pp73-macosx_14_0_arm64.whl", hash = "sha256:a4b9159734b326535f4dd01d947f919c6eefd2d9827466a696c44ced82dfbc18", size = 5285635, upload-time = "2025-11-16T22:52:29.266Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/2f/37eeb9014d9c8b3e9c55bc599c68263ca44fdbc12a93e45a21d1d56df737/numpy-2.3.5-pp311-pypy311_pp73-macosx_14_0_x86_64.whl", hash = "sha256:2feae0d2c91d46e59fcd62784a3a83b3fb677fead592ce51b5a6fbb4f95965ff", size = 6801770, upload-time = "2025-11-16T22:52:31.421Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/e4/68d2f474df2cb671b2b6c2986a02e520671295647dad82484cde80ca427b/numpy-2.3.5-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ffac52f28a7849ad7576293c0cb7b9f08304e8f7d738a8cb8a90ec4c55a998eb", size = 14391768, upload-time = "2025-11-16T22:52:33.593Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/50/94ccd8a2b141cb50651fddd4f6a48874acb3c91c8f0842b08a6afc4b0b21/numpy-2.3.5-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:63c0e9e7eea69588479ebf4a8a270d5ac22763cc5854e9a7eae952a3908103f7", size = 16729263, upload-time = "2025-11-16T22:52:36.369Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/ee/346fa473e666fe14c52fcdd19ec2424157290a032d4c41f98127bfb31ac7/numpy-2.3.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:f16417ec91f12f814b10bafe79ef77e70113a2f5f7018640e7425ff979253425", size = 12967213, upload-time = "2025-11-16T22:52:39.38Z" },
]
[[package]]
@@ -1542,7 +1593,7 @@ wheels = [
[[package]]
name = "openai"
-version = "1.109.1"
+version = "2.8.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@@ -1554,9 +1605,9 @@ dependencies = [
{ name = "tqdm" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c6/a1/a303104dc55fc546a3f6914c842d3da471c64eec92043aef8f652eb6c524/openai-1.109.1.tar.gz", hash = "sha256:d173ed8dbca665892a6db099b4a2dfac624f94d20a93f46eb0b56aae940ed869", size = 564133, upload-time = "2025-09-24T13:00:53.075Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/d5/e4/42591e356f1d53c568418dc7e30dcda7be31dd5a4d570bca22acb0525862/openai-2.8.1.tar.gz", hash = "sha256:cb1b79eef6e809f6da326a7ef6038719e35aa944c42d081807bfa1be8060f15f", size = 602490, upload-time = "2025-11-17T22:39:59.549Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/1d/2a/7dd3d207ec669cacc1f186fd856a0f61dbc255d24f6fdc1a6715d6051b0f/openai-1.109.1-py3-none-any.whl", hash = "sha256:6bcaf57086cf59159b8e27447e4e7dd019db5d29a438072fbd49c290c7e65315", size = 948627, upload-time = "2025-09-24T13:00:50.754Z" },
+ { url = "https://files.pythonhosted.org/packages/55/4f/dbc0c124c40cb390508a82770fb9f6e3ed162560181a85089191a851c59a/openai-2.8.1-py3-none-any.whl", hash = "sha256:c6c3b5a04994734386e8dad3c00a393f56d3b68a27cd2e8acae91a59e4122463", size = 1022688, upload-time = "2025-11-17T22:39:57.675Z" },
]
[[package]]
@@ -1816,20 +1867,20 @@ wheels = [
[[package]]
name = "polyfactory"
-version = "2.22.3"
+version = "3.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "faker" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/64/5a/c9105c974e03d78dc6d5642bee97f075156a28ad344428e562c6c86526b9/polyfactory-2.22.3.tar.gz", hash = "sha256:ae57d07408d1f7609031a83827c7980ce32104535e146cac2253988d0a7665e1", size = 263543, upload-time = "2025-10-18T14:04:54.901Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/45/c6/2f795ede27e82bf6ed10a6b689f975017e36bf497915bfeafc3932b1a613/polyfactory-3.0.0.tar.gz", hash = "sha256:c1be54fc10fe738415eef2248bcfc687e3d74634f5e73b70e4ba58952df31363", size = 304234, upload-time = "2025-11-15T11:23:02.003Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c4/f7/244a5b1dd298650e4092c501197dad45036b1c31309ad4d01af430071a0f/polyfactory-2.22.3-py3-none-any.whl", hash = "sha256:0bfd5fe2fb2e5db39ded6aee8e923d1961095d4ebb44185cceee4654cb85e0b1", size = 63715, upload-time = "2025-10-18T14:04:52.657Z" },
+ { url = "https://files.pythonhosted.org/packages/54/86/742c4eb237cdb5e7e6dfb1474f5f0dac967896185f832c012dff4d7bdce2/polyfactory-3.0.0-py3-none-any.whl", hash = "sha256:ad9ba3b3f04c292d30673587946e1c5d18e249074368b09abb7c9455288cfc37", size = 61237, upload-time = "2025-11-15T11:23:00.524Z" },
]
[[package]]
name = "posthog"
-version = "6.9.1"
+version = "7.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "backoff" },
@@ -1839,9 +1890,25 @@ dependencies = [
{ name = "six" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/36/5e/137aaf1d45cc6fafa5573d24dfae795ceae75fdf3232d298828f2e54d688/posthog-6.9.1.tar.gz", hash = "sha256:0bf1115261369b76e2f643d04805cec434236f23fb69972ed5d1bd49b5a9a6fe", size = 126229, upload-time = "2025-11-07T15:57:26.347Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/d4/b9afe855a8a7a1bf4459c28ae4c300b40338122dc850acabefcf2c3df24d/posthog-7.0.1.tar.gz", hash = "sha256:21150562c2630a599c1d7eac94bc5c64eb6f6acbf3ff52ccf1e57345706db05a", size = 126985, upload-time = "2025-11-15T12:44:22.465Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/91/72/ad1961cc3423f679bceb6c098ec67c5db7ab55dbafc71c5a4faf4ec99d68/posthog-6.9.1-py3-none-any.whl", hash = "sha256:a8e33fef54275c32077afea4b2a0e2ca554b226b63d6fcd319447c81154faa1f", size = 144481, upload-time = "2025-11-07T15:57:25.183Z" },
+ { url = "https://files.pythonhosted.org/packages/05/0c/8b6b20b0be71725e6e8a32dcd460cdbf62fe6df9bc656a650150dc98fedd/posthog-7.0.1-py3-none-any.whl", hash = "sha256:efe212d8d88a9ba80a20c588eab4baf4b1a5e90e40b551160a5603bb21e96904", size = 145234, upload-time = "2025-11-15T12:44:21.247Z" },
+]
+
+[[package]]
+name = "pre-commit"
+version = "4.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cfgv" },
+ { name = "identify" },
+ { name = "nodeenv" },
+ { name = "pyyaml" },
+ { name = "virtualenv" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a6/49/7845c2d7bf6474efd8e27905b51b11e6ce411708c91e829b93f324de9929/pre_commit-4.4.0.tar.gz", hash = "sha256:f0233ebab440e9f17cabbb558706eb173d19ace965c68cdce2c081042b4fab15", size = 197501, upload-time = "2025-11-08T21:12:11.607Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/27/11/574fe7d13acf30bfd0a8dd7fa1647040f2b8064f13f43e8c963b1e65093b/pre_commit-4.4.0-py2.py3-none-any.whl", hash = "sha256:b35ea52957cbf83dcc5d8ee636cbead8624e3a15fbfa61a370e42158ac8a5813", size = 226049, upload-time = "2025-11-08T21:12:10.228Z" },
]
[[package]]
@@ -2065,7 +2132,7 @@ wheels = [
[[package]]
name = "pylint"
-version = "4.0.2"
+version = "4.0.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "astroid" },
@@ -2077,22 +2144,22 @@ dependencies = [
{ name = "tomli", marker = "python_full_version < '3.11'" },
{ name = "tomlkit" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/b6/f8/2feda2bc72654811f2596e856c33c2a98225fba717df2b55c8d6a1f5cdad/pylint-4.0.2.tar.gz", hash = "sha256:9c22dfa52781d3b79ce86ab2463940f874921a3e5707bcfc98dd0c019945014e", size = 1572401, upload-time = "2025-10-20T13:02:34.975Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/dd/9c/0500020a5446031220f487ca0c762713c6f3ddad7231b811aaf1d473f6aa/pylint-4.0.3.tar.gz", hash = "sha256:a427fe76e0e5355e9fb9b604fd106c419cafb395886ba7f3cebebb03f30e081d", size = 1570368, upload-time = "2025-11-13T15:54:41.394Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/1e/8b/2e814a255436fc6d604a60f1e8b8a186e05082aa3c0cabfd9330192496a2/pylint-4.0.2-py3-none-any.whl", hash = "sha256:9627ccd129893fb8ee8e8010261cb13485daca83e61a6f854a85528ee579502d", size = 536019, upload-time = "2025-10-20T13:02:32.778Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/01/b8acd4087102c774d432a6663bac4857405c64771445c0a3110828bc5c88/pylint-4.0.3-py3-none-any.whl", hash = "sha256:896d09afb0e78bbf2e030cd1f3d8dc92771a51f7e46828cbc3948a89cd03433a", size = 536199, upload-time = "2025-11-13T15:54:39.734Z" },
]
[[package]]
name = "pymdown-extensions"
-version = "10.16.1"
+version = "10.17.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "markdown" },
{ name = "pyyaml" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/55/b3/6d2b3f149bc5413b0a29761c2c5832d8ce904a1d7f621e86616d96f505cc/pymdown_extensions-10.16.1.tar.gz", hash = "sha256:aace82bcccba3efc03e25d584e6a22d27a8e17caa3f4dd9f207e49b787aa9a91", size = 853277, upload-time = "2025-07-28T16:19:34.167Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/e7/d9/a987e4d549c6c82353fce5fa5f650229bb60ea4c0d1684a2714a509aef58/pymdown_extensions-10.17.1.tar.gz", hash = "sha256:60d05fe55e7fb5a1e4740fc575facad20dc6ee3a748e8d3d36ba44142e75ce03", size = 845207, upload-time = "2025-11-11T21:44:58.815Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e4/06/43084e6cbd4b3bc0e80f6be743b2e79fbc6eed8de9ad8c629939fa55d972/pymdown_extensions-10.16.1-py3-none-any.whl", hash = "sha256:d6ba157a6c03146a7fb122b2b9a121300056384eafeec9c9f9e584adfdb2a32d", size = 266178, upload-time = "2025-07-28T16:19:31.401Z" },
+ { url = "https://files.pythonhosted.org/packages/81/40/b2d7b9fdccc63e48ae4dbd363b6b89eb7ac346ea49ed667bb71f92af3021/pymdown_extensions-10.17.1-py3-none-any.whl", hash = "sha256:1f160209c82eecbb5d8a0d8f89a4d9bd6bdcbde9a8537761844cfc57ad5cd8a6", size = 266310, upload-time = "2025-11-11T21:44:56.809Z" },
]
[[package]]
@@ -2321,16 +2388,15 @@ wheels = [
[[package]]
name = "rich"
-version = "13.9.4"
+version = "14.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "markdown-it-py" },
{ name = "pygments" },
- { name = "typing-extensions", marker = "python_full_version < '3.11'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/ab/3a/0316b28d0761c6734d6bc14e770d85506c986c85ffb239e688eeaab2c2bc/rich-13.9.4.tar.gz", hash = "sha256:439594978a49a09530cff7ebc4b5c7103ef57baf48d5ea3184f21d9a2befa098", size = 223149, upload-time = "2024-11-01T16:43:57.873Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/19/71/39c7c0d87f8d4e6c020a393182060eaefeeae6c01dab6a84ec346f2567df/rich-13.9.4-py3-none-any.whl", hash = "sha256:6049d5e6ec054bf2779ab3358186963bac2ea89175919d699e378b99738c2a90", size = 242424, upload-time = "2024-11-01T16:43:55.817Z" },
+ { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" },
]
[[package]]
@@ -2347,28 +2413,28 @@ wheels = [
[[package]]
name = "ruff"
-version = "0.14.4"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/df/55/cccfca45157a2031dcbb5a462a67f7cf27f8b37d4b3b1cd7438f0f5c1df6/ruff-0.14.4.tar.gz", hash = "sha256:f459a49fe1085a749f15414ca76f61595f1a2cc8778ed7c279b6ca2e1fd19df3", size = 5587844, upload-time = "2025-11-06T22:07:45.033Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/17/b9/67240254166ae1eaa38dec32265e9153ac53645a6c6670ed36ad00722af8/ruff-0.14.4-py3-none-linux_armv6l.whl", hash = "sha256:e6604613ffbcf2297cd5dcba0e0ac9bd0c11dc026442dfbb614504e87c349518", size = 12606781, upload-time = "2025-11-06T22:07:01.841Z" },
- { url = "https://files.pythonhosted.org/packages/46/c8/09b3ab245d8652eafe5256ab59718641429f68681ee713ff06c5c549f156/ruff-0.14.4-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:d99c0b52b6f0598acede45ee78288e5e9b4409d1ce7f661f0fa36d4cbeadf9a4", size = 12946765, upload-time = "2025-11-06T22:07:05.858Z" },
- { url = "https://files.pythonhosted.org/packages/14/bb/1564b000219144bf5eed2359edc94c3590dd49d510751dad26202c18a17d/ruff-0.14.4-py3-none-macosx_11_0_arm64.whl", hash = "sha256:9358d490ec030f1b51d048a7fd6ead418ed0826daf6149e95e30aa67c168af33", size = 11928120, upload-time = "2025-11-06T22:07:08.023Z" },
- { url = "https://files.pythonhosted.org/packages/a3/92/d5f1770e9988cc0742fefaa351e840d9aef04ec24ae1be36f333f96d5704/ruff-0.14.4-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:81b40d27924f1f02dfa827b9c0712a13c0e4b108421665322218fc38caf615c2", size = 12370877, upload-time = "2025-11-06T22:07:10.015Z" },
- { url = "https://files.pythonhosted.org/packages/e2/29/e9282efa55f1973d109faf839a63235575519c8ad278cc87a182a366810e/ruff-0.14.4-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f5e649052a294fe00818650712083cddc6cc02744afaf37202c65df9ea52efa5", size = 12408538, upload-time = "2025-11-06T22:07:13.085Z" },
- { url = "https://files.pythonhosted.org/packages/8e/01/930ed6ecfce130144b32d77d8d69f5c610e6d23e6857927150adf5d7379a/ruff-0.14.4-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:aa082a8f878deeba955531f975881828fd6afd90dfa757c2b0808aadb437136e", size = 13141942, upload-time = "2025-11-06T22:07:15.386Z" },
- { url = "https://files.pythonhosted.org/packages/6a/46/a9c89b42b231a9f487233f17a89cbef9d5acd538d9488687a02ad288fa6b/ruff-0.14.4-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:1043c6811c2419e39011890f14d0a30470f19d47d197c4858b2787dfa698f6c8", size = 14544306, upload-time = "2025-11-06T22:07:17.631Z" },
- { url = "https://files.pythonhosted.org/packages/78/96/9c6cf86491f2a6d52758b830b89b78c2ae61e8ca66b86bf5a20af73d20e6/ruff-0.14.4-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a9f3a936ac27fb7c2a93e4f4b943a662775879ac579a433291a6f69428722649", size = 14210427, upload-time = "2025-11-06T22:07:19.832Z" },
- { url = "https://files.pythonhosted.org/packages/71/f4/0666fe7769a54f63e66404e8ff698de1dcde733e12e2fd1c9c6efb689cb5/ruff-0.14.4-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:95643ffd209ce78bc113266b88fba3d39e0461f0cbc8b55fb92505030fb4a850", size = 13658488, upload-time = "2025-11-06T22:07:22.32Z" },
- { url = "https://files.pythonhosted.org/packages/ee/79/6ad4dda2cfd55e41ac9ed6d73ef9ab9475b1eef69f3a85957210c74ba12c/ruff-0.14.4-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:456daa2fa1021bc86ca857f43fe29d5d8b3f0e55e9f90c58c317c1dcc2afc7b5", size = 13354908, upload-time = "2025-11-06T22:07:24.347Z" },
- { url = "https://files.pythonhosted.org/packages/b5/60/f0b6990f740bb15c1588601d19d21bcc1bd5de4330a07222041678a8e04f/ruff-0.14.4-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:f911bba769e4a9f51af6e70037bb72b70b45a16db5ce73e1f72aefe6f6d62132", size = 13587803, upload-time = "2025-11-06T22:07:26.327Z" },
- { url = "https://files.pythonhosted.org/packages/c9/da/eaaada586f80068728338e0ef7f29ab3e4a08a692f92eb901a4f06bbff24/ruff-0.14.4-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:76158a7369b3979fa878612c623a7e5430c18b2fd1c73b214945c2d06337db67", size = 12279654, upload-time = "2025-11-06T22:07:28.46Z" },
- { url = "https://files.pythonhosted.org/packages/66/d4/b1d0e82cf9bf8aed10a6d45be47b3f402730aa2c438164424783ac88c0ed/ruff-0.14.4-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:f3b8f3b442d2b14c246e7aeca2e75915159e06a3540e2f4bed9f50d062d24469", size = 12357520, upload-time = "2025-11-06T22:07:31.468Z" },
- { url = "https://files.pythonhosted.org/packages/04/f4/53e2b42cc82804617e5c7950b7079d79996c27e99c4652131c6a1100657f/ruff-0.14.4-py3-none-musllinux_1_2_i686.whl", hash = "sha256:c62da9a06779deecf4d17ed04939ae8b31b517643b26370c3be1d26f3ef7dbde", size = 12719431, upload-time = "2025-11-06T22:07:33.831Z" },
- { url = "https://files.pythonhosted.org/packages/a2/94/80e3d74ed9a72d64e94a7b7706b1c1ebaa315ef2076fd33581f6a1cd2f95/ruff-0.14.4-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:5a443a83a1506c684e98acb8cb55abaf3ef725078be40237463dae4463366349", size = 13464394, upload-time = "2025-11-06T22:07:35.905Z" },
- { url = "https://files.pythonhosted.org/packages/54/1a/a49f071f04c42345c793d22f6cf5e0920095e286119ee53a64a3a3004825/ruff-0.14.4-py3-none-win32.whl", hash = "sha256:643b69cb63cd996f1fc7229da726d07ac307eae442dd8974dbc7cf22c1e18fff", size = 12493429, upload-time = "2025-11-06T22:07:38.43Z" },
- { url = "https://files.pythonhosted.org/packages/bc/22/e58c43e641145a2b670328fb98bc384e20679b5774258b1e540207580266/ruff-0.14.4-py3-none-win_amd64.whl", hash = "sha256:26673da283b96fe35fa0c939bf8411abec47111644aa9f7cfbd3c573fb125d2c", size = 13635380, upload-time = "2025-11-06T22:07:40.496Z" },
- { url = "https://files.pythonhosted.org/packages/30/bd/4168a751ddbbf43e86544b4de8b5c3b7be8d7167a2a5cb977d274e04f0a1/ruff-0.14.4-py3-none-win_arm64.whl", hash = "sha256:dd09c292479596b0e6fec8cd95c65c3a6dc68e9ad17b8f2382130f87ff6a75bb", size = 12663065, upload-time = "2025-11-06T22:07:42.603Z" },
+version = "0.14.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/82/fa/fbb67a5780ae0f704876cb8ac92d6d76da41da4dc72b7ed3565ab18f2f52/ruff-0.14.5.tar.gz", hash = "sha256:8d3b48d7d8aad423d3137af7ab6c8b1e38e4de104800f0d596990f6ada1a9fc1", size = 5615944, upload-time = "2025-11-13T19:58:51.155Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/68/31/c07e9c535248d10836a94e4f4e8c5a31a1beed6f169b31405b227872d4f4/ruff-0.14.5-py3-none-linux_armv6l.whl", hash = "sha256:f3b8248123b586de44a8018bcc9fefe31d23dda57a34e6f0e1e53bd51fd63594", size = 13171630, upload-time = "2025-11-13T19:57:54.894Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/5c/283c62516dca697cd604c2796d1487396b7a436b2f0ecc3fd412aca470e0/ruff-0.14.5-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:f7a75236570318c7a30edd7f5491945f0169de738d945ca8784500b517163a72", size = 13413925, upload-time = "2025-11-13T19:57:59.181Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/f3/aa319f4afc22cb6fcba2b9cdfc0f03bbf747e59ab7a8c5e90173857a1361/ruff-0.14.5-py3-none-macosx_11_0_arm64.whl", hash = "sha256:6d146132d1ee115f8802356a2dc9a634dbf58184c51bff21f313e8cd1c74899a", size = 12574040, upload-time = "2025-11-13T19:58:02.056Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/7f/cb5845fcc7c7e88ed57f58670189fc2ff517fe2134c3821e77e29fd3b0c8/ruff-0.14.5-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e2380596653dcd20b057794d55681571a257a42327da8894b93bbd6111aa801f", size = 13009755, upload-time = "2025-11-13T19:58:05.172Z" },
+ { url = "https://files.pythonhosted.org/packages/21/d2/bcbedbb6bcb9253085981730687ddc0cc7b2e18e8dc13cf4453de905d7a0/ruff-0.14.5-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2d1fa985a42b1f075a098fa1ab9d472b712bdb17ad87a8ec86e45e7fa6273e68", size = 12937641, upload-time = "2025-11-13T19:58:08.345Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/58/e25de28a572bdd60ffc6bb71fc7fd25a94ec6a076942e372437649cbb02a/ruff-0.14.5-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:88f0770d42b7fa02bbefddde15d235ca3aa24e2f0137388cc15b2dcbb1f7c7a7", size = 13610854, upload-time = "2025-11-13T19:58:11.419Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/24/43bb3fd23ecee9861970978ea1a7a63e12a204d319248a7e8af539984280/ruff-0.14.5-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:3676cb02b9061fee7294661071c4709fa21419ea9176087cb77e64410926eb78", size = 15061088, upload-time = "2025-11-13T19:58:14.551Z" },
+ { url = "https://files.pythonhosted.org/packages/23/44/a022f288d61c2f8c8645b24c364b719aee293ffc7d633a2ca4d116b9c716/ruff-0.14.5-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b595bedf6bc9cab647c4a173a61acf4f1ac5f2b545203ba82f30fcb10b0318fb", size = 14734717, upload-time = "2025-11-13T19:58:17.518Z" },
+ { url = "https://files.pythonhosted.org/packages/58/81/5c6ba44de7e44c91f68073e0658109d8373b0590940efe5bd7753a2585a3/ruff-0.14.5-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f55382725ad0bdb2e8ee2babcbbfb16f124f5a59496a2f6a46f1d9d99d93e6e2", size = 14028812, upload-time = "2025-11-13T19:58:20.533Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/ef/41a8b60f8462cb320f68615b00299ebb12660097c952c600c762078420f8/ruff-0.14.5-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7497d19dce23976bdaca24345ae131a1d38dcfe1b0850ad8e9e6e4fa321a6e19", size = 13825656, upload-time = "2025-11-13T19:58:23.345Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/00/207e5de737fdb59b39eb1fac806904fe05681981b46d6a6db9468501062e/ruff-0.14.5-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:410e781f1122d6be4f446981dd479470af86537fb0b8857f27a6e872f65a38e4", size = 13959922, upload-time = "2025-11-13T19:58:26.537Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/7e/fa1f5c2776db4be405040293618846a2dece5c70b050874c2d1f10f24776/ruff-0.14.5-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:c01be527ef4c91a6d55e53b337bfe2c0f82af024cc1a33c44792d6844e2331e1", size = 12932501, upload-time = "2025-11-13T19:58:29.822Z" },
+ { url = "https://files.pythonhosted.org/packages/67/d8/d86bf784d693a764b59479a6bbdc9515ae42c340a5dc5ab1dabef847bfaa/ruff-0.14.5-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:f66e9bb762e68d66e48550b59c74314168ebb46199886c5c5aa0b0fbcc81b151", size = 12927319, upload-time = "2025-11-13T19:58:32.923Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/de/ee0b304d450ae007ce0cb3e455fe24fbcaaedae4ebaad6c23831c6663651/ruff-0.14.5-py3-none-musllinux_1_2_i686.whl", hash = "sha256:d93be8f1fa01022337f1f8f3bcaa7ffee2d0b03f00922c45c2207954f351f465", size = 13206209, upload-time = "2025-11-13T19:58:35.952Z" },
+ { url = "https://files.pythonhosted.org/packages/33/aa/193ca7e3a92d74f17d9d5771a765965d2cf42c86e6f0fd95b13969115723/ruff-0.14.5-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:c135d4b681f7401fe0e7312017e41aba9b3160861105726b76cfa14bc25aa367", size = 13953709, upload-time = "2025-11-13T19:58:39.002Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/f1/7119e42aa1d3bf036ffc9478885c2e248812b7de9abea4eae89163d2929d/ruff-0.14.5-py3-none-win32.whl", hash = "sha256:c83642e6fccfb6dea8b785eb9f456800dcd6a63f362238af5fc0c83d027dd08b", size = 12925808, upload-time = "2025-11-13T19:58:42.779Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/9d/7c0a255d21e0912114784e4a96bf62af0618e2190cae468cd82b13625ad2/ruff-0.14.5-py3-none-win_amd64.whl", hash = "sha256:9d55d7af7166f143c94eae1db3312f9ea8f95a4defef1979ed516dbb38c27621", size = 14331546, upload-time = "2025-11-13T19:58:45.691Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/80/69756670caedcf3b9be597a6e12276a6cf6197076eb62aad0c608f8efce0/ruff-0.14.5-py3-none-win_arm64.whl", hash = "sha256:4b700459d4649e2594b31f20a9de33bc7c19976d4746d8d0798ad959621d64a4", size = 13433331, upload-time = "2025-11-13T19:58:48.434Z" },
]
[[package]]
@@ -2491,6 +2557,31 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl", hash = "sha256:26445eca388f82e72884e0d580d5464cd801a3ea01e63e5601bdff9ba6a48de2", size = 78540, upload-time = "2024-11-24T20:12:19.698Z" },
]
+[[package]]
+name = "ty"
+version = "0.0.1a27"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8f/65/3592d7c73d80664378fc90d0a00c33449a99cbf13b984433c883815245f3/ty-0.0.1a27.tar.gz", hash = "sha256:d34fe04979f2c912700cbf0919e8f9b4eeaa10c4a2aff7450e5e4c90f998bc28", size = 4516059, upload-time = "2025-11-18T21:55:18.381Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e6/05/7945aa97356446fd53ed3ddc7ee02a88d8ad394217acd9428f472d6b109d/ty-0.0.1a27-py3-none-linux_armv6l.whl", hash = "sha256:3cbb735f5ecb3a7a5f5b82fb24da17912788c109086df4e97d454c8fb236fbc5", size = 9375047, upload-time = "2025-11-18T21:54:31.577Z" },
+ { url = "https://files.pythonhosted.org/packages/69/4e/89b167a03de0e9ec329dc89bc02e8694768e4576337ef6c0699987681342/ty-0.0.1a27-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:4a6367236dc456ba2416563301d498aef8c6f8959be88777ef7ba5ac1bf15f0b", size = 9169540, upload-time = "2025-11-18T21:54:34.036Z" },
+ { url = "https://files.pythonhosted.org/packages/38/07/e62009ab9cc242e1becb2bd992097c80a133fce0d4f055fba6576150d08a/ty-0.0.1a27-py3-none-macosx_11_0_arm64.whl", hash = "sha256:8e93e231a1bcde964cdb062d2d5e549c24493fb1638eecae8fcc42b81e9463a4", size = 8711942, upload-time = "2025-11-18T21:54:36.3Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/43/f35716ec15406f13085db52e762a3cc663c651531a8124481d0ba602eca0/ty-0.0.1a27-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c5b6a8166b60117da1179851a3d719cc798bf7e61f91b35d76242f0059e9ae1d", size = 8984208, upload-time = "2025-11-18T21:54:39.453Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/79/486a3374809523172379768de882c7a369861165802990177fe81489b85f/ty-0.0.1a27-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bfbe8b0e831c072b79a078d6c126d7f4d48ca17f64a103de1b93aeda32265dc5", size = 9157209, upload-time = "2025-11-18T21:54:42.664Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/08/9a7c8efcb327197d7d347c548850ef4b54de1c254981b65e8cd0672dc327/ty-0.0.1a27-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:90e09678331552e7c25d7eb47868b0910dc5b9b212ae22c8ce71a52d6576ddbb", size = 9519207, upload-time = "2025-11-18T21:54:45.311Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/9d/7b4680683e83204b9edec551bb91c21c789ebc586b949c5218157ee474b7/ty-0.0.1a27-py3-none-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:88c03e4beeca79d85a5618921e44b3a6ea957e0453e08b1cdd418b51da645939", size = 10148794, upload-time = "2025-11-18T21:54:48.329Z" },
+ { url = "https://files.pythonhosted.org/packages/89/21/8b961b0ab00c28223f06b33222427a8e31aa04f39d1b236acc93021c626c/ty-0.0.1a27-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3ece5811322789fefe22fc088ed36c5879489cd39e913f9c1ff2a7678f089c61", size = 9900563, upload-time = "2025-11-18T21:54:51.214Z" },
+ { url = "https://files.pythonhosted.org/packages/85/eb/95e1f0b426c2ea8d443aa923fcab509059c467bbe64a15baaf573fea1203/ty-0.0.1a27-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2f2ccb4f0fddcd6e2017c268dfce2489e9a36cb82a5900afe6425835248b1086", size = 9926355, upload-time = "2025-11-18T21:54:53.927Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/78/40e7f072049e63c414f2845df780be3a494d92198c87c2ffa65e63aecf3f/ty-0.0.1a27-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33450528312e41d003e96a1647780b2783ab7569bbc29c04fc76f2d1908061e3", size = 9480580, upload-time = "2025-11-18T21:54:56.617Z" },
+ { url = "https://files.pythonhosted.org/packages/18/da/f4a2dfedab39096808ddf7475f35ceb750d9a9da840bee4afd47b871742f/ty-0.0.1a27-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:a0a9ac635deaa2b15947701197ede40cdecd13f89f19351872d16f9ccd773fa1", size = 8957524, upload-time = "2025-11-18T21:54:59.085Z" },
+ { url = "https://files.pythonhosted.org/packages/21/ea/26fee9a20cf77a157316fd3ab9c6db8ad5a0b20b2d38a43f3452622587ac/ty-0.0.1a27-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:797fb2cd49b6b9b3ac9f2f0e401fb02d3aa155badc05a8591d048d38d28f1e0c", size = 9201098, upload-time = "2025-11-18T21:55:01.845Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/53/e14591d1275108c9ae28f97ac5d4b93adcc2c8a4b1b9a880dfa9d07c15f8/ty-0.0.1a27-py3-none-musllinux_1_2_i686.whl", hash = "sha256:7fe81679a0941f85e98187d444604e24b15bde0a85874957c945751756314d03", size = 9275470, upload-time = "2025-11-18T21:55:04.23Z" },
+ { url = "https://files.pythonhosted.org/packages/37/44/e2c9acecac70bf06fb41de285e7be2433c2c9828f71e3bf0e886fc85c4fd/ty-0.0.1a27-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:355f651d0cdb85535a82bd9f0583f77b28e3fd7bba7b7da33dcee5a576eff28b", size = 9592394, upload-time = "2025-11-18T21:55:06.542Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/a7/4636369731b24ed07c2b4c7805b8d990283d677180662c532d82e4ef1a36/ty-0.0.1a27-py3-none-win32.whl", hash = "sha256:61782e5f40e6df622093847b34c366634b75d53f839986f1bf4481672ad6cb55", size = 8783816, upload-time = "2025-11-18T21:55:09.648Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/1d/b76487725628d9e81d9047dc0033a5e167e0d10f27893d04de67fe1a9763/ty-0.0.1a27-py3-none-win_amd64.whl", hash = "sha256:c682b238085d3191acddcf66ef22641562946b1bba2a7f316012d5b2a2f4de11", size = 9616833, upload-time = "2025-11-18T21:55:12.457Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/db/c7cd5276c8f336a3cf87992b75ba9d486a7cf54e753fcd42495b3bc56fb7/ty-0.0.1a27-py3-none-win_arm64.whl", hash = "sha256:e146dfa32cbb0ac6afb0cb65659e87e4e313715e68d76fe5ae0a4b3d5b912ce8", size = 9137796, upload-time = "2025-11-18T21:55:15.897Z" },
+]
+
[[package]]
name = "typer"
version = "0.20.0"
@@ -2531,39 +2622,39 @@ bedrock-runtime = [
[[package]]
name = "types-aiobotocore"
-version = "2.25.1"
+version = "2.25.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "botocore-stubs" },
{ name = "typing-extensions", marker = "python_full_version < '3.12'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/f4/33/bded03f2eac48036cbd03421ae0163ff10ceb57969d1617763bb1405bc88/types_aiobotocore-2.25.1.tar.gz", hash = "sha256:133d3db91356e53e4b22cdf745a392d8c0527383273f04d4c6bd0b0a44320cca", size = 85673, upload-time = "2025-10-29T01:54:06.19Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/77/b3/15e487a2a84ca827cbd80af1de4485d25dc3014f3122d8069665bee725bc/types_aiobotocore-2.25.2.tar.gz", hash = "sha256:6b2458ac1c0999de0ba4a1dcfbf3bc0d5c7ca2ea0c72f6732abe67d08ebf0240", size = 85272, upload-time = "2025-11-12T01:58:58.83Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/96/6c/033dcbfb6beedb4576b3732cdcf78ed3676e3d3bed8b1b3648c560dbdbe1/types_aiobotocore-2.25.1-py3-none-any.whl", hash = "sha256:72bf7d0eee076d179987f0eafb16deabeebe0845abbae72738ffcbba3d4afee7", size = 53741, upload-time = "2025-10-29T01:54:01.373Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/1f/0115f843090b969ef4902eb8ef128c613fdc91787c34a4b9e4140cf763fa/types_aiobotocore-2.25.2-py3-none-any.whl", hash = "sha256:b6027aa2dc020a9e2eeaa0ab9178f5c2cb6bb5e07613b7ea26174cc491d33b39", size = 53655, upload-time = "2025-11-12T01:58:55.575Z" },
]
[[package]]
name = "types-aiobotocore-bedrock"
-version = "2.25.1"
+version = "2.25.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions", marker = "python_full_version < '3.12'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/47/ac/5ea3704ef61041c851a89412473b623d73418b546a9e26b72deeb9ae6c56/types_aiobotocore_bedrock-2.25.1.tar.gz", hash = "sha256:ab0820893fa8481909f5dcd5020aa3ee529874193f30db0ade3db0c9ad7f33b3", size = 60006, upload-time = "2025-10-29T01:44:10.008Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/69/99/b986a2dcd19828b7d79d237d9c6154327bfcc4fd6126b67ea6343322a8b5/types_aiobotocore_bedrock-2.25.2.tar.gz", hash = "sha256:524043f7cf95d4670706b05d48835d8ad3e401fa4095000d107ca444ea3c4e7f", size = 59660, upload-time = "2025-11-12T01:42:00.672Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/fd/d2/87a0f382432fb0fb6a0e8ea6ae0901f15209d0fd5422d376d00f5b05a5b4/types_aiobotocore_bedrock-2.25.1-py3-none-any.whl", hash = "sha256:98b9597c94846e8a977b3a6cc6a9e5474769bc0c56a2d6ec2c34eff02d1fe17a", size = 67594, upload-time = "2025-10-29T01:44:07.861Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/d3/1d32ce456037008fa0878832f80347d029887a7ea20922745f003def4ab9/types_aiobotocore_bedrock-2.25.2-py3-none-any.whl", hash = "sha256:d06ebd7a8e8ffe22567719136bd73b9a804f77bd83ec7ad02a8077438c5436d3", size = 67366, upload-time = "2025-11-12T01:41:59.369Z" },
]
[[package]]
name = "types-aiobotocore-bedrock-runtime"
-version = "2.25.1"
+version = "2.25.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions", marker = "python_full_version < '3.12'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/bb/29/aa80b34c0c481c9d44e5764e3323703eddf9d20952b6d951f92c663526e7/types_aiobotocore_bedrock_runtime-2.25.1.tar.gz", hash = "sha256:28e61628dc5d73b00b35f45d20b048fe2b2d85a0feb7d2aa7e3d3d1647c8d336", size = 28550, upload-time = "2025-10-29T01:49:10.904Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/54/af/47f695dd846b3898feb0a858020ac86f088c65826d0d60f9680a1177837d/types_aiobotocore_bedrock_runtime-2.25.2.tar.gz", hash = "sha256:7068ac2f4cda13852739db2dc822ef7498c6fe81fc031b496dc35bdcb7cc44f3", size = 28451, upload-time = "2025-11-12T01:42:02.961Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/cd/89/f079b72b61fd12c358352bb1c96541ad47070319f5bec92db62e7464e53a/types_aiobotocore_bedrock_runtime-2.25.1-py3-none-any.whl", hash = "sha256:762f3a1b65bdb5389f46f3a42537b9dc95f02424d518454024c0ef89566c73a3", size = 34655, upload-time = "2025-10-29T01:44:10.854Z" },
+ { url = "https://files.pythonhosted.org/packages/01/ba/baeb40c98b0a2be60b5e6c704ed16646f6981618a543087832877aa7b170/types_aiobotocore_bedrock_runtime-2.25.2-py3-none-any.whl", hash = "sha256:8a2e881e0922c6866b5fac1835abfa8b623a2046db9125ea20e2832248ed55a0", size = 34804, upload-time = "2025-11-12T01:42:01.759Z" },
]
[[package]]
@@ -2577,11 +2668,11 @@ wheels = [
[[package]]
name = "types-awscrt"
-version = "0.28.2"
+version = "0.28.4"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/86/65/f92debc7c9ff9e6e51cf1495248f0edd2fa7123461acf5d07ec1688d8ac1/types_awscrt-0.28.2.tar.gz", hash = "sha256:4349b6fc7b1cd9c9eb782701fb213875db89ab1781219c0e947dd7c4d9dcd65e", size = 17438, upload-time = "2025-10-19T06:39:11.202Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/f7/6f/d4f2adb086e8f5cd2ae83cf8dbb192057d8b5025120e5b372468292db67f/types_awscrt-0.28.4.tar.gz", hash = "sha256:15929da84802f27019ee8e4484fb1c102e1f6d4cf22eb48688c34a5a86d02eb6", size = 17692, upload-time = "2025-11-11T02:56:53.516Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/27/23/535c2b3492fb31286a6adad45af3367eba3c23edc2fa24824d9526626012/types_awscrt-0.28.2-py3-none-any.whl", hash = "sha256:d08916fa735cfc032e6a8cfdac92785f1c4e88623999b224ea4e6267d5de5fcb", size = 41929, upload-time = "2025-10-19T06:39:10.042Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/ae/9acc4adf1d5d7bb7d09b6f9ff5d4d04a72eb64700d104106dd517665cd57/types_awscrt-0.28.4-py3-none-any.whl", hash = "sha256:2d453f9e27583fcc333771b69a5255a5a4e2c52f86e70f65f3c5a6789d3443d0", size = 42307, upload-time = "2025-11-11T02:56:52.231Z" },
]
[[package]]
@@ -2599,7 +2690,7 @@ version = "3.5.0.20251106"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
- { name = "numpy", version = "2.3.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
+ { name = "numpy", version = "2.3.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/da/0d/0657390fb7e4fffb9e0a20d51b9b63451771c238ad7ca2eb31de12827a87/types_networkx-3.5.0.20251106.tar.gz", hash = "sha256:2f17a53a35cf6dbbb031039c19bf0f127ec1b37f3530c57a517a0dd67a42e343", size = 72307, upload-time = "2025-11-06T03:06:52.263Z" }
wheels = [
@@ -2676,6 +2767,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/a7/c2/fe1e52489ae3122415c51f387e221dd0773709bad6c6cdaa599e8a2c5185/urllib3-2.5.0-py3-none-any.whl", hash = "sha256:e6b01673c0fa6a13e374b50871808eb3bf7046c4b125b216f6bf1cc604cff0dc", size = 129795, upload-time = "2025-06-18T14:07:40.39Z" },
]
+[[package]]
+name = "virtualenv"
+version = "20.35.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "distlib" },
+ { name = "filelock" },
+ { name = "platformdirs" },
+ { name = "typing-extensions", marker = "python_full_version < '3.11'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/20/28/e6f1a6f655d620846bd9df527390ecc26b3805a0c5989048c210e22c5ca9/virtualenv-20.35.4.tar.gz", hash = "sha256:643d3914d73d3eeb0c552cbb12d7e82adf0e504dbf86a3182f8771a153a1971c", size = 6028799, upload-time = "2025-10-29T06:57:40.511Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/79/0c/c05523fa3181fdf0c9c52a6ba91a23fbf3246cc095f26f6516f9c60e6771/virtualenv-20.35.4-py3-none-any.whl", hash = "sha256:c21c9cede36c9753eeade68ba7d523529f228a403463376cf821eaae2b650f1b", size = 6005095, upload-time = "2025-10-29T06:57:37.598Z" },
+]
+
[[package]]
name = "watchdog"
version = "6.0.0"