Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Refactor/cleanups (#360)
Louis Choquel72df421f7197
diff --git a/.cursor/rules/coding_standards.mdc b/.cursor/rules/coding_standards.mdcdeleted file mode 100644index 73c0f02db..000000000--- a/.cursor/rules/coding_standards.mdc+++ /dev/null@@ -1,130 +0,0 @@-----description:-globs:-alwaysApply: true-----# Coding Standards & Best Practices--This document outlines the core coding standards, best practices, and quality control procedures for the codebase.--## Type Hints--1. **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tupele types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use the `|` syntax for union types (e.g `str | int`) and `| None` for optionals- - Use Field(default_factory=...) for mutable defaults and if it's a list of something else than str, use `empty_list_factory_of()` to make a factory: `number_list: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")`- - Use `BaseModel` and respect Pydantic v2 standards, in particular use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose--2. **StrEnum**- - Import from `pipelex.types`:- ```python- from pipelex.types import StrEnum- ```--3. **Self type**- - Import from `pipelex.types`:- ```python- from pipelex.types import Self- ```--## Factory Pattern-- - Use Factory Pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such--## Error Handling-- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones- - Never catch Exception, only catch specific exceptions- - Always add `from exc` to the exception-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```-- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.--## Documentation--1. **Docstring Format**- ```python- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:- """Process and resize an image.-- Args:- image_path: Path to the source image- size: Tuple of (width, height) for resizing-- Returns:- Processed image as bytes- """- pass- ```--2. **Class Documentation**- ```python- class ImageProcessor:- """Handles image processing operations.-- Provides methods for resizing, converting, and optimizing images.- """- ```--## Code Quality Checks--### Linting and Type Checking--Before finalizing a task, run:-```bash-make fix-unused-imports-make check-```--This runs multiple code quality tools:-- Pyright: Static type checking-- Ruff: Fast Python linter-- Mypy: Static type checker--Always fix any issues reported by these tools before proceeding.--### Running Tests--1. **Quick Test Run** (no LLM/image generation):- ```bash- make tp- ```- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)`--2. **Specific Tests**:- ```bash- make tp TEST=TestClassName- # or- make tp TEST=test_function_name- ```- Note: Matches names starting with the provided string.--**Important**: Never run `make ti`, `make test-inference`, `make te`, `make test-extract`, `make tg`, or `make test-img-gen` - these use costly inference.--## Pipelines--- All pipeline definitions go in `pipelex/libraries/pipelines/`-- Always validate pipelines after creation/edit with `make validate`.- Iterate if there are errors.--## Project Structure--- **Pipelines**: `pipelex/libraries/pipelines/`-- **Tests**: `tests/` directory-- **Documentation**: `docs/` directory\ No newline at end of filediff --git a/.cursor/rules/commands.mdc b/.cursor/rules/commands.mdcnew file mode 100644index 000000000..6c017d195--- /dev/null+++ b/.cursor/rules/commands.mdc@@ -0,0 +1,8 @@+---+alwaysApply: true+description: Guidelines for running commands+---+# Commands++ - When you want to run commands such as `python`, `pytest` or any of our CLI such as `pipelex` or `cocode`, ALWAYS use the current obvious virtual env. If the installaton is standard, the venv is named `.venv` so always check that first.+\ No newline at end of filediff --git a/.cursor/rules/docs.mdc b/.cursor/rules/docs.mdcindex ac9c5191e..1a166508b 100644--- a/.cursor/rules/docs.mdc+++ b/.cursor/rules/docs.mdc@@ -1,8 +1,11 @@----description:-globs: docs/**/*.mdalwaysApply: false+description: Guidelines for writing documentation+globs:+- docs/**/*.md---+# Writing Docs+Write docs and answer questions about writing docs.We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.@@ -10,3 +13,4 @@ We use Material for MkDocs. All markdown in our docs must be compatible with Mat## MkDocs Markdown Requirements- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.+diff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdcindex d93ee4a08..a21831a30 100644--- a/.cursor/rules/llms.mdc+++ b/.cursor/rules/llms.mdc@@ -1,6 +1,9 @@----globs: *.plx,*.tomlalwaysApply: false+description: LLM configuration and usage guidelines+globs:+- '*.plx'+- '*.toml'---# Rules to choose LLM models used in PipeLLMs.@@ -39,7 +42,7 @@ type = "PipeLLM"description = "Write text about Hello World."output = "Text"model = { model = "gpt-5", temperature = 0.9 }-prompt_template = """+prompt = """Write a haiku about Hello World."""```@@ -65,7 +68,7 @@ description = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"model = "llm_to_extract_invoice"-prompt_template = """+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -79,3 +82,4 @@ You must not use an LLM preset in a PipeLLM that does not exist in the deck. IfYou can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+diff --git a/.cursor/rules/pytest.mdc b/.cursor/rules/pytest_standards.mdcsimilarity index 56%rename from .cursor/rules/pytest.mdcrename to .cursor/rules/pytest_standards.mdcindex 89a009295..b1c2f11e2 100644--- a/.cursor/rules/pytest.mdc+++ b/.cursor/rules/pytest_standards.mdc@@ -1,7 +1,8 @@----description:-globs: tests/**/*.pyalwaysApply: false+description: Guidelines for writing unit tests+globs:+- tests/**/*.py---# Writing unit tests@@ -9,13 +10,9 @@ alwaysApply: falseNEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.-These rules apply when writing unit tests.-- Always use pytest--## Test file structure+### Test file structure- Name test files with `test_` prefix-- Use descriptive names that match the functionality being tested- Place test files in the appropriate test category directory:- `tests/unit/` - for unit tests that test individual functions/classes in isolation- `tests/integration/` - for integration tests that test component interactions@@ -26,12 +23,12 @@ These rules apply when writing unit tests.- Always put test inside Test classes.- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`-## Markers+### MarkersApply the appropriate markers:- "llm: uses an LLM to generate text or objects"- "img_gen: uses an image generation AI"-- "extract: uses ocr models"+- "extract: uses text/image extraction from documents"- "inference: uses either an LLM or an image generation AI"- "gha_disabled: will not be able to run properly on GitHub Actions"@@ -41,9 +38,9 @@ Several markers may be applied. For instance, if the test uses an LLM, then it u- Never use the unittest.mock. Use pytest-mock.-## Test Class Structure+### Test Class Structure-Always group the tests of a module into a test class:+- Always group the tests of a module into a test class:```python@pytest.mark.llm@@ -51,7 +48,7 @@ Always group the tests of a module into a test class:@pytest.mark.asyncio(loop_scope="class")class TestFooBar:@pytest.mark.parametrize(- "topic test_case_blueprint",+ "topic, test_case_blueprint",[TestCases.CASE_1,TestCases.CASE_2,@@ -66,70 +63,13 @@ class TestFooBar:# Test implementation```-Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.--## Writing integration test to test pipes--### Required imports for pipe tests--```python-import pytest-from pytest import FixtureRequest-from pipelex import log, pretty_print-from pipelex.core.stuffs.stuff_factory import StuffBlueprint, StuffFactory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.hub import get_report_delegate-from pipelex.libraries.pipelines.base_library.retrieve import RetrievedExcerpt-from pipelex.config_pipelex import get_config--from pipelex.core.pipe import PipeAbstract, update_job_metadata_for_pipe-from pipelex.core.pipes.pipe_output import PipeOutput, PipeOutputType-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.pipe_works.pipe_router_protocol import PipeRouterProtocol-```--### Pipe test implementation steps--1. Create Stuff from blueprint:--```python-stuff = StuffFactory.make_stuff(- concept_code="RetrievedExcerpt",- domain="retrieve",- content=RetrievedExcerpt(text="<Some retrieved text>", justification="<Some justification>")- name="retrieved_text",-)-```--2. Create Working Memory:--```python-working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)-```--3. Run the pipe:--```python-pipe_output = await pipe_router.run_pipe(- pipe_code="pipe_name",- pipe_run_params=PipeRunParamsFactory.make_run_params(),- working_memory=working_memory,- job_metadata=JobMetadata(),-)-```--4. Basic assertions:--```python-assert pipe_output is not None-assert pipe_output.working_memory is not None-assert pipe_output.main_stuff is not None-```+- Never more than 1 class per test file.+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.### Test Data Organization-- If it's not already there, create a `test_data.py` file in the test directory+- If it's not already there, create a `test_data.py` file in the proper test directory- Define test cases using `StuffBlueprint`:```python@@ -145,7 +85,7 @@ class TestCases:value="test_value")- CASE_BLUEPRINTS: ClassVar[List[Tuple[str, str]]] = [ # topic, blueprint"+ CASE_BLUEPRINTS: ClassVar[list[tuple[str, str]]] = [ # topic, blueprint"("topic1", CASE_BLUEPRINT_1),("topic2", CASE_BLUEPRINT_2),]@@ -156,11 +96,12 @@ Also note that we provide a topic for the test case, which is purely for conveni## Best Practices for Testing+- Whenever possible, use strong asserts to test value, not just type and presence.- Use parametrize for multiple test cases- Test both success and failure cases- Verify working memory state- Check output structure and content- Use meaningful test case names-- Include docstrings explaining test purpose+- Include docstrings explaining test purpose but not on top of the file- Log outputs for debugging- Generate reports for cost trackingdiff --git a/.cursor/rules/python_standards.mdc b/.cursor/rules/python_standards.mdcnew file mode 100644index 000000000..13d3c5a92--- /dev/null+++ b/.cursor/rules/python_standards.mdc@@ -0,0 +1,175 @@+---+alwaysApply: false+description: Python coding standards and best practices+globs:+- '**/*.py'+---+# Coding Standards & Best Practices for Python Code++This document outlines the core coding standards, best practices, and quality control procedures for the codebase.++## Variables, loops and indexes++ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.++## Enums and tests++ - When defining enums related to string values, always inherit from `StrEnum`+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.++## Imports++### **Imports at the top of the file**++ - Import all necessary libraries at the top of the file+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.++- **Logging and Pretty Printing**:++ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:+ ```python+ from pipelex import log, pretty_print++ log.info("Hello, world!")+ ```+ - Both have a title arg which is handy when logging/printing objects:++ ```python+ log.debug("Hello, world!", title="Your first Pipelex log")+ pretty_print(output_object, title="Your first Pipelex output")+ ```+ - Both handle formatting json using Rich, pretty_print makes it prettier.++- **StrEnum and Self type**:++ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import Self, StrEnum+ ```++## Typing++### **Always Use Type Hints**++ - Every function parameter must be typed+ - Every function return must be typed+ - Use type hints for all variables where type is not obvious+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]+ - Use type hints for all fields+ - Use Field(default_factory=...) for mutable defaults+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.++### **BaseModel / Pydantic Standards**++ - Use `BaseModel` and respect Pydantic v2 standards+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`+ - Keep models focused and single-purpose+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:+ ```python+ from pydantic import BaseModel, Field+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++ class MyModel(BaseModel):+ names: list[str] = Field(default_factory=list) # OK for strings+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")+ ```++## Factory Pattern++ - Use Factory Pattern for object creation when dealing with multiple implementations+ - Our factory methods are named `make_from_...` and such++## Error Handling++ - Always catch exceptions at the place where you can add useful context to it.+ - Use try/except blocks with specific exceptions+ - Convert third-party exceptions to our custom ones+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands+ - NEVER raise generic exceptions like ValueError or TypeError, create new error classes and raise them instead+ - Always add `from exc` to the exception++ ```python+ try:+ self.models_manager.setup()+ except RoutingProfileLibraryNotFoundError as exc:+ msg = "The routing library could not be found, please call `pipelex init config` to create it"+ raise PipelexSetupError(msg) from exc+ ```++ **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.++## Documentation++1. **Docstring Format**+ ```python+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:+ """Process and resize an image.++ Args:+ image_path: Path to the source image+ size: Tuple of (width, height) for resizing++ Returns:+ Processed image as bytes+ """+ pass+ ```++2. **Class Documentation**+ ```python+ class ImageProcessor:+ """Handles image processing operations.++ Provides methods for resizing, converting, and optimizing images.+ """+ ```++## Code Quality Checks++### Linting and Type Checking++Before finalizing a task, run:+```bash+make fix-unused-imports+make check+```++This runs multiple code quality tools:+- Pyright: Static type checking+- Ruff: Fast Python linter+- Mypy: Static type checker++Always fix any issues reported by these tools before proceeding.++### Running Tests++1. **Quick Test Run** (no LLM/image generation):+ ```bash+ make tp+ ```+ Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)`++2. **Specific Tests**:+ ```bash+ make tp TEST=TestClassName+ # or+ make tp TEST=test_function_name+ ```+ Note: Matches names starting with the provided string.++**Important**: Never run `make ti`, `make test-inference`, `make te`, `make test-extract`, `make tg`, or `make test-img-gen` - these use costly inference.++## Pipelines++- Always validate pipelines after creation/edit with `make validate`.+ Iterate if there are errors.++## Project Structure++- **Tests**: `tests/` directory+- **Documentation**: `docs/` directorydiff --git a/.cursor/rules/run_pipelex.mdc b/.cursor/rules/run_pipelex.mdcnew file mode 100644index 000000000..5f29c45ab--- /dev/null+++ b/.cursor/rules/run_pipelex.mdc@@ -0,0 +1,231 @@+---+alwaysApply: false+description: Guidelines for running Pipelex pipelines+globs:+- examples/**/*.py+---+# Guide to execute a pipeline and write example code++## Example to execute a pipeline with text output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+++async def hello_world() -> str:+ """+ This function demonstrates the use of a super simple Pipelex pipeline to generate text.+ """+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="hello_world",+ )++ return pipe_output.main_stuff_as_str+++# start Pipelex+Pipelex.make()+# run sample using asyncio+output_text = asyncio.run(hello_world())+pretty_print(output_text, title="Your first Pipelex output")+```++## Example to execute a pipeline with structured output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent++from my_project.gantt.gantt_struct import GanttChart++SAMPLE_NAME = "extract_gantt"+IMAGE_URL = "assets/gantt/gantt_tree_house.png"+++async def extract_gantt(image_url: str) -> GanttChart:+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )+ # Output the result+ return pipe_output.main_stuff_as(content_type=GanttChart)+++# start Pipelex+Pipelex.make()++# run sample using asyncio+gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))+pretty_print(gantt_chart, title="Gantt Chart")+```++## Setting up the input memory++### Explanation of input memory++The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:+```python+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]+```+As you can seen, we made it so different ways can be used to define that stuff using structured content or data.++### Different ways to set up the input memory++So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:++```python+# Here we have a single input and it's a Text.+# If you assign a string, by default it will be considered as a TextContent.+ pipe_output = await execute_pipeline(+ pipe_code="master_advisory_orchestrator",+ input_memory={+ "user_input": problem_description,+ },+ )++# Here we have a single input and it's a PDF.+# Because PDFContent is a native concept, we can use it directly as a value,+# the system knows what content it corresponds to:+ pipe_output = await execute_pipeline(+ pipe_code="power_extractor_dpe",+ input_memory={+ "document": PDFContent(url=pdf_url),+ },+ )++# Here we have a single input and it's an Image.+# Because ImageContent is a native concept, we can use it directly as a value:+ pipe_output = await execute_pipeline(+ pipe_code="fashion_variation_pipeline",+ input_memory={+ "fashion_photo": ImageContent(url=image_url),+ },+ )++# Here we have a single input, it's an image but+# its actually a more specific concept gantt.GanttImage which refines Image,+# so we must provide it using a dict with the concept and the content:+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )++# Here is a more complex example with multiple inputs assigned using different ways:+ pipe_output = await execute_pipeline(+ pipe_code="retrieve_then_answer",+ dynamic_output_concept_code="contracts.Fees",+ input_memory={+ "text": load_text_from_path(path=text_path),+ "question": {+ "concept": "answer.Question",+ "content": question,+ },+ "client_instructions": client_instructions,+ },+ )+```++## Using the outputs of a pipeline++All pipe executions return a `PipeOutput` object.+It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.+It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:++```python++class PipeOutput(BaseModel):+ working_memory: WorkingMemory = Field(default_factory=WorkingMemory)+ pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)++ @property+ def main_stuff(self) -> Stuff:+ ...++ def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:+ ...++ def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:+ ...++ def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:+ ...++ @property+ def main_stuff_as_text(self) -> TextContent:+ ...++ @property+ def main_stuff_as_str(self) -> str:+ ...++ @property+ def main_stuff_as_image(self) -> ImageContent:+ ...++ @property+ def main_stuff_as_text_and_image(self) -> TextAndImagesContent:+ ...++ @property+ def main_stuff_as_number(self) -> NumberContent:+ ...++ @property+ def main_stuff_as_html(self) -> HtmlContent:+ ...++ @property+ def main_stuff_as_mermaid(self) -> MermaidContent:+ ...+```++As you can see, you can extract any variable from the output working memory.++### Getting the main stuff as a specific type++Simple text as a string:++```python+result = pipe_output.main_stuff_as_str+```+Structured object (BaseModel):++```python+result = pipe_output.main_stuff_as(content_type=GanttChart)+```++If it's a list, you can get a `ListContent` of the specific type.++```python+result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)+```++or if you want, you can get the actual items as a regular python list:++```python+result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)+```++---+diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdcindex 575544a6f..4b4f058b5 100644--- a/.cursor/rules/tdd.mdc+++ b/.cursor/rules/tdd.mdc@@ -1,7 +1,6 @@----description:-globs:alwaysApply: false+description: Guidelines for writing test-driven development code---# Test-Driven Development Guide@@ -26,3 +25,4 @@ If the code needs refactoring, with the best practices [coding_standards.mdc](co5. **Validate tests**Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.+diff --git a/.cursor/rules/write_pipelex.mdc b/.cursor/rules/write_pipelex.mdcnew file mode 100644index 000000000..6e3339c1e--- /dev/null+++ b/.cursor/rules/write_pipelex.mdc@@ -0,0 +1,837 @@+---+alwaysApply: false+description: Guidelines for writing Pipelex pipelines+globs:+- '**/*.plx'+- '**/pipelines/**/*.py'+---+# Guide to write or edit pipelines using the Pipelex language in .plx files++- Always first write your "plan" in natural language, then transcribe it in pipelex.+- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)++## Pipeline File Naming+- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)+- Files must be `.py` for code defining the data structures+- Use descriptive names in `snake_case`++## Pipeline File Outline+A pipeline file has three main sections:+1. Domain statement+2. Concept definitions+3. Pipe definitions++### Domain Statement+```plx+domain = "domain_name"+description = "Description of the domain" # Optional+```+Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.++### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.++```plx+[concept]+ConceptName = "Description of the concept"+```++**Naming Rules:**+- Use PascalCase for concept names+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)++**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.++**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:++```plx+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"+```++For details on how to structure concepts with fields, see the "Structuring Models" section below.++### Pipe Definitions++## Pipe Base Definition++```plx+[pipe.your_pipe_name]+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"+```++The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).+So If you have this error:+`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •+variable='['invoice']'``+That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.++NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:+<NEVER DO THIS>+```plx+inputs = {+ input_1 = "ConceptName1",+ input_2 = "ConceptName2"+}+```+</NEVER DO THIS>++- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:++## Structuring Models++Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.++### Three Ways to Structure Concepts++**1. No Structure Needed**++If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.++**2. Inline Structure Definition (RECOMMENDED for most cases)**++For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }+```++**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):+```plx+field_name = "Field description"+```++**Detailed syntax** (with explicit properties):+```plx+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }+```++**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator++class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```++**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.++### Decision Rules for Agents++**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class++**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)++**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required++### Inline Structure Limitations++Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes+++## Pipe Controllers and Pipe Operators++Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:++1. **Controllers** - For flow control:+ - `PipeSequence` - For creating a sequence of multiple steps+ - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory+ - `PipeParallel` - For parallelizing pipes++2. **Operators** - For specific tasks:+ - `PipeLLM` - Generate Text and Objects (include Vision LLM)+ - `PipeExtract` - Extract text and images from an image or a PDF+ - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.+ - `PipeImgGen` - Generate Images+ - `PipeFunc` - For running classic python scripts++## PipeSequence controller++Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.++### Basic Definition+```plx+[pipe.your_sequence_name]+type = "PipeSequence"+description = "Description of what this sequence does"+inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps+output = "OutputType"+steps = [+ { pipe = "first_pipe", result = "first_result" },+ { pipe = "second_pipe", result = "second_result" },+ { pipe = "final_pipe", result = "final_result" }+]+```++### Key Components++1. **Steps Array**: List of pipes to execute in sequence+ - `pipe`: Name of the pipe to execute+ - `result`: Name to assign to the pipe's output that will be in the working memory++### Using PipeBatch in Steps++You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:++```plx+steps = [+ { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"+ }+]+```++1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.+ - Must be a `ListContent` type containing the items to process+ - Can reference inputs or results from previous steps++2. **batch_as**: Defines the name that will be used to reference the current item being processed+ - This name can be used in the pipe's input mappings+ - Makes each item from the batch available as a single element++The result of a batched step will be a `ListContent` containing the outputs from processing each item.++## PipeCondition controller++The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.++### Basic usage++```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression = "input_data.category"+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```+or+```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```++### Key Parameters++- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)+- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"++[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++## PipeLLM operator++PipeLLM is used to:+1. Generate text or objects with LLMs+2. Process images with Vision LLMs++### Basic Usage++Simple Text Generation:+```plx+[pipe.write_story]+type = "PipeLLM"+description = "Write a short story"+output = "Text"+prompt = """+Write a short story about a programmer.+"""+```++Structured Data Extraction:+```plx+[pipe.extract_info]+type = "PipeLLM"+description = "Extract information"+inputs = { text = "Text" }+output = "PersonInfo"+prompt = """+Extract person information from this text:+@text+"""+```++Supports system instructions:+```plx+[pipe.expert_analysis]+type = "PipeLLM"+description = "Expert analysis"+output = "Analysis"+system_prompt = "You are a data analysis expert"+prompt = "Analyze this data"+```++### Multiple Outputs++Generate multiple outputs (fixed number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+nb_output = 3 # Generate exactly 3 ideas+```++Generate multiple outputs (variable number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+multiple_output = true # Let the LLM decide how many to generate+```++### Vision++Process images with VLMs (image inputs must be tagged in the prompt):+```plx+[pipe.analyze_image]+type = "PipeLLM"+description = "Analyze image"+inputs = { image = "Image" }+output = "ImageAnalysis"+prompt = """+Describe what you see in this image:++$image+"""+```++You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++### Writing prompts for PipeLLM++**Insert stuff inside a tagged block**++If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.++Example template:+```plx+prompt = """+Match the expense with its corresponding invoice:++@expense++@invoices+"""+```+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.++DO NOT write things like "Here is the expense: @expense".+DO write simply "@expense" alone in an isolated line.++**Insert stuff inline**++If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.++Example template:+```plx+prompt = """+Your goal is to summarize everything related to $topic in the provided text:++@text++Please provide only the summary, with no additional text or explanations.+Your summary should not be longer than 2 sentences.+"""+```++In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.+Be sure to make the proper choice of prefix for each insertion.++DO NOT write "$topic" alone in an isolated line.+DO write things like "Write an essay about $topic" to include text into an actual sentence.+++## PipeExtract operator++The PipeExtract operator is used to extract text and images from an image or a PDF++### Simple Text Extraction+```plx+[pipe.extract_info]+type = "PipeExtract"+description = "extract the information"+inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.+output = "Page"+```++Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.++The output concept `Page` is a native concept, with the structure `PageContent`:+It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`++```python+class TextAndImagesContent(StuffContent):+ text: TextContent | None+ images: list[ImageContent] | None++class PageContent(StructuredContent): # CONCEPT IS "Page"+ text_and_images: TextAndImagesContent+ page_view: ImageContent | None = None+```+- `text_and_images` are the text, and the related images found in the input image or PDF.+- `page_view` is the screenshot of the whole pdf page/image.++## PipeCompose operator++The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.++### Basic Usage++Simple Template Composition:+```plx+[pipe.compose_report]+type = "PipeCompose"+description = "Compose a report using template"+inputs = { data = "ReportData" }+output = "Text"+template = """+# Report Summary++Based on the analysis:+$data++Generated on: {{ current_date }}+"""+```++Using Named Templates:+```plx+[pipe.use_template]+type = "PipeCompose"+description = "Use a predefined template"+inputs = { content = "Text" }+output = "Text"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+```++CRM Email Template:+```plx+[pipe.compose_follow_up_email]+type = "PipeCompose"+description = "Compose a personalized follow-up email for CRM"+inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }+output = "Text"+template_category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+template = """+Subject: Following up on our $deal.product_name discussion++Hi $customer.first_name,++I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.++Based on our discussion, I understand that your key requirements are: $deal.customer_requirements++I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:++**Next Steps:**+- Schedule a demo tailored to your specific needs+- Provide you with a customized quote based on your requirements+- Connect you with our implementation team++Would you be available for a 30-minute call this week? I have openings on:+{% for slot in available_slots %}+- {{ slot }}+{% endfor %}++Looking forward to moving this forward together!++Best regards,+$sales_rep.name+$sales_rep.title+$sales_rep.phone | $sales_rep.email+"""+```++### Key Parameters++- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)+- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)+- `templating_style`: Styling options for template rendering+- `extra_context`: Additional context variables for template++For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++### Template Variables++Use the same variable insertion rules as PipeLLM:+- `@variable` for block insertion (multi-line content)+- `$variable` for inline insertion (short text)++## PipeImgGen operator++The PipeImgGen operator is used to generate images using AI image generation models.++### Basic Usage++Simple Image Generation:+```plx+[pipe.generate_image]+type = "PipeImgGen"+description = "Generate an image from prompt"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+```++Using Image Generation Settings:+```plx+[pipe.generate_photo]+type = "PipeImgGen"+description = "Generate a high-quality photo"+inputs = { prompt = "ImgGenPrompt" }+output = "Photo"+model = { model = "fast-img-gen" }+aspect_ratio = "16:9"+quality = "hd"+```++Multiple Image Generation:+```plx+[pipe.generate_variations]+type = "PipeImgGen"+description = "Generate multiple image variations"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+nb_output = 3+seed = "auto"+```++Advanced Configuration:+```plx+[pipe.generate_custom]+type = "PipeImgGen"+description = "Generate image with custom settings"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+model = "img_gen_preset_name" # Use predefined preset+aspect_ratio = "1:1"+quality = "hd"+background = "transparent"+output_format = "png"+is_raw = false+safety_tolerance = 3+```++### Key Parameters++**Image Generation Settings:**+- `model`: Model choice (preset name or inline settings with model name)+- `quality`: Image quality ("standard", "hd")++**Output Configuration:**+- `nb_output`: Number of images to generate+- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)+- `output_format`: File format ("png", "jpeg", "webp")+- `background`: Background type ("default", "transparent")++**Generation Control:**+- `seed`: Random seed (integer or "auto")+- `is_raw`: Whether to apply post-processing+- `is_moderated`: Enable content moderation+- `safety_tolerance`: Content safety level (1-6)++### Input Requirements++PipeImgGen requires exactly one input that must be either:+- An `ImgGenPrompt` concept+- A concept that refines `ImgGenPrompt`++The input can be named anything but must contain the prompt text for image generation.++## PipeFunc operator++The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.++### Basic Usage++Simple Function Call:+```plx+[pipe.process_data]+type = "PipeFunc"+description = "Process data using custom function"+inputs = { input_data = "DataType" }+output = "ProcessedData"+function_name = "process_data_function"+```++File Processing Example:+```plx+[pipe.read_file]+type = "PipeFunc"+description = "Read file content"+inputs = { file_path = "FilePath" }+output = "FileContent"+function_name = "read_file_content"+```++### Key Parameters++- `function_name`: Name of the Python function to call (must be registered in func_registry)++### Function Requirements++The Python function must:++1. **Be registered** in the `func_registry`+2. **Accept `working_memory`** as a parameter:+ ```python+ async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:+ # Function implementation+ pass+ ```++3. **Return appropriate types**:+ - `StuffContent`: Single content object+ - `list[StuffContent]`: Multiple content objects (becomes ListContent)+ - `str`: Simple string (becomes TextContent)++### Function Registration++Functions must be registered in the function registry before use:++```python+from pipelex.system.registries.func_registry import func_registry++@func_registry.register("my_function_name")+async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:+ # Access inputs from working memory+ input_data = working_memory.get_stuff("input_name")++ # Process data+ result = process_logic(input_data.content)++ # Return result+ return MyResultContent(data=result)+```++### Working Memory Access++Inside the function, access pipeline inputs through working memory:++```python+async def process_function(working_memory: WorkingMemory) -> TextContent:+ # Get input stuff by name+ input_stuff = working_memory.get_stuff("input_name")++ # Access the content+ input_content = input_stuff.content++ # Process and return+ processed_text = f"Processed: {input_content.text}"+ return TextContent(text=processed_text)+```++---++## Rules to choose LLM models used in PipeLLMs.++### LLM Configuration System++In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:++- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`++### LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++### Using an LLM Handle in a PipeLLM++Here is an example of using a model to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""+```++As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").++### LLM Presets++Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:++```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:++The category of this invoice is: $invoice_details.category.++@invoice_text+"""+```++The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.++---++ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+Then, create an example file to run the pipeline in the `examples` folder.+But don't write documentation unless asked explicitly to.diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.mdindex f54bd0b20..4d612f7da 100644--- a/.github/copilot-instructions.md+++ b/.github/copilot-instructions.md@@ -1,195 +1,71 @@-# Coding Standards & Best Practices+<!-- BEGIN_PIPELEX_RULES -->+## Guide to write or edit pipelines using the Pipelex language in .plx files-This document outlines the core coding standards, best practices, and quality control procedures for the codebase.--## Type Hints--1. **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tupele types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use the `|` syntax for union types (e.g `str | int`) and `| None` for optionals- - Use Field(default_factory=...) for mutable defaults and if it's a list of something else than str, use `empty_list_factory_of()` to make a factory: `number_list: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")`- - Use `BaseModel` and respect Pydantic v2 standards, in particular use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose--2. **StrEnum**- - Import from `pipelex.types`:- ```python- from pipelex.types import StrEnum- ```--3. **Self type**- - Import from `pipelex.types`:- ```python- from pipelex.types import Self- ```--## Factory Pattern-- - Use Factory Pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such--## Error Handling-- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones- - Never catch Exception, only catch specific exceptions- - Always add `from exc` to the exception-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```-- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.--## Documentation--1. **Docstring Format**- ```python- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:- """Process and resize an image.-- Args:- image_path: Path to the source image- size: Tuple of (width, height) for resizing-- Returns:- Processed image as bytes- """- pass- ```--2. **Class Documentation**- ```python- class ImageProcessor:- """Handles image processing operations.-- Provides methods for resizing, converting, and optimizing images.- """- ```--## Code Quality Checks--### Linting and Type Checking--Before finalizing a task, run:-```bash-make fix-unused-imports-make check-```--This runs multiple code quality tools:-- Pyright: Static type checking-- Ruff: Fast Python linter-- Mypy: Static type checker--Always fix any issues reported by these tools before proceeding.--### Running Tests--1. **Quick Test Run** (no LLM/image generation):- ```bash- make tp- ```- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or ocr)) and not (needs_output or pipelex_api)`--2. **Specific Tests**:- ```bash- make tp TEST=TestClassName- # or- make tp TEST=test_function_name- ```- Note: Matches names starting with the provided string.--**Important**: Never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-img-gen` - these use costly inference.--## Pipelines--- All pipeline definitions go in `pipelex/libraries/pipelines/`-- Always validate pipelines after creation/edit with `make validate`.- Iterate if there are errors.--## Project Structure--- **Pipelines**: `pipelex/libraries/pipelines/`-- **Tests**: `tests/` directory-- **Documentation**: `docs/` directory-------# Guide to write or edit pipelines using the Pipelex language in .plx files--- Always first write your "plan" in natural langage, then transcribe it in pipelex.+- Always first write your "plan" in natural language, then transcribe it in pipelex.- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.-- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)-## Pipeline File Naming+### Pipeline File Naming- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)-- Files must be `.py` for structures+- Files must be `.py` for code defining the data structures- Use descriptive names in `snake_case`-## Pipeline File Structure+### Pipeline File OutlineA pipeline file has three main sections:1. Domain statement2. Concept definitions3. Pipe definitions-### Domain Statement+#### Domain Statement```plxdomain = "domain_name"description = "Description of the domain" # Optional```Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.-### Concept Definitions+#### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.+```plx[concept]-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output+ConceptName = "Description of the concept"```-Important Rules:+**Naming Rules:**- Use PascalCase for concept names-- Never use plurals (no "Stories", use "Story")-- Avoid adjectives (no "LargeText", use "Text")-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number)-yes-### Pipe Definitions+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)-## Pipe Base Structure+**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.-```plx-[pipe.your_pipe_name]-type = "PipeLLM"-description = "A description of what your pipe does"-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }-output = "ConceptName"-```+**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:-DO NOT WRITE:```plx-[pipe.your_pipe_name]-type = "pipe_sequence"+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"```-But it should be:+For details on how to structure concepts with fields, see the "Structuring Models" section below.++#### Pipe Definitions++### Pipe Base Definition```plx[pipe.your_pipe_name]-type = "PipeSequence"-description = "....."+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"```-The pipes will all have at least this base structure.-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditionnal pipes (if PipeCondition).+The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).So If you have this error:`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •variable='['invoice']'``@@ -207,74 +83,108 @@ inputs = {- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:-## Structuring Models+### Structuring Models-### Model Location and Registration+Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.-- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`-- Models must inherit from `StructuredContent` or appropriate content type+#### Three Ways to Structure Concepts-## Model Structure+**1. No Structure Needed**-Concepts and their structure classes are meant to indicate an idea.-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.+If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.-**IMPORTANT: Never create unnecessary structure classes that only refine native concepts without adding fields.**+**2. Inline Structure Definition (RECOMMENDED for most cases)**-DO NOT create structures like:-```python-class Joke(TextContent):- """A humorous text that makes people laugh."""- pass+For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }```-If a concept only refines a native concept (like Text, Image, etc.) without adding new fields, simply declare it in the .plx file:+**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):```plx-[concept]-Joke = "A humorous text that makes people laugh."+field_name = "Field description"```-If you simply need to refine another native concept, construct it like this:++**Detailed syntax** (with explicit properties):```plx-[concept.Landscape]-refines = "Image"+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }```-Only create a Python structure class when you need to add specific fields:-```python-from datetime import datetime-from typing import List, Optional-from pydantic import Field+**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains+```pythonfrom pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator-# IMPORTANT: THE CLASS MUST BE A SUBCLASS OF StructuredContent-class YourModel(StructuredContent): # Always be a subclass of StructuredContent- # Required fields- field1: str- field2: int+class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```- # Optional fields with defaults- field3: Optional[str] = Field(None, "Description of field3")- field4: List[str] = Field(default_factory=list)+**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.- # Date fields should remove timezone- date_field: Optional[datetime] = None-```-### Usage+#### Decision Rules for Agents-Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.+**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class-Structure classes defined within `pipelex_libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.+**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)+**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required-### Best Practices for structures+#### Inline Structure Limitations-- Respect Pydantic v2 standards-- Use type hints for all fields-- Use `Field` declaration and write the description+Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes-## Pipe Controllers and Pipe Operators+### Pipe Controllers and Pipe OperatorsLook at the Pipes we have in order to adapt it. Pipes are organized in two categories:@@ -290,11 +200,11 @@ Look at the Pipes we have in order to adapt it. Pipes are organized in two categ- `PipeImgGen` - Generate Images- `PipeFunc` - For running classic python scripts-## PipeSequence controller+### PipeSequence controllerPurpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.-### Basic Structure+#### Basic Definition```plx[pipe.your_sequence_name]type = "PipeSequence"@@ -308,13 +218,13 @@ steps = []```-### Key Components+#### Key Components1. **Steps Array**: List of pipes to execute in sequence- `pipe`: Name of the pipe to execute- `result`: Name to assign to the pipe's output that will be in the working memory-### Using PipeBatch in Steps+#### Using PipeBatch in StepsYou can use PipeBatch functionality within steps using `batch_over` and `batch_as`:@@ -335,21 +245,22 @@ steps = [The result of a batched step will be a `ListContent` containing the outputs from processing each item.-## PipeCondition controller+### PipeCondition controllerThe PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.-### Basic usage+#### Basic usage```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"@@ -358,32 +269,49 @@ or```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"```-### Key Parameters+#### Key Parameters- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)-- `pipe_map`: Dictionary mapping expression results to pipe codes :-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"-## PipeLLM operator+[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++### PipeLLM operatorPipeLLM is used to:1. Generate text or objects with LLMs2. Process images with Vision LLMs-### Basic Usage+#### Basic UsageSimple Text Generation:```plx@@ -391,7 +319,7 @@ Simple Text Generation:type = "PipeLLM"description = "Write a short story"output = "Text"-prompt_template = """+prompt = """Write a short story about a programmer."""```@@ -403,7 +331,7 @@ type = "PipeLLM"description = "Extract information"inputs = { text = "Text" }output = "PersonInfo"-prompt_template = """+prompt = """Extract person information from this text:@text"""@@ -416,10 +344,10 @@ type = "PipeLLM"description = "Expert analysis"output = "Analysis"system_prompt = "You are a data analysis expert"-prompt_template = "Analyze this data"+prompt = "Analyze this data"```-### Multiple Outputs+#### Multiple OutputsGenerate multiple outputs (fixed number):```plx@@ -439,19 +367,33 @@ output = "Idea"multiple_output = true # Let the LLM decide how many to generate```-### Vision+#### Vision-Process images with VLMs:+Process images with VLMs (image inputs must be tagged in the prompt):```plx[pipe.analyze_image]type = "PipeLLM"description = "Analyze image"-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in an attribute within a stuff, you can add something like `{ "page.image": "Image" }+inputs = { image = "Image" }output = "ImageAnalysis"-prompt_template = "Describe what you see in this image"+prompt = """+Describe what you see in this image:++$image+"""```-### Writing prompts for PipeLLM+You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++#### Writing prompts for PipeLLM**Insert stuff inside a tagged block**@@ -459,7 +401,7 @@ If the inserted text is supposedly a long text, made of several lines or paragraExample template:```plx-prompt_template = """+prompt = """Match the expense with its corresponding invoice:@expense@@ -467,7 +409,7 @@ Match the expense with its corresponding invoice:@invoices"""```-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.DO NOT write things like "Here is the expense: @expense".DO write simply "@expense" alone in an isolated line.@@ -478,7 +420,7 @@ If the inserted text is short text and it makes sense to have it inserted directExample template:```plx-prompt_template = """+prompt = """Your goal is to summarize everything related to $topic in the provided text:@text@@ -495,11 +437,11 @@ DO NOT write "$topic" alone in an isolated line.DO write things like "Write an essay about $topic" to include text into an actual sentence.-## PipeExtract operator+### PipeExtract operatorThe PipeExtract operator is used to extract text and images from an image or a PDF-### Simple Text Extraction+#### Simple Text Extraction```plx[pipe.extract_info]type = "PipeExtract"@@ -508,28 +450,38 @@ inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This isoutput = "Page"```-Only one input is allowed and it must either be an `Image` or a `PDF`.+Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.The output concept `Page` is a native concept, with the structure `PageContent`:It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page````pythonclass TextAndImagesContent(StuffContent):- text: Optional[TextContent]- images: Optional[List[ImageContent]]+ text: TextContent | None+ images: list[ImageContent] | Noneclass PageContent(StructuredContent): # CONCEPT IS "Page"text_and_images: TextAndImagesContent- page_view: Optional[ImageContent] = None+ page_view: ImageContent | None = None```- `text_and_images` are the text, and the related images found in the input image or PDF.- `page_view` is the screenshot of the whole pdf page/image.-## PipeCompose operator+### PipeCompose operatorThe PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.-### Basic Usage+#### Basic UsageSimple Template Composition:```plx@@ -538,8 +490,8 @@ type = "PipeCompose"description = "Compose a report using template"inputs = { data = "ReportData" }output = "Text"-jinja2 = """-# Report Summary+template = """+## Report SummaryBased on the analysis:$data@@ -555,7 +507,21 @@ type = "PipeCompose"description = "Use a predefined template"inputs = { content = "Text" }output = "Text"-jinja2_name = "standard_report_template"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }```CRM Email Template:@@ -567,7 +533,7 @@ inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }output = "Text"template_category = "html"templating_style = { tag_style = "square_brackets", text_format = "html" }-jinja2 = """+template = """Subject: Following up on our $deal.product_name discussionHi $customer.first_name,@@ -597,25 +563,30 @@ $sales_rep.phone | $sales_rep.email"""```-### Key Parameters+#### Key Parameters-- `jinja2`: Inline Jinja2 template (mutually exclusive with jinja2_name)-- `jinja2_name`: Name of a predefined template (mutually exclusive with jinja2)+- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)- `templating_style`: Styling options for template rendering- `extra_context`: Additional context variables for template-### Template Variables+For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++#### Template VariablesUse the same variable insertion rules as PipeLLM:- `@variable` for block insertion (multi-line content)- `$variable` for inline insertion (short text)-## PipeImgGen operator+### PipeImgGen operatorThe PipeImgGen operator is used to generate images using AI image generation models.-### Basic Usage+#### Basic UsageSimple Image Generation:```plx@@ -633,9 +604,9 @@ type = "PipeImgGen"description = "Generate a high-quality photo"inputs = { prompt = "ImgGenPrompt" }output = "Photo"-model = { model = "flux-pro/v1.1-ultra", quality = "hd" }+model = { model = "fast-img-gen" }aspect_ratio = "16:9"-nb_steps = 8+quality = "hd"```Multiple Image Generation:@@ -665,14 +636,11 @@ is_raw = falsesafety_tolerance = 3```-### Key Parameters+#### Key Parameters**Image Generation Settings:**-- `img_gen`: ImgGenChoice (preset name or inline settings)-- `img_gen_handle`: Direct model handle (legacy)+- `model`: Model choice (preset name or inline settings with model name)- `quality`: Image quality ("standard", "hd")-- `nb_steps`: Number of generation steps-- `guidance_scale`: How closely to follow the prompt**Output Configuration:**- `nb_output`: Number of images to generate@@ -686,7 +654,7 @@ safety_tolerance = 3- `is_moderated`: Enable content moderation- `safety_tolerance`: Content safety level (1-6)-### Input Requirements+#### Input RequirementsPipeImgGen requires exactly one input that must be either:- An `ImgGenPrompt` concept@@ -694,11 +662,11 @@ PipeImgGen requires exactly one input that must be either:The input can be named anything but must contain the prompt text for image generation.-## PipeFunc operator+### PipeFunc operatorThe PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.-### Basic Usage+#### Basic UsageSimple Function Call:```plx@@ -720,11 +688,11 @@ output = "FileContent"function_name = "read_file_content"```-### Key Parameters+#### Key Parameters- `function_name`: Name of the Python function to call (must be registered in func_registry)-### Function Requirements+#### Function RequirementsThe Python function must:@@ -741,12 +709,12 @@ The Python function must:- `list[StuffContent]`: Multiple content objects (becomes ListContent)- `str`: Simple string (becomes TextContent)-### Function Registration+#### Function RegistrationFunctions must be registered in the function registry before use:```python-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registry@func_registry.register("my_function_name")async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:@@ -760,7 +728,7 @@ async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:return MyResultContent(data=result)```-### Working Memory Access+#### Working Memory AccessInside the function, access pipeline inputs through working memory:@@ -779,9 +747,9 @@ async def process_function(working_memory: WorkingMemory) -> TextContent:----## Rules to choose LLM models used in PipeLLMs.+### Rules to choose LLM models used in PipeLLMs.-### LLM Configuration System+#### LLM Configuration SystemIn order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:@@ -790,7 +758,7 @@ LLM configurations are managed through the new inference backend system with fil- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`- **Routing**: `.pipelex/inference/routing_profiles.toml`-### LLM Handles+#### LLM HandlesAn llm_handle can be either:1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system@@ -806,9 +774,9 @@ base-mistral = "mistral-medium"The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-### Using an LLM Handle in a PipeLLM+#### Using an LLM Handle in a PipeLLM-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:+Here is an example of using a model to specify which LLM to use in a PipeLLM:```plx[pipe.hello_world]@@ -816,14 +784,14 @@ type = "PipeLLM"description = "Write text about Hello World."output = "Text"model = { model = "gpt-5", temperature = 0.9 }-prompt_template = """+prompt = """Write a haiku about Hello World."""```As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-### LLM Presets+#### LLM PresetsPresets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.@@ -842,7 +810,7 @@ description = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"model = "llm_to_extract_invoice"-prompt_template = """+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -862,11 +830,9 @@ ALWAYS RUN `make validate` when you are finished writing pipelines: This checksThen, create an example file to run the pipeline in the `examples` folder.But don't write documentation unless asked explicitly to.------# Guide to write an example to execute a pipeline+## Guide to execute a pipeline and write example code-## Example to execute a pipeline with text output+### Example to execute a pipeline with text output```pythonimport asyncio@@ -888,14 +854,14 @@ async def hello_world() -> str:return pipe_output.main_stuff_as_str-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciooutput_text = asyncio.run(hello_world())pretty_print(output_text, title="Your first Pipelex output")```-## Example to execute a pipeline with structured output+### Example to execute a pipeline with structured output```pythonimport asyncio@@ -903,8 +869,9 @@ import asynciofrom pipelex import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent-from pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart+from my_project.gantt.gantt_struct import GanttChartSAMPLE_NAME = "extract_gantt"IMAGE_URL = "assets/gantt/gantt_tree_house.png"@@ -925,32 +892,32 @@ async def extract_gantt(image_url: str) -> GanttChart:return pipe_output.main_stuff_as(content_type=GanttChart)-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciogantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))pretty_print(gantt_chart, title="Gantt Chart")```-## Setting up the input memory+### Setting up the input memory-### Explanation of input memory+#### Explanation of input memoryThe input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:```python-StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str-ImplicitMemory = Dict[str, StuffContentOrData]+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]```As you can seen, we made it so different ways can be used to define that stuff using structured content or data.-### Different ways to set up the input memory+#### Different ways to set up the input memorySo here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:```python-# Here we have a single input and it's a Text.-# If you assign a string, by default it will be considered as a TextContent.+## Here we have a single input and it's a Text.+## If you assign a string, by default it will be considered as a TextContent.pipe_output = await execute_pipeline(pipe_code="master_advisory_orchestrator",input_memory={@@ -958,9 +925,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's a PDF.-# Because PDFContent is a native concept, we can use it directly as a value,-# the system knows what content it corresponds to:+## Here we have a single input and it's a PDF.+## Because PDFContent is a native concept, we can use it directly as a value,+## the system knows what content it corresponds to:pipe_output = await execute_pipeline(pipe_code="power_extractor_dpe",input_memory={@@ -968,8 +935,8 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's an Image.-# Because ImageContent is a native concept, we can use it directly as a value:+## Here we have a single input and it's an Image.+## Because ImageContent is a native concept, we can use it directly as a value:pipe_output = await execute_pipeline(pipe_code="fashion_variation_pipeline",input_memory={@@ -977,9 +944,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input, it's an image but-# its actually a more specific concept gantt.GanttImage which refines Image,-# so we must provide it using a dict with the concept and the content:+## Here we have a single input, it's an image but+## its actually a more specific concept gantt.GanttImage which refines Image,+## so we must provide it using a dict with the concept and the content:pipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",input_memory={@@ -990,7 +957,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here is a more complex example with multiple inputs assigned using different ways:+## Here is a more complex example with multiple inputs assigned using different ways:pipe_output = await execute_pipeline(pipe_code="retrieve_then_answer",dynamic_output_concept_code="contracts.Fees",@@ -1005,7 +972,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa)```-## Using the outputs of a pipeline+### Using the outputs of a pipelineAll pipe executions return a `PipeOutput` object.It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.@@ -1059,9 +1026,9 @@ class PipeOutput(BaseModel):...```-As you can see, you can extarct any variable from the output working memory.+As you can see, you can extract any variable from the output working memory.-### Getting the main stuff as a specific type+#### Getting the main stuff as a specific typeSimple text as a string:@@ -1088,161 +1055,81 @@ result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)----# Writing unit tests--## Unit test generalities--NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.--### Test file structure--- Name test files with `test_` prefix-- Use descriptive names that match the functionality being tested-- Place test files in the appropriate test category directory:- - `tests/unit/` - for unit tests that test individual functions/classes in isolation- - `tests/integration/` - for integration tests that test component interactions- - `tests/e2e/` - for end-to-end tests that test complete workflows- - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)-- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest-- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.-- Always put test inside Test classes.-- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`--### Markers--Apply the appropriate markers:-- "llm: uses an LLM to generate text or objects"-- "img_gen: uses an image generation AI"-- "extract: uses ocr models"-- "inference: uses either an LLM or an image generation AI"-- "gha_disabled: will not be able to run properly on GitHub Actions"--Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.--### Important rules--- Never use the unittest.mock. Use pytest-mock.--### Test Class Structure+## Rules to choose LLM models used in PipeLLMs.-Always group the tests of a module into a test class:+### LLM Configuration System-```python[email protected](loop_scope="class")-class TestFooBar:- @pytest.mark.parametrize(- "topic test_case_blueprint",- [- TestCases.CASE_1,- TestCases.CASE_2,- ],- )- async def test_pipe_processing(- self,- request: FixtureRequest,- topic: str,- test_case_blueprint: StuffBlueprint,- ):- # Test implementation-```+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:-Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`-## Writing integration test to test pipes+### LLM Handles-### Required imports for pipe tests+An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:-```python-import pytest-from pytest import FixtureRequest-from pipelex import log, pretty_print-from pipelex.core.stuffs.stuff_factory import StuffBlueprint, StuffFactory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.hub import get_report_delegate-from pipelex.libraries.pipelines.base_library.retrieve import RetrievedExcerpt-from pipelex.config_pipelex import get_config--from pipelex.core.pipe import PipeAbstract, update_job_metadata_for_pipe-from pipelex.core.pipes.pipe_output import PipeOutput, PipeOutputType-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.pipe_works.pipe_router_protocol import PipeRouterProtocol+```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"```-### Pipe test implementation steps--1. Create Stuff from blueprint:+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-```python-stuff = StuffFactory.make_stuff(- concept_code="RetrievedExcerpt",- domain="retrieve",- content=RetrievedExcerpt(text="<Some retrieved text>", justification="<Some justification>")- name="retrieved_text",-)-```+### Using an LLM Handle in a PipeLLM-2. Create Working Memory:+Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:-```python-working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""```-3. Run the pipe:+As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-```python-pipe_output = await pipe_router.run_pipe(- pipe_code="pipe_name",- pipe_run_params=PipeRunParamsFactory.make_run_params(),- working_memory=working_memory,- job_metadata=JobMetadata(),-)-```+### LLM Presets-4. Basic assertions:+Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.-```python-assert pipe_output is not None-assert pipe_output.working_memory is not None-assert pipe_output.main_stuff is not None+Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```-### Test Data Organization+The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:-- If it's not already there, create a `test_data.py` file in the test directory-- Define test cases using `StuffBlueprint`:+```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:-```python-class TestCases:- CASE_BLUEPRINT_1 = StuffBlueprint(- name="test_case_1",- concept_code="domain.ConceptName1",- value="test_value"- )- CASE_BLUEPRINT_2 = StuffBlueprint(- name="test_case_2",- concept_code="domain.ConceptName2",- value="test_value"- )+The category of this invoice is: $invoice_details.category.- CASE_BLUEPRINTS: ClassVar[List[Tuple[str, str]]] = [ # topic, blueprint"- ("topic1", CASE_BLUEPRINT_1),- ("topic2", CASE_BLUEPRINT_2),- ]+@invoice_text+"""```-Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.-Also note that we provide a topic for the test case, which is purely for convenience.+The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.-## Best Practices for Testing-- Use parametrize for multiple test cases-- Test both success and failure cases-- Verify working memory state-- Check output structure and content-- Use meaningful test case names-- Include docstrings explaining test purpose-- Log outputs for debugging-- Generate reports for cost tracking+You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+<!-- END_PIPELEX_RULES -->diff --git a/.gitignore b/.gitignoreindex 7b442388d..00a8bc1ba 100644--- a/.gitignore+++ b/.gitignore@@ -43,9 +43,6 @@ Thumbs.dbreports/results/-# Pipelex libraries, duplicated from pipelex/libraries/ by `pipelex init-libraries`-/pipelex_libraries-# personnal pipelex config file that overrides the default onepipelex_super.tomldiff --git a/.pipelex/inference/backends/pipelex_inference.toml b/.pipelex/inference/backends/pipelex_inference.tomlindex d850714e8..eb82d8c71 100644--- a/.pipelex/inference/backends/pipelex_inference.toml+++ b/.pipelex/inference/backends/pipelex_inference.toml@@ -101,6 +101,13 @@ inputs = ["text", "images"]outputs = ["text", "structured"]costs = { input = 0.10, output = 0.40 }+["gemini-2.5-pro"]+model_id = "gemini-2.5-pro"+inputs = ["text", "images"]+outputs = ["text", "structured"]+max_prompt_images = 3000+costs = { input = 1.25, output = 10.0 }+["gemini-2.5-flash"]model_id = "gemini/gemini-2.5-flash"inputs = ["text", "images"]diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex dddcbf095..e246c84bb 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -1,25 +1,23 @@-[pipelex]-[pipelex.observer_config]-observer_dir = "results/observer"+[pipelex.log_config]+default_log_level = "INFO"-[pipelex.aws_config]-api_key_method = "env"-# The possible values are "env" and "secret_provider".-# "env" means means that the env var are stored in your .env file.-# "secret_provider" means that the env var are stored in your Secret Manager (See the doc for injecting a secret provider).+[pipelex.log_config.package_log_levels]+pipelex = "INFO"-[cogt]+[cogt.llm_config]+# is_dump_text_prompts_enabled = true+# is_dump_response_text_enabled = true-####################################################################################################-# OCR config-####################################################################################################+[cogt.llm_config.instructor_config]+# is_dump_kwargs_enabled = true+# is_dump_response_enabled = true+# is_dump_error_enabled = true-[cogt.extract_config]-page_output_text_file_name = "page_text.md"+[pipelex.observer_config]+observer_dir = "results/observer"[pipelex.feature_config]# WIP/Experimental feature flags-is_pipeline_tracking_enabled = false-is_activity_tracking_enabled = false+# is_pipeline_tracking_enabled = true+# is_activity_tracking_enabled = trueis_reporting_enabled = true-diff --git a/.vscode/launch.json b/.vscode/launch.jsonindex 942c9c079..a9da599ad 100644--- a/.vscode/launch.json+++ b/.vscode/launch.json@@ -16,14 +16,14 @@"justMyCode": false},{- "name": "Debug test without inference",+ "name": "Debug test boot without inference","type": "debugpy","request": "launch","module": "pytest","args": ["-s","-vv",- "-k test_pipe_llm_with_external_llm_handle"+ "-k test_boot"],"console": "integratedTerminal"},diff --git a/.cursor/rules/pipelex.mdc b/.windsurfrules.mdsimilarity index 57%rename from .cursor/rules/pipelex.mdcrename to .windsurfrules.mdindex ca99c0a6d..4d612f7da 100644--- a/.cursor/rules/pipelex.mdc+++ b/.windsurfrules.md@@ -1,72 +1,71 @@-----alwaysApply: false-----# Pipeline Guide+<!-- BEGIN_PIPELEX_RULES -->+## Guide to write or edit pipelines using the Pipelex language in .plx files-- Always first write your "plan" in natural langage, then transcribe it in pipelex.-- You should ALWAYS RUN the terminal command `make validate` when you are writing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.-- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)+- Always first write your "plan" in natural language, then transcribe it in pipelex.+- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)-# Pipeline Structure Guide--## Pipeline File Naming+### Pipeline File Naming- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)-- Files must be `.py` for structures+- Files must be `.py` for code defining the data structures- Use descriptive names in `snake_case`-## Pipeline File Structure+### Pipeline File OutlineA pipeline file has three main sections:1. Domain statement2. Concept definitions3. Pipe definitions-### Domain Statement+#### Domain Statement```plxdomain = "domain_name"description = "Description of the domain" # Optional```Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.-### Concept Definitions+#### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.+```plx[concept]-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output+ConceptName = "Description of the concept"```-Important Rules:+**Naming Rules:**- Use PascalCase for concept names-- Never use plurals (no "Stories", use "Story")-- Avoid adjectives (no "LargeText", use "Text")-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number)-yes-### Pipe Definitions+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)-## Pipe Base Structure+**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.-```plx-[pipe.your_pipe_name]-type = "PipeLLM"-description = "A description of what your pipe does"-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }-output = "ConceptName"-```+**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:-DO NOT WRITE:```plx-[pipe.your_pipe_name]-type = "pipe_sequence"+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"```-But it should be:+For details on how to structure concepts with fields, see the "Structuring Models" section below.++#### Pipe Definitions++### Pipe Base Definition```plx[pipe.your_pipe_name]-type = "PipeSequence"-description = "....."+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"```-The pipes will all have at least this base structure.-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditionnal pipes (if PipeCondition).+The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).So If you have this error:`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •variable='['invoice']'``@@ -84,75 +83,108 @@ inputs = {- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:-# Structured Models Rules+### Structuring Models-## Model Location and Registration+Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.-- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`-- Models must inherit from `StructuredContent` or appropriate content type+#### Three Ways to Structure Concepts-## Model Structure+**1. No Structure Needed**-Concepts and their structure classes are meant to indicate an idea.-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.+If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.-**IMPORTANT: Never create unnecessary structure classes that only refine native concepts without adding fields.**+**2. Inline Structure Definition (RECOMMENDED for most cases)**-DO NOT create structures like:-```python-class Joke(TextContent):- """A humorous text that makes people laugh."""- pass+For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }```-If a concept only refines a native concept (like Text, Image, etc.) without adding new fields, simply declare it in the .plx file:+**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):```plx-[concept]-Joke = "A humorous text that makes people laugh."+field_name = "Field description"```-If you simply need to refine another native concept, construct it like this:++**Detailed syntax** (with explicit properties):```plx-[concept.Landscape]-refines = "Image"+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }```-Only create a Python structure class when you need to add specific fields:+**3. Python StructuredContent Class (For Advanced Features)**-```python-from datetime import datetime-from typing import List, Optional-from pydantic import Field+Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains+```pythonfrom pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator-# IMPORTANT: THE CLASS MUST BE A SUBCLASS OF StructuredContent-class YourModel(StructuredContent): # Always be a subclass of StructuredContent- # Required fields- field1: str- field2: int+class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```- # Optional fields with defaults- field3: Optional[str] = Field(None, "Description of field3")- field4: List[str] = Field(default_factory=list)+**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.- # Date fields should remove timezone- date_field: Optional[datetime] = None-```-## Usage+#### Decision Rules for Agents-Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.+**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class-Structure classes defined within `pipelex_libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.+**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)+**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required-## Best Practices for structures+#### Inline Structure Limitations-- Respect Pydantic v2 standards-- Use type hints for all fields-- Use `Field` declaration and write the description+Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes-## Pipe Controllers and Pipe Operator+### Pipe Controllers and Pipe OperatorsLook at the Pipes we have in order to adapt it. Pipes are organized in two categories:@@ -160,7 +192,6 @@ Look at the Pipes we have in order to adapt it. Pipes are organized in two categ- `PipeSequence` - For creating a sequence of multiple steps- `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory- `PipeParallel` - For parallelizing pipes- - `PipeBatch` - For running pipes in Batch over a ListContent2. **Operators** - For specific tasks:- `PipeLLM` - Generate Text and Objects (include Vision LLM)@@ -169,12 +200,11 @@ Look at the Pipes we have in order to adapt it. Pipes are organized in two categ- `PipeImgGen` - Generate Images- `PipeFunc` - For running classic python scripts-# PipeSequence Guide+### PipeSequence controller-## Purpose-PipeSequence executes multiple pipes in a defined order, where each step can use results from previous steps.+Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.-## Basic Structure+#### Basic Definition```plx[pipe.your_sequence_name]type = "PipeSequence"@@ -188,13 +218,13 @@ steps = []```-## Key Components+#### Key Components1. **Steps Array**: List of pipes to execute in sequence- `pipe`: Name of the pipe to execute- `result`: Name to assign to the pipe's output that will be in the working memory-## Using PipeBatch in Steps+#### Using PipeBatch in StepsYou can use PipeBatch functionality within steps using `batch_over` and `batch_as`:@@ -215,23 +245,22 @@ steps = [The result of a batched step will be a `ListContent` containing the outputs from processing each item.-# PipeCondition Controller+### PipeCondition controllerThe PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.-## Usage in PLX Configuration--### Basic Usage with Direct Expression+#### Basic usage```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"@@ -240,126 +269,179 @@ or```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"```-## Key Parameters+#### Key Parameters- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)-- `pipe_map`: Dictionary mapping expression results to pipe codes :-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger--# PipeBatch Controller--The PipeBatch controller allows you to apply a pipe operation to each element in a list of inputs in parallele. It is created via a PipeSequence.--## Usage in PLX Configuration+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found+Example with fail as default:```plx-[pipe.sequence_with_batch]-type = "PipeSequence"-description = "A Sequence of pipes"-inputs = { input_data = "ConceptName" }-output = "OutputConceptName"-steps = [- { pipe = "pipe_to_apply", batch_over = "input_list", batch_as = "current_item", result = "batch_results" }-]-```--## Key Parameters--- `pipe`: The pipe operation to apply to each element in the batch-- `batch_over`: The name of the list in the context to iterate over-- `batch_as`: The name to use for the current element in the pipe's context-- `result`: Where to store the results of the batch operation+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"-# PipeLLM Guide+[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```-## Purpose+### PipeLLM operatorPipeLLM is used to:1. Generate text or objects with LLMs2. Process images with Vision LLMs-## Basic Usage+#### Basic Usage-### Simple Text Generation+Simple Text Generation:```plx[pipe.write_story]type = "PipeLLM"description = "Write a short story"output = "Text"-prompt_template = """+prompt = """Write a short story about a programmer."""```-### Structured Data Extraction+Structured Data Extraction:```plx[pipe.extract_info]type = "PipeLLM"description = "Extract information"inputs = { text = "Text" }output = "PersonInfo"-prompt_template = """+prompt = """Extract person information from this text:@text"""```-### System Prompts-Add system-level instructions:+Supports system instructions:```plx[pipe.expert_analysis]type = "PipeLLM"description = "Expert analysis"output = "Analysis"system_prompt = "You are a data analysis expert"-prompt_template = "Analyze this data"+prompt = "Analyze this data"```-### Multiple Outputs-Generate multiple results:+#### Multiple Outputs++Generate multiple outputs (fixed number):```plx[pipe.generate_ideas]type = "PipeLLM"description = "Generate ideas"output = "Idea"nb_output = 3 # Generate exactly 3 ideas-# OR+```++Generate multiple outputs (variable number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"multiple_output = true # Let the LLM decide how many to generate```-### Vision Tasks-Process images with VLMs:+#### Vision++Process images with VLMs (image inputs must be tagged in the prompt):```plx[pipe.analyze_image]type = "PipeLLM"description = "Analyze image"-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in a stuff, you can add something like `{ "page.image": "Image" }+inputs = { image = "Image" }output = "ImageAnalysis"-prompt_template = "Describe what you see in this image"+prompt = """+Describe what you see in this image:++$image+"""```-# PipeOCR Guide+You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```-## Purpose+#### Writing prompts for PipeLLM-The PipeExtract operator is used to extract text and images from an image or a PDF+**Insert stuff inside a tagged block**++If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.++Example template:+```plx+prompt = """+Match the expense with its corresponding invoice:++@expense++@invoices+"""+```+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.++DO NOT write things like "Here is the expense: @expense".+DO write simply "@expense" alone in an isolated line.++**Insert stuff inline**++If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.++Example template:+```plx+prompt = """+Your goal is to summarize everything related to $topic in the provided text:++@text++Please provide only the summary, with no additional text or explanations.+Your summary should not be longer than 2 sentences.+"""+```++In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.+Be sure to make the proper choice of prefix for each insertion.++DO NOT write "$topic" alone in an isolated line.+DO write things like "Write an essay about $topic" to include text into an actual sentence.+++### PipeExtract operator-## Basic Usage+The PipeExtract operator is used to extract text and images from an image or a PDF-### Simple Text Extraction+#### Simple Text Extraction```plx[pipe.extract_info]type = "PipeExtract"@@ -368,59 +450,81 @@ inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This isoutput = "Page"```-Only one input is allowed and it must either be an `Image` or a `PDF`.+Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.The output concept `Page` is a native concept, with the structure `PageContent`:It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page````pythonclass TextAndImagesContent(StuffContent):- text: Optional[TextContent]- images: Optional[List[ImageContent]]+ text: TextContent | None+ images: list[ImageContent] | Noneclass PageContent(StructuredContent): # CONCEPT IS "Page"text_and_images: TextAndImagesContent- page_view: Optional[ImageContent] = None+ page_view: ImageContent | None = None```- `text_and_images` are the text, and the related images found in the input image or PDF.- `page_view` is the screenshot of the whole pdf page/image.-# PipeCompose Guide--## Purpose+### PipeCompose operatorThe PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.-## Basic Usage+#### Basic Usage-### Simple Template Composition+Simple Template Composition:```plx[pipe.compose_report]type = "PipeCompose"description = "Compose a report using template"inputs = { data = "ReportData" }output = "Text"-jinja2 = """-# Report Summary+template = """+## Report SummaryBased on the analysis:-@data+$dataGenerated on: {{ current_date }}"""```-### Using Named Templates+Using Named Templates:```plx[pipe.use_template]type = "PipeCompose"description = "Use a predefined template"inputs = { content = "Text" }output = "Text"-jinja2_name = "standard_report_template"+template_name = "standard_report_template"```-### CRM Email Template+Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+```++CRM Email Template:```plx[pipe.compose_follow_up_email]type = "PipeCompose"@@ -429,7 +533,7 @@ inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }output = "Text"template_category = "html"templating_style = { tag_style = "square_brackets", text_format = "html" }-jinja2 = """+template = """Subject: Following up on our $deal.product_name discussionHi $customer.first_name,@@ -459,29 +563,32 @@ $sales_rep.phone | $sales_rep.email"""```-## Key Parameters+#### Key Parameters-- `jinja2`: Inline Jinja2 template (mutually exclusive with jinja2_name)-- `jinja2_name`: Name of a predefined template (mutually exclusive with jinja2)+- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)- `templating_style`: Styling options for template rendering- `extra_context`: Additional context variables for template-## Template Variables+For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++#### Template VariablesUse the same variable insertion rules as PipeLLM:- `@variable` for block insertion (multi-line content)- `$variable` for inline insertion (short text)-# PipeImgGen Guide--## Purpose+### PipeImgGen operatorThe PipeImgGen operator is used to generate images using AI image generation models.-## Basic Usage+#### Basic Usage-### Simple Image Generation+Simple Image Generation:```plx[pipe.generate_image]type = "PipeImgGen"@@ -490,19 +597,19 @@ inputs = { prompt = "ImgGenPrompt" }output = "Image"```-### Using Image Generation Settings+Using Image Generation Settings:```plx[pipe.generate_photo]type = "PipeImgGen"description = "Generate a high-quality photo"inputs = { prompt = "ImgGenPrompt" }output = "Photo"-model = { model = "flux-pro/v1.1-ultra", quality = "hd" }+model = { model = "fast-img-gen" }aspect_ratio = "16:9"-nb_steps = 8+quality = "hd"```-### Multiple Image Generation+Multiple Image Generation:```plx[pipe.generate_variations]type = "PipeImgGen"@@ -513,7 +620,7 @@ nb_output = 3seed = "auto"```-### Advanced Configuration+Advanced Configuration:```plx[pipe.generate_custom]type = "PipeImgGen"@@ -529,28 +636,25 @@ is_raw = falsesafety_tolerance = 3```-## Key Parameters+#### Key Parameters-### Image Generation Settings-- `img_gen`: ImgGenChoice (preset name or inline settings)-- `img_gen_handle`: Direct model handle (legacy)+**Image Generation Settings:**+- `model`: Model choice (preset name or inline settings with model name)- `quality`: Image quality ("standard", "hd")-- `nb_steps`: Number of generation steps-- `guidance_scale`: How closely to follow the prompt-### Output Configuration+**Output Configuration:**- `nb_output`: Number of images to generate- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)- `output_format`: File format ("png", "jpeg", "webp")- `background`: Background type ("default", "transparent")-### Generation Control+**Generation Control:**- `seed`: Random seed (integer or "auto")- `is_raw`: Whether to apply post-processing- `is_moderated`: Enable content moderation- `safety_tolerance`: Content safety level (1-6)-## Input Requirements+#### Input RequirementsPipeImgGen requires exactly one input that must be either:- An `ImgGenPrompt` concept@@ -558,15 +662,13 @@ PipeImgGen requires exactly one input that must be either:The input can be named anything but must contain the prompt text for image generation.-# PipeFunc Guide--## Purpose+### PipeFunc operatorThe PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.-## Basic Usage+#### Basic Usage-### Simple Function Call+Simple Function Call:```plx[pipe.process_data]type = "PipeFunc"@@ -576,7 +678,7 @@ output = "ProcessedData"function_name = "process_data_function"```-### File Processing Example+File Processing Example:```plx[pipe.read_file]type = "PipeFunc"@@ -586,11 +688,11 @@ output = "FileContent"function_name = "read_file_content"```-## Key Parameters+#### Key Parameters- `function_name`: Name of the Python function to call (must be registered in func_registry)-## Function Requirements+#### Function RequirementsThe Python function must:@@ -607,12 +709,12 @@ The Python function must:- `list[StuffContent]`: Multiple content objects (becomes ListContent)- `str`: Simple string (becomes TextContent)-## Function Registration+#### Function RegistrationFunctions must be registered in the function registry before use:```python-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registry@func_registry.register("my_function_name")async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:@@ -626,7 +728,7 @@ async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:return MyResultContent(data=result)```-## Working Memory Access+#### Working Memory AccessInside the function, access pipeline inputs through working memory:@@ -643,52 +745,94 @@ async def process_function(working_memory: WorkingMemory) -> TextContent:return TextContent(text=processed_text)```-This rule explains how to write prompt templates in PipeLLM definitions.+----## Insert stuff inside a tagged block+### Rules to choose LLM models used in PipeLLMs.-If the inserted text is supposedly long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.+#### LLM Configuration System-Example template:-```plx-prompt_template = """-Match the expense with its corresponding invoice:+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:-@expense+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`-@invoices+#### LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++#### Using an LLM Handle in a PipeLLM++Here is an example of using a model to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World."""```-In this example, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.-**DO NOT write things like "Here is the expense: @expense".**-**DO write simply "@expense" alone in an isolated line.**+As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-## Insert stuff inline+#### LLM Presets-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.+Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:-Example template:```plx-prompt_template = """-Your goal is to summarize everything related to $topic in the provided text:+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:-@text+The category of this invoice is: $invoice_details.category.-Please provide only the summary, with no additional text or explanations.-Your summary should not be longer than 2 sentences.+@invoice_text"""```-Here, $topic will be inserted inline, whereas @text will be a a delimited block.-Be sure to make the proper choice of prefix for each insertion.+The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.++----**DO NOT write "$topic" alone in an isolated line.**-**DO write things like "Write an essay about $topic" included in an actual sentence.**+ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+Then, create an example file to run the pipeline in the `examples` folder.+But don't write documentation unless asked explicitly to.-# Guide to write an example to execute a pipeline+## Guide to execute a pipeline and write example code-## Example to execute a pipeline with text output+### Example to execute a pipeline with text output```pythonimport asyncio@@ -710,14 +854,14 @@ async def hello_world() -> str:return pipe_output.main_stuff_as_str-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciooutput_text = asyncio.run(hello_world())pretty_print(output_text, title="Your first Pipelex output")```-## Example to execute a pipeline with structured output+### Example to execute a pipeline with structured output```pythonimport asyncio@@ -725,8 +869,9 @@ import asynciofrom pipelex import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent-from pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart+from my_project.gantt.gantt_struct import GanttChartSAMPLE_NAME = "extract_gantt"IMAGE_URL = "assets/gantt/gantt_tree_house.png"@@ -747,30 +892,32 @@ async def extract_gantt(image_url: str) -> GanttChart:return pipe_output.main_stuff_as(content_type=GanttChart)-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciogantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))pretty_print(gantt_chart, title="Gantt Chart")```-## Setting up the input memory+### Setting up the input memory-### Explanation of input memory+#### Explanation of input memory-The input memory is a dictionary of key-value pairs, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:+The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:```python-StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str-ImplicitMemory = Dict[str, StuffContentOrData]+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]```As you can seen, we made it so different ways can be used to define that stuff using structured content or data.+#### Different ways to set up the input memory+So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:```python-# Here we have a single input and it's a Text.-# If you assign a string, by default it will be considered as a TextContent.+## Here we have a single input and it's a Text.+## If you assign a string, by default it will be considered as a TextContent.pipe_output = await execute_pipeline(pipe_code="master_advisory_orchestrator",input_memory={@@ -778,9 +925,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's a PDF.-# Because PDFContent is a native concept, we can use it directly as a value,-# the system knows what content it corresponds to:+## Here we have a single input and it's a PDF.+## Because PDFContent is a native concept, we can use it directly as a value,+## the system knows what content it corresponds to:pipe_output = await execute_pipeline(pipe_code="power_extractor_dpe",input_memory={@@ -788,8 +935,8 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's an Image.-# Because ImageContent is a native concept, we can use it directly as a value:+## Here we have a single input and it's an Image.+## Because ImageContent is a native concept, we can use it directly as a value:pipe_output = await execute_pipeline(pipe_code="fashion_variation_pipeline",input_memory={@@ -797,9 +944,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input, it's an image but-# its actually a more specific concept gantt.GanttImage which refines Image,-# so we must provide it using a dict with the concept and the content:+## Here we have a single input, it's an image but+## its actually a more specific concept gantt.GanttImage which refines Image,+## so we must provide it using a dict with the concept and the content:pipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",input_memory={@@ -810,7 +957,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here is a more complex example with multiple inputs assigned using different ways:+## Here is a more complex example with multiple inputs assigned using different ways:pipe_output = await execute_pipeline(pipe_code="retrieve_then_answer",dynamic_output_concept_code="contracts.Fees",@@ -825,7 +972,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa)```-## Using the outputs of a pipeline+### Using the outputs of a pipelineAll pipe executions return a `PipeOutput` object.It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.@@ -879,9 +1026,9 @@ class PipeOutput(BaseModel):...```-As you can see, you can extarct any variable from the output working memory.+As you can see, you can extract any variable from the output working memory.-### Getting the main stuff as a specific type+#### Getting the main stuff as a specific typeSimple text as a string:@@ -945,7 +1092,7 @@ type = "PipeLLM"description = "Write text about Hello World."output = "Text"model = { model = "gpt-5", temperature = 0.9 }-prompt_template = """+prompt = """Write a haiku about Hello World."""```@@ -971,7 +1118,7 @@ description = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"model = "llm_to_extract_invoice"-prompt_template = """+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -983,10 +1130,6 @@ The category of this invoice is: $invoice_details.category.The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.-You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.------ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.-Then, create an example file to run the pipeline in the `examples` folder.-But don't write documentation unless asked explicitly to.+You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+<!-- END_PIPELEX_RULES -->diff --git a/AGENTS.md b/AGENTS.mdindex 6cc4503be..4d612f7da 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -1,195 +1,71 @@-# Coding Standards & Best Practices+<!-- BEGIN_PIPELEX_RULES -->+## Guide to write or edit pipelines using the Pipelex language in .plx files-This document outlines the core coding standards, best practices, and quality control procedures for the codebase.--## Type Hints--1. **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tupele types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use the `|` syntax for union types (e.g `str | int`) and `| None` for optionals- - Use Field(default_factory=...) for mutable defaults and if it's a list of something else than str, use `empty_list_factory_of()` to make a factory: `number_list: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")`- - Use `BaseModel` and respect Pydantic v2 standards, in particular use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose--2. **StrEnum**- - Import from `pipelex.types`:- ```python- from pipelex.types import StrEnum- ```--3. **Self type**- - Import from `pipelex.types`:- ```python- from pipelex.types import Self- ```--## Factory Pattern-- - Use Factory Pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such--## Error Handling-- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones- - Never catch Exception, only catch specific exceptions- - Always add `from exc` to the exception-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```-- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.--## Documentation--1. **Docstring Format**- ```python- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:- """Process and resize an image.-- Args:- image_path: Path to the source image- size: Tuple of (width, height) for resizing-- Returns:- Processed image as bytes- """- pass- ```--2. **Class Documentation**- ```python- class ImageProcessor:- """Handles image processing operations.-- Provides methods for resizing, converting, and optimizing images.- """- ```--## Code Quality Checks--### Linting and Type Checking--Before finalizing a task, run:-```bash-make fix-unused-imports-make check-```--This runs multiple code quality tools:-- Pyright: Static type checking-- Ruff: Fast Python linter-- Mypy: Static type checker--Always fix any issues reported by these tools before proceeding.--### Running Tests--1. **Quick Test Run** (no LLM/image generation):- ```bash- make tp- ```- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or ocr)) and not (needs_output or pipelex_api)`--2. **Specific Tests**:- ```bash- make tp TEST=TestClassName- # or- make tp TEST=test_function_name- ```- Note: Matches names starting with the provided string.--**Important**: Never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-img-gen` - these use costly inference.--## Pipelines--- All pipeline definitions go in `pipelex/libraries/pipelines/`-- Always validate pipelines after creation/edit with `make validate`.- Iterate if there are errors.--## Project Structure--- **Pipelines**: `pipelex/libraries/pipelines/`-- **Tests**: `tests/` directory-- **Documentation**: `docs/` directory-------# Guide to write or edit pipelines using the Pipelex language in .plx files--- Always first write your "plan" in natural langage, then transcribe it in pipelex.+- Always first write your "plan" in natural language, then transcribe it in pipelex.- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.-- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)-## Pipeline File Naming+### Pipeline File Naming- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)-- Files must be `.py` for structures+- Files must be `.py` for code defining the data structures- Use descriptive names in `snake_case`-## Pipeline File Structure+### Pipeline File OutlineA pipeline file has three main sections:1. Domain statement2. Concept definitions3. Pipe definitions-### Domain Statement+#### Domain Statement```plxdomain = "domain_name"description = "Description of the domain" # Optional```Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.-### Concept Definitions+#### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.+```plx[concept]-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output+ConceptName = "Description of the concept"```-Important Rules:+**Naming Rules:**- Use PascalCase for concept names-- Never use plurals (no "Stories", use "Story")-- Avoid adjectives (no "LargeText", use "Text")-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number)-yes-### Pipe Definitions+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)-## Pipe Base Structure+**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.-```plx-[pipe.your_pipe_name]-type = "PipeLLM"-description = "A description of what your pipe does"-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }-output = "ConceptName"-```+**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:-DO NOT WRITE:```plx-[pipe.your_pipe_name]-type = "pipe_sequence"+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"```-But it should be:+For details on how to structure concepts with fields, see the "Structuring Models" section below.++#### Pipe Definitions++### Pipe Base Definition```plx[pipe.your_pipe_name]-type = "PipeSequence"-description = "....."+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"```-The pipes will all have at least this base structure.-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditionnal pipes (if PipeCondition).+The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).So If you have this error:`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •variable='['invoice']'``@@ -207,74 +83,108 @@ inputs = {- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:-## Structuring Models+### Structuring Models-### Model Location and Registration+Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.-- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`-- Models must inherit from `StructuredContent` or appropriate content type+#### Three Ways to Structure Concepts-## Model Structure+**1. No Structure Needed**-Concepts and their structure classes are meant to indicate an idea.-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.+If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.-**IMPORTANT: Never create unnecessary structure classes that only refine native concepts without adding fields.**+**2. Inline Structure Definition (RECOMMENDED for most cases)**-DO NOT create structures like:-```python-class Joke(TextContent):- """A humorous text that makes people laugh."""- pass+For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }```-If a concept only refines a native concept (like Text, Image, etc.) without adding new fields, simply declare it in the .plx file:+**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):```plx-[concept]-Joke = "A humorous text that makes people laugh."+field_name = "Field description"```-If you simply need to refine another native concept, construct it like this:++**Detailed syntax** (with explicit properties):```plx-[concept.Landscape]-refines = "Image"+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }```-Only create a Python structure class when you need to add specific fields:-```python-from datetime import datetime-from typing import List, Optional-from pydantic import Field+**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains+```pythonfrom pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator-# IMPORTANT: THE CLASS MUST BE A SUBCLASS OF StructuredContent-class YourModel(StructuredContent): # Always be a subclass of StructuredContent- # Required fields- field1: str- field2: int+class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```- # Optional fields with defaults- field3: Optional[str] = Field(None, "Description of field3")- field4: List[str] = Field(default_factory=list)+**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.- # Date fields should remove timezone- date_field: Optional[datetime] = None-```-### Usage+#### Decision Rules for Agents-Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.+**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class-Structure classes defined within `pipelex_libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.+**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)+**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required-### Best Practices for structures+#### Inline Structure Limitations-- Respect Pydantic v2 standards-- Use type hints for all fields-- Use `Field` declaration and write the description+Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes-## Pipe Controllers and Pipe Operators+### Pipe Controllers and Pipe OperatorsLook at the Pipes we have in order to adapt it. Pipes are organized in two categories:@@ -290,11 +200,11 @@ Look at the Pipes we have in order to adapt it. Pipes are organized in two categ- `PipeImgGen` - Generate Images- `PipeFunc` - For running classic python scripts-## PipeSequence controller+### PipeSequence controllerPurpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.-### Basic Structure+#### Basic Definition```plx[pipe.your_sequence_name]type = "PipeSequence"@@ -308,13 +218,13 @@ steps = []```-### Key Components+#### Key Components1. **Steps Array**: List of pipes to execute in sequence- `pipe`: Name of the pipe to execute- `result`: Name to assign to the pipe's output that will be in the working memory-### Using PipeBatch in Steps+#### Using PipeBatch in StepsYou can use PipeBatch functionality within steps using `batch_over` and `batch_as`:@@ -335,21 +245,22 @@ steps = [The result of a batched step will be a `ListContent` containing the outputs from processing each item.-## PipeCondition controller+### PipeCondition controllerThe PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.-### Basic usage+#### Basic usage```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"@@ -358,32 +269,49 @@ or```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"```-### Key Parameters+#### Key Parameters- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)-- `pipe_map`: Dictionary mapping expression results to pipe codes :-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"-## PipeLLM operator+[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++### PipeLLM operatorPipeLLM is used to:1. Generate text or objects with LLMs2. Process images with Vision LLMs-### Basic Usage+#### Basic UsageSimple Text Generation:```plx@@ -391,7 +319,7 @@ Simple Text Generation:type = "PipeLLM"description = "Write a short story"output = "Text"-prompt_template = """+prompt = """Write a short story about a programmer."""```@@ -403,7 +331,7 @@ type = "PipeLLM"description = "Extract information"inputs = { text = "Text" }output = "PersonInfo"-prompt_template = """+prompt = """Extract person information from this text:@text"""@@ -416,10 +344,10 @@ type = "PipeLLM"description = "Expert analysis"output = "Analysis"system_prompt = "You are a data analysis expert"-prompt_template = "Analyze this data"+prompt = "Analyze this data"```-### Multiple Outputs+#### Multiple OutputsGenerate multiple outputs (fixed number):```plx@@ -439,19 +367,33 @@ output = "Idea"multiple_output = true # Let the LLM decide how many to generate```-### Vision+#### Vision-Process images with VLMs:+Process images with VLMs (image inputs must be tagged in the prompt):```plx[pipe.analyze_image]type = "PipeLLM"description = "Analyze image"-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in an attribute within a stuff, you can add something like `{ "page.image": "Image" }+inputs = { image = "Image" }output = "ImageAnalysis"-prompt_template = "Describe what you see in this image"+prompt = """+Describe what you see in this image:++$image+"""```-### Writing prompts for PipeLLM+You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++#### Writing prompts for PipeLLM**Insert stuff inside a tagged block**@@ -459,7 +401,7 @@ If the inserted text is supposedly a long text, made of several lines or paragraExample template:```plx-prompt_template = """+prompt = """Match the expense with its corresponding invoice:@expense@@ -467,7 +409,7 @@ Match the expense with its corresponding invoice:@invoices"""```-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.DO NOT write things like "Here is the expense: @expense".DO write simply "@expense" alone in an isolated line.@@ -478,7 +420,7 @@ If the inserted text is short text and it makes sense to have it inserted directExample template:```plx-prompt_template = """+prompt = """Your goal is to summarize everything related to $topic in the provided text:@text@@ -495,11 +437,11 @@ DO NOT write "$topic" alone in an isolated line.DO write things like "Write an essay about $topic" to include text into an actual sentence.-## PipeExtract operator+### PipeExtract operatorThe PipeExtract operator is used to extract text and images from an image or a PDF-### Simple Text Extraction+#### Simple Text Extraction```plx[pipe.extract_info]type = "PipeExtract"@@ -508,28 +450,38 @@ inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This isoutput = "Page"```-Only one input is allowed and it must either be an `Image` or a `PDF`.+Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.The output concept `Page` is a native concept, with the structure `PageContent`:It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page````pythonclass TextAndImagesContent(StuffContent):- text: Optional[TextContent]- images: Optional[List[ImageContent]]+ text: TextContent | None+ images: list[ImageContent] | Noneclass PageContent(StructuredContent): # CONCEPT IS "Page"text_and_images: TextAndImagesContent- page_view: Optional[ImageContent] = None+ page_view: ImageContent | None = None```- `text_and_images` are the text, and the related images found in the input image or PDF.- `page_view` is the screenshot of the whole pdf page/image.-## PipeCompose operator+### PipeCompose operatorThe PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.-### Basic Usage+#### Basic UsageSimple Template Composition:```plx@@ -538,8 +490,8 @@ type = "PipeCompose"description = "Compose a report using template"inputs = { data = "ReportData" }output = "Text"-jinja2 = """-# Report Summary+template = """+## Report SummaryBased on the analysis:$data@@ -555,7 +507,21 @@ type = "PipeCompose"description = "Use a predefined template"inputs = { content = "Text" }output = "Text"-jinja2_name = "standard_report_template"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }```CRM Email Template:@@ -567,7 +533,7 @@ inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }output = "Text"template_category = "html"templating_style = { tag_style = "square_brackets", text_format = "html" }-jinja2 = """+template = """Subject: Following up on our $deal.product_name discussionHi $customer.first_name,@@ -597,25 +563,30 @@ $sales_rep.phone | $sales_rep.email"""```-### Key Parameters+#### Key Parameters-- `jinja2`: Inline Jinja2 template (mutually exclusive with jinja2_name)-- `jinja2_name`: Name of a predefined template (mutually exclusive with jinja2)+- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)- `templating_style`: Styling options for template rendering- `extra_context`: Additional context variables for template-### Template Variables+For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++#### Template VariablesUse the same variable insertion rules as PipeLLM:- `@variable` for block insertion (multi-line content)- `$variable` for inline insertion (short text)-## PipeImgGen operator+### PipeImgGen operatorThe PipeImgGen operator is used to generate images using AI image generation models.-### Basic Usage+#### Basic UsageSimple Image Generation:```plx@@ -633,9 +604,9 @@ type = "PipeImgGen"description = "Generate a high-quality photo"inputs = { prompt = "ImgGenPrompt" }output = "Photo"-model = { model = "fast-img-gen", quality = "hd" }+model = { model = "fast-img-gen" }aspect_ratio = "16:9"-nb_steps = 8+quality = "hd"```Multiple Image Generation:@@ -665,14 +636,11 @@ is_raw = falsesafety_tolerance = 3```-### Key Parameters+#### Key Parameters**Image Generation Settings:**-- `img_gen`: ImgGenChoice (preset name or inline settings)-- `img_gen_handle`: Direct model handle (legacy)+- `model`: Model choice (preset name or inline settings with model name)- `quality`: Image quality ("standard", "hd")-- `nb_steps`: Number of generation steps-- `guidance_scale`: How closely to follow the prompt**Output Configuration:**- `nb_output`: Number of images to generate@@ -686,7 +654,7 @@ safety_tolerance = 3- `is_moderated`: Enable content moderation- `safety_tolerance`: Content safety level (1-6)-### Input Requirements+#### Input RequirementsPipeImgGen requires exactly one input that must be either:- An `ImgGenPrompt` concept@@ -694,11 +662,11 @@ PipeImgGen requires exactly one input that must be either:The input can be named anything but must contain the prompt text for image generation.-## PipeFunc operator+### PipeFunc operatorThe PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.-### Basic Usage+#### Basic UsageSimple Function Call:```plx@@ -720,11 +688,11 @@ output = "FileContent"function_name = "read_file_content"```-### Key Parameters+#### Key Parameters- `function_name`: Name of the Python function to call (must be registered in func_registry)-### Function Requirements+#### Function RequirementsThe Python function must:@@ -741,12 +709,12 @@ The Python function must:- `list[StuffContent]`: Multiple content objects (becomes ListContent)- `str`: Simple string (becomes TextContent)-### Function Registration+#### Function RegistrationFunctions must be registered in the function registry before use:```python-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registry@func_registry.register("my_function_name")async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:@@ -760,7 +728,7 @@ async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:return MyResultContent(data=result)```-### Working Memory Access+#### Working Memory AccessInside the function, access pipeline inputs through working memory:@@ -779,9 +747,9 @@ async def process_function(working_memory: WorkingMemory) -> TextContent:----## Rules to choose LLM models used in PipeLLMs.+### Rules to choose LLM models used in PipeLLMs.-### LLM Configuration System+#### LLM Configuration SystemIn order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:@@ -790,7 +758,7 @@ LLM configurations are managed through the new inference backend system with fil- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`- **Routing**: `.pipelex/inference/routing_profiles.toml`-### LLM Handles+#### LLM HandlesAn llm_handle can be either:1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system@@ -806,9 +774,9 @@ base-mistral = "mistral-medium"The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-### Using an LLM Handle in a PipeLLM+#### Using an LLM Handle in a PipeLLM-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:+Here is an example of using a model to specify which LLM to use in a PipeLLM:```plx[pipe.hello_world]@@ -816,14 +784,14 @@ type = "PipeLLM"description = "Write text about Hello World."output = "Text"model = { model = "gpt-5", temperature = 0.9 }-prompt_template = """+prompt = """Write a haiku about Hello World."""```As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-### LLM Presets+#### LLM PresetsPresets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.@@ -842,7 +810,7 @@ description = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"model = "llm_to_extract_invoice"-prompt_template = """+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -862,11 +830,9 @@ ALWAYS RUN `make validate` when you are finished writing pipelines: This checksThen, create an example file to run the pipeline in the `examples` folder.But don't write documentation unless asked explicitly to.------# Guide to write an example to execute a pipeline+## Guide to execute a pipeline and write example code-## Example to execute a pipeline with text output+### Example to execute a pipeline with text output```pythonimport asyncio@@ -888,14 +854,14 @@ async def hello_world() -> str:return pipe_output.main_stuff_as_str-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciooutput_text = asyncio.run(hello_world())pretty_print(output_text, title="Your first Pipelex output")```-## Example to execute a pipeline with structured output+### Example to execute a pipeline with structured output```pythonimport asyncio@@ -903,8 +869,9 @@ import asynciofrom pipelex import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent-from pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart+from my_project.gantt.gantt_struct import GanttChartSAMPLE_NAME = "extract_gantt"IMAGE_URL = "assets/gantt/gantt_tree_house.png"@@ -925,32 +892,32 @@ async def extract_gantt(image_url: str) -> GanttChart:return pipe_output.main_stuff_as(content_type=GanttChart)-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciogantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))pretty_print(gantt_chart, title="Gantt Chart")```-## Setting up the input memory+### Setting up the input memory-### Explanation of input memory+#### Explanation of input memoryThe input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:```python-StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str-ImplicitMemory = Dict[str, StuffContentOrData]+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]```As you can seen, we made it so different ways can be used to define that stuff using structured content or data.-### Different ways to set up the input memory+#### Different ways to set up the input memorySo here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:```python-# Here we have a single input and it's a Text.-# If you assign a string, by default it will be considered as a TextContent.+## Here we have a single input and it's a Text.+## If you assign a string, by default it will be considered as a TextContent.pipe_output = await execute_pipeline(pipe_code="master_advisory_orchestrator",input_memory={@@ -958,9 +925,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's a PDF.-# Because PDFContent is a native concept, we can use it directly as a value,-# the system knows what content it corresponds to:+## Here we have a single input and it's a PDF.+## Because PDFContent is a native concept, we can use it directly as a value,+## the system knows what content it corresponds to:pipe_output = await execute_pipeline(pipe_code="power_extractor_dpe",input_memory={@@ -968,8 +935,8 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's an Image.-# Because ImageContent is a native concept, we can use it directly as a value:+## Here we have a single input and it's an Image.+## Because ImageContent is a native concept, we can use it directly as a value:pipe_output = await execute_pipeline(pipe_code="fashion_variation_pipeline",input_memory={@@ -977,9 +944,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input, it's an image but-# its actually a more specific concept gantt.GanttImage which refines Image,-# so we must provide it using a dict with the concept and the content:+## Here we have a single input, it's an image but+## its actually a more specific concept gantt.GanttImage which refines Image,+## so we must provide it using a dict with the concept and the content:pipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",input_memory={@@ -990,7 +957,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here is a more complex example with multiple inputs assigned using different ways:+## Here is a more complex example with multiple inputs assigned using different ways:pipe_output = await execute_pipeline(pipe_code="retrieve_then_answer",dynamic_output_concept_code="contracts.Fees",@@ -1005,7 +972,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa)```-## Using the outputs of a pipeline+### Using the outputs of a pipelineAll pipe executions return a `PipeOutput` object.It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.@@ -1059,9 +1026,9 @@ class PipeOutput(BaseModel):...```-As you can see, you can extarct any variable from the output working memory.+As you can see, you can extract any variable from the output working memory.-### Getting the main stuff as a specific type+#### Getting the main stuff as a specific typeSimple text as a string:@@ -1088,160 +1055,81 @@ result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)----# Writing unit tests--## Unit test generalities--NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.--### Test file structure--- Name test files with `test_` prefix-- Use descriptive names that match the functionality being tested-- Place test files in the appropriate test category directory:- - `tests/unit/` - for unit tests that test individual functions/classes in isolation- - `tests/integration/` - for integration tests that test component interactions- - `tests/e2e/` - for end-to-end tests that test complete workflows- - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)-- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest-- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.-- Always put test inside Test classes.-- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`--### Markers--Apply the appropriate markers:-- "llm: uses an LLM to generate text or objects"-- "img_gen: uses an image generation AI"-- "inference: uses either an LLM or an image generation AI"-- "gha_disabled: will not be able to run properly on GitHub Actions"--Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.--### Important rules--- Never use the unittest.mock. Use pytest-mock.--### Test Class Structure+## Rules to choose LLM models used in PipeLLMs.-Always group the tests of a module into a test class:+### LLM Configuration System-```python[email protected](loop_scope="class")-class TestFooBar:- @pytest.mark.parametrize(- "topic test_case_blueprint",- [- TestCases.CASE_1,- TestCases.CASE_2,- ],- )- async def test_pipe_processing(- self,- request: FixtureRequest,- topic: str,- test_case_blueprint: StuffBlueprint,- ):- # Test implementation-```+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:-Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`-## Writing integration test to test pipes+### LLM Handles-### Required imports for pipe tests+An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:-```python-import pytest-from pytest import FixtureRequest-from pipelex import log, pretty_print-from pipelex.core.stuffs.stuff_factory import StuffBlueprint, StuffFactory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.hub import get_report_delegate-from pipelex.libraries.pipelines.base_library.retrieve import RetrievedExcerpt-from pipelex.config_pipelex import get_config--from pipelex.core.pipe import PipeAbstract, update_job_metadata_for_pipe-from pipelex.core.pipes.pipe_output import PipeOutput, PipeOutputType-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.pipe_works.pipe_router_protocol import PipeRouterProtocol+```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"```-### Pipe test implementation steps--1. Create Stuff from blueprint:+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-```python-stuff = StuffFactory.make_stuff(- concept_code="RetrievedExcerpt",- domain="retrieve",- content=RetrievedExcerpt(text="<Some retrieved text>", justification="<Some justification>")- name="retrieved_text",-)-```+### Using an LLM Handle in a PipeLLM-2. Create Working Memory:+Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:-```python-working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""```-3. Run the pipe:+As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-```python-pipe_output = await pipe_router.run_pipe(- pipe_code="pipe_name",- pipe_run_params=PipeRunParamsFactory.make_run_params(),- working_memory=working_memory,- job_metadata=JobMetadata(),-)-```+### LLM Presets-4. Basic assertions:+Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.-```python-assert pipe_output is not None-assert pipe_output.working_memory is not None-assert pipe_output.main_stuff is not None+Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```-### Test Data Organization+The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:-- If it's not already there, create a `test_data.py` file in the test directory-- Define test cases using `StuffBlueprint`:+```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:-```python-class TestCases:- CASE_BLUEPRINT_1 = StuffBlueprint(- name="test_case_1",- concept_code="domain.ConceptName1",- value="test_value"- )- CASE_BLUEPRINT_2 = StuffBlueprint(- name="test_case_2",- concept_code="domain.ConceptName2",- value="test_value"- )+The category of this invoice is: $invoice_details.category.- CASE_BLUEPRINTS: ClassVar[List[Tuple[str, str]]] = [ # topic, blueprint"- ("topic1", CASE_BLUEPRINT_1),- ("topic2", CASE_BLUEPRINT_2),- ]+@invoice_text+"""```-Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.-Also note that we provide a topic for the test case, which is purely for convenience.+The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.-## Best Practices for Testing-- Use parametrize for multiple test cases-- Test both success and failure cases-- Verify working memory state-- Check output structure and content-- Use meaningful test case names-- Include docstrings explaining test purpose-- Log outputs for debugging-- Generate reports for cost tracking+You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+<!-- END_PIPELEX_RULES -->diff --git a/BLACKBOX_RULES.md b/BLACKBOX_RULES.mdnew file mode 100644index 000000000..4d612f7da--- /dev/null+++ b/BLACKBOX_RULES.md@@ -0,0 +1,1135 @@+<!-- BEGIN_PIPELEX_RULES -->+## Guide to write or edit pipelines using the Pipelex language in .plx files++- Always first write your "plan" in natural language, then transcribe it in pipelex.+- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)++### Pipeline File Naming+- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)+- Files must be `.py` for code defining the data structures+- Use descriptive names in `snake_case`++### Pipeline File Outline+A pipeline file has three main sections:+1. Domain statement+2. Concept definitions+3. Pipe definitions++#### Domain Statement+```plx+domain = "domain_name"+description = "Description of the domain" # Optional+```+Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.++#### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.++```plx+[concept]+ConceptName = "Description of the concept"+```++**Naming Rules:**+- Use PascalCase for concept names+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)++**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.++**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:++```plx+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"+```++For details on how to structure concepts with fields, see the "Structuring Models" section below.++#### Pipe Definitions++### Pipe Base Definition++```plx+[pipe.your_pipe_name]+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"+```++The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).+So If you have this error:+`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •+variable='['invoice']'``+That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.++NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:+<NEVER DO THIS>+```plx+inputs = {+ input_1 = "ConceptName1",+ input_2 = "ConceptName2"+}+```+</NEVER DO THIS>++- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:++### Structuring Models++Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.++#### Three Ways to Structure Concepts++**1. No Structure Needed**++If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.++**2. Inline Structure Definition (RECOMMENDED for most cases)**++For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }+```++**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):+```plx+field_name = "Field description"+```++**Detailed syntax** (with explicit properties):+```plx+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }+```++**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator++class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```++**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.++#### Decision Rules for Agents++**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class++**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)++**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required++#### Inline Structure Limitations++Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes+++### Pipe Controllers and Pipe Operators++Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:++1. **Controllers** - For flow control:+ - `PipeSequence` - For creating a sequence of multiple steps+ - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory+ - `PipeParallel` - For parallelizing pipes++2. **Operators** - For specific tasks:+ - `PipeLLM` - Generate Text and Objects (include Vision LLM)+ - `PipeExtract` - Extract text and images from an image or a PDF+ - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.+ - `PipeImgGen` - Generate Images+ - `PipeFunc` - For running classic python scripts++### PipeSequence controller++Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.++#### Basic Definition+```plx+[pipe.your_sequence_name]+type = "PipeSequence"+description = "Description of what this sequence does"+inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps+output = "OutputType"+steps = [+ { pipe = "first_pipe", result = "first_result" },+ { pipe = "second_pipe", result = "second_result" },+ { pipe = "final_pipe", result = "final_result" }+]+```++#### Key Components++1. **Steps Array**: List of pipes to execute in sequence+ - `pipe`: Name of the pipe to execute+ - `result`: Name to assign to the pipe's output that will be in the working memory++#### Using PipeBatch in Steps++You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:++```plx+steps = [+ { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"+ }+]+```++1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.+ - Must be a `ListContent` type containing the items to process+ - Can reference inputs or results from previous steps++2. **batch_as**: Defines the name that will be used to reference the current item being processed+ - This name can be used in the pipe's input mappings+ - Makes each item from the batch available as a single element++The result of a batched step will be a `ListContent` containing the outputs from processing each item.++### PipeCondition controller++The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.++#### Basic usage++```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression = "input_data.category"+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```+or+```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```++#### Key Parameters++- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)+- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"++[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++### PipeLLM operator++PipeLLM is used to:+1. Generate text or objects with LLMs+2. Process images with Vision LLMs++#### Basic Usage++Simple Text Generation:+```plx+[pipe.write_story]+type = "PipeLLM"+description = "Write a short story"+output = "Text"+prompt = """+Write a short story about a programmer.+"""+```++Structured Data Extraction:+```plx+[pipe.extract_info]+type = "PipeLLM"+description = "Extract information"+inputs = { text = "Text" }+output = "PersonInfo"+prompt = """+Extract person information from this text:+@text+"""+```++Supports system instructions:+```plx+[pipe.expert_analysis]+type = "PipeLLM"+description = "Expert analysis"+output = "Analysis"+system_prompt = "You are a data analysis expert"+prompt = "Analyze this data"+```++#### Multiple Outputs++Generate multiple outputs (fixed number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+nb_output = 3 # Generate exactly 3 ideas+```++Generate multiple outputs (variable number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+multiple_output = true # Let the LLM decide how many to generate+```++#### Vision++Process images with VLMs (image inputs must be tagged in the prompt):+```plx+[pipe.analyze_image]+type = "PipeLLM"+description = "Analyze image"+inputs = { image = "Image" }+output = "ImageAnalysis"+prompt = """+Describe what you see in this image:++$image+"""+```++You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++#### Writing prompts for PipeLLM++**Insert stuff inside a tagged block**++If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.++Example template:+```plx+prompt = """+Match the expense with its corresponding invoice:++@expense++@invoices+"""+```+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.++DO NOT write things like "Here is the expense: @expense".+DO write simply "@expense" alone in an isolated line.++**Insert stuff inline**++If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.++Example template:+```plx+prompt = """+Your goal is to summarize everything related to $topic in the provided text:++@text++Please provide only the summary, with no additional text or explanations.+Your summary should not be longer than 2 sentences.+"""+```++In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.+Be sure to make the proper choice of prefix for each insertion.++DO NOT write "$topic" alone in an isolated line.+DO write things like "Write an essay about $topic" to include text into an actual sentence.+++### PipeExtract operator++The PipeExtract operator is used to extract text and images from an image or a PDF++#### Simple Text Extraction+```plx+[pipe.extract_info]+type = "PipeExtract"+description = "extract the information"+inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.+output = "Page"+```++Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.++The output concept `Page` is a native concept, with the structure `PageContent`:+It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`++```python+class TextAndImagesContent(StuffContent):+ text: TextContent | None+ images: list[ImageContent] | None++class PageContent(StructuredContent): # CONCEPT IS "Page"+ text_and_images: TextAndImagesContent+ page_view: ImageContent | None = None+```+- `text_and_images` are the text, and the related images found in the input image or PDF.+- `page_view` is the screenshot of the whole pdf page/image.++### PipeCompose operator++The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.++#### Basic Usage++Simple Template Composition:+```plx+[pipe.compose_report]+type = "PipeCompose"+description = "Compose a report using template"+inputs = { data = "ReportData" }+output = "Text"+template = """+## Report Summary++Based on the analysis:+$data++Generated on: {{ current_date }}+"""+```++Using Named Templates:+```plx+[pipe.use_template]+type = "PipeCompose"+description = "Use a predefined template"+inputs = { content = "Text" }+output = "Text"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+```++CRM Email Template:+```plx+[pipe.compose_follow_up_email]+type = "PipeCompose"+description = "Compose a personalized follow-up email for CRM"+inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }+output = "Text"+template_category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+template = """+Subject: Following up on our $deal.product_name discussion++Hi $customer.first_name,++I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.++Based on our discussion, I understand that your key requirements are: $deal.customer_requirements++I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:++**Next Steps:**+- Schedule a demo tailored to your specific needs+- Provide you with a customized quote based on your requirements+- Connect you with our implementation team++Would you be available for a 30-minute call this week? I have openings on:+{% for slot in available_slots %}+- {{ slot }}+{% endfor %}++Looking forward to moving this forward together!++Best regards,+$sales_rep.name+$sales_rep.title+$sales_rep.phone | $sales_rep.email+"""+```++#### Key Parameters++- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)+- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)+- `templating_style`: Styling options for template rendering+- `extra_context`: Additional context variables for template++For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++#### Template Variables++Use the same variable insertion rules as PipeLLM:+- `@variable` for block insertion (multi-line content)+- `$variable` for inline insertion (short text)++### PipeImgGen operator++The PipeImgGen operator is used to generate images using AI image generation models.++#### Basic Usage++Simple Image Generation:+```plx+[pipe.generate_image]+type = "PipeImgGen"+description = "Generate an image from prompt"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+```++Using Image Generation Settings:+```plx+[pipe.generate_photo]+type = "PipeImgGen"+description = "Generate a high-quality photo"+inputs = { prompt = "ImgGenPrompt" }+output = "Photo"+model = { model = "fast-img-gen" }+aspect_ratio = "16:9"+quality = "hd"+```++Multiple Image Generation:+```plx+[pipe.generate_variations]+type = "PipeImgGen"+description = "Generate multiple image variations"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+nb_output = 3+seed = "auto"+```++Advanced Configuration:+```plx+[pipe.generate_custom]+type = "PipeImgGen"+description = "Generate image with custom settings"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+model = "img_gen_preset_name" # Use predefined preset+aspect_ratio = "1:1"+quality = "hd"+background = "transparent"+output_format = "png"+is_raw = false+safety_tolerance = 3+```++#### Key Parameters++**Image Generation Settings:**+- `model`: Model choice (preset name or inline settings with model name)+- `quality`: Image quality ("standard", "hd")++**Output Configuration:**+- `nb_output`: Number of images to generate+- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)+- `output_format`: File format ("png", "jpeg", "webp")+- `background`: Background type ("default", "transparent")++**Generation Control:**+- `seed`: Random seed (integer or "auto")+- `is_raw`: Whether to apply post-processing+- `is_moderated`: Enable content moderation+- `safety_tolerance`: Content safety level (1-6)++#### Input Requirements++PipeImgGen requires exactly one input that must be either:+- An `ImgGenPrompt` concept+- A concept that refines `ImgGenPrompt`++The input can be named anything but must contain the prompt text for image generation.++### PipeFunc operator++The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.++#### Basic Usage++Simple Function Call:+```plx+[pipe.process_data]+type = "PipeFunc"+description = "Process data using custom function"+inputs = { input_data = "DataType" }+output = "ProcessedData"+function_name = "process_data_function"+```++File Processing Example:+```plx+[pipe.read_file]+type = "PipeFunc"+description = "Read file content"+inputs = { file_path = "FilePath" }+output = "FileContent"+function_name = "read_file_content"+```++#### Key Parameters++- `function_name`: Name of the Python function to call (must be registered in func_registry)++#### Function Requirements++The Python function must:++1. **Be registered** in the `func_registry`+2. **Accept `working_memory`** as a parameter:+ ```python+ async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:+ # Function implementation+ pass+ ```++3. **Return appropriate types**:+ - `StuffContent`: Single content object+ - `list[StuffContent]`: Multiple content objects (becomes ListContent)+ - `str`: Simple string (becomes TextContent)++#### Function Registration++Functions must be registered in the function registry before use:++```python+from pipelex.system.registries.func_registry import func_registry++@func_registry.register("my_function_name")+async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:+ # Access inputs from working memory+ input_data = working_memory.get_stuff("input_name")++ # Process data+ result = process_logic(input_data.content)++ # Return result+ return MyResultContent(data=result)+```++#### Working Memory Access++Inside the function, access pipeline inputs through working memory:++```python+async def process_function(working_memory: WorkingMemory) -> TextContent:+ # Get input stuff by name+ input_stuff = working_memory.get_stuff("input_name")++ # Access the content+ input_content = input_stuff.content++ # Process and return+ processed_text = f"Processed: {input_content.text}"+ return TextContent(text=processed_text)+```++---++### Rules to choose LLM models used in PipeLLMs.++#### LLM Configuration System++In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:++- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`++#### LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++#### Using an LLM Handle in a PipeLLM++Here is an example of using a model to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""+```++As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").++#### LLM Presets++Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:++```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:++The category of this invoice is: $invoice_details.category.++@invoice_text+"""+```++The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.++---++ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+Then, create an example file to run the pipeline in the `examples` folder.+But don't write documentation unless asked explicitly to.++## Guide to execute a pipeline and write example code++### Example to execute a pipeline with text output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+++async def hello_world() -> str:+ """+ This function demonstrates the use of a super simple Pipelex pipeline to generate text.+ """+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="hello_world",+ )++ return pipe_output.main_stuff_as_str+++## start Pipelex+Pipelex.make()+## run sample using asyncio+output_text = asyncio.run(hello_world())+pretty_print(output_text, title="Your first Pipelex output")+```++### Example to execute a pipeline with structured output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent++from my_project.gantt.gantt_struct import GanttChart++SAMPLE_NAME = "extract_gantt"+IMAGE_URL = "assets/gantt/gantt_tree_house.png"+++async def extract_gantt(image_url: str) -> GanttChart:+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )+ # Output the result+ return pipe_output.main_stuff_as(content_type=GanttChart)+++## start Pipelex+Pipelex.make()++## run sample using asyncio+gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))+pretty_print(gantt_chart, title="Gantt Chart")+```++### Setting up the input memory++#### Explanation of input memory++The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:+```python+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]+```+As you can seen, we made it so different ways can be used to define that stuff using structured content or data.++#### Different ways to set up the input memory++So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:++```python+## Here we have a single input and it's a Text.+## If you assign a string, by default it will be considered as a TextContent.+ pipe_output = await execute_pipeline(+ pipe_code="master_advisory_orchestrator",+ input_memory={+ "user_input": problem_description,+ },+ )++## Here we have a single input and it's a PDF.+## Because PDFContent is a native concept, we can use it directly as a value,+## the system knows what content it corresponds to:+ pipe_output = await execute_pipeline(+ pipe_code="power_extractor_dpe",+ input_memory={+ "document": PDFContent(url=pdf_url),+ },+ )++## Here we have a single input and it's an Image.+## Because ImageContent is a native concept, we can use it directly as a value:+ pipe_output = await execute_pipeline(+ pipe_code="fashion_variation_pipeline",+ input_memory={+ "fashion_photo": ImageContent(url=image_url),+ },+ )++## Here we have a single input, it's an image but+## its actually a more specific concept gantt.GanttImage which refines Image,+## so we must provide it using a dict with the concept and the content:+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )++## Here is a more complex example with multiple inputs assigned using different ways:+ pipe_output = await execute_pipeline(+ pipe_code="retrieve_then_answer",+ dynamic_output_concept_code="contracts.Fees",+ input_memory={+ "text": load_text_from_path(path=text_path),+ "question": {+ "concept": "answer.Question",+ "content": question,+ },+ "client_instructions": client_instructions,+ },+ )+```++### Using the outputs of a pipeline++All pipe executions return a `PipeOutput` object.+It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.+It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:++```python++class PipeOutput(BaseModel):+ working_memory: WorkingMemory = Field(default_factory=WorkingMemory)+ pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)++ @property+ def main_stuff(self) -> Stuff:+ ...++ def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:+ ...++ def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:+ ...++ def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:+ ...++ @property+ def main_stuff_as_text(self) -> TextContent:+ ...++ @property+ def main_stuff_as_str(self) -> str:+ ...++ @property+ def main_stuff_as_image(self) -> ImageContent:+ ...++ @property+ def main_stuff_as_text_and_image(self) -> TextAndImagesContent:+ ...++ @property+ def main_stuff_as_number(self) -> NumberContent:+ ...++ @property+ def main_stuff_as_html(self) -> HtmlContent:+ ...++ @property+ def main_stuff_as_mermaid(self) -> MermaidContent:+ ...+```++As you can see, you can extract any variable from the output working memory.++#### Getting the main stuff as a specific type++Simple text as a string:++```python+result = pipe_output.main_stuff_as_str+```+Structured object (BaseModel):++```python+result = pipe_output.main_stuff_as(content_type=GanttChart)+```++If it's a list, you can get a `ListContent` of the specific type.++```python+result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)+```++or if you want, you can get the actual items as a regular python list:++```python+result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)+```++---++## Rules to choose LLM models used in PipeLLMs.++### LLM Configuration System++In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:++- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`++### LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++### Using an LLM Handle in a PipeLLM++Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""+```++As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").++### LLM Presets++Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:++```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:++The category of this invoice is: $invoice_details.category.++@invoice_text+"""+```++The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.+++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+<!-- END_PIPELEX_RULES -->diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f2bdec807..78556d0fe 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,12 +2,80 @@## [Unreleased]-### Highlights--- In the `PipeLLM`, the image inputs can now be used and tagged in the prompt like all other concepts.-- Use claude-4.5-sonnet instead of claude-4-sonnet in the base deck.+### Highlights - Moving fast and breaking things++- Added the new builder pipeline system for auto-generating Pipelex bundles from user briefs+ - it's a pipeline to generate pipelines, and it works!+ - the pipeline definitions are in `pipelex_libraries/pipelines/base_library/builder/`+ - removed the previous draft which was named `meta_pipeline.plx`++**Breaking changes... for good!**++We tried to group all the renamings we wanted to do which impact our language, so that you get one migration to apply and then we will be way more stable in the future releases.++This is all in the spirit of making Pipelex a declarative language, where you express what you want to do, and the system will figure out how to do it. So our focus inwas to make the Pipelex language easier to understand and use for non-technical users, and at the same time use more consistent and obvious words that developers are used to.++**💡 Pro tip:** To make migration easier, pass the [migration guide](https://github.com/PipelexLab/pipelex/blob/main/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md) to your favorite SWE agent (Cursor, Claude Code, github copilot, etc.) and let it handle the bulk of the changes!++- **Removed centralized `pipelex_libraries` folder system**+ - Pipelines are now auto-discovered from anywhere in your project—no special directory required+ - No config path parameters needed in `Pipelex.make()` or CLI commands (just call `Pipelex.make()`)+ - Custom functions require `@pipe_func()` decorator for auto-discovery+ - Structure classes auto-discovered (must inherit from `StructuredContent`)+ - Configuration stays at repository root in `.pipelex/` directory+ - See [migration guide](https://github.com/PipelexLab/pipelex/blob/main/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md) for details on reorganizing your project structure++- General changes+ - renamed `definition` fields to `description` across all cases++- Renamed **PipeJinja2** to **PipeCompose**+ - the fact that our templating engine is Jinja2 is a technnical detail, not fundamental to the language, especially since we included a pre-processor enabling insertion of variables in prompts using `@variable` or `$variable`, in addition to the jinja2 syntax `{{ variable }}`+ - renamed `jinja2` field to `template` for the same reason+ - for more control, instead of providing a string for the `template` field, you can also use a nested `template` section with `template`, `category` and `templating_style` fields++- Renamed **PipeOCR** to **PipeExtract**+ - this is to account for various text extraction techniques from images and docs, including but not only OCR; e.g. we now have integrated the `pypdfium2` package which can extract text and images from PDF, when it's actually real text (not an image), and soon we'll add support for other document extraction models solutions+ - removed obligation to name your document input `ocr_input`, it can now be named whatever you want as long as it's a single input and it's either an `Image` or a `PDF` or some concept refining PDF or Image+ - renamed `ocr_page_contents_from_pdf` to `extract_page_contents_from_pdf`+ - renamed `ocr_page_contents_and_views_from_pdf` to `extract_page_contents_and_views_from_pdf`+ - introduced model settings and presets for extract models like we had for LLMs+ - renamed `ocr_model` to `model` for choice of model, preset, or explicit setting and introduced `base_ocr_mistral` as an alias to `mistral-ocr`++- **PipeLLM** field renames+ - image inputs must now be tagged in the prompt like all other inputs; you can just drop their names at the beginning or end of the prompt, or you can reference them in meaningful sentences to guide the Visual LLM, e.g. "Analyze the colors in $some_photo and the shapes in $some_painting."+ - renamed `prompt_template` field to `prompt`+ - renamed `llm` field to `model`+ - renamed `llm_to_structure` field to `model_to_structure`++- **PipeImgGen** field renames+ - renamed `img_gen` field to `model` for choice of model, preset, or explicit setting+ - removed some technical settings such as `nb_steps` from the pipe attributes, instead you can set these as model settings or model presets+ - introduced model settings and presets for image generation models like we had for LLMs++- **PipeCondition** field renames+ - renamed `pipe_map` to `outcomes`+ - renamed `default_pipe_code` to `default_outcome` and it's now a required field, because we need to know what to do if the expression doesn't match any key in the outcomes map; if you don't know what to do in that case, then it's a failure and you can use the `fail` value++- **Configuration file changes** (`.pipelex/` directory)+ - Renamed parameter `llm_handle` to `model` across all LLM presets in deck files+ - Renamed parameter `img_gen_handle` to `model` across all image generation presets in deck files+ - Renamed parameter `ocr_handle` to `model` in extraction presets+ - Renamed `ocr` section to `extract` throughout configuration files+ - Renamed `ocr_config` to `extract_config` in `pipelex.toml`+ - Renamed `base_ocr_pypdfium2` to `base_extract_pypdfium2`+ - Renamed `is_auto_setup_preset_ocr` to `is_auto_setup_preset_extract`+ - Renamed `nb_ocr_pages` to `nb_extract_pages`+ - Updated pytest marker from 'ocr' to 'extract'### Added+ - Added `cheap-gpt` model alias for `gpt-4o-mini`+ - Added `cheap_llm_for_vision` preset using `gemini-2.5-flash-lite`+ - Added `llm_for_testing_vision` and `llm_for_testing_vision_structured` presets for vision testing+ - Added `is_dump_text_prompts_enabled` and `is_dump_response_text_enabled` configuration flags to have the console display everything that goes in and out of the LLMs+ - Added `generic_templates` section in `llm_config` with structure extraction prompts+ - Added useful error messages with migration configuration maps pin-pointing the fields to rename for config and plx files+ - Added improved error message for `PipeFunc` when function not found in registry, mentioning `@pipe_func()` decorator requirement since v0.12.0+ - Added pytest filterwarnings to ignore deprecated class-based config warnings- Added `Flow` class that represents the flow of pipe signatures- Added `pipe-builder` command `flow` to generate flow view from pipeline brief- Added `FlowFactory` class to create Flow from PipelexBundleSpec or PLX files@@ -26,10 +94,15 @@- Added `MissingDependencyError` exception for missing optional dependencies### Changed+ - Replaced package `toml` by `tomli` which is more modern and faster+ - Updated Gemini 2.0 model from `gemini-2.0-flash-exp` to `gemini-2.0-flash` with new pricing (input: $0.10, output: $0.40 per million tokens)+ - Updated Gemini 2.5 Series comment from '(when available)' to stable release+ - Updated `base-claude` from `claude-4-sonnet` to `claude-4.5-sonnet` across all presets+ - Updated kajson dependency from version `0.3.0` to `0.3.1`- Cleanup env example and better explain how to set up keys in README and docs- - Changed Gemini model configuration from `gemini-2.0-flash-exp` (free tier) to `gemini-2.0-flash` with pricing ($0.10 input, $0.40 output per million tokens)- - Removed Gemini 1.5 series models (gemini-1.5-pro, gemini-1.5-flash, gemini-1.5-flash-8b) from configuration- Changed Gemini routing from `google` backend to `pipelex_inference` backend+ - **BREAKING:** Split `pipelex.core.stuffs.stuff_content` module into individual files per content type (affects imports: `StructuredContent`, `TextContent`, `ImageContent`, `ListContent`, `PDFContent`, `PageContent`, `NumberContent`, `HtmlContent`, `MermaidContent`, `TextAndImagesContent`)+ - **BREAKING:** Renamed package `pipelex.pipe_works` to `pipelex.pipe_run` and moved `PipeRunParams` classes into it- Renamed `ConceptProviderAbstract` to `ConceptLibraryAbstract`- Renamed `DomainProviderAbstract` to `DomainLibraryAbstract`- Renamed `PipeProviderAbstract` to `PipeLibraryAbstract`@@ -43,12 +116,13 @@- Changed `PipeLLM` validation to check all inputs are in required variables- Updated `LLMPromptSpec` to handle image collections (lists/tuples) in addition to single images- Changed Mermaid diagram URL generation from `/img/` to `/svg/` endpoint- - Updated kajson dependency from 0.3.0 to 0.3.1- Changed `PipeLLMPromptTemplate.make_llm_prompt()` to private method `_make_llm_prompt()`- Updated pipe-builder prompts to include concept specs for better context- Updated `PipelexBundleSpec.to_blueprint()` to sort pipes by dependencies before creating bundle- Changed exception base class from `PipelexError` to `PipelexException` throughout codebase- Updated Makefile pyright target to use `--pythonpath` flag correctly+ - Updated PipeFunc documentation to reflect `@pipe_func()` decorator requirement and auto-discovery from anywhere in project+ - Added warnings about module-level code execution during auto-discovery to PipeFunc and StructuredContent documentation### Fixed- Fixed Makefile target `pyright` to use correct pythonpath flag@@ -59,12 +133,17 @@- Updated README badge URL to point to main branch instead of feature/pipe-builder branch### Removed+ - Removed centralized `pipelex_libraries` folder system and `pipelex init libraries` command+ - Removed config path parameters from `Pipelex.make()` (`relative_config_folder_path`, `config_folder_path`, `from_file`)+ - Removed Gemini 1.5 series models: `gemini-1.5-pro`, `gemini-1.5-flash`, and `gemini-1.5-flash-8b`+ - Removed `base_templates.toml` file (generic prompts moved to `pipelex.toml`)- Removed `gpt-5-mini` from possible models in pipe-builder- Removed useless functions in `LLMJobFactory`: `make_llm_job_from_prompt_factory()`, `make_llm_job_from_prompt_template()`, `make_llm_job_from_prompt_contents()`- Removed `add_or_update_pipe()` method from PipeLibrary- Removed `get_optional_library_manager()` method from PipelexHub- Removed `get_optional_domain_provider()` and `get_optional_concept_provider()` methods from hub- Removed unused test fixtures (apple, cherry, blueberry, concept_provider, pretty) from conftest.py+ - Removed some Vision/Image description pipes from the base library, because we doubt they were useful as they were## [v0.11.0] - 2025-10-01diff --git a/CLAUDE.md b/CLAUDE.mdindex 7339ec32a..4d612f7da 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,195 +1,71 @@-# Coding Standards & Best Practices+<!-- BEGIN_PIPELEX_RULES -->+## Guide to write or edit pipelines using the Pipelex language in .plx files-This document outlines the core coding standards, best practices, and quality control procedures for the codebase.--## Type Hints--1. **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tupele types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use the `|` syntax for union types (e.g `str | int`) and `| None` for optionals- - Use Field(default_factory=...) for mutable defaults and if it's a list of something else than str, use `empty_list_factory_of()` to make a factory: `number_list: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")`- - Use `BaseModel` and respect Pydantic v2 standards, in particular use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose--2. **StrEnum**- - Import from `pipelex.types`:- ```python- from pipelex.types import StrEnum- ```--3. **Self type**- - Import from `pipelex.types`:- ```python- from pipelex.types import Self- ```--## Factory Pattern-- - Use Factory Pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such--## Error Handling-- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones- - Never catch Exception, only catch specific exceptions- - Always add `from exc` to the exception-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```-- **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.--## Documentation--1. **Docstring Format**- ```python- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:- """Process and resize an image.-- Args:- image_path: Path to the source image- size: Tuple of (width, height) for resizing-- Returns:- Processed image as bytes- """- pass- ```--2. **Class Documentation**- ```python- class ImageProcessor:- """Handles image processing operations.-- Provides methods for resizing, converting, and optimizing images.- """- ```--## Code Quality Checks--### Linting and Type Checking--Before finalizing a task, run:-```bash-make fix-unused-imports-make check-```--This runs multiple code quality tools:-- Pyright: Static type checking-- Ruff: Fast Python linter-- Mypy: Static type checker--Always fix any issues reported by these tools before proceeding.--### Running Tests--1. **Quick Test Run** (no LLM/image generation):- ```bash- make tp- ```- Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or ocr)) and not (needs_output or pipelex_api)`--2. **Specific Tests**:- ```bash- make tp TEST=TestClassName- # or- make tp TEST=test_function_name- ```- Note: Matches names starting with the provided string.--**Important**: Never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-img-gen` - these use costly inference.--## Pipelines--- All pipeline definitions go in `pipelex/libraries/pipelines/`-- Always validate pipelines after creation/edit with `make validate`.- Iterate if there are errors.--## Project Structure--- **Pipelines**: `pipelex/libraries/pipelines/`-- **Tests**: `tests/` directory-- **Documentation**: `docs/` directory-------# Guide to write or edit pipelines using the Pipelex language in .plx files--- Always first write your "plan" in natural langage, then transcribe it in pipelex.+- Always first write your "plan" in natural language, then transcribe it in pipelex.- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.-- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)-## Pipeline File Naming+### Pipeline File Naming- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)-- Files must be `.py` for structures+- Files must be `.py` for code defining the data structures- Use descriptive names in `snake_case`-## Pipeline File Structure+### Pipeline File OutlineA pipeline file has three main sections:1. Domain statement2. Concept definitions3. Pipe definitions-### Domain Statement+#### Domain Statement```plxdomain = "domain_name"description = "Description of the domain" # Optional```Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.-### Concept Definitions+#### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.+```plx[concept]-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output+ConceptName = "Description of the concept"```-Important Rules:+**Naming Rules:**- Use PascalCase for concept names-- Never use plurals (no "Stories", use "Story")-- Avoid adjectives (no "LargeText", use "Text")-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number)-yes-### Pipe Definitions+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)-## Pipe Base Structure+**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.-```plx-[pipe.your_pipe_name]-type = "PipeLLM"-description = "A description of what your pipe does"-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }-output = "ConceptName"-```+**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:-DO NOT WRITE:```plx-[pipe.your_pipe_name]-type = "pipe_sequence"+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"```-But it should be:+For details on how to structure concepts with fields, see the "Structuring Models" section below.++#### Pipe Definitions++### Pipe Base Definition```plx[pipe.your_pipe_name]-type = "PipeSequence"-description = "....."+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"```-The pipes will all have at least this base structure.-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditionnal pipes (if PipeCondition).+The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).So If you have this error:`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •variable='['invoice']'``@@ -207,74 +83,108 @@ inputs = {- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:-## Structuring Models+### Structuring Models-### Model Location and Registration+Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.-- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`-- Models must inherit from `StructuredContent` or appropriate content type+#### Three Ways to Structure Concepts-## Model Structure+**1. No Structure Needed**-Concepts and their structure classes are meant to indicate an idea.-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.+If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.-**IMPORTANT: Never create unnecessary structure classes that only refine native concepts without adding fields.**+**2. Inline Structure Definition (RECOMMENDED for most cases)**-DO NOT create structures like:-```python-class Joke(TextContent):- """A humorous text that makes people laugh."""- pass+For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }```-If a concept only refines a native concept (like Text, Image, etc.) without adding new fields, simply declare it in the .plx file:+**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):```plx-[concept]-Joke = "A humorous text that makes people laugh."+field_name = "Field description"```-If you simply need to refine another native concept, construct it like this:++**Detailed syntax** (with explicit properties):```plx-[concept.Landscape]-refines = "Image"+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }```-Only create a Python structure class when you need to add specific fields:-```python-from datetime import datetime-from typing import List, Optional-from pydantic import Field+**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains+```pythonfrom pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator-# IMPORTANT: THE CLASS MUST BE A SUBCLASS OF StructuredContent-class YourModel(StructuredContent): # Always be a subclass of StructuredContent- # Required fields- field1: str- field2: int+class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```- # Optional fields with defaults- field3: Optional[str] = Field(None, "Description of field3")- field4: List[str] = Field(default_factory=list)+**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.- # Date fields should remove timezone- date_field: Optional[datetime] = None-```-### Usage+#### Decision Rules for Agents-Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.+**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class-Structure classes defined within `pipelex_libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.+**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)+**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required-### Best Practices for structures+#### Inline Structure Limitations-- Respect Pydantic v2 standards-- Use type hints for all fields-- Use `Field` declaration and write the description+Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes-## Pipe Controllers and Pipe Operators+### Pipe Controllers and Pipe OperatorsLook at the Pipes we have in order to adapt it. Pipes are organized in two categories:@@ -290,11 +200,11 @@ Look at the Pipes we have in order to adapt it. Pipes are organized in two categ- `PipeImgGen` - Generate Images- `PipeFunc` - For running classic python scripts-## PipeSequence controller+### PipeSequence controllerPurpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.-### Basic Structure+#### Basic Definition```plx[pipe.your_sequence_name]type = "PipeSequence"@@ -308,13 +218,13 @@ steps = []```-### Key Components+#### Key Components1. **Steps Array**: List of pipes to execute in sequence- `pipe`: Name of the pipe to execute- `result`: Name to assign to the pipe's output that will be in the working memory-### Using PipeBatch in Steps+#### Using PipeBatch in StepsYou can use PipeBatch functionality within steps using `batch_over` and `batch_as`:@@ -335,21 +245,22 @@ steps = [The result of a batched step will be a `ListContent` containing the outputs from processing each item.-## PipeCondition controller+### PipeCondition controllerThe PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.-### Basic usage+#### Basic usage```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"@@ -358,32 +269,49 @@ or```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+description = "A conditional pipe to decide whether..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"-[pipe.conditional_operation.pipe_map]+[pipe.conditional_operation.outcomes]small = "process_small"medium = "process_medium"large = "process_large"```-### Key Parameters+#### Key Parameters- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)-- `pipe_map`: Dictionary mapping expression results to pipe codes :-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"-## PipeLLM operator+[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++### PipeLLM operatorPipeLLM is used to:1. Generate text or objects with LLMs2. Process images with Vision LLMs-### Basic Usage+#### Basic UsageSimple Text Generation:```plx@@ -391,7 +319,7 @@ Simple Text Generation:type = "PipeLLM"description = "Write a short story"output = "Text"-prompt_template = """+prompt = """Write a short story about a programmer."""```@@ -403,7 +331,7 @@ type = "PipeLLM"description = "Extract information"inputs = { text = "Text" }output = "PersonInfo"-prompt_template = """+prompt = """Extract person information from this text:@text"""@@ -416,10 +344,10 @@ type = "PipeLLM"description = "Expert analysis"output = "Analysis"system_prompt = "You are a data analysis expert"-prompt_template = "Analyze this data"+prompt = "Analyze this data"```-### Multiple Outputs+#### Multiple OutputsGenerate multiple outputs (fixed number):```plx@@ -439,19 +367,33 @@ output = "Idea"multiple_output = true # Let the LLM decide how many to generate```-### Vision+#### Vision-Process images with VLMs:+Process images with VLMs (image inputs must be tagged in the prompt):```plx[pipe.analyze_image]type = "PipeLLM"description = "Analyze image"-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in an attribute within a stuff, you can add something like `{ "page.image": "Image" }+inputs = { image = "Image" }output = "ImageAnalysis"-prompt_template = "Describe what you see in this image"+prompt = """+Describe what you see in this image:++$image+"""```-### Writing prompts for PipeLLM+You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++#### Writing prompts for PipeLLM**Insert stuff inside a tagged block**@@ -459,7 +401,7 @@ If the inserted text is supposedly a long text, made of several lines or paragraExample template:```plx-prompt_template = """+prompt = """Match the expense with its corresponding invoice:@expense@@ -467,7 +409,7 @@ Match the expense with its corresponding invoice:@invoices"""```-In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.DO NOT write things like "Here is the expense: @expense".DO write simply "@expense" alone in an isolated line.@@ -478,7 +420,7 @@ If the inserted text is short text and it makes sense to have it inserted directExample template:```plx-prompt_template = """+prompt = """Your goal is to summarize everything related to $topic in the provided text:@text@@ -495,11 +437,11 @@ DO NOT write "$topic" alone in an isolated line.DO write things like "Write an essay about $topic" to include text into an actual sentence.-## PipeExtract operator+### PipeExtract operatorThe PipeExtract operator is used to extract text and images from an image or a PDF-### Simple Text Extraction+#### Simple Text Extraction```plx[pipe.extract_info]type = "PipeExtract"@@ -508,28 +450,38 @@ inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This isoutput = "Page"```-Only one input is allowed and it must either be an `Image` or a `PDF`.+Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.The output concept `Page` is a native concept, with the structure `PageContent`:It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page````pythonclass TextAndImagesContent(StuffContent):- text: Optional[TextContent]- images: Optional[List[ImageContent]]+ text: TextContent | None+ images: list[ImageContent] | Noneclass PageContent(StructuredContent): # CONCEPT IS "Page"text_and_images: TextAndImagesContent- page_view: Optional[ImageContent] = None+ page_view: ImageContent | None = None```- `text_and_images` are the text, and the related images found in the input image or PDF.- `page_view` is the screenshot of the whole pdf page/image.-## PipeCompose operator+### PipeCompose operatorThe PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.-### Basic Usage+#### Basic UsageSimple Template Composition:```plx@@ -538,8 +490,8 @@ type = "PipeCompose"description = "Compose a report using template"inputs = { data = "ReportData" }output = "Text"-jinja2 = """-# Report Summary+template = """+## Report SummaryBased on the analysis:$data@@ -555,7 +507,21 @@ type = "PipeCompose"description = "Use a predefined template"inputs = { content = "Text" }output = "Text"-jinja2_name = "standard_report_template"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }```CRM Email Template:@@ -567,7 +533,7 @@ inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }output = "Text"template_category = "html"templating_style = { tag_style = "square_brackets", text_format = "html" }-jinja2 = """+template = """Subject: Following up on our $deal.product_name discussionHi $customer.first_name,@@ -597,25 +563,30 @@ $sales_rep.phone | $sales_rep.email"""```-### Key Parameters+#### Key Parameters-- `jinja2`: Inline Jinja2 template (mutually exclusive with jinja2_name)-- `jinja2_name`: Name of a predefined template (mutually exclusive with jinja2)+- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)- `templating_style`: Styling options for template rendering- `extra_context`: Additional context variables for template-### Template Variables+For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++#### Template VariablesUse the same variable insertion rules as PipeLLM:- `@variable` for block insertion (multi-line content)- `$variable` for inline insertion (short text)-## PipeImgGen operator+### PipeImgGen operatorThe PipeImgGen operator is used to generate images using AI image generation models.-### Basic Usage+#### Basic UsageSimple Image Generation:```plx@@ -624,7 +595,6 @@ type = "PipeImgGen"description = "Generate an image from prompt"inputs = { prompt = "ImgGenPrompt" }output = "Image"-model = "flux-pro/v1.1-ultra"```Using Image Generation Settings:@@ -634,9 +604,9 @@ type = "PipeImgGen"description = "Generate a high-quality photo"inputs = { prompt = "ImgGenPrompt" }output = "Photo"-model = { model = "flux-pro/v1.1-ultra", quality = "hd" }+model = { model = "fast-img-gen" }aspect_ratio = "16:9"-nb_steps = 8+quality = "hd"```Multiple Image Generation:@@ -666,14 +636,11 @@ is_raw = falsesafety_tolerance = 3```-### Key Parameters+#### Key Parameters**Image Generation Settings:**-- `img_gen`: ImgGenChoice (preset name or inline settings)-- `img_gen_handle`: Direct model handle (legacy)+- `model`: Model choice (preset name or inline settings with model name)- `quality`: Image quality ("standard", "hd")-- `nb_steps`: Number of generation steps-- `guidance_scale`: How closely to follow the prompt**Output Configuration:**- `nb_output`: Number of images to generate@@ -687,7 +654,7 @@ safety_tolerance = 3- `is_moderated`: Enable content moderation- `safety_tolerance`: Content safety level (1-6)-### Input Requirements+#### Input RequirementsPipeImgGen requires exactly one input that must be either:- An `ImgGenPrompt` concept@@ -695,11 +662,11 @@ PipeImgGen requires exactly one input that must be either:The input can be named anything but must contain the prompt text for image generation.-## PipeFunc operator+### PipeFunc operatorThe PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.-### Basic Usage+#### Basic UsageSimple Function Call:```plx@@ -721,11 +688,11 @@ output = "FileContent"function_name = "read_file_content"```-### Key Parameters+#### Key Parameters- `function_name`: Name of the Python function to call (must be registered in func_registry)-### Function Requirements+#### Function RequirementsThe Python function must:@@ -742,12 +709,12 @@ The Python function must:- `list[StuffContent]`: Multiple content objects (becomes ListContent)- `str`: Simple string (becomes TextContent)-### Function Registration+#### Function RegistrationFunctions must be registered in the function registry before use:```python-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registry@func_registry.register("my_function_name")async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:@@ -761,7 +728,7 @@ async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:return MyResultContent(data=result)```-### Working Memory Access+#### Working Memory AccessInside the function, access pipeline inputs through working memory:@@ -780,9 +747,9 @@ async def process_function(working_memory: WorkingMemory) -> TextContent:----## Rules to choose LLM models used in PipeLLMs.+### Rules to choose LLM models used in PipeLLMs.-### LLM Configuration System+#### LLM Configuration SystemIn order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:@@ -791,7 +758,7 @@ LLM configurations are managed through the new inference backend system with fil- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`- **Routing**: `.pipelex/inference/routing_profiles.toml`-### LLM Handles+#### LLM HandlesAn llm_handle can be either:1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system@@ -807,9 +774,9 @@ base-mistral = "mistral-medium"The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-### Using an LLM Handle in a PipeLLM+#### Using an LLM Handle in a PipeLLM-Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:+Here is an example of using a model to specify which LLM to use in a PipeLLM:```plx[pipe.hello_world]@@ -817,14 +784,14 @@ type = "PipeLLM"description = "Write text about Hello World."output = "Text"model = { model = "gpt-5", temperature = 0.9 }-prompt_template = """+prompt = """Write a haiku about Hello World."""```As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-### LLM Presets+#### LLM PresetsPresets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.@@ -843,7 +810,7 @@ description = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"model = "llm_to_extract_invoice"-prompt_template = """+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -863,11 +830,9 @@ ALWAYS RUN `make validate` when you are finished writing pipelines: This checksThen, create an example file to run the pipeline in the `examples` folder.But don't write documentation unless asked explicitly to.------# Guide to write an example to execute a pipeline+## Guide to execute a pipeline and write example code-## Example to execute a pipeline with text output+### Example to execute a pipeline with text output```pythonimport asyncio@@ -889,14 +854,14 @@ async def hello_world() -> str:return pipe_output.main_stuff_as_str-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciooutput_text = asyncio.run(hello_world())pretty_print(output_text, title="Your first Pipelex output")```-## Example to execute a pipeline with structured output+### Example to execute a pipeline with structured output```pythonimport asyncio@@ -904,8 +869,9 @@ import asynciofrom pipelex import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent-from pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart+from my_project.gantt.gantt_struct import GanttChartSAMPLE_NAME = "extract_gantt"IMAGE_URL = "assets/gantt/gantt_tree_house.png"@@ -926,32 +892,32 @@ async def extract_gantt(image_url: str) -> GanttChart:return pipe_output.main_stuff_as(content_type=GanttChart)-# start Pipelex+## start PipelexPipelex.make()-# run sample using asyncio+## run sample using asynciogantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))pretty_print(gantt_chart, title="Gantt Chart")```-## Setting up the input memory+### Setting up the input memory-### Explanation of input memory+#### Explanation of input memoryThe input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:```python-StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str-ImplicitMemory = Dict[str, StuffContentOrData]+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]```As you can seen, we made it so different ways can be used to define that stuff using structured content or data.-### Different ways to set up the input memory+#### Different ways to set up the input memorySo here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:```python-# Here we have a single input and it's a Text.-# If you assign a string, by default it will be considered as a TextContent.+## Here we have a single input and it's a Text.+## If you assign a string, by default it will be considered as a TextContent.pipe_output = await execute_pipeline(pipe_code="master_advisory_orchestrator",input_memory={@@ -959,9 +925,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's a PDF.-# Because PDFContent is a native concept, we can use it directly as a value,-# the system knows what content it corresponds to:+## Here we have a single input and it's a PDF.+## Because PDFContent is a native concept, we can use it directly as a value,+## the system knows what content it corresponds to:pipe_output = await execute_pipeline(pipe_code="power_extractor_dpe",input_memory={@@ -969,8 +935,8 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input and it's an Image.-# Because ImageContent is a native concept, we can use it directly as a value:+## Here we have a single input and it's an Image.+## Because ImageContent is a native concept, we can use it directly as a value:pipe_output = await execute_pipeline(pipe_code="fashion_variation_pipeline",input_memory={@@ -978,9 +944,9 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here we have a single input, it's an image but-# its actually a more specific concept gantt.GanttImage which refines Image,-# so we must provide it using a dict with the concept and the content:+## Here we have a single input, it's an image but+## its actually a more specific concept gantt.GanttImage which refines Image,+## so we must provide it using a dict with the concept and the content:pipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",input_memory={@@ -991,7 +957,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa},)-# Here is a more complex example with multiple inputs assigned using different ways:+## Here is a more complex example with multiple inputs assigned using different ways:pipe_output = await execute_pipeline(pipe_code="retrieve_then_answer",dynamic_output_concept_code="contracts.Fees",@@ -1006,7 +972,7 @@ So here are a few concrete examples of calls to execute_pipeline with various wa)```-## Using the outputs of a pipeline+### Using the outputs of a pipelineAll pipe executions return a `PipeOutput` object.It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.@@ -1060,9 +1026,9 @@ class PipeOutput(BaseModel):...```-As you can see, you can extarct any variable from the output working memory.+As you can see, you can extract any variable from the output working memory.-### Getting the main stuff as a specific type+#### Getting the main stuff as a specific typeSimple text as a string:@@ -1089,160 +1055,81 @@ result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)----# Writing unit tests--## Unit test generalities--NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.--### Test file structure--- Name test files with `test_` prefix-- Use descriptive names that match the functionality being tested-- Place test files in the appropriate test category directory:- - `tests/unit/` - for unit tests that test individual functions/classes in isolation- - `tests/integration/` - for integration tests that test component interactions- - `tests/e2e/` - for end-to-end tests that test complete workflows- - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)-- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest-- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.-- Always put test inside Test classes.-- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`--### Markers--Apply the appropriate markers:-- "llm: uses an LLM to generate text or objects"-- "img_gen: uses an image generation AI"-- "inference: uses either an LLM or an image generation AI"-- "gha_disabled: will not be able to run properly on GitHub Actions"--Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.--### Important rules--- Never use the unittest.mock. Use pytest-mock.--### Test Class Structure+## Rules to choose LLM models used in PipeLLMs.-Always group the tests of a module into a test class:+### LLM Configuration System-```python[email protected](loop_scope="class")-class TestFooBar:- @pytest.mark.parametrize(- "topic test_case_blueprint",- [- TestCases.CASE_1,- TestCases.CASE_2,- ],- )- async def test_pipe_processing(- self,- request: FixtureRequest,- topic: str,- test_case_blueprint: StuffBlueprint,- ):- # Test implementation-```+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:-Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`-## Writing integration test to test pipes+### LLM Handles-### Required imports for pipe tests+An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:-```python-import pytest-from pytest import FixtureRequest-from pipelex import log, pretty_print-from pipelex.core.stuffs.stuff_factory import StuffBlueprint, StuffFactory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.hub import get_report_delegate-from pipelex.libraries.pipelines.base_library.retrieve import RetrievedExcerpt-from pipelex.config_pipelex import get_config--from pipelex.core.pipe import PipeAbstract, update_job_metadata_for_pipe-from pipelex.core.pipes.pipe_output import PipeOutput, PipeOutputType-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.core.pipes.pipe_run_params import PipeRunParams-from pipelex.pipe_works.pipe_router_protocol import PipeRouterProtocol+```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"```-### Pipe test implementation steps--1. Create Stuff from blueprint:+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.-```python-stuff = StuffFactory.make_stuff(- concept_code="RetrievedExcerpt",- domain="retrieve",- content=RetrievedExcerpt(text="<Some retrieved text>", justification="<Some justification>")- name="retrieved_text",-)-```+### Using an LLM Handle in a PipeLLM-2. Create Working Memory:+Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:-```python-working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""```-3. Run the pipe:+As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").-```python-pipe_output = await pipe_router.run_pipe(- pipe_code="pipe_name",- pipe_run_params=PipeRunParamsFactory.make_run_params(),- working_memory=working_memory,- job_metadata=JobMetadata(),-)-```+### LLM Presets-4. Basic assertions:+Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.-```python-assert pipe_output is not None-assert pipe_output.working_memory is not None-assert pipe_output.main_stuff is not None+Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }```-### Test Data Organization+The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:-- If it's not already there, create a `test_data.py` file in the test directory-- Define test cases using `StuffBlueprint`:+```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:-```python-class TestCases:- CASE_BLUEPRINT_1 = StuffBlueprint(- name="test_case_1",- concept_code="domain.ConceptName1",- value="test_value"- )- CASE_BLUEPRINT_2 = StuffBlueprint(- name="test_case_2",- concept_code="domain.ConceptName2",- value="test_value"- )+The category of this invoice is: $invoice_details.category.- CASE_BLUEPRINTS: ClassVar[List[Tuple[str, str]]] = [ # topic, blueprint"- ("topic1", CASE_BLUEPRINT_1),- ("topic2", CASE_BLUEPRINT_2),- ]+@invoice_text+"""```-Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.-Also note that we provide a topic for the test case, which is purely for convenience.+The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.-## Best Practices for Testing-- Use parametrize for multiple test cases-- Test both success and failure cases-- Verify working memory state-- Check output structure and content-- Use meaningful test case names-- Include docstrings explaining test purpose-- Log outputs for debugging-- Generate reports for cost tracking+You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+<!-- END_PIPELEX_RULES -->diff --git a/Makefile b/Makefileindex e578a9da8..16085ff07 100644--- a/Makefile+++ b/Makefile@@ -7,14 +7,15 @@ PROJECT_NAME := $(shell grep '^name = ' pyproject.toml | sed -E 's/name = "(.*)"# The "?" is used to make the variable optional, so that it can be overridden by the user.PYTHON_VERSION ?= 3.11-VENV_PYTHON := $(VIRTUAL_ENV)/bin/python-VENV_PYTEST := $(VIRTUAL_ENV)/bin/pytest-VENV_RUFF := $(VIRTUAL_ENV)/bin/ruff-VENV_PYRIGHT := $(VIRTUAL_ENV)/bin/pyright-VENV_MYPY := $(VIRTUAL_ENV)/bin/mypy-VENV_PIPELEX := $(VIRTUAL_ENV)/bin/pipelex-VENV_MKDOCS := $(VIRTUAL_ENV)/bin/mkdocs-VENV_PYLINT := $(VIRTUAL_ENV)/bin/pylint+# Note: VENV_* variables include quotes to handle paths with spaces (e.g., "My Projects/pipelex")+VENV_PYTHON := "$(VIRTUAL_ENV)/bin/python"+VENV_PYTEST := "$(VIRTUAL_ENV)/bin/pytest"+VENV_RUFF := "$(VIRTUAL_ENV)/bin/ruff"+VENV_PYRIGHT := "$(VIRTUAL_ENV)/bin/pyright"+VENV_MYPY := "$(VIRTUAL_ENV)/bin/mypy"+VENV_PIPELEX := "$(VIRTUAL_ENV)/bin/pipelex"+VENV_MKDOCS := "$(VIRTUAL_ENV)/bin/mkdocs"+VENV_PYLINT := "$(VIRTUAL_ENV)/bin/pylint"UV_MIN_VERSION = $(shell grep -m1 'required-version' pyproject.toml | sed -E 's/.*= *"([^<>=, ]+).*/\1/')@@ -57,8 +58,7 @@ make cft - Shorthand -> config-templatemake cleanenv - Remove virtual env and lock filesmake cleanderived - Remove extraneous compiled files, caches, logs, etc.-make cleanlibraries - Remove pipelex_libraries-make cleanall - Remove all -> cleanenv + cleanderived + cleanlibraries+make cleanall - Remove all -> cleanenv + cleanderivedmake merge-check-ruff-lint - Run ruff merge check without updating filesmake merge-check-ruff-format - Run ruff merge check without updating files@@ -106,7 +106,7 @@ export HELP.PHONY: \all help env lock install update build \format lint pyright mypy pylint \- cleanderived cleanenv cleanlibraries cleanall \+ cleanderived cleanenv cleanall \test test-xdist t test-quiet tq test-with-prints tp test-inference ti \test-llm tl test-img-gen tg test-extract te codex-tests gha-tests \run-all-tests run-manual-trigger-gha-tests run-gha_disabled-tests \@@ -135,17 +135,17 @@ check-uv:env: check-uv$(call PRINT_TITLE,"Creating virtual environment")- @if [ ! -d $(VIRTUAL_ENV) ]; then \+ @if [ ! -d "$(VIRTUAL_ENV)" ]; then \echo "Creating Python virtual env in \`${VIRTUAL_ENV}\`"; \- uv venv $(VIRTUAL_ENV) --python $(PYTHON_VERSION); \+ uv venv "$(VIRTUAL_ENV)" --python $(PYTHON_VERSION); \else \echo "Python virtual env already exists in \`${VIRTUAL_ENV}\`"; \fi- @echo "Using Python: $$($(VENV_PYTHON) --version) from $$(which $$(readlink -f $(VENV_PYTHON)))"+ @echo "Using Python: $$($(VENV_PYTHON) --version) from $$(readlink $(VENV_PYTHON) 2>/dev/null || echo $(VENV_PYTHON))"install: env$(call PRINT_TITLE,"Installing dependencies")- @. $(VIRTUAL_ENV)/bin/activate && \+ @. "$(VIRTUAL_ENV)/bin/activate" && \uv sync --all-extras && \echo "Installed Pipelex dependencies in ${VIRTUAL_ENV} with all extras.";@@ -162,7 +162,7 @@ update: envvalidate: env$(call PRINT_TITLE,"Running setup sequence")- $(VENV_PIPELEX) validate all -c pipelex/libraries+ $(VENV_PIPELEX) validate allbuild: env$(call PRINT_TITLE,"Building the wheels")@@ -199,17 +199,12 @@ cleanenv:find . -type d -wholename './.venv' -exec rm -rf {} + && \echo "Cleaned up virtual env and dependency lock files";-cleanlibraries:- $(call PRINT_TITLE,"Erasing derived files and directories")- @find . -type d -wholename './pipelex_libraries' -exec rm -rf {} + && \- echo "Cleaned up pipelex_libraries";-cleanconfig:$(call PRINT_TITLE,"Erasing config files and directories")@find . -type d -wholename './.pipelex' -exec rm -rf {} + && \echo "Cleaned up .pipelex";-cleanall: cleanderived cleanenv cleanlibraries cleanconfig+cleanall: cleanderived cleanenv cleanconfig@echo "Cleaned up all derived files and directories";##########################################################################################@@ -425,7 +420,7 @@ merge-check-ruff-lint: env check-unused-importsmerge-check-pyright: env$(call PRINT_TITLE,"Typechecking with pyright")- $(VENV_PYRIGHT) --pythonpath $(VIRTUAL_ENV)/bin/python3+ $(VENV_PYRIGHT) --pythonpath $(VENV_PYTHON)merge-check-mypy: env$(call PRINT_TITLE,"Typechecking with mypy")diff --git a/PLAN.md b/PLAN.mdnew file mode 100644index 000000000..900544b65--- /dev/null+++ b/PLAN.md@@ -0,0 +1,292 @@+# Goal++Add a `kit` command group that:++1. Exports each `pipelex.kit/agents/*.md` to **Cursor** as `.cursor/rules/*.mdc` (adds YAML front-matter from `index.toml`).+2. Builds a **merged single-file rules block** from ordered fragments (with `demote=1`) and inserts/updates it in targets (`AGENTS.md`, `CLAUDE.md`, etc.) using **marker spans**.+ All parsing/merging via **markdown-parser-py**; config via **TOML → dict** (`toml_utils`) → **Pydantic v2** models.++---++# Package layout (as agreed)++```+pipelex/kit/+ __init__.py+ index.toml+ agents/+ pytest_standards.md+ python_standards.md+ run_pipelines.md+ write_pipelex.md+ configs/+ ... (as provided)+ migrations/+ ... (optional)+```++---++# Pydantic v2 models (dict → object via `model_validate`)++```python+# pipelex/kit/index_models.py+from pydantic import BaseModel, Field+from typing import Optional, List, Dict, Any++class AgentsMerge(BaseModel):+ order: List[str]+ demote: int = 1++class CursorFileOverride(BaseModel):+ front_matter: Dict[str, Any] = Field(default_factory=dict)++class CursorSpec(BaseModel):+ front_matter: Dict[str, Any] = Field(default_factory=dict)+ files: Dict[str, CursorFileOverride] = Field(default_factory=dict)++class Target(BaseModel):+ id: str+ path: str+ strategy: str # "merge" (for now)+ marker_begin: str+ marker_end: str+ parent: Optional[str] = None++class KitIndex(BaseModel):+ meta: Dict[str, Any] = Field(default_factory=dict)+ agents: AgentsMerge+ cursor: CursorSpec+ targets: List[Target]+```++**Load `index.toml` using your utils:**++```python+# pipelex/kit/index_loader.py+from pipelex.tools.misc.toml_utils import load_toml_from_path, TomlError+from importlib.resources import files+from .index_models import KitIndex++def load_index() -> KitIndex:+ path = files("pipelex.kit") / "index.toml"+ data = load_toml_from_path(str(path))+ return KitIndex.model_validate(data)+```++---++# CLI integration (Typer)++Add a **`kit`** sub-app under your existing CLI:++```python+# pipelex/cli/commands/kit_cmd.py+import typer+from pathlib import Path+from typing_extensions import Annotated+from pipelex.kit.index_loader import load_index+from pipelex.kit.paths import get_agents_dir # small helper shown below+from pipelex.kit.cursor_export import export_cursor_rules+from pipelex.kit.targets_update import build_merged_rules, update_targets++kit_app = typer.Typer(help="Manage kit assets: export Cursor rules and merge agent docs")++@kit_app.command("sync")+def sync(+ repo_root: Annotated[Path, typer.Option("--repo-root", dir_okay=True, writable=True)] = Path("."),+ cursor: Annotated[bool, typer.Option("--cursor/--no-cursor")] = True,+ single_files: Annotated[bool, typer.Option("--single-files/--no-single-files")] = True,+ dry_run: Annotated[bool, typer.Option("--dry-run")] = False,+ diff: Annotated[bool, typer.Option("--diff")] = False,+ backup: Annotated[str | None, typer.Option("--backup")] = None,+ force: Annotated[bool, typer.Option("--force")] = False,+):+ idx = load_index()+ agents_dir = get_agents_dir()++ if cursor:+ export_cursor_rules(agents_dir, repo_root / ".cursor" / "rules", idx, dry_run=dry_run)++ if single_files:+ merged_md = build_merged_rules(agents_dir, idx) # ordered + demoted+ update_targets(repo_root, merged_md, idx.targets, dry_run=dry_run, diff=diff, backup=backup, force=force)+```++Wire it into your main CLI:++```python+# in your main cli module+from pipelex.cli.commands.kit_cmd import kit_app+app.add_typer(kit_app, name="kit", help="Manage kit assets")+```++---++# Core helpers++## Paths++```python+# pipelex/kit/paths.py+from importlib.resources import files+from typing import Any+from importlib.abc import Traversable++def get_kit_root() -> Traversable:+ return files("pipelex.kit")++def get_agents_dir() -> Traversable:+ return get_kit_root() / "agents"+```++## Cursor export (.md → .mdc with YAML front matter)++```python+# pipelex/kit/cursor_export.py+from typing import Iterable+from pathlib import Path+import textwrap+import yaml # pyyaml (MIT)+from importlib.abc import Traversable+from .index_models import KitIndex+from pipelex.tools.misc.toml_utils import TomlError # (for symmetry if needed)++def _iter_agent_files(agents_dir: Traversable) -> Iterable[tuple[str, str]]:+ for child in agents_dir.iterdir():+ if child.name.endswith(".md") and child.is_file():+ yield child.name, child.read_text(encoding="utf-8")++def _front_matter_for(name: str, idx: KitIndex) -> dict:+ base = dict(idx.cursor.front_matter)+ key = name.removesuffix(".md")+ if key in idx.cursor.files:+ base |= idx.cursor.files[key].front_matter+ return base++def export_cursor_rules(agents_dir: Traversable, out_dir: Path, idx: KitIndex, dry_run: bool=False) -> None:+ out_dir.mkdir(parents=True, exist_ok=True)+ for fname, body in _iter_agent_files(agents_dir):+ fm = _front_matter_for(fname, idx)+ yaml_block = "---\n" + yaml.safe_dump(fm, sort_keys=False).rstrip() + "\n---\n"+ mdc = yaml_block + body+ out_path = out_dir / (fname.removesuffix(".md") + ".mdc")+ if dry_run:+ typer.echo(f"[DRY] write {out_path}")+ else:+ out_path.write_text(mdc, encoding="utf-8")+```++## Merging for single-file targets (markdown-parser-py)++```python+# pipelex/kit/targets_update.py+from dataclasses import dataclass+from pathlib import Path+from importlib.abc import Traversable+from typing import List+import difflib+import typer++from markdown_parser_py import Doc # replace with actual import names+from markdown_parser_py import parse as md_parse, render as md_render # adjust to real API++from .index_models import KitIndex, Target+from .markers import find_span, wrap, replace_span+from pipelex.tools.misc.toml_utils import load_toml_from_path # (not strictly needed here)++def _read_agent_file(agents_dir: Traversable, name: str) -> str:+ return (agents_dir / name).read_text(encoding="utf-8")++def _demote(doc: Doc, n: int) -> Doc:+ # Walk headings and +n to their level (implementation per markdown-parser-py API)+ # pseudo:+ # for h in doc.headings(): h.level += n+ return doc++def build_merged_rules(agents_dir: Traversable, idx: KitIndex) -> str:+ parts: List[str] = []+ for name in idx.agents.order:+ md = _read_agent_file(agents_dir, name)+ d = md_parse(md)+ d = _demote(d, idx.agents.demote)+ parts.append(md_render(d).rstrip())+ return ("\n\n".join(parts)).strip() + "\n"++def _insert_block_with_ast(target_md: str, block_md: str, parent: str | None) -> str:+ # Parse both; locate parent heading (if any) or choose heuristic (after first H1 or end)+ # Attach block as a new section. Return rendered string.+ tdoc = md_parse(target_md or "")+ bdoc = md_parse(block_md)+ # ... attach via library’s attach API (level handling already demoted)+ return md_render(tdoc)++def _diff(before: str, after: str, path: str) -> str:+ return "".join(difflib.unified_diff(before.splitlines(True), after.splitlines(True), fromfile=path, tofile=path))++def update_targets(repo_root: Path, merged_rules: str, targets: List[Target],+ dry_run: bool, diff: bool, backup: str | None, force: bool) -> None:+ for t in targets:+ p = repo_root / t.path+ before = p.read_text(encoding="utf-8") if p.exists() else ""+ span = find_span(before, t.marker_begin, t.marker_end)++ if span:+ after_block = wrap(t.marker_begin, t.marker_end, merged_rules)+ after = replace_span(before, span, after_block)+ else:+ inserted = _insert_block_with_ast(before, merged_rules, t.parent)+ after = inserted if inserted.endswith("\n") else inserted + "\n"+ # wrap only the inserted region — simplest approach is to wrap merged_rules before insert+ # or insert markers during AST attach (implementation detail)++ if dry_run:+ typer.echo(f"[DRY] update {p}")+ if diff:+ typer.echo(_diff(before, after, str(p)))+ else:+ if backup and p.exists():+ (p.with_suffix(p.suffix + backup)).write_text(before, encoding="utf-8")+ p.parent.mkdir(parents=True, exist_ok=True)+ p.write_text(after, encoding="utf-8")+```++## Markers++```python+# pipelex/kit/markers.py+def find_span(text: str, begin: str, end: str):+ s = text.find(begin)+ if s == -1: return None+ e = text.find(end, s)+ if e == -1: return None+ e += len(end)+ return (s, e)++def wrap(begin: str, end: str, md: str) -> str:+ return f"{begin}\n{md.rstrip()}\n{end}"++def replace_span(text: str, span: tuple[int,int], replacement: str) -> str:+ s, e = span+ return text[:s] + replacement + text[e:]+```++---++# Behavior rules (quick)++* **Cursor**: overwrite `.cursor/rules/*.mdc` each run (we own them).+* **Single-file**: only mutate content within our markers; if absent, **insert via AST** and add markers.+* **Demote before concat** (per `index.toml`).+* Preserve front-matter/encoding/line endings.+* `--dry-run` prints plan; `--diff` shows unified diff; `--backup` writes `*.bak`.++---++# Exit codes++* `0` success / no changes+* `1` failures (I/O, parse, invalid index, write blocked)+* `2` ambiguous parent resolution (if you choose to enforce)++This plugs directly into your existing CLI, uses your TOML loader + Pydantic v2, and scopes the implementation clearly for another SWE to take over.diff --git a/README.md b/README.mdindex 5d57c07b6..fabf80da6 100644--- a/README.md+++ b/README.md@@ -79,7 +79,7 @@ description = "Extract buyer from purchase document"inputs = { purchase_document_text = "PurchaseDocumentText" }output = "Buyer"model = "llm_to_extract_info"-prompt_template = """+prompt = """Extract the first and last name of the buyer from this purchase document:@purchase_document_text"""@@ -257,7 +257,7 @@ system_prompt = """You are an expert in social media optimization, particularly for tech content on Twitter/X.Your role is to analyze tech tweets and check if they display typical startup communication pitfalls."""-prompt_template = """+prompt = """Evaluate the tweet for these key issues:**Fluffiness** - Overuse of buzzwords without concrete meaning (e.g., "synergizing disruptive paradigms")@@ -286,7 +286,7 @@ system_prompt = """You are an expert in writing engaging tech tweets that drive meaningful discussions and engagement.Your goal is to rewrite tweets to be impactful and avoid the pitfalls identified in the analysis."""-prompt_template = """+prompt = """Rewrite this tech tweet to be more engaging and effective, based on the analysis:Original tweet:diff --git a/docs/pages/advanced-customization/index.md b/docs/pages/advanced-customization/index.mdindex ceb53e5d6..af83f63f8 100644--- a/docs/pages/advanced-customization/index.md+++ b/docs/pages/advanced-customization/index.md@@ -13,15 +13,11 @@ There are two main ways to inject custom implementations:```pythonfrom pipelex import Pipelex-pipelex = Pipelex(- config_dir_path="./pipelex_libraries",+pipelex = Pipelex.make(template_provider=MyTemplateProvider(),pipeline_tracker=MyPipelineTracker(),activity_manager=MyActivityManager(),- reporting_delegate=MyReportingDelegate()-)--pipelex.setup(+ reporting_delegate=MyReportingDelegate(),secrets_provider=MySecretsProvider(),content_generator=MyContentGenerator(),pipe_router=MyPipeRouter()diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.mdindex 73408ce72..1ca81085b 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md@@ -53,7 +53,7 @@ Those concepts will be Text-based by default. If you want to use sutrctured outpGroup concepts that naturally belong together in the same domain. A domain acts as a namespace for a set of related concepts and pipes, helping you organize and reuse your pipeline components. You can learn more about them in [Kick off a Knowledge Pipeline Project](kick-off-a-knowledge-pipeline-project.md#what-are-domains).```plx-# pipelex_libraries/pipelines/finance.plx+# finance.plxdomain = "finance"description = "Financial document processing"@@ -65,20 +65,64 @@ PaymentTerms = "Conditions under which payment is to be made"LineItem = "An individual item or service listed in a financial document"```+## How to Structure Your Concepts++Once you've defined your concepts semantically, you may need to add structure if they have specific fields. Pipelex offers three approaches:++### 1. No Structure (Concept Only)++For concepts that only refine native concepts without adding fields, just declare them with a description. They default to text-based content.++```plx+[concept]+ProductReview = "A customer's evaluation of a product or service"+```++### 2. Inline TOML Structures (Recommended)++Define structured concepts directly in your `.plx` files using TOML syntax. This is the fastest and simplest approach for most use cases.++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+total_amount = { type = "number", description = "Total invoice amount", required = true }+vendor_name = "The name of the vendor"+```++Behind the scenes, Pipelex generates a complete Pydantic model with validation.++### 3. Python StructuredContent Classes++Create explicit Python classes when you need custom validation, computed properties, or advanced features.++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field++class Invoice(StructuredContent):+ invoice_number: str+ total_amount: float = Field(ge=0, description="Total invoice amount")+ vendor_name: str+```++**For detailed guidance on choosing and implementing these approaches, see [Structuring Concepts](structuring-concepts.md).**+## Adding Structure with Python Models-While text definitions help LLMs understand your concepts, Python models ensure structured, validated outputs. This combination gives you the best of both worlds: AI flexibility with software reliability.+This section covers the Python class approach for structured concepts. For a simpler alternative using inline TOML syntax, see [Structuring Concepts](structuring-concepts.md).-**Important**: If you don't create a Python class for a concept, it defaults to text-based content. Only create Python models when you need structured output with specific fields.+While text definitions help LLMs understand your concepts, Python models ensure structured, validated outputs when you need custom validation, computed properties, or advanced features.-### Creating Your First Structured Model+### Creating a Python Structured Model-For each concept that needs structured output, create a corresponding Python class:+For concepts that need custom logic, create a Python class that inherits from `StructuredContent`:```python-# pipelex_libraries/pipelines/finance.py+# finance.pyfrom datetime import datetime-from typing import List, Optionalfrom pydantic import Fieldfrom pipelex.core.stuffs.structured_content import StructuredContent@@ -90,14 +134,20 @@ class Invoice(StructuredContent):customer_name: strtotal_amount: float = Field(ge=0, description="Total invoice amount")currency: str = Field(default="USD", description="Three-letter currency code")- line_items: List[str] = Field(default_factory=list)```The model name must match the concept name exactly: `Invoice` concept → `Invoice` class.-### Basic Validation Examples+Python classes are automatically discovered and registered by Pipelex.++!!! warning "Module Execution During Auto-Discovery"+ When Pipelex discovers `StructuredContent` classes, it imports the module containing them. **Any code at the module level (outside functions/classes) will be executed during import.** This can have unintended side effects.++ **Best practice:** Keep your `StructuredContent` classes in dedicated modules (e.g., `*_struct.py` files) with minimal module-level code, or ensure module-level code is safe to execute during discovery.++### With Custom Validation-Use Pydantic's validation features to ensure data quality:+Use Pydantic's validation features for complex rules:```pythonfrom pydantic import field_validator@@ -115,56 +165,27 @@ class Employee(StructuredContent):if "@" not in v:raise ValueError("Invalid email format")return v.lower()--class ProductReview(StructuredContent):- product_name: str- reviewer_name: str- rating: int = Field(ge=1, le=5, description="Rating from 1 to 5 stars")- review_text: str- verified_purchase: bool = False-```--### Working with Optional Fields--Not all data is always available. Use Optional fields with sensible defaults:--```python-from typing import Optional-from datetime import datetime-from pipelex.core.stuffs.structured_content import StructuredContent--class Meeting(StructuredContent):- title: str- scheduled_date: datetime- duration_minutes: int = Field(ge=15, le=480, description="Meeting duration")- location: Optional[str] = None- attendees: List[str] = Field(default_factory=list)- notes: Optional[str] = None- is_recurring: bool = False```### Linking Concepts to Models-The connection between PLX definitions and Python models happens automatically through naming:+The connection between `.plx` definitions and Python models happens automatically through naming:```plx-# pipelex_libraries/pipelines/hr.plx+# hr.plxdomain = "hr"[concept]Employee = "A person employed by an organization"Meeting = "A scheduled gathering of people for discussion"-PerformanceReview = "An evaluation of an employee's work performance"Department = "An organizational unit within a company" # No Python model => text-based``````python-# pipelex_libraries/pipelines/hr.py+# hr.pyfrom pipelex.core.stuffs.structured_content import StructuredContentfrom datetime import datetime-from typing import List, Optional-# Only define models for concepts that need structureclass Employee(StructuredContent):name: stremail: str@@ -175,18 +196,13 @@ class Meeting(StructuredContent):title: strscheduled_date: datetimeduration_minutes: int- attendees: List[str]--class PerformanceReview(StructuredContent):- employee_name: str- review_period: str- rating: int = Field(ge=1, le=5)- strengths: List[str]- areas_for_improvement: List[str]+ attendees: list[str]# Note: Department concept has no Python model, so it's text-based```+**For more examples, advanced features, and guidance on when to use Python classes vs inline structures, see [Structuring Concepts](structuring-concepts.md).**+## Concept Refinement and InheritanceSometimes concepts build on each other. A `Contract` is a kind of `Document`. A `NonCompeteClause` is a specific part of a `Contract`. Pipelex lets you express these relationships.diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.mdindex 265e5b4df..49f14c4fa 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md@@ -40,7 +40,7 @@ type = "PipeLLM"description = "Generate a catchy tagline for a product"inputs = { description = "ProductDescription" }output = "Tagline"-prompt_template = """+prompt = """Product Description:@description@@ -76,7 +76,7 @@ description = "Extract keywords from a product description"inputs = { description = "ProductDescription" }output = "Keyword"multiple_output = true-prompt_template = """+prompt = """Please extract the most relevant keywords from the following product description:@description@@ -89,7 +89,7 @@ type = "PipeLLM"description = "Generate a tagline from keywords"inputs = { keywords = "Keyword" }output = "Tagline"-prompt_template = """+prompt = """Here are the key product keywords:@keywordsdiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.mdindex 26a39347f..f5ddefddd 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md@@ -2,10 +2,10 @@## Creating Your First Pipeline-A pipeline in Pipelex is a collection of related concepts and pipes. Start by creating a PLX file in the `pipelines` directory:+A pipeline in Pipelex is a collection of related concepts and pipes. Start by creating a PLX file in your project:```plx-# pipelex_libraries/pipelines/tutorial.plx+# tutorial.plxdomain = "tutorial"description = "My first Pipelex library"@@ -21,7 +21,7 @@ type = "PipeLLM"description = "Answer a question"inputs = { question = "tutorial.Question" }output = "tutorial.Answer"-prompt_template = """+prompt = """Please answer the following question:@question@@ -99,8 +99,10 @@ A domain consists of:2. **Organization**- One domain per topic/functionality.- - Match Python file names with domain names (`finance.plx` -> `finance.py`).+ - Use `_struct.py` suffix for structure files (`finance.plx` -> `finance_struct.py`).- Keep related concepts within the same domain.+ - Place your `.plx` files anywhere in your project - Pipelex automatically discovers them.+ - Keep `.pipelex/` configuration directory at repository root.3. **Documentation**- Always add a description to your domain.@@ -132,25 +134,63 @@ Consistent naming makes your pipeline code discoverable and maintainable:- For multi-word domains, use underscores: domain "customer_service" → `customer_service.plx`### Python Model Files-- It is recommended to match the PLX filename exactly: `legal.plx` → `legal.py`-- But in any case, Pipelex will load models from all python modules in the `pipelines` directory or its subdirectories.+- It is recommended to name structure files with a `_struct.py` suffix: `legal.plx` → `legal_struct.py`+- Pipelex will automatically discover and load structure classes from all Python files in your project (excluding common directories like `.venv`, `.git`, etc.)## Project Structure-Every Pipelex project follows a simple directory structure that keeps your knowledge pipelines organized and maintainable:+**Key principle:** Put `.plx` files where they belong in YOUR codebase. Pipelex automatically finds them.+### Recommended Patterns++**Topic-Based (Best for organized codebases):**+```+your-project/+├── my_project/ # Your Python package+│ ├── finance/+│ │ ├── models.py+│ │ ├── services.py+│ │ ├── invoices.plx # Pipeline with finance code+│ │ └── invoices_struct.py # Structure classes+│ └── legal/+│ ├── models.py+│ ├── contracts.plx # Pipeline with legal code+│ └── contracts_struct.py+├── .pipelex/ # Config at repo root+│ ├── pipelex.toml+│ └── inference/+└── requirements.txt+```++**Centralized (If you prefer grouping pipelines):**```your-project/-├── pipelex_libraries/ # All your pipeline code lives here-│ ├── pipelines/ # Pipeline definitions and models-│ │ ├── __init__.py-│ │ ├── characters.plx # Domain definitions-│ │ └── characters.py # Python models for concepts-│ ├── templates/ # Reusable prompt templates-│ ├── llm_integrations/ # LLM provider configurations-│ └── llm_deck/ # LLM model presets-├── main.py # Your application code-└── requirements.txt # Python dependencies+├── my_project/+│ ├── pipelines/ # All pipelines together+│ │ ├── finance.plx+│ │ ├── finance_struct.py+│ │ ├── legal.plx+│ │ └── legal_struct.py+│ └── core/+│ └── (your code)+└── .pipelex/```-The `pipelex_libraries/pipelines` directory is where Pipelex looks for your pipeline definitions. This standardized structure means you can share libraries between projects, version control them separately, and maintain clean separation between your pipeline logic and application code.+**Flat (Small projects):**+```+your-project/+├── my_project/+│ ├── invoice_pipeline.plx+│ ├── invoice_struct.py+│ └── main.py+└── .pipelex/+```++### Key Points++- **Flexible placement**: `.plx` files work anywhere in your project+- **Automatic discovery**: Pipelex scans and finds them automatically+- **Configuration location**: `.pipelex/` stays at repository root+- **Naming convention**: Use `_struct.py` suffix for structure files+- **Excluded directories**: `.venv`, `.git`, `__pycache__`, `node_modules` are skipped+- **Best practice**: Keep related pipelines with their related codediff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.mdindex e71addcf0..ece06b821 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md@@ -61,7 +61,7 @@ large = "process_large"type = "PipeLLM"description = "Handle small category"output = "native.Text"-prompt_template = """+prompt = """Output this only: "small""""@@ -69,7 +69,7 @@ Output this only: "small"type = "PipeLLM"description = "Handle medium category"output = "native.Text"-prompt_template = """+prompt = """Output this only: "medium""""@@ -77,7 +77,7 @@ Output this only: "medium"type = "PipeLLM"description = "Handle large category"output = "native.Text"-prompt_template = """+prompt = """Output this only: "large""""```@@ -109,7 +109,7 @@ type = "PipeLLM"description = "Process invoice documents"inputs = { classification = "DocumentType" }output = "ProcessedDocument"-prompt_template = """+prompt = """Process this invoice document..."""@@ -118,7 +118,7 @@ type = "PipeLLM"description = "Process receipt documents"inputs = { classification = "DocumentType" }output = "ProcessedDocument"-prompt_template = """+prompt = """Process this receipt document..."""@@ -127,7 +127,7 @@ type = "PipeLLM"description = "Handle unknown document types"inputs = { classification = "DocumentType" }output = "ProcessedDocument"-prompt_template = """+prompt = """Process this unknown document type..."""```diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.mdindex 99b40db85..cedf88c12 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md@@ -4,23 +4,25 @@ The `PipeFunc` operator provides an essential escape hatch, allowing you to exec## How it works-`PipeFunc` operates by calling a Python function that has been automatically registered with Pipelex's central function registry.+`PipeFunc` operates by calling a Python function that has been registered with Pipelex's central function registry.-1. **Automatic Registration**: Functions are automatically discovered and registered from Python files in the `pipelex/libraries/` directory when Pipelex starts up.-2. **Function Signature**: Eligible functions are automatically registered using their function name as the registry key.-3. **Execution**: When the `PipeFunc` pipe is executed, it looks up your function by name and calls it, passing in the current `working_memory`.-4. **Returning Data**: The function returns data, which `PipeFunc` places back into the working memory, associated with the pipe's `output` concept.+1. **Decorator Required**: Functions must be decorated with `@pipe_func()` to be discovered and registered (since v0.12.0).+2. **Automatic Discovery**: Functions with the `@pipe_func()` decorator are automatically discovered from anywhere in your project when Pipelex starts up.+3. **Function Signature**: Eligible functions are registered using their function name (or a custom name) as the registry key.+4. **Execution**: When the `PipeFunc` pipe is executed, it looks up your function by name and calls it, passing in the current `working_memory`.+5. **Returning Data**: The function returns data, which `PipeFunc` places back into the working memory, associated with the pipe's `output` concept.## Function Eligibility RequirementsFor a function to be automatically registered and available to `PipeFunc`, it **must** meet all of the following criteria:!!! warning "Function Eligibility Requirements"+ - **Must be decorated with** `@pipe_func()` (required since v0.12.0)- **Must be an async function** (defined with `async def`)- **Must have exactly 1 parameter** named `working_memory`- **Parameter type must be** `WorkingMemory`- **Return type must be** a subclass of `StuffContent` (or a generic type like `ListContent[SomeType]`)- - **Must be defined in a Pipelex library file** within the `pipelines/` directory+ - **Must be discoverable** (not in excluded directories like `.venv`, `__pycache__`, etc.)### Return values@@ -30,16 +32,27 @@ Your async Python function can return:## How to Create a Function-To make a Python function available to `PipeFunc`, simply create it in any Python file within the `pipelex/libraries/` directory structure.+To make a Python function available to `PipeFunc`:++1. Add the `@pipe_func()` decorator to your function+2. Place the function anywhere in your project (it will be auto-discovered)+3. Ensure it meets all eligibility requirements++!!! warning "Module Execution During Auto-Discovery"+ When Pipelex discovers functions with `@pipe_func()`, it imports the module containing them. **Any code at the module level (outside functions/classes) will be executed during import.** This can have unintended side effects.++ **Best practice:** Keep your `@pipe_func()` functions in dedicated modules with minimal module-level code, or ensure module-level code is safe to execute during discovery.Here is an example of an eligible function:```python-# in a file like pipelex/libraries/my_custom_functions.py+# in any Python file in your project (e.g., my_project/custom_functions.py)+from pipelex.system.registries.func_registry import pipe_funcfrom pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.stuffs.stuff_content import TextContent+from pipelex.core.stuffs.text_content import TextContent+@pipe_func() # Required decorator for auto-discoveryasync def concatenate_texts(working_memory: WorkingMemory) -> TextContent:"""Retrieves two text stuffs, concatenates them, and returns a new text stuff.@@ -55,6 +68,19 @@ async def concatenate_texts(working_memory: WorkingMemory) -> TextContent:The function will be automatically registered with the name `concatenate_texts` (the function name) when Pipelex starts up.+### Custom Registration Name++You can optionally specify a custom name for registration:++```python+@pipe_func(name="custom_concat")+async def concatenate_texts(working_memory: WorkingMemory) -> TextContent:+ # Implementation...+ pass+```++Then use `function_name = "custom_concat"` in your `.plx` file.+## ConfigurationOnce the function is registered, you can use it in your `.plx` file.diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.mdindex 456859fd8..8ce99bf60 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md@@ -76,7 +76,7 @@ ImagePrompt = "A text prompt for generating an image"[pipe.generate_logo_variations]type = "PipeImgGen"description = "Generate three logo variations from a prompt"-inputs = { prompt = "images.ImgGenPrompt" }+inputs = { prompt = "ImgGenPrompt" }output = "Image"nb_output = 3model = "base_img_gen"diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.mdindex e02a41e93..ec8b5ee67 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md@@ -27,7 +27,7 @@ type = "PipeLLM"description = "Describe an image"inputs = { image = "Image" }output = "VisualDescription"-prompt_template = """+prompt = """Describe the provided image in great detail."""```@@ -42,9 +42,9 @@ You can use any concept that refines `Image` as an input, and choose descriptive[pipe.analyze_wedding]type = "PipeLLM"description = "Analyze wedding photo"-inputs = { wedding_photo = "images.Photo" }+inputs = { wedding_photo = "Photo" }output = "PhotoAnalysis"-prompt_template = """+prompt = """Analyze this wedding photo and describe the key moments captured."""```@@ -59,7 +59,7 @@ type = "PipeLLM"description = "Analyze the visual layout of a page"inputs = { "page_content.page_view" = "Image" }output = "LayoutAnalysis"-prompt_template = """+prompt = """Analyze the visual layout and design elements of this page.Focus on typography, spacing, and overall composition."""@@ -83,7 +83,7 @@ inputs = {second_image = "Image"}output = "ImageComparison"-prompt_template = """+prompt = """Compare these two images and describe their similarities and differences."""```@@ -101,7 +101,7 @@ inputs = {document.page_view = "Image"}output = "DocumentAnalysis"-prompt_template = """+prompt = """Given this context: $contextAnalyze the document page shown in the image and explain how it relates to the provided context.@@ -167,7 +167,7 @@ type = "PipeLLM"description = "Summarize a text"inputs = { text = "TextToSummarize" }output = "TextSummary"-prompt_template = """+prompt = """Please provide a concise summary of the following text:@text@@ -186,7 +186,7 @@ type = "PipeLLM"description = "Extract table data from an image"inputs = { image = "TableScreenshot" }output = "TableData"-prompt_template = """+prompt = """Extract the table data from this image and format it as a structured table."""```@@ -204,7 +204,7 @@ type = "PipeLLM"description = "Process an expense report"inputs = { report = "ExpenseReport" }output = "ProcessedExpenseReport"-prompt_template = """+prompt = """Analyze this expense report and extract the following information:- Total amount- Datediff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.mdindex a6404d251..56b64c95d 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md@@ -4,39 +4,120 @@ Pipelex provides powerful tools to automatically generate complete, working pipe## Overview-The pipeline creation system creates a fully working pipeline that has been both statically and dynamically validated. The system automatically handles all aspects of pipeline generation, from understanding requirements to producing executable code.--!!! warning "Execution Time"- Pipeline generation can take 4 to 7 minutes to complete, as the system performs comprehensive validation and testing to ensure the pipeline works correctly.+The pipeline creation system can generate pipelines in different modes depending on your needs - from quick one-shot generation to validated, production-ready pipelines that have been automatically tested and fixed.## Core Commands-### Build Blueprint+### Build Pipe (Recommended)-Generate a complete pipeline blueprint from requirements:+Generate a validated pipeline with automatic fixing of deterministic issues:```bash-pipelex build blueprint "BRIEF IN NATURAL LANGUAGE" [OPTIONS]+pipelex build pipe "BRIEF IN NATURAL LANGUAGE" [OPTIONS]```+This command runs a validation/fix loop to ensure the generated pipeline is correct and runnable. It automatically detects and corrects common issues.+**Example:**+```bash-pipelex build blueprint "Take a photo as input, and render the opposite of the photo" \- -c your/path/to/pipelex/libraries \- -o output/pipeline/file/path+pipelex build pipe "Take a photo as input, and render the opposite of the photo" \+ -o results/photo_inverter.plx```**Options:**-- `--output, -o`: Output path for generated files-- `--config, -c`: Path to libraries folder-## Complete Workflow+- `--output, -o`: Output path for generated PLX file (default: `./results/generated_pipeline.plx`)+- `--no-output`: Skip saving the pipeline to file++### Build One-Shot (Fast)++Generate a pipeline quickly without validation:++```bash+pipelex build one-shot "BRIEF IN NATURAL LANGUAGE" [OPTIONS]+```++This command generates the pipeline in a single pass without validation. It's faster but may produce pipelines that need manual fixes.++**Example:**++```bash+pipelex build one-shot "Extract invoice data from PDF documents" \+ -o results/invoice_extractor.plx+```++**Use when:** You want to quickly iterate on ideas or plan to manually review/modify the pipeline.++### Build Partial (Debug)++Generate a partial pipeline specification as JSON for debugging:++```bash+pipelex build partial "BRIEF IN NATURAL LANGUAGE" [OPTIONS]+```++This outputs the internal pipeline specification in JSON format, useful for debugging and understanding how Pipelex interprets your requirements.++**Example:**++```bash+pipelex build partial "Analyze sentiment from customer reviews" \+ -o results/debug_spec.json+```-### 1. Requirements Analysis+## Quick Start Example-Start with clear, specific requirements:+The simplest way to create a pipeline is to use the `build pipe` command with a clear description:-```text-Take a photo as input, and render the opposite of the photo+```bash+pipelex build pipe "Given an expense report, apply company rules and validate compliance"```+This will:++1. Analyze your requirements+2. Generate a complete pipeline with appropriate concepts and pipes+3. Validate the pipeline for correctness+4. Fix any deterministic issues automatically+5. Save the working pipeline to `./results/generated_pipeline.plx`++## Best Practices++When creating pipelines with natural language:++**Be Specific About Inputs and Outputs:**++- ✅ Good: "Take a PDF invoice as input and extract the total amount, vendor name, and date"+- ❌ Vague: "Process invoices"++**Describe the Transformation:**++- ✅ Good: "Analyze sentiment of customer reviews and categorize as positive, negative, or neutral"+- ❌ Vague: "Do something with reviews"++**Mention Data Types When Relevant:**++- ✅ Good: "Extract text from a PDF, then summarize it into 3 bullet points"+- ❌ Unclear: "Summarize documents"++## What Gets Generated++When you run a build command, Pipelex automatically creates:++- **Domain definition**: The namespace for your pipeline+- **Concepts**: Structured data types for inputs and outputs+- **Pipes**: The processing steps and LLM operations+- **Python structures**: When structured output is needed (saved alongside the `.plx` file with `_struct.py` suffix)++All generated pipelines follow Pipelex best practices and conventions automatically.++## Next Steps++After generating your pipeline:++1. **Review the generated `.plx` file** to understand the structure+2. **Test the pipeline** using the generated example code+3. **Iterate if needed** by modifying the natural language description and regenerating+4. **Customize** the pipeline by editing the `.plx` file directly for fine-tuning+diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/structuring-concepts.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/structuring-concepts.mdnew file mode 100644index 000000000..19d1c9b60--- /dev/null+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/structuring-concepts.md@@ -0,0 +1,431 @@+# Structuring Concepts++Pipelex offers two ways to add structure to your concepts: **inline TOML structures** and **Python classes**. This guide helps you choose the right approach and shows you how to use both effectively.++For an introduction to concepts themselves, see [Define Your Concepts](define_your_concepts.md).++## Quick Comparison++| Approach | Best For | Advantages | Limitations |+|----------|----------|------------|-------------|+| **Inline TOML** | Most use cases, prototyping | Fast, single-file, no boilerplate | No custom validation or methods |+| **Python Class** | Complex validation, computed properties | Full Pydantic power, IDE support | More files, more code |++## Inline Structure Definition++Define structured concepts directly in your `.plx` files using TOML syntax. This is the **recommended approach** for most use cases.++### Quick Example++```plx+domain = "finance"++[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items in the invoice", required = false }+```++Behind the scenes, Pipelex automatically generates a fully-typed Pydantic model with validation—all from TOML.++### Supported Field Types++Inline structures support these field types:++- **text**: String values+- **integer**: Whole numbers+- **boolean**: True/false values+- **number**: Numeric values (integers or floats)+- **date**: Date and datetime values+- **list**: Arrays/lists (specify `item_type`)+- **dict**: Dictionaries (specify `key_type` and `value_type`)++### Field Properties++Each field can specify:++- **type**: The data type (required for detailed definitions)+- **description**: Human-readable description+- **required**: Whether the field is mandatory (default: `true`)+- **default_value**: Default value if not provided+- **choices**: For enum-like fields, a list of valid values+- **item_type**: For list fields, the type of list items+- **key_type** and **value_type**: For dict fields, the types of keys and values++### Simple vs Detailed Syntax++**Simple syntax** (creates required text field):++```plx+[concept.Person.structure]+name = "The person's full name"+email = "The person's email address"+```++**Detailed syntax** (explicit properties):++```plx+[concept.Employee.structure]+employee_id = { type = "integer", description = "Unique employee identifier", required = true }+department = { type = "text", description = "Department name", required = false, default_value = "General" }+is_active = { type = "boolean", description = "Employment status", required = false, default_value = true }+```++You can mix both styles in the same structure.++### Complex Field Examples++**List Fields:**++```plx+[concept.Project.structure]+name = "Project name"+tags = { type = "list", item_type = "text", description = "Project tags", required = false }+team_members = { type = "list", item_type = "text", description = "Team member names", required = true }+```++**Dictionary Fields:**++```plx+[concept.Configuration.structure]+app_name = "Application name"+settings = { type = "dict", key_type = "text", value_type = "text", description = "Configuration settings", required = false }+```++**Choice Fields (Enums):**++```plx+[concept.Task.structure]+title = "Task title"+priority = { choices = ["low", "medium", "high"], description = "Task priority level", required = true }+status = { choices = ["todo", "in_progress", "done"], description = "Current status", default_value = "todo" }+```++## Python StructuredContent Classes++For advanced features, create explicit Python classes that inherit from `StructuredContent`.++### When to Use Python Classes++Use Python classes when you need:++1. **Custom validation logic** - Cross-field validation, complex rules+2. **Computed properties** - Derived values, formatted outputs+3. **Custom methods** - Business logic, helper functions+4. **Reusability** - Shared structures across multiple domains+5. **Advanced typing** - `Literal`, complex unions, etc.+6. **Better IDE support** - Full autocomplete and type checking++### Basic Python Class Example++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field++class Invoice(StructuredContent):+ """A commercial invoice."""++ invoice_number: str = Field(description="Unique invoice identifier")+ issue_date: datetime = Field(description="Date the invoice was issued")+ total_amount: float = Field(ge=0, description="Total invoice amount")+ vendor_name: str = Field(description="Name of the vendor")+ line_items: list[str] = Field(default_factory=list, description="List of items")+```++Classes inheriting from `StructuredContent` are automatically discovered and registered by Pipelex.++!!! warning "Module Execution During Auto-Discovery"+ When Pipelex discovers `StructuredContent` classes, it imports the module containing them. **Any code at the module level (outside functions/classes) will be executed during import.** This can have unintended side effects.++ **Best practice:** Keep your `StructuredContent` classes in dedicated modules (e.g., `*_struct.py` files) with minimal module-level code, or ensure module-level code is safe to execute during discovery.++### With Custom Validation++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator++class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ total_amount: float = Field(ge=0, description="Total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")+ net_amount: float = Field(ge=0, description="Net amount before tax")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```++### With Computed Properties++```python+from datetime import datetime+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field++class Subscription(StructuredContent):+ """A subscription with computed properties."""++ start_date: datetime = Field(description="Subscription start date")+ end_date: datetime = Field(description="Subscription end date")+ monthly_price: float = Field(ge=0, description="Monthly subscription price")++ @property+ def duration_days(self) -> int:+ """Calculate subscription duration in days."""+ return (self.end_date - self.start_date).days++ @property+ def total_cost(self) -> float:+ """Calculate total subscription cost."""+ months = self.duration_days / 30.0+ return months * self.monthly_price+```++## Advantages of Inline Structures++Inline structures offer several benefits:++### Rapid Development++- **Single File**: Keep concepts, structures, and pipes all in one `.plx` file+- **No Context Switching**: No need to jump between `.plx` and `.py` files+- **Quick Iteration**: Modify structures instantly without managing separate files++### Simplicity++- **Declarative Syntax**: Straightforward TOML that's easy to read and write+- **No Boilerplate**: No need for Python imports, class definitions, or field declarations+- **Automatic Registration**: Generated classes are automatically discovered++### Type Safety++- **Pydantic Models**: Full Pydantic v2 models behind the scenes+- **Runtime Validation**: Automatic validation of structured outputs from LLMs+- **Type Hints**: Generated classes include proper type annotations++## Limitations of Inline Structures++### What Inline Structures Cannot Do++Inline structures cannot provide:++- **Custom validators** - No `@field_validator` decorators+- **Computed properties** - No `@property` methods+- **Custom methods** - No class methods or instance methods+- **Complex validation** - No cross-field validation logic+- **Nested custom concepts** - Cannot reference other custom concepts as field types (coming soon)+- **Advanced Pydantic features** - No custom serializers, model validators, etc.++### Concept Refinement Restrictions++Currently, inline structures can only be used for concepts that:++- Don't refine other concepts, OR+- Refine native concepts only: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`++### Tooling Limitations++- **Limited IDE autocomplete** compared to explicit Python classes+- **No static type checking** with `mypy` or `pyright`+- **Less refactoring support** in IDEs++## Using AI to Create Python Classes++Modern AI coding assistants like Cursor AI and GitHub Copilot can generate `StructuredContent` classes instantly, making the transition from inline structures to Python classes fast and easy.++### Recommended Workflow++Follow this pragmatic approach:++1. **Prototype Fast**: Start with inline structures for rapid development+2. **Validate Quickly**: Test your pipelines and iterate on the structure+3. **Upgrade When Needed**: Convert to Python classes when you need advanced features+4. **Let AI Help**: Use AI assistants to generate the Python code automatically++### Example: AI-Assisted Migration++**Step 1: Start with inline structure**++```plx+[concept.UserProfile]+description = "A user profile"++[concept.UserProfile.structure]+username = "The user's username"+email = "The user's email address"+age = { type = "integer", description = "User's age", required = false }+```++**Step 2: Ask your AI assistant**++> "Convert this inline UserProfile structure to a Python StructuredContent class with email validation"++**Step 3: AI generates the class**++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator+import re++class UserProfile(StructuredContent):+ """A user profile with validation."""++ username: str = Field(description="The user's username")+ email: str = Field(description="The user's email address")+ age: int | None = Field(default=None, description="User's age")++ @field_validator('email')+ @classmethod+ def validate_email(cls, v):+ """Validate email format."""+ pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'+ if not re.match(pattern, v):+ raise ValueError('Invalid email format')+ return v+```++**Step 4: Update your .plx file**++```plx+[concept]+UserProfile = "A user profile" # Structure now defined in Python+```++The Python class is automatically discovered and registered.++## Migration Guide++### From Inline Structure to Python Class++Here's how to migrate when you need more advanced features:++**1. You have this inline structure:**++```plx+domain = "ecommerce"++[concept.Product]+description = "A product in the catalog"++[concept.Product.structure]+product_id = { type = "integer", description = "Unique product ID", required = true }+name = "Product name"+price = { type = "number", description = "Product price", required = true }+in_stock = { type = "boolean", description = "Stock availability", default_value = true }+```++**2. Create a Python file** (e.g., `ecommerce_struct.py`):++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field++class Product(StructuredContent):+ """A product in the catalog."""++ product_id: int = Field(description="Unique product ID")+ name: str = Field(description="Product name")+ price: float = Field(ge=0, description="Product price")+ in_stock: bool = Field(default=True, description="Stock availability")+```++**3. Update your `.plx` file:**++```plx+domain = "ecommerce"++[concept]+Product = "A product in the catalog"++# Structure section removed - now defined in ecommerce_struct.py+```++**4. Test your pipeline** - The behavior should be identical.++**5. Add enhancements** (optional):++```python+class Product(StructuredContent):+ """A product in the catalog."""++ product_id: int = Field(description="Unique product ID")+ name: str = Field(description="Product name")+ price: float = Field(ge=0, description="Product price")+ in_stock: bool = Field(default=True, description="Stock availability")++ @field_validator('price')+ @classmethod+ def validate_price(cls, v):+ """Ensure price is positive and reasonable."""+ if v < 0:+ raise ValueError('Price cannot be negative')+ if v > 1_000_000:+ raise ValueError('Price seems unreasonably high')+ return v++ @property+ def display_price(self) -> str:+ """Format price for display."""+ return f"${self.price:.2f}"+```++## Recommendations++The inline structure feature is a **practical solution for the majority of use cases**. Use it to:++- Get started quickly without Python overhead+- Keep all pipeline logic in one place+- Iterate rapidly during development+- Still get full type safety and validation++When your needs grow, **Python `StructuredContent` classes offer more power and flexibility**.++### Guidelines++- ✅ **Use inline structures** for straightforward data models+- ✅ **Use inline structures** during prototyping and early development+- ✅ **Use inline structures** for domain-specific models with simple validation+- ✅ **Use Python classes** when you need custom validation logic+- ✅ **Use Python classes** for reusable, shared data models+- ✅ **Use Python classes** when you need computed properties or methods+- ✅ **Use Python classes** for complex type relationships++Remember: You can always start with inline structures and migrate to Python classes later. The migration is straightforward, and AI assistants can help you make the transition quickly.++## Future Roadmap++### Nested Custom Concepts (Coming Soon)++Soon, you'll be able to reference other custom concepts as field types in inline structures:++```plx+[concept.Address.structure]+street = "Street address"+city = "City name"+postal_code = "Postal or ZIP code"++[concept.Company.structure]+name = "Company name"+headquarters = { type = "Address", description = "Company headquarters", required = true }+```++This will enable building complex, nested data models entirely within `.plx` files.++## Related Documentation++- [Define Your Concepts](define_your_concepts.md) - Learn about concept semantics and naming+- [Quick Start](../quick-start/index.md) - Get started with structured outputs+- [Design and Run Pipelines](design_and_run_pipelines.md) - Use structured concepts in pipes+diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.mdindex 54f85a6ea..2aef0687c 100644--- a/docs/pages/configuration/config-technical/library-config.md+++ b/docs/pages/configuration/config-technical/library-config.md@@ -1,107 +1,287 @@-# Library Configuration+# Pipeline Discovery and Loading-The Library Configuration manages how Pipelex organizes, loads, and handles libraries in your project. Libraries in Pipelex include pipelines and templates.+Pipelex automatically discovers and loads pipeline files (`.plx`) and structure classes from your project. This page explains how the discovery system works and how to organize your pipelines effectively.-## Directory Structure+## How Pipeline Discovery Works-The library system uses two main root directories:+When you initialize Pipelex with `Pipelex.make()`, the system:-- Internal library root (`pipelex/libraries`): Contains the base libraries shipped with Pipelex-- Exported library root (`pipelex_libraries`): Contains your project's libraries, including copies of base libraries+1. **Scans your project directory** for all `.plx` files+2. **Discovers Python structure classes** that inherit from `StructuredContent`+3. **Loads pipeline definitions** including domains, concepts, and pipes+4. **Registers custom functions** decorated with `@pipe_func()`-### Standard Paths+All of this happens automatically - no configuration needed.++## Excluded Directories++To improve performance and avoid loading unnecessary files, Pipelex automatically excludes common directories from discovery:++- `.venv` - Virtual environments+- `.git` - Git repository data+- `__pycache__` - Python bytecode cache+- `.pytest_cache` - Pytest cache+- `.mypy_cache` - Mypy type checker cache+- `.ruff_cache` - Ruff linter cache+- `node_modules` - Node.js dependencies+- `.env` - Environment files+- `results` - Common output directory++Files in these directories will not be scanned, even if they contain `.plx` files or structure classes.++## Project Organization++**Golden rule:** Put `.plx` files where they make sense in YOUR project. Pipelex finds them automatically.++### Common Patterns++**1. Topic-Based (Recommended for structured projects)**++Keep pipelines with related code:```-pipelex_libraries/ # Exported library root-├── pipelines/ # Pipeline definitions-│ ├── base_library/ # Base pipelines from Pipelex-│ └── your_pipelines/ # Your custom pipelines-└── templates/ # Template files+your_project/+├── my_project/+│ ├── finance/+│ │ ├── models.py+│ │ ├── services.py+│ │ ├── invoices.plx # With finance code+│ │ └── invoices_struct.py+│ └── legal/+│ ├── models.py+│ ├── contracts.plx # With legal code+│ └── contracts_struct.py+├── .pipelex/+└── requirements.txt```-## Library Loading Process+**Benefits:**+- Related things stay together+- Easy to find pipeline for a specific module+- Natural code organization-1. **Domain Loading**:+**2. Centralized (For simpler discovery)**- - Loads domain definitions first- - Each domain must be defined exactly once- - Supports system prompts and structure templates+Group all pipelines in one place:-2. **Concept Loading**:+```+your_project/+├── my_project/+│ ├── pipelines/ # All pipelines here+│ │ ├── finance.plx+│ │ ├── finance_struct.py+│ │ ├── legal.plx+│ │ └── legal_struct.py+│ └── core/+└── .pipelex/+```- - Loads native concepts first- - Loads custom concepts from PLX files- - Validates concept definitions and relationships+**Benefits:**+- All pipelines in one location+- Simple structure for small projects-3. **Pipe Loading**:+## Alternative Structures- - Loads pipe definitions after concepts- - Validates pipe configurations- - Links pipes with their respective domains+Pipelex supports flexible organization. Here are other valid approaches:-### Library Initialization+### Feature-Based Organization-Use the CLI command to initialize libraries:-```bash-pipelex init libraries+```+your_project/+├── my_project/+│ ├── features/+│ │ ├── document_processing/+│ │ │ ├── extract.plx+│ │ │ └── extract_struct.py+│ │ └── image_generation/+│ │ ├── generate.plx+│ │ └── generate_struct.py+│ └── main.py+└── .pipelex/```-This will:+### Domain-Driven Organization-1. Create the necessary directory structure-2. Copy base libraries to your project-3. Set up initial configuration files+```+your_project/+├── my_project/+│ ├── finance/+│ │ ├── pipelines/+│ │ │ └── invoices.plx+│ │ └── invoice_struct.py+│ ├── legal/+│ │ ├── pipelines/+│ │ │ └── contracts.plx+│ │ └── contract_struct.py+│ └── main.py+└── .pipelex/+```-### Library Export Options+### Flat Organization (Small Projects)-When exporting libraries to your project:+```+your_project/+├── my_project/+│ ├── invoice_processing.plx+│ ├── invoice_struct.py+│ └── main.py+└── .pipelex/+```-- Use `overwrite=True` to force update existing files-- Maintain directory structure and initialization files+## Loading Process-## Validation+Pipelex loads your pipelines in a specific order to ensure dependencies are resolved correctly:++### 1. Domain Loading++- Loads domain definitions from all `.plx` files+- Each domain must be defined exactly once+- Supports system prompts and structure templates per domain++### 2. Concept Loading++- Loads native concepts (Text, Image, PDF, etc.)+- Loads custom concepts from `.plx` files+- Validates concept definitions and relationships+- Links concepts to Python structure classes by name++### 3. Structure Class Registration++- Discovers all classes inheriting from `StructuredContent`+- Registers them in the class registry+- Makes them available for structured output generation++### 4. Pipe Loading++- Loads pipe definitions from `.plx` files+- Validates pipe configurations+- Links pipes with their respective domains+- Resolves input/output concept references-The library manager performs several validation steps:+### 5. Function Registration-1. **Concept Library Validation**:+- Discovers functions decorated with `@pipe_func()`+- Registers them in the function registry+- Makes them available for `PipeFunc` operators- - Checks concept relationships- - Validates concept definitions+## Custom Function Registration-2. **Pipe Library Validation**:+For custom functions used in `PipeFunc` operators, add the `@pipe_func()` decorator:- - Verifies pipe configurations- - Checks domain relationships+```python+from pipelex.system.registries.func_registry import pipe_func+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.text_content import TextContent-3. **Domain Library Validation**:+@pipe_func()+async def my_custom_function(working_memory: WorkingMemory) -> TextContent:+ """+ This function is automatically discovered and registered.+ """+ input_data = working_memory.get_stuff("input_name")+ # Process data+ return TextContent(text=f"Processed: {input_data.content.text}")- - Ensures domain completeness- - Validates domain relationships+# Optional: specify a custom name+@pipe_func(name="custom_processor")+async def another_function(working_memory: WorkingMemory) -> TextContent:+ # Implementation+ pass+```++## Validation-## Error Handling+After making changes to your pipelines, validate them:-The library system includes specific error types:+```bash+# Validate all pipelines+pipelex validate all-- `LibraryError`: Base error for library issues-- `LibraryParsingError`: For PLX parsing issues-- `ConceptLibraryError`: For concept-related issues-- `PipeLibraryError`: For pipe-related issues+# Validate a specific pipe+pipelex validate pipe YOUR_PIPE_CODE++# Show all available pipes+pipelex show pipes++# Show details of a specific pipe+pipelex show pipe YOUR_PIPE_CODE+```## Best Practices-1. **Organization**:+### 1. Organization++- Keep related concepts and pipes in the same `.plx` file+- Use meaningful domain names that reflect functionality+- Match Python file names with PLX file names (`finance.plx` → `finance.py`)+- Group complex pipelines using subdirectories++### 2. Structure Classes++- Only create Python classes when you need structured output+- Name classes to match concept names exactly+- Use `_struct.py` suffix for files containing structure classes (e.g., `finance_struct.py`)+- Inherit from `StructuredContent` or its subclasses+- Place structure class files near their corresponding `.plx` files+- **Keep modules clean**: Avoid module-level code that executes on import (Pipelex imports modules during auto-discovery)++### 3. Custom Functions++- Always use the `@pipe_func()` decorator+- Use descriptive function names+- Document function parameters and return types+- Keep functions focused and testable+- **Keep modules clean**: Avoid module-level code that executes on import (Pipelex imports modules during auto-discovery)++### 4. Validation++- Run `pipelex validate all` after making changes+- Check for domain consistency+- Verify concept relationships+- Test pipes individually before composing them++## Troubleshooting++### Pipelines Not Found++**Problem:** Pipelex doesn't find your `.plx` files.++**Solutions:**++1. Ensure files have the `.plx` extension+2. Check that files are not in excluded directories+3. Verify file permissions allow reading+4. Run `pipelex show pipes` to see what was discovered++### Structure Classes Not Registered++**Problem:** Your Python classes aren't recognized.++**Solutions:**++1. Ensure classes inherit from `StructuredContent`+2. Check class names match concept names exactly+3. Verify files are not in excluded directories+4. Make sure class definitions are valid Python++### Custom Functions Not Found++**Problem:** `PipeFunc` can't find your function.++**Solutions:**- - Keep related concepts and pipes in the same PLX file- - Use meaningful domain names- - Structure complex libraries using subdirectories+1. Add the `@pipe_func()` decorator+2. Ensure function signature matches requirements+3. Check function is `async` and accepts `working_memory`+4. Verify function is in a discoverable location-2. **Validation**:+### Validation Errors- - Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes- - Check for domain consistency- - Verify concept relationships+**Problem:** `pipelex validate all` shows errors.-3. **Customization**:+**Solutions:**- - Keep custom pipelines separate from base library- - Document domain-specific configurations+1. Read error messages carefully - they indicate the problem+2. Check concept references are spelled correctly+3. Verify pipe configurations match expected format+4. Ensure all required fields are presentdiff --git a/docs/pages/configuration/index.md b/docs/pages/configuration/index.mdindex 2c0cf18e2..e43037c86 100644--- a/docs/pages/configuration/index.md+++ b/docs/pages/configuration/index.md@@ -69,5 +69,5 @@ NB: The run_mode unit_test is used for testing purposes.1. **Version Control**: Include your base `pipelex.toml` in version control2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings3. **Documentation**: Comment any custom settings for team reference-4. **Validation**: Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making configuration changes+4. **Validation**: Run `pipelex validate all` after making configuration changes5. **Gitignore**: Add local and sensitive override files to `.gitignore`diff --git a/docs/pages/cookbook-examples/extract-dpe.md b/docs/pages/cookbook-examples/extract-dpe.mdindex b185974c5..4af51b595 100644--- a/docs/pages/cookbook-examples/extract-dpe.md+++ b/docs/pages/cookbook-examples/extract-dpe.md@@ -12,14 +12,11 @@ The pipeline `power_extractor_dpe` is designed to recognize and extract the key```pythonasync def extract_dpe(pdf_url: str) -> Dpe:- working_memory = WorkingMemoryFactory.make_from_pdf(- pdf_url=pdf_url,- concept_string="PDF",- name="pdf",- )pipe_output = await execute_pipeline(pipe_code="power_extractor_dpe",- working_memory=working_memory,+ input_memory={+ "document": PDFContent(url=pdf_url),+ },)working_memory = pipe_output.working_memorydpe: Dpe = working_memory.get_list_stuff_first_item_as(name="dpe", item_type=Dpe)@@ -62,21 +59,26 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informa[pipe.write_markdown_from_page_content_dpe]type = "PipeLLM"description = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"-inputs = { page_content = "Page" }-output = "Dpe" # The output is structured as a Dpe object+inputs = { "page_content.page_view" = "Image", page_content = "Page" }+output = "Dpe"model = "llm_for_img_to_text"structuring_method = "preliminary_text"-system_prompt = """You are a multimodal LLM, expert in converting images into perfect markdown."""-prompt_template = """-You are given an image of a French 'Diagnostic de Performance Energetique'.+system_prompt = """You are a multimodal LLM, expert at converting images into perfect markdown."""+prompt = """+You are given an image of a French 'Diagnostic de Performance Energetique': $page_content.page_viewYour role is to convert the image into perfect markdown.To help you do so, you are given the text extracted from the page by an OCR model.@page_content.text_and_images.text.text- It is very important that you collect every element, especially if they are related to the energy performance of the building.+- Pay attention to all the pieces of information that may be included in images, graphs, charts, or tables.- We value letters like "A, B, C, D, E, F, G" as they are energy performance classes.-# ... (prompt continues)+- Pay attention to the text alignment, it might have been misaligned by the OCR.+- The OCR extraction may be highly incomplete. It is your job to complete the text and add the missing information using the image.+- Output only the markdown, nothing else. No need for "```markdown" or "```".+- You can use HTML if it helps you.+- You can use tables if it is relevant."""```This is a great example of how to create a highly specialized extraction pipeline by combining a custom data model with a detailed, guiding prompt.\ No newline at end of filediff --git a/docs/pages/cookbook-examples/extract-gantt.md b/docs/pages/cookbook-examples/extract-gantt.mdindex bb29bdbc1..9d8a5f99e 100644--- a/docs/pages/cookbook-examples/extract-gantt.md+++ b/docs/pages/cookbook-examples/extract-gantt.md@@ -12,19 +12,16 @@ The pipeline takes an image as input, creates a working memory, and then execute```pythonasync def extract_gantt(image_url: str) -> GanttChart:- # Create Working Memory- working_memory = WorkingMemoryFactory.make_from_image(- image_url=image_url,- concept_string="gantt.GanttImage",- name="gantt_chart_image",- )-# Run the pipepipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",- working_memory=working_memory,+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttChartImage",+ "content": ImageContent(url=image_url),+ }+ },)-# Output the resultreturn pipe_output.main_stuff_as(content_type=GanttChart)```@@ -74,16 +71,15 @@ steps = [{ pipe = "gather_in_a_gantt_chart", result = "gantt_chart" },]-# This is the pipe that extracts the details for a single task[pipe.extract_details_of_task]type = "PipeLLM"description = "Extract the precise dates of the task, start_date and end_date"inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }-output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails object+output = "GanttTaskDetails"structuring_method = "preliminary_text"model = "llm_to_extract_diagram"-prompt_template = """-I am sharing an image of a Gantt chart.+prompt = """+I am sharing an image of a Gantt chart: $gantt_chart_image.Please analyse the image and for a given task name (and only this task), extract the information of the task, if relevant.Be careful, the time unit is this:diff --git a/docs/pages/cookbook-examples/extract-generic.md b/docs/pages/cookbook-examples/extract-generic.mdindex a08e5e586..e62da82cc 100644--- a/docs/pages/cookbook-examples/extract-generic.md+++ b/docs/pages/cookbook-examples/extract-generic.md@@ -12,14 +12,11 @@ The `power_extractor` pipeline is at the heart of this example. After its execut```pythonasync def extract_generic(pdf_url: str) -> TextAndImagesContent:- working_memory = WorkingMemoryFactory.make_from_pdf(- pdf_url=pdf_url,- concept_string="PDF",- name="pdf",- )pipe_output = await execute_pipeline(pipe_code="power_extractor",- working_memory=working_memory,+ input_memory={+ "document": PDFContent(url=pdf_url),+ },)working_memory = pipe_output.working_memorymarkdown_and_images: TextAndImagesContent = merge_markdown_and_images(working_memory)@@ -38,7 +35,7 @@ def merge_markdown_and_images(working_memory: WorkingMemory) -> TextAndImagesCon# ... (check for length equality)# Concatenate the markdown text- concatenated_markdown_text: str = "\\n".join([page_markdown.text for page_markdown in page_markdown_list.items])+ concatenated_markdown_text: str = "\n".join([page_markdown.text for page_markdown in page_markdown_list.items])# Aggregate the images from the page contentsimage_contents: List[ImageContent] = []diff --git a/docs/pages/cookbook-examples/extract-proof-of-purchase.md b/docs/pages/cookbook-examples/extract-proof-of-purchase.mdindex f17a75672..141dbf294 100644--- a/docs/pages/cookbook-examples/extract-proof-of-purchase.md+++ b/docs/pages/cookbook-examples/extract-proof-of-purchase.md@@ -12,14 +12,11 @@ The pipeline `power_extractor_proof_of_purchase` is specifically designed to han```pythonasync def extract_proof_of_purchase(pdf_url: str) -> ProofOfPurchase:- working_memory = WorkingMemoryFactory.make_from_pdf(- pdf_url=pdf_url,- concept_string="PDF",- name="pdf",- )pipe_output = await execute_pipeline(pipe_code="power_extractor_proof_of_purchase",- working_memory=working_memory,+ input_memory={+ "document": PDFContent(url=pdf_url),+ },)working_memory = pipe_output.working_memoryproof_of_purchase: ProofOfPurchase = working_memory.get_list_stuff_first_item_as(name="proof_of_purchase", item_type=ProofOfPurchase)@@ -58,12 +55,12 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d[pipe.write_markdown_from_page_content_proof_of_purchase]type = "PipeLLM"description = "Write markdown from page content"-inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the page-output = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase object+inputs = { "page_content.page_view" = "Image", page_content = "Page" }+output = "ProofOfPurchase"model = "llm_for_img_to_text"structuring_method = "preliminary_text"system_prompt = """You are a multimodal LLM, expert at converting images into perfect markdown."""-prompt_template = """+prompt = """You are given an image of a proof of purchase.Your role is to convert the image into perfect markdown.diff --git a/docs/pages/cookbook-examples/extract-table.md b/docs/pages/cookbook-examples/extract-table.mdindex 4243ab8f1..80791a2ad 100644--- a/docs/pages/cookbook-examples/extract-table.md+++ b/docs/pages/cookbook-examples/extract-table.md@@ -12,14 +12,14 @@ The pipeline `extract_html_table_and_review` takes an image of a table, processe```pythonasync def extract_table(table_screenshot: str) -> HtmlTable:- working_memory = WorkingMemoryFactory.make_from_image(- image_url=table_screenshot,- concept_string="tables.TableScreenshot",- name="table_screenshot",- )pipe_output = await execute_pipeline(pipe_code="extract_html_table_and_review",- working_memory=working_memory,+ input_memory={+ "table_screenshot": {+ "concept": "tables.TableScreenshot",+ "content": ImageContent(url=table_screenshot),+ }+ },)html_table = pipe_output.main_stuff_as(content_type=HtmlTable)return html_table@@ -77,7 +77,7 @@ type = "PipeLLM"description = "Review an HTML table"inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }output = "HtmlTable"-prompt_template = """+prompt = """Your role is to correct an html_table to make sure that it matches the one in the provided image.@html_tablediff --git a/docs/pages/cookbook-examples/hello-world.md b/docs/pages/cookbook-examples/hello-world.mdindex 443690e5f..e9fd824b4 100644--- a/docs/pages/cookbook-examples/hello-world.md+++ b/docs/pages/cookbook-examples/hello-world.md@@ -17,14 +17,16 @@ The `hello_world` function demonstrates the simplest possible Pipelex pipeline.```pythonimport asyncio-from pipelex.tools.misc.pretty import pretty_print+from pipelex import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipelineasync def hello_world():-- # Execute the pipeline+ """+ This function demonstrates the use of a super simple Pipelex pipeline to generate text.+ """+ # Run the pipepipe_output = await execute_pipeline(pipe_code="hello_world",)diff --git a/docs/pages/cookbook-examples/invoice-extractor.md b/docs/pages/cookbook-examples/invoice-extractor.mdindex 0aacf4bbb..8e37231ff 100644--- a/docs/pages/cookbook-examples/invoice-extractor.md+++ b/docs/pages/cookbook-examples/invoice-extractor.md@@ -11,17 +11,12 @@ This example provides a comprehensive pipeline for processing invoices. It takesThe `process_invoice` pipeline is a complete workflow for invoice processing.```python-async def process_expense_report() -> ListContent[Invoice]:- invoice_pdf_path = "assets/invoice_extractor/invoice_1.pdf"-- # Create Stuff objects- working_memory = WorkingMemoryFactory.make_from_pdf(- pdf_url=invoice_pdf_path,- name="invoice_pdf",- )+async def process_invoice(pdf_url: str) -> ListContent[Invoice]:pipe_output = await execute_pipeline(pipe_code="process_invoice",- working_memory=working_memory,+ input_memory={+ "document": PDFContent(url=pdf_url),+ },)return pipe_output.main_stuff_as_list(item_type=Invoice)@@ -62,28 +57,31 @@ class Invoice(StructuredContent):The entire workflow is defined in a PLX file. This declarative approach makes the pipeline easy to understand and modify. Here's a snippet from `invoice.plx`:```plx-# The main pipeline, a sequence of steps[pipe.process_invoice]type = "PipeSequence"description = "Process relevant information from an invoice"-inputs = { invoice_pdf = "PDF" }+inputs = { document = "PDF" }output = "Invoice"steps = [- # First, run OCR on the PDF{ pipe = "extract_text_from_image", result = "invoice_pages" },- # Then, run the invoice extraction on each page{ pipe = "extract_invoice", batch_over = "invoice_pages", batch_as = "invoice_page", result = "invoice" },]-# A sub-pipeline that uses an LLM to extract the data+[pipe.extract_text_from_image]+type = "PipeExtract"+description = "Extract page contents from an image"+inputs = { document = "PDF" }+output = "Page"+page_views = true+model = "base_extract_mistral"+[pipe.extract_invoice_data]type = "PipeLLM"description = "Extract invoice information from an invoice text transcript"-inputs = { "invoice_page.page_view" = "Page", invoice_details = "InvoiceDetails" }+inputs = { "invoice_page.page_view" = "Image", invoice_details = "InvoiceDetails", invoice_page = "Page" }output = "Invoice"-# The output is constrained to the "Invoice" model-model = "llm_to_extract_invoice"-prompt_template = """+model = "llm_to_extract_invoice"+prompt = """Extract invoice information from this invoice:The category of this invoice is: $invoice_details.category.@@ -91,7 +89,8 @@ The category of this invoice is: $invoice_details.category.@invoice_page.text_and_images.text.text"""```-This shows how a complex workflow, including OCR and LLM calls, can be defined in a simple, readable format. The `model = "llm_to_extract_invoice"` line is particularly powerful, as it tells the LLM to structure its output according to the `Invoice` model.++This shows how a complex workflow, including text extraction with `PipeExtract` and LLM calls, can be defined in a simple, readable format. The `model = "llm_to_extract_invoice"` line is particularly powerful, as it tells the LLM to structure its output according to the `Invoice` model.## The Pipeline Flowchartdiff --git a/docs/pages/cookbook-examples/simple-ocr.md b/docs/pages/cookbook-examples/simple-ocr.mdindex f606cffb2..5ec655a87 100644--- a/docs/pages/cookbook-examples/simple-ocr.md+++ b/docs/pages/cookbook-examples/simple-ocr.md@@ -6,25 +6,22 @@ This is a fundamental building block for many document processing workflows.## Get the code-[**➡️ View on GitHub: examples/simple_extract.py**](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/simple_ocr.py)+[**➡️ View on GitHub: examples/_quick_start/simple_ocr.py**](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/_quick_start/simple_ocr.py)## The Pipeline Explained-The core of this example is a simple function that creates a "working memory" from a PDF and then executes a pre-defined pipeline called `extract_page_contents_from_pdf`.+The core of this example is a simple function that executes a pre-defined pipeline called `extract_page_contents_from_pdf`.```python-async def simple_ocr(pdf_url: str):- working_memory = WorkingMemoryFactory.make_from_pdf(- pdf_url=pdf_url,- concept_string="PDF",- name="pdf",- )+async def simple_ocr(pdf_url: str) -> ListContent[PageContent]:pipe_output = await execute_pipeline(pipe_code="extract_page_contents_from_pdf",- working_memory=working_memory,+ input_memory={+ "document": PDFContent(url=pdf_url),+ },)page_content_list: ListContent[PageContent] = pipe_output.main_stuff_as_list(item_type=PageContent)return page_content_list```-This showcases how easy it is to kick off a complex process with just a few lines of code.\ No newline at end of file+This showcases how easy it is to kick off a complex process with just a few lines of code. The `input_memory` dictionary simply maps the input name to the PDF content, and the pipeline handles the rest.\ No newline at end of filediff --git a/docs/pages/cookbook-examples/write-screenplay.md b/docs/pages/cookbook-examples/write-screenplay.mdindex 0f1ef8b19..b9c809949 100644--- a/docs/pages/cookbook-examples/write-screenplay.md+++ b/docs/pages/cookbook-examples/write-screenplay.md@@ -8,26 +8,20 @@ This example demonstrates how to use Pipelex for creative text generation. It ta## The Pipeline Explained-The `generate_screenplay` function takes a pitch as a string, creates a `Stuff` object with the `screenplay.Pitch` concept, and then runs the `generate_screenplay` pipeline.+The `generate_screenplay` function takes a pitch as a string and executes the `generate_screenplay` pipeline, passing the pitch through the `input_memory` dictionary with the concept specification.```pythonasync def generate_screenplay(pitch: str):"""Generate a screenplay from a pitch using the pipeline."""- # Create Stuff object for the pitch- pitch_stuff = StuffFactory.make_from_concept_string(- concept_string="screenplay.Pitch",- content=TextContent(text=pitch),- name="pitch",- )-- # Create Working Memory- working_memory = WorkingMemoryFactory.make_from_single_stuff(pitch_stuff)-- # Run the pipepipe_output = await execute_pipeline(pipe_code="generate_screenplay",- working_memory=working_memory,+ input_memory={+ "pitch": {+ "concept": "screenplay.Pitch",+ "content": pitch,+ }+ },)pretty_print(pipe_output, title="Pipe Output")```diff --git a/docs/pages/cookbook-examples/write-tweet.md b/docs/pages/cookbook-examples/write-tweet.mdindex 067c3449a..f91d853f3 100644--- a/docs/pages/cookbook-examples/write-tweet.md+++ b/docs/pages/cookbook-examples/write-tweet.md@@ -8,38 +8,26 @@ This example demonstrates how to create a pipeline that takes a draft of a tweet## The Pipeline Explained-The `optimize_tweet` function is the core of this example. It takes two strings, `draft_tweet_str` and `writing_style_str`, creates two `Stuff` objects with the concepts `tech_tweet.DraftTweet` and `tech_tweet.WritingStyle`, and then runs the `optimize_tweet_sequence` pipeline.+The `optimize_tweet` function is the core of this example. It takes two strings, `draft_tweet` and `writing_style`, and executes the `optimize_tweet_sequence` pipeline, passing both inputs through the `input_memory` dictionary with their respective concept specifications.```python-async def optimize_tweet(draft_tweet_str: str, writing_style_str: str) -> OptimizedTweet:- # Create the draft tweet stuff- draft_tweet = StuffFactory.make_from_concept_string(- concept_string="tech_tweet.DraftTweet",- content=TextContent(text=draft_tweet_str),- name="draft_tweet",- )- writing_style = StuffFactory.make_from_concept_string(- concept_string="tech_tweet.WritingStyle",- content=TextContent(text=writing_style_str),- name="writing_style",- )-- # Create working memory- working_memory = WorkingMemoryFactory.make_from_multiple_stuffs(- [- draft_tweet,- writing_style,- ]- )-- # Run the sequence pipe+async def optimize_tweet(draft_tweet: str, writing_style: str) -> str:pipe_output = await execute_pipeline(pipe_code="optimize_tweet_sequence",- working_memory=working_memory,+ input_memory={+ "draft_tweet": {+ "concept": "tech_tweet.DraftTweet",+ "content": draft_tweet,+ },+ "writing_style": {+ "concept": "tech_tweet.WritingStyle",+ "content": writing_style,+ },+ },)# Get the optimized tweet- optimized_tweet = pipe_output.main_stuff_as(content_type=OptimizedTweet)+ optimized_tweet = pipe_output.main_stuff_as_strreturn optimized_tweet```@@ -82,7 +70,7 @@ system_prompt = """You are an expert in social media optimization, particularly for tech content on Twitter/X.Your role is to analyze tech tweets and check if they display typical startup communication pitfalls."""-prompt_template = """+prompt = """Evaluate the tweet for these key issues:**Fluffiness** - Overuse of buzzwords without concrete meaning...diff --git a/docs/pages/installation/index.md b/docs/pages/installation/index.mdindex 906ae2301..480262e9b 100644--- a/docs/pages/installation/index.md+++ b/docs/pages/installation/index.md@@ -1,65 +1,212 @@-# Installation+# Installation & Configuration-## Prerequisites--Pipelex requires `python` version `3.10` or above, and access to an LLM, via an API key or a custom endpoint.+## Installation-## Getting Started+Install Pipelex with pip:-Along with our [Quick Start Guide](../quick-start/index.md), we recommend you check out our [Cookbook](https://github.com/Pipelex/pipelex-cookbook) for practical examples.+```bash+pip install pipelex+```-- **Create a virtual environment** (recommended)+Or use [uv](https://github.com/astral-sh/uv) for faster installs:```bash-python3 -m venv .venv && source .venv/bin/activate+uv pip install pipelex```- - **Install Pipelex**+That's it! Pipelex is now installed. Continue to the [Quick-start Guide](../quick-start/index.md) to generate your first pipeline.++---++## API Configuration++To run pipelines with LLMs, you need to configure API access. **You have three options** - choose what works best for you:++### Option 1: Pipelex Inference (Easiest for Getting Started)++Get **free credits** for testing and development with a single API key that works with all major LLM providers:++**Benefits:**++- No credit card required+- Access to OpenAI, Anthropic, Google, Mistral, and more+- Perfect for development and testing+- Single API key for all models++**Setup:**++1. Join our Discord community to get your free API key:+ - Visit [https://go.pipelex.com/discord](https://go.pipelex.com/discord)+ - Request your key in the appropriate channel-Pipelex can be installed from PyPI. We encourage the use of [uv](https://github.com/astral-sh/uv) for faster installs and dependency management:+2. Create a `.env` file in your project root:+ ```bash+ echo "PIPELEX_INFERENCE_API_KEY=your-key-here" > .env+ ```++That's it! Your pipelines can now access any supported LLM.++### Option 2: Bring Your Own API Keys++Use your existing API keys from LLM providers. This is ideal if you:++- Already have API keys from providers+- Need to use specific accounts for billing+- Have negotiated rates or enterprise agreements++**Setup:**++Create a `.env` file in your project root with your provider keys:```bash-uv pip install pipelex+# OpenAI+OPENAI_API_KEY=sk-...++# Anthropic+ANTHROPIC_API_KEY=sk-ant-...++GOOGLE_API_KEY=...++# Mistral+MISTRAL_API_KEY=...++# FAL (for image generation)+FAL_API_KEY=...++# XAI+XAI_API_KEY=...++# Azure OpenAI+AZURE_API_KEY=...+AZURE_API_BASE=...+AZURE_API_VERSION=...++# AWS Bedrock+AWS_ACCESS_KEY_ID=...+AWS_SECRET_ACCESS_KEY=...+AWS_REGION=...```-Otherwise use pip:-```bash-pip install pipelex+You only need to add keys for the providers you plan to use.++**Configure the Inference Backend:**++When using your own keys, you need to tell Pipelex which backends to enable:++1. Initialize your configuration:+ ```bash+ pipelex init config+ ```++2. Edit `.pipelex/inference/backends.toml` to enable your providers++For example, to use Google models:++```toml+[google]+enabled = true```-- **Make sure you have a .env** file at the root of your project that contains the following fields+Learn more in our [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) guide.++### Option 3: Local AI (No API Keys Required)++Run AI models locally without any API keys. This is perfect if you:++- Want complete privacy and control+- Have capable hardware (GPU recommended)+- Need offline capabilities+- Want to avoid API costs++**Supported Local Options:**++**Ollama** (Recommended):++1. Install [Ollama](https://ollama.ai/)+2. Pull a model: `ollama pull llama2`+3. No API key needed! Configure Ollama backend in `.pipelex/inference/backends.toml`++**Other Local Providers:**++- **vLLM**: High-performance inference server+- **LM Studio**: User-friendly local model interface+- **llama.cpp**: Lightweight C++ inference++Configure these in `.pipelex/inference/backends.toml`. See our [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) for details.++---++## Initialize Configuration++To set up Pipelex configuration files, run:```bash-OPENAI_API_KEY=sk_...+pipelex init config```-All the secret keys used by `pipelex` are specified in the `.env.example` file. However, by default, only the `OPENAI_API_KEY` is required.+This creates a `.pipelex/` directory with:+- `pipelex.toml`: Feature flags, logging, cost reporting+- `inference/`: LLM configuration and model presets-- **Make sure you run the init commands:**+Learn more in our [Configuration documentation](../configuration/index.md).-In order to set the pipelex configuration files, you need to run 2 commands using the CLI (we recommend to run it at the root of your project):+----- `pipelex init libraries`: This will create a `pipelex_libraries` folder, with the base llm configuration and the base pipelines.-This is the directory where you should add your pipelines.+## Project Organization-The structure is like this:+Pipelex automatically discovers `.plx` pipeline files anywhere in your project (excluding `.venv`, `.git`, `node_modules`, etc.).++**Recommended: Keep pipelines with related code**```bash-├── pipelex_libraries-│ ├── __init__.py-│ ├── pipelines/ # The pipelines and the structured output are stored here-│ │ ├── __init__.py-│ │ └── base_library/ # The base library with basic pipelines-│ ├── templates/ # Those are template prompt libraries-│ ├── llm_deck/ # A llm deck is a simple way to name a llm and its configuration.-│ └── llm_integrations/ # This directory regroups the configuration of the different models+your_project/+├── my_project/ # Your Python package+│ ├── finance/+│ │ ├── services.py+│ │ ├── invoices.plx # Pipeline with finance code+│ │ └── invoices_struct.py # Structure classes+│ └── legal/+│ ├── services.py+│ ├── contracts.plx # Pipeline with legal code+│ └── contracts_struct.py+├── .pipelex/ # Config at repo root+│ └── pipelex.toml+├── .env # API keys (git-ignored)+└── requirements.txt```-Learn more about pipelex_libraries in our [Libraries documentation](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md)+**Alternative: Centralize pipelines**++```bash+your_project/+├── pipelines/+│ ├── invoices.plx+│ ├── contracts.plx+│ └── structures.py+└── .pipelex/+ └── pipelex.toml+```++Learn more in our [Project Structure documentation](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md).++---++## Prerequisites++- **Python**: Version 3.10 or above+- **API Access**: One of the three options above (Pipelex Inference, your own keys, or local AI)++---++## Next Steps-- `pipelex init config`: This cli command will create a `pipelex.toml` file at the root of the project, with basic configuration. This configuration file gathers all configuration for feature flags, logging, cost reporting, and so on... Learn more in our [Configuration documentation](../configuration/index.md)+Now that you have Pipelex installed and configured:+1. **Start generating pipelines**: [Quick-start Guide](../quick-start/index.md)+2. **Explore examples**: [Cookbook Repository](https://github.com/Pipelex/pipelex-cookbook)+3. **Learn the concepts**: [The Pipelex Paradigm](../pipelex-paradigm-for-repeatable-ai-workflows/index.md)+4. **Deep dive**: [Build Reliable AI Workflows](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md)-💡 _Any troubles? Have a look at our [Cookbook](https://github.com/Pipelex/pipelex-cookbook)!-<!-- and come ask for help on our [Discord](https://go.pipelex.com/discord)_ -->+💡 Need help? Check out our [Cookbook](https://github.com/Pipelex/pipelex-cookbook) for practical examples!diff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.mdindex f913699ea..7a6ea8258 100644--- a/docs/pages/quick-start/index.md+++ b/docs/pages/quick-start/index.md@@ -1,79 +1,139 @@# Quick-start-This guide shows the basics of Pipelex for the simplest use-cases: LLM calling and structured outputs.+Welcome to Pipelex! This guide will get you started building AI workflows in minutes.-You can **find more powerful examples** in the [Cookbook Examples](../cookbook-examples/index.md) section of the docs or dive directly into the [Cookbook repository](https://github.com/Pipelex/pipelex-cookbook). [](https://github.com/Pipelex/pipelex-cookbook/)+## What is Pipelex?++Pipelex is an open-source Python framework for building **repeatable AI workflows**. Instead of cramming everything into one complex prompt, you break tasks into focused steps—each pipe handling one clear transformation. These workflows process information from **extraction** to **analysis** to **decision making**. The result? **Deterministic structure with adaptive intelligence**: the reliability of software with the flexibility of AI.++Each pipe processes information: it takes structured inputs and produces structured outputs. Pipelex uses **Concepts** (typing with meaning) to ensure your pipelines make sense. A "non-compete clause" and a "flower description" are both text, but they represent different concepts. Pipelex validates that pipes connect properly, catching errors before they happen.++The `.plx` language is simple and readable—even for non-technical users. You can generate pipelines with AI or write them yourself.++Learn more about the philosophy in [The Pipelex Paradigm](../pipelex-paradigm-for-repeatable-ai-workflows/index.md).----## Setting up API Keys+## Installation++Install Pipelex with pip:-Before you can make LLM calls with Pipelex, you need to configure API keys. You have two options:+```bash+pip install pipelex+```-### Option 1: Use Pipelex Inference (Recommended for Getting Started)+That's it! Now let's generate your first pipeline.-Get **free access** to all well-known commercial and open-source LLMs with a single API key:+----1. **Join our Discord community to get your free Pipelex Inference key** (no credit card required, limited time offer)- - Visit [https://go.pipelex.com/discord](https://go.pipelex.com/discord) to join- - Request your API key in the appropriate channel once you're in+## Generate Your First Pipeline with AI-2. **Set up your environment**:- ```bash- # Create a .env file in your project root- echo "PIPELEX_INFERENCE_API_KEY=your-key-here" > .env- ```+The fastest way to get started is to generate pipelines using natural language. Pipelex can create complete, multi-step workflows for you.-With Pipelex Inference, you get instant access to models from OpenAI, Anthropic, Google, Mistral, and more - all through a single API key!+### Step 1: Generate with Pipe Builder-### Option 2: Use Your Own API Keys+The Pipe Builder transforms natural language descriptions into working `.plx` files. Here are some real-world examples:-If you already have API keys from LLM providers, you can use them directly:+**Example 1: Expense Report Validation**```bash-# Add to your .env file-# To directly use models on OpenAI, you will need to set the following variable-OPENAI_API_KEY=your-openai-key-# To directly use models on Anthropic, you will need to set the following variable-ANTHROPIC_API_KEY=your-anthropic-key-# To directly use models on Google, you will need to set the following variable-GOOGLE_API_KEY=your-google-key-# To directly use models on Mistral, you will need to set the following variable-MISTRAL_API_KEY=your-mistral-key-# To directly use models on FAL, you will need to set the following variable-FAL_API_KEY=your-fal-key-# To directly use models on XAI, you will need to set the following variable-XAI_API_KEY=your-xai-key--# To use models via Ollama, you will need to set the following variables-OLLAMA_API_KEY=your-ollama-key-# To use models via BlackboxAI, you will need to set the following variables-BLACKBOX_API_KEY=your-blackboxai-key-# To use models via Azure OpenAI, you will need to set the following variables-AZURE_API_KEY=your-azure-key-AZURE_API_BASE=your-azure-endpoint-AZURE_API_VERSION=your-azure-version-# To use models via AWS Bedrock, you will need to set the following variables-AWS_ACCESS_KEY_ID=your-aws-key-AWS_SECRET_ACCESS_KEY=your-aws-secret-AWS_REGION=your-aws-region+pipelex build pipe "Given an expense report, apply company rules" -o expense_pipeline.plx```-Adding those env variables is not enough. You also need to configure the inference backend to choose where to route the AI calls.-See the [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md#inference-backends) documentation.+**Example 2: CV/Job Matching**++```bash+pipelex build pipe "Take a CV in a PDF file, a Job offer text, and analyze if they match" -o cv_matcher.plx+```++**Example 3: Compliance Analysis**++```bash+pipelex build pipe "Given an RFP PDF, build a compliance matrix" -o compliance_pipeline.plx+```++These commands generate complete `.plx` files containing:++- Domain definitions+- Concept declarations+- Multi-step pipe workflows++The Pipe Builder handles complexity automatically, creating pipelines with proper structure and validation.++!!! info "Pipe Builder is in Beta"+ The Pipe Builder is currently in beta and improving rapidly. Expect frequent enhancements!++### Step 2: Install Agent Rules for Iteration++Once you have a generated pipeline, you can refine it using any AI coding assistant:++```bash+pipelex kit rules+```++This command installs Pipelex rules for:++- **Cursor**+- **Claude Code**+- **OpenAI Codex**+- **GitHub Copilot**+- **Windsurf**+- **Blackbox AI**++These rules teach your AI assistant Pipelex syntax, concepts, and best practices.++### Step 3: Iterate with Your AI Assistant++With the rules installed, use natural language to refine your pipeline:++- "Add a validation step to check for negative amounts"+- "Include confidence scores in the match analysis"+- "Add error handling for missing PDF files"+- "Create a summary report at the end"++Your AI assistant will understand Pipelex and make the changes correctly.++### Step 4: Test Your Pipeline++Run your pipeline from the command line or Python:++**CLI:**++```bash+pipelex run <pipe_code> --input-memory-from-json input.json+```++**Python:**++```python+import asyncio+from pipelex.pipeline.execute import execute_pipeline+from pipelex.pipelex import Pipelex++async def run_pipeline():+ pipe_output = await execute_pipeline(pipe_code="your_pipe_code")+ print(pipe_output.main_stuff_as_str)++Pipelex.make()+asyncio.run(run_pipeline())+```++!!! tip "Need API Access?"+ To run these pipelines with LLMs, you'll need API access. We offer **free credits** for testing and developing, or you can bring your own keys, or run local AI. See the [Installation & Configuration](../installation/index.md#api-configuration) guide for all options.-For example, if you want to use a gemini model via GOOGLE, enable the google backend in `.pipelex/inference/backends.toml` and set the API key in your env.----## Your first LLM call with Pipelex+## Understanding How It Works++Ready to dive deeper? This section shows you how to manually create pipelines and understand the `.plx` language.++### Your First LLM Call with Pipelex-Let's start by running your very first LLM call using Pipelex.-For illustration purposes, let's build **a character generator**. Each example relies on asynchronous execution and typed models for reliable prompts.+Let's build a **character generator** to understand the basics.-### Write your first pipeline+#### Write Your First Pipeline-First, create a `.plx` library file in the `pipelex_libraries/pipelines` directory to store your pipe definition.-Run `pipelex init libraries` to create this directory if it doesn't exist. For now, keep all your pipeline definitions inside that folder only.+Create a `.plx` file anywhere in your project (we recommend a `pipelines` directory):`character.plx````plx@@ -84,13 +144,20 @@ domain = "characters"type = "PipeLLM"description = "Creates a character."output = "Text"-prompt_template = """You are a book writer. Your task is to create a character.+prompt = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""```-### Run your first Pipelex script+This pipeline:-Now, create a `.py` python file to run your script. You can save it anywhere in your repository.+- Declares a `characters` domain+- Defines a `create_character` pipe of type `PipeLLM`+- Outputs plain `Text`+- Uses a simple prompt++#### Run Your First Pipelex Script++Create a Python file to execute the pipeline:`character.py````python@@ -113,7 +180,7 @@ Pipelex.make()asyncio.run(create_character())```-### Get your first Pipelex result+#### Get Your First Pipelex Result```bashpython character.py@@ -121,9 +188,9 @@ python character.py-## How to use a specific LLM or LLM provider+### Using Specific LLMs-### Indicate your LLM selection explicitly using the `llm` attribute+#### Indicate Your LLM Selection Explicitly```plx[pipe.create_character]@@ -131,11 +198,11 @@ type = "PipeLLM"description = "Create a character."output = "Text"model = { model = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }-prompt_template = """You are a book writer. Your task is to create a character.+prompt = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""```-### Or use an LLM preset from the LLM deck+#### Or Use an LLM Preset from the LLM Deck```plx[pipe.create_character]@@ -143,7 +210,7 @@ type = "PipeLLM"description = "Create a character."output = "Text"model = "llm_for_creative_writing"-prompt_template = """You are a book writer. Your task is to create a character.+prompt = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""# The llm deck above is defined in `.pipelex/inference/deck/base_deck.toml` as:@@ -151,20 +218,19 @@ Think of it and then output the character description."""# it's a base preset that we provide. you can add your own presets, too.```-💡 We have a lot of [LLM presets available by default](https://github.com/Pipelex/pipelex/tree/main/.pipelex/inference/deck/base_deck.toml).-Make sure you have credentials for the underlying LLM provider (and added your API key to the `.env`) and select the one you want!+💡 We have many [LLM presets available by default](https://github.com/Pipelex/pipelex/tree/main/.pipelex/inference/deck/base_deck.toml).-Learn more about LLM presets, LLM handles and LLM deck in our [LLM Configuration Guide](../build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md)+Learn more in our [LLM Configuration Guide](../build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md).-### Generate a structured output+### Generate Structured Outputs-Let's say that we no longer want plain text as output but a rigorously structured Character object.+Let's create a rigorously structured `Character` object instead of plain text.-### Define the model+#### Define the Structure-Using the [Pydantic BaseModel](https://docs.pydantic.dev/latest/) syntax, define your object structure as a Python class, in the `pipelex_libraries/pipelines` directory:+Using [Pydantic BaseModel](https://docs.pydantic.dev/latest/) syntax:-`pipelex_libraries/pipelines/characters.py`+`characters.py````pythonfrom pipelex.core.stuffs.structured_content import StructuredContent@@ -177,50 +243,66 @@ class Character(StructuredContent):description: str```-### Improve the pipeline+!!! tip "Keep Structure Files Clean"+ Keep your `StructuredContent` classes in dedicated files with minimal module-level code. Pipelex imports these modules during auto-discovery, so any module-level code will be executed.++💡 **Alternative: Inline Structure Definition**++Define structures directly in your `.plx` file:++```plx+[concept.Character]+description = "A character in a fiction story"++[concept.Character.structure]+name = "The character's name"+age = { type = "integer", description = "The character's age", required = true }+gender = "The character's gender"+description = "A description of the character"+```++Learn more in [Structuring Concepts](../build-reliable-ai-workflows-with-pipelex/structuring-concepts.md).-It's time to specify that your output be a `Character` instance. Use the `output` field for that purpose.+#### Improve the Pipeline-💡 Here, the concept name matches the class name (ie. `Character`), the `Character` class will automatically be considered as the structure to output.+Specify that your output is a `Character` instance:-`pipelex_libraries/pipelines/characters.plx`+`characters.plx````plxdomain = "characters"[concept]-Character = "A character is a fiction story" # <- Define here your output concept so that it is linked to the class name+Character = "A character in a fiction story" # <- Define your output concept[pipe][pipe.create_character]type = "PipeLLM"description = "Create a character. Get a structured result."output = "Character" # <- This is the output concept for your pipe-prompt_template = """You are a book writer. Your task is to create a character.+prompt = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""```-💡 Defining the `Character` concept as "A character is a fiction story" might seem obvious but… think of it: "character" can also mean a letter or symbol in a text. Defining concepts is the best way to avoid any ambiguity and make sure the LLMs understand what you mean.+💡 The concept name matches the class name (`Character`), so Pipelex automatically links them.-### Run your pipeline+💡 Defining concepts removes ambiguity—"character" could mean a letter or symbol, but here it clearly means a fictional person.-As you can see, the output is a `Character` instance.+#### Run Your Pipeline-+The output is now a structured `Character` instance:+-## Generate using information in a prompt template--What if you want to pass some data into a prompt?-You can do that using a prompt template.+### Using Prompt Templates-In this example, we no longer want to generate characters. We want to process existing ones, especially their description attributes.+Pass data into prompts using templates.-We want to extract structured information from the description field. Thus we have a `Character` input and a `CharacterMetadata` output.+Let's process existing characters and extract metadata from their descriptions.-### Define the output structure+#### Define the Output Structure```python-# pipelex_libraries/pipelines/character_model.py+# character_model.pyfrom pipelex.core.stuffs.structured_content import StructuredContent# input class@@ -238,9 +320,9 @@ class CharacterMetadata(StructuredContent):height: float```-### **Let's use a template to fill prompts with data**+#### Use a Template to Fill Prompts with Data-💡 Our template syntax is based on [Jinja2 syntax](https://jinja.palletsprojects.com/en/stable/). You can include a variable using the **classic** `{{ double.curly.braces }}`, and to make it simpler, we've added the possibility to just prefix your variable with the `@` symbol (recommended). Pipes declare their required inputs explicitly with the `inputs` table:+💡 Our template syntax is based on [Jinja2](https://jinja.palletsprojects.com/en/stable/). Use `{{ double.curly.braces }}` or the simpler `@` prefix (recommended).```plx[concept]@@ -251,9 +333,9 @@ CharacterMetadata = "Metadata regarding a character."[pipe.extract_character_1]type = "PipeLLM"description = "Get character information from a description."-inputs = { character = "Character" } # <- These are the inputs of your pipe, usable in the prompt_template+inputs = { character = "Character" } # <- These inputs are usable in the promptoutput = "CharacterMetadata"-prompt_template = """+prompt = """You are given a text description of a character.Your task is to extract specific data from the following description.@@ -261,11 +343,11 @@ Your task is to extract specific data from the following description."""```-💡 `@character.description` is substituted by grabbing the stuff named `character`in the working memory and using its `description`attribute+💡 `@character.description` grabs the `character` stuff from working memory and uses its `description` attribute.-Learn more about how we use Jinja in the [PipeLLM documentation](../build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md).+Learn more about Jinja in the [PipeLLM documentation](../build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md).-### **This is how you do it from the code side**+#### Execute from Python```pythonimport asyncio@@ -275,7 +357,7 @@ from pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline-from pipelex.libraries.pipelines.screenplay import Character, CharacterMetadata+from character_model import Character, CharacterMetadataasync def process_existing_character():@@ -309,7 +391,7 @@ async def process_existing_character():working_memory=working_memory,)- # Get the result as a porperly typed instance+ # Get the result as a properly typed instanceextracted_metadata = pipe_output.main_stuff_as(content_type=CharacterMetadata) # <- This is the output of your pipe, properly typedprint(extracted_metadata)@@ -319,6 +401,32 @@ Pipelex.make()asyncio.run(process_existing_character())```-### **Get result**+#### Get Result++---++## Next Steps++Now that you understand the basics, explore more:++**Learn More:**++- [Cookbook Examples](../cookbook-examples/index.md) - Real-world examples and patterns+- [Build Reliable AI Workflows](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md) - Deep dive into pipeline design+- [Pipe Operators](../build-reliable-ai-workflows-with-pipelex/pipe-operators/index.md) - PipeLLM, PipeExtract, PipeCompose, and more+- [Pipe Controllers](../build-reliable-ai-workflows-with-pipelex/pipe-controllers/index.md) - PipeSequence, PipeParallel, PipeBatch, PipeCondition++**Explore Tools:**++- [Pipe Builder](../tools/pipe-builder.md) - Generate pipelines from natural language+- [Kit Commands](../tools/kit.md) - Manage agent rules and migrations+- [CLI Commands](../tools/cli.md) - Command-line interface reference++**Configure:**++- [LLM Configuration](../build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md) - Optimize cost and quality+- [Inference Backend](../configuration/config-technical/inference-backend-config.md) - Configure model providers++[](https://github.com/Pipelex/pipelex-cookbook/)diff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.mdindex ce7b65f53..557131cfe 100644--- a/docs/pages/tools/cli.md+++ b/docs/pages/tools/cli.md@@ -6,10 +6,9 @@ The Pipelex CLI provides a command-line interface for managing and interacting w### Init group-Initialize project assets.+Initialize project configuration.```bash-pipelex init libraries [DIRECTORY] [--overwrite/-o]pipelex init config [--reset/-r]```@@ -18,8 +17,8 @@ pipelex init config [--reset/-r]Validate configuration and pipelines.```bash-pipelex validate all [-c/--config-folder-path PATH]-pipelex validate pipe PIPE_CODE [-c/--config-folder-path PATH]+pipelex validate all+pipelex validate pipe PIPE_CODE```### Show group@@ -28,15 +27,16 @@ Inspect configuration and pipes.```bashpipelex show config-pipelex show pipes [-c/--config-folder-path PATH]-pipelex show pipe PIPE_CODE [-c/--config-folder-path PATH]+pipelex show pipes+pipelex show pipe PIPE_CODE```## Usage Tips-1. Always run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes to your configuration or pipelines+1. Always run `pipelex validate all` after making changes to your configuration or pipelines2. Use `pipelex show config` to debug configuration issues-3. When initializing a new project:- - Start with `pipelex init config`- - Then run `pipelex init libraries`- - Finally, validate your setup with `pipelex validate all -c path/to/your/pipelex/config/folder`+3. Use `pipelex show pipes` to see all discovered pipelines+4. When initializing a new project:+ - Run `pipelex init config` to create configuration files+ - Create your `.plx` pipeline files anywhere in your project+ - Validate your setup with `pipelex validate all`diff --git a/docs/pages/tools/kit.md b/docs/pages/tools/kit.mdnew file mode 100644index 000000000..7787ad8f9--- /dev/null+++ b/docs/pages/tools/kit.md@@ -0,0 +1,136 @@+# Pipelex Kit Commands++The Pipelex Kit provides commands for managing agent rules and migration instructions. These commands help you integrate Pipelex guidelines into your AI coding assistants and keep track of breaking changes across versions.++## Available Commands++### Install Agent Rules++Install Pipelex agent rules for AI coding assistants:++```bash+pipelex kit rules+```++This command:++1. Exports agent markdown files to Cursor `.mdc` files with YAML front-matter in `.cursor/rules`+2. Builds merged agent documentation and updates target files for other AI assistants++**Supported AI Assistants:**++- **Cursor** (`.cursor/rules/`)+- **Claude Code** (`CLAUDE.md`)+- **OpenAI Codex** (`AGENTS.md`)+- **GitHub Copilot** (`.github/copilot-instructions.md`)+- **Windsurf** (`.windsurfrules.md`)+- **Blackbox AI** (`BLACKBOX_RULES.md`)++**Options:**++- `--repo-root PATH`: Repository root directory (default: current directory)+- `--cursor/--no-cursor`: Export Cursor rules (default: enabled)+- `--single-files/--no-single-files`: Update single-file agent documentation targets (default: enabled)+- `--dry-run`: Show what would be done without making changes+- `--diff`: Show unified diff of changes+- `--backup SUFFIX`: Create backups with the specified suffix (e.g., `.bak`)++**Examples:**++```bash+# Install rules for all supported AI assistants+pipelex kit rules++# Preview changes without applying them+pipelex kit rules --dry-run --diff++# Install only Cursor rules+pipelex kit rules --no-single-files++# Create backups before updating+pipelex kit rules --backup .bak+```++### Remove Agent Rules++Remove Pipelex agent rules from your project:++```bash+pipelex kit remove-rules+```++This command:++1. Deletes agent markdown files from Cursor `.mdc` files in `.cursor/rules`+2. Removes marked sections from target files (or deletes entire files with `--delete-files`)++**Options:**++- `--repo-root PATH`: Repository root directory (default: current directory)+- `--cursor/--no-cursor`: Remove Cursor rules (default: enabled)+- `--single-files/--no-single-files`: Remove agent documentation from target files (default: enabled)+- `--delete-files`: Delete entire target files instead of just removing marked sections+- `--dry-run`: Show what would be done without making changes+- `--diff`: Show unified diff of changes+- `--backup SUFFIX`: Create backups with the specified suffix (e.g., `.bak`)++**Examples:**++```bash+# Remove all agent rules+pipelex kit remove-rules++# Preview what would be removed+pipelex kit remove-rules --dry-run++# Remove only Cursor rules+pipelex kit remove-rules --no-single-files++# Delete entire target files+pipelex kit remove-rules --delete-files+```++### Sync Migration Instructions++Sync migration instructions from the Pipelex kit to your project:++```bash+pipelex kit migrations+```++This command copies migration documentation files from the `pipelex.kit` package to your project's `.pipelex/migrations` directory. These files provide detailed instructions for migrating between Pipelex versions.++**Options:**++- `--repo-root PATH`: Repository root directory (default: current directory)+- `--dry-run`: Show what would be done without making changes++**Examples:**++```bash+# Sync migration instructions+pipelex kit migrations++# Preview what would be copied+pipelex kit migrations --dry-run+```++## Agent Rules Overview++The agent rules installed by `pipelex kit rules` include:++- **`write_pipelex.md`**: Guidelines for writing Pipelex pipelines, including syntax, best practices, and common patterns+- **`run_pipelex.md`**: Guidelines for running and testing Pipelex pipelines+- **`llms.md`**: LLM configuration and usage guidelines+- **`python_standards.md`**: Python coding standards and best practices+- **`docs.md`**: Guidelines for writing documentation+- **`pytest_standards.md`**: Guidelines for writing unit tests+- **`tdd.md`**: Test-driven development guidelines++These rules are configured in `pipelex/kit/index.toml`.++## Related Documentation++- [Pipe Builder](pipe-builder.md) - Generate pipelines from natural language+- [CLI Commands](cli.md) - General Pipelex CLI documentation+diff --git a/docs/pages/tools/pipe-builder.md b/docs/pages/tools/pipe-builder.mdnew file mode 100644index 000000000..1896659e8--- /dev/null+++ b/docs/pages/tools/pipe-builder.md@@ -0,0 +1,124 @@+# Pipe Builder++!!! warning "Beta Feature"+ The Pipe Builder is currently in beta and progressing fast. Expect frequent improvements and changes.++The Pipe Builder is an AI-powered tool that generates Pipelex pipelines from natural language descriptions. It helps you quickly prototype pipelines by describing what you want to achieve, and the builder translates your requirements into working `.plx` files.++## Overview++The Pipe Builder uses AI to:++- Understand your pipeline requirements from a brief description+- Generate domain concepts, pipe specifications, and complete pipeline structure+- Validate the generated pipeline for common errors+- Automatically fix certain deterministic issues++## Usage++Generate a pipeline with one validation/fix loop that automatically corrects deterministic issues:++```bash+pipelex build pipe "Brief description of what the pipeline should do" -o path/to/output.plx+```++**Example:**++```bash+pipelex build pipe "Given an expense report, apply company rules" -o results/expense_pipeline.plx+```++This command:++1. Generates a complete pipeline from your brief+2. Validates the pipeline structure+3. Attempts to automatically fix common errors+4. Saves the final pipeline to the specified path++## Options++The build command supports the following options:++- `--output`, `-o`: Path to save the generated file (default: `./results/generated_pipeline.plx`)+- `--no-output`: Skip saving the file (useful for testing)++## How It Works++The Pipe Builder follows this process:++1. **Analysis**: Analyzes your brief to understand the domain and requirements+2. **Concept Generation**: Creates appropriate domain concepts for your workflow+3. **Pipe Generation**: Generates pipe operators and controllers to implement the logic+4. **Validation**: Validates the generated pipeline structure+5. **Automatic Fixes**: Fixes common errors like missing inputs or incorrect pipe connections++## Example Use Cases++**Document Processing:**++```bash+pipelex build pipe "Take a CV in a PDF file and a Job offer text, and analyze if they match"+```++**Data Transformation:**++```bash+pipelex build pipe "Extract structured data from invoice images"+```++**Multi-step Workflows:**++```bash+pipelex build pipe "Given an RFP PDF, build a compliance matrix"+```++## Current Limitations++The Pipe Builder is in active development and currently:++- Can automatically fix input/output connection errors+- May require manual adjustments for complex conditional logic or custom functions+- Validation focuses on structural correctness, not business logic++## Tips for Best Results++- You can be specific in your brief about inputs, outputs, data formats, or structures if you know what you need+- If you're uncertain about the details, let the AI figure it out and see what it generates+- Include any domain-specific requirements you're aware of upfront++## Iterating on Generated Pipelines++After generating a pipeline, you can continue refining it using any Software Engineering (SWE) agent. The generated `.plx` file can be iteratively improved through natural language instructions.++Pipelex provides specialized agent rules (`write_pipelex.md` and `run_pipelex.md`) that guide AI assistants in working with pipelines. You can install these rules for your preferred AI coding assistant using:++```bash+pipelex kit rules+```++This command installs the rules for:++- **Cursor**+- **Claude Code**+- **OpenAI Codex**+- **GitHub Copilot**+- **Windsurf**+- **Blackbox AI**++These rules help AI assistants understand Pipelex syntax, best practices, and common patterns, making it easier to iterate and refine your generated pipelines.++## Next Steps++After generating a pipeline:++1. Review the generated `.plx` file+2. Test it with sample inputs: `pipelex run <pipe_code> --input-memory-from-json input.json`+3. Continue iterating using your preferred SWE agent with the Pipelex agent rules+4. Adjust concepts or pipe configurations as needed++For more information on pipeline structure and customization, see:++- [Design and Run Pipelines](../build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md)+- [Pipe Operators](../build-reliable-ai-workflows-with-pipelex/pipe-operators/index.md)+- [Pipe Controllers](../build-reliable-ai-workflows-with-pipelex/pipe-controllers/index.md)+diff --git a/mkdocs.yml b/mkdocs.ymlindex 0f76fd2da..a400b27e0 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -85,8 +85,8 @@ markdown_extensions:nav:- Home:- Welcome: index.md- - Installation: pages/installation/index.md- Quick-start: pages/quick-start/index.md+ - Installation: pages/installation/index.md- Understand Pipelex:- The Knowledge Pipeline Manifesto: manifesto.md- The Pipelex Paradigm: pages/pipelex-paradigm-for-repeatable-ai-workflows/index.md@@ -108,6 +108,7 @@ nav:- Build Reliable AI Workflows:- Kick off a Pipeline Project: pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md- Define Your Concepts: pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md+ - Structuring Concepts: pages/build-reliable-ai-workflows-with-pipelex/structuring-concepts.md- Design and Run Pipelines: pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md- Pipeline Creation: pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md- Pipe Operators:@@ -147,6 +148,8 @@ nav:- Changelog: changelog.md- Tools:- CLI: pages/tools/cli.md+ - Kit Commands: pages/tools/kit.md+ - Pipe Builder: pages/tools/pipe-builder.md- Logging: pages/tools/logging.md- Advanced Customizations:- Overview: pages/advanced-customization/index.mddiff --git a/pipelex/__init__.py b/pipelex/__init__.pyindex 98ce4968c..567519bf5 100644--- a/pipelex/__init__.py+++ b/pipelex/__init__.py@@ -1,4 +1,3 @@-from pipelex._bootstrap_user_libs import activate as _px_bootstrap_user_libsfrom pipelex.tools.log.log import logfrom pipelex.tools.misc.pretty import pretty_print, pretty_print_md@@ -7,9 +6,3 @@ __all__ = ["pretty_print","pretty_print_md",]--# -------------------------------------------------------------# Keep <project>/pipelex_libraries on sys.path for every installer (Fix for uv)-# --------------------------------------------------------------_px_bootstrap_user_libs()diff --git a/pipelex/_bootstrap_user_libs.py b/pipelex/_bootstrap_user_libs.pydeleted file mode 100644index 1f6b0a62d..000000000--- a/pipelex/_bootstrap_user_libs.py+++ /dev/null@@ -1,30 +0,0 @@-import sys-from pathlib import Path--# -------------------------------------------------------------------------# Public helper: run once, keep user libs import-able-# ---------------------------------------------------------------------------def activate() -> None:- # 1) Re-create Poetry’s behaviour: put <cwd> itself on sys.path- root = Path.cwd()- root_str = str(root)- if root_str not in sys.path:- sys.path.insert(0, root_str)-- # 2) If a pipelex_libraries folder already exists, also touch __init__.py- # (helps IDE & static type-checkers) and ensure the *exact* folder is- # on sys.path. When it does **not** exist yet, nothing else to do —- # as soon as the user runs `pipelex init-libraries`, the directory will be- # created *inside* <root>, which is already import-able thanks to (1).- for parent in (root, *root.parents):- lib_dir = parent / "pipelex_libraries"- if lib_dir.is_dir():- # 1) make it a *real* package so editors & type-checkers see it- (lib_dir / "__init__.py").touch(exist_ok=True)- # 2) put it at the front of sys.path exactly once- lib_path = str(lib_dir)- if lib_path not in sys.path:- sys.path.insert(0, lib_path)- breakdiff --git a/pipelex/libraries/pipelines/builder/__init__.py b/pipelex/builder/__init__.pysimilarity index 100%rename from pipelex/libraries/pipelines/builder/__init__.pyrename to pipelex/builder/__init__.pydiff --git a/pipelex/libraries/pipelines/builder/builder.plx b/pipelex/builder/builder.plxsimilarity index 100%rename from pipelex/libraries/pipelines/builder/builder.plxrename to pipelex/builder/builder.plxdiff --git a/pipelex/libraries/pipelines/builder/builder.py b/pipelex/builder/builder.pysimilarity index 94%rename from pipelex/libraries/pipelines/builder/builder.pyrename to pipelex/builder/builder.pyindex f96943eed..bca9d8647 100644--- a/pipelex/libraries/pipelines/builder/builder.py+++ b/pipelex/builder/builder.py@@ -2,6 +2,31 @@ from typing import TYPE_CHECKING, Annotated, castfrom pydantic import ConfigDict, Field, ValidationError, field_validator+from pipelex.builder.builder_errors import (+ ConceptDefinitionErrorData,+ ConceptFailure,+ ConceptSpecError,+ DomainFailure,+ PipeBuilderError,+ PipeDefinitionErrorData,+ PipeFailure,+ PipelexBundleError,+ PipelexBundleUnexpectedError,+ PipeSpecError,+ StaticValidationErrorData,+ ValidateDryRunError,+)+from pipelex.builder.concept.concept_spec import ConceptSpec+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec+from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.core.bundles.pipe_sorter import sort_pipes_by_dependenciesfrom pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion, PipelexBundleBlueprintfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprint@@ -16,32 +41,8 @@ from pipelex.exceptions import (StaticValidationError,)from pipelex.hub import get_library_manager-from pipelex.libraries.pipelines.builder.builder_errors import (- ConceptDefinitionErrorData,- ConceptFailure,- ConceptSpecError,- DomainFailure,- PipeBuilderError,- PipeDefinitionErrorData,- PipeFailure,- PipelexBundleError,- PipelexBundleUnexpectedError,- PipeSpecError,- StaticValidationErrorData,- ValidateDryRunError,-)-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_batch_spec import PipeBatchSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_compose_spec import PipeComposeSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_condition_spec import PipeConditionSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_extract_spec import PipeExtractSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_func_spec import PipeFuncSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_llm_spec import PipeLLMSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_parallel_spec import PipeParallelSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_sequence_spec import PipeSequenceSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_run.dry_run import dry_run_pipes+from pipelex.system.registries.func_registry import pipe_funcfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorif TYPE_CHECKING:@@ -189,6 +190,7 @@ def _convert_pipe_spec(pipe_spec: PipeSpecUnion) -> PipeSpecUnion:return cast("PipeSpecUnion", pipe_class.model_validate(pipe_spec.model_dump(serialize_as_any=True)))+@pipe_func()async def assemble_pipelex_bundle_spec(working_memory: WorkingMemory) -> PipelexBundleSpec:"""Construct a PipelexBundleSpec from working memory containing concept and pipe blueprints.diff --git a/pipelex/libraries/pipelines/builder/builder_errors.py b/pipelex/builder/builder_errors.pysimilarity index 89%rename from pipelex/libraries/pipelines/builder/builder_errors.pyrename to pipelex/builder/builder_errors.pyindex 2cab3ed0f..c30895a3d 100644--- a/pipelex/libraries/pipelines/builder/builder_errors.py+++ b/pipelex/builder/builder_errors.py@@ -2,21 +2,21 @@from pydantic import Field+from pipelex.builder.concept.concept_spec import ConceptSpec+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpec+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpec+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpec+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpec+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpec+from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpec+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpecfrom pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.exceptions import (PipelexException,StaticValidationErrorType,)-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_batch_spec import PipeBatchSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_compose_spec import PipeComposeSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_condition_spec import PipeConditionSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_extract_spec import PipeExtractSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_func_spec import PipeFuncSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_llm_spec import PipeLLMSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_parallel_spec import PipeParallelSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_sequence_spec import PipeSequenceSpecfrom pipelex.types import Self# Type alias for pipe spec uniondiff --git a/pipelex/libraries/pipelines/builder/builder_loop.py b/pipelex/builder/builder_loop.pysimilarity index 97%rename from pipelex/libraries/pipelines/builder/builder_loop.pyrename to pipelex/builder/builder_loop.pyindex 0ccca249b..c7d113f29 100644--- a/pipelex/libraries/pipelines/builder/builder_loop.py+++ b/pipelex/builder/builder_loop.py@@ -1,20 +1,20 @@from pipelex import pretty_print-from pipelex.client.protocol import ImplicitMemory-from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories-from pipelex.exceptions import StaticValidationErrorType-from pipelex.hub import get_required_pipe-from pipelex.language.plx_factory import PlxFactory-from pipelex.libraries.pipelines.builder.builder import (+from pipelex.builder.builder import (PipelexBundleSpec,PipeSpecUnion,reconstruct_bundle_with_pipe_fixes,validate_bundle_spec,)-from pipelex.libraries.pipelines.builder.builder_errors import (+from pipelex.builder.builder_errors import (PipelexBundleError,PipelexBundleNoFixForError,PipelexBundleUnexpectedError,)+from pipelex.client.protocol import ImplicitMemory+from pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories+from pipelex.exceptions import StaticValidationErrorType+from pipelex.hub import get_required_pipe+from pipelex.language.plx_factory import PlxFactoryfrom pipelex.pipeline.execute import execute_pipelinefrom pipelex.tools.misc.file_utils import save_text_to_path@@ -59,7 +59,7 @@ class BuilderLoop:raise PipelexBundleUnexpectedError(message=msg) from bundle_errormatch static_error.error_type:case StaticValidationErrorType.MISSING_INPUT_VARIABLE | StaticValidationErrorType.EXTRANEOUS_INPUT_VARIABLE:- if not AllowedPipeCategories.is_controller_by_str(category_str=pipe_spec.category):+ if not AllowedPipeCategories.is_controller_by_str(category_str=pipe_spec.pipe_category):msg = (f"Static validation error: pipelex_bundle_spec had an input requirement error for a pipe spec of type "f"{pipe_spec.type} for considered pipe code: '{static_error.pipe_code}' but it was not a PipeController. "diff --git a/pipelex/libraries/pipelines/builder/concept/__init__.py b/pipelex/builder/concept/__init__.pysimilarity index 100%rename from pipelex/libraries/pipelines/builder/concept/__init__.pyrename to pipelex/builder/concept/__init__.pydiff --git a/pipelex/libraries/pipelines/builder/concept/concept.plx b/pipelex/builder/concept/concept.plxsimilarity index 100%rename from pipelex/libraries/pipelines/builder/concept/concept.plxrename to pipelex/builder/concept/concept.plxdiff --git a/pipelex/libraries/pipelines/builder/concept/concept_spec.py b/pipelex/builder/concept/concept_spec.pysimilarity index 99%rename from pipelex/libraries/pipelines/builder/concept/concept_spec.pyrename to pipelex/builder/concept/concept_spec.pyindex f418fa125..107a08b27 100644--- a/pipelex/libraries/pipelines/builder/concept/concept_spec.py+++ b/pipelex/builder/concept/concept_spec.py@@ -16,6 +16,7 @@ from pipelex.core.concepts.exceptions import ConceptCodeError, ConceptStringOrCofrom pipelex.core.domains.domain_blueprint import DomainBlueprintfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.system.registries.func_registry import pipe_funcfrom pipelex.tools.misc.string_utils import is_pascal_case, normalize_to_ascii, snake_to_pascal_casefrom pipelex.types import Self, StrEnum@@ -250,6 +251,7 @@ class ConceptSpec(StructuredContent):return ConceptBlueprint(description=self.description, structure=converted_structure, refines=self.refines)+@pipe_func()async def create_concept_spec(working_memory: WorkingMemory) -> ConceptSpec:concept_spec_draft = working_memory.get_stuff_as(name="concept_spec_draft", content_type=ConceptSpecDraft)concept_spec_structures_stuff = working_memory.get_stuff_as_list(name="concept_spec_structures", item_type=ConceptStructureSpec)diff --git a/pipelex/libraries/pipelines/builder/flow.py b/pipelex/builder/flow.pysimilarity index 95%rename from pipelex/libraries/pipelines/builder/flow.pyrename to pipelex/builder/flow.pyindex 7e866599b..ee5b67dfe 100644--- a/pipelex/libraries/pipelines/builder/flow.py+++ b/pipelex/builder/flow.py@@ -1,7 +1,7 @@from pydantic import ConfigDict, Field+from pipelex.builder.pipe.pipe_signature import PipeSignaturefrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSignaturefrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprintfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprintfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintdiff --git a/pipelex/libraries/pipelines/builder/flow_factory.py b/pipelex/builder/flow_factory.pysimilarity index 92%rename from pipelex/libraries/pipelines/builder/flow_factory.pyrename to pipelex/builder/flow_factory.pyindex f70701270..32752c54e 100644--- a/pipelex/libraries/pipelines/builder/flow_factory.py+++ b/pipelex/builder/flow_factory.py@@ -1,12 +1,12 @@from pathlib import Pathfrom typing import Any+from pipelex.builder.builder import PipelexBundleSpec+from pipelex.builder.flow import Flow, FlowElementUnion+from pipelex.builder.pipe.pipe_signature import PipeSignaturefrom pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.core.interpreter import PipelexInterpreterfrom pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories-from pipelex.libraries.pipelines.builder.builder import PipelexBundleSpec-from pipelex.libraries.pipelines.builder.flow import Flow, FlowElementUnion-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSignaturefrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprintfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprintfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint@@ -48,7 +48,7 @@ class FlowFactory:if bundle_blueprint.pipe:for pipe_code, pipe_blueprint in bundle_blueprint.pipe.items():- if pipe_blueprint.category == AllowedPipeCategories.PIPE_CONTROLLER:+ if pipe_blueprint.pipe_category == AllowedPipeCategories.PIPE_CONTROLLER:# Keep controllers as-is (they are already blueprints which match spec structure)# Type check to ensure we only assign controller blueprintsif isinstance(@@ -88,7 +88,7 @@ class FlowFactory:return PipeSignature(code=pipe_code,- category="PipeSignature",+ pipe_category="PipeSignature",type=pipe_blueprint.type,description=pipe_blueprint.description or "",inputs=inputs,diff --git a/pipelex/libraries/pipelines/builder/pipe/__init__.py b/pipelex/builder/pipe/__init__.pysimilarity index 100%rename from pipelex/libraries/pipelines/builder/pipe/__init__.pyrename to pipelex/builder/pipe/__init__.pydiff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_batch_spec.py b/pipelex/builder/pipe/pipe_batch_spec.pysimilarity index 92%rename from pipelex/libraries/pipelines/builder/pipe/pipe_batch_spec.pyrename to pipelex/builder/pipe/pipe_batch_spec.pyindex 56b8127ca..e1332b50c 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_batch_spec.py+++ b/pipelex/builder/pipe/pipe_batch_spec.py@@ -3,7 +3,7 @@ from typing import Literalfrom pydantic import Fieldfrom typing_extensions import override-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint@@ -26,7 +26,7 @@ class PipeBatchSpec(PipeSpec):"""type: Literal["PipeBatch"] = "PipeBatch"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"branch_pipe_code: str = Field(description="The pipe code to execute for each item in the input list. This pipe is instantiated once per item in parallel.")@@ -44,7 +44,7 @@ class PipeBatchSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,branch_pipe_code=self.branch_pipe_code,input_list_name=self.input_list_name,input_item_name=self.input_item_name,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_compose_spec.py b/pipelex/builder/pipe/pipe_compose_spec.pysimilarity index 94%rename from pipelex/libraries/pipelines/builder/pipe/pipe_compose_spec.pyrename to pipelex/builder/pipe/pipe_compose_spec.pyindex d578720bd..2650043b8 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_compose_spec.py+++ b/pipelex/builder/pipe/pipe_compose_spec.py@@ -4,10 +4,10 @@ from pydantic import Field, field_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprintfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormat-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprintfrom pipelex.types import StrEnum@@ -77,7 +77,7 @@ class PipeComposeSpec(PipeSpec):"""PipeComposeSpec defines a templating operation based on a Jinja2 template."""type: SkipJsonSchema[Literal["PipeCompose"]] = "PipeCompose"- category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"+ pipe_category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"template: str = Field(description="Jinja2 template string")target_format: TargetFormat | str = Field(description="Target format for the output")@@ -95,7 +95,7 @@ class PipeComposeSpec(PipeSpec):category = target_format.categorytemplate_blueprint = TemplateBlueprint(- source=self.template,+ template=self.template,templating_style=templating_style,category=category,extra_context=None,@@ -106,6 +106,6 @@ class PipeComposeSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,template=template_blueprint,)diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_condition_spec.py b/pipelex/builder/pipe/pipe_condition_spec.pysimilarity index 90%rename from pipelex/libraries/pipelines/builder/pipe/pipe_condition_spec.pyrename to pipelex/builder/pipe/pipe_condition_spec.pyindex 44d6dd7c9..23b763f13 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_condition_spec.py+++ b/pipelex/builder/pipe/pipe_condition_spec.py@@ -4,7 +4,7 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprintfrom pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome@@ -21,7 +21,7 @@ class PipeConditionSpec(PipeSpec):"""type: SkipJsonSchema[Literal["PipeCondition"]] = "PipeCondition"- category: SkipJsonSchema[Literal["PipeController"]] = "PipeController"+ pipe_category: SkipJsonSchema[Literal["PipeController"]] = "PipeController"jinja2_expression_template: str = Field(description="Jinja2 expression to evaluate.")outcomes: dict[str, str] = Field(..., description="Mapping `dict[str, str]` of condition to outcomes.")default_outcome: str | SpecialOutcome = Field(description="The fallback outcome if the expression result does not match any key in outcome map.")@@ -34,7 +34,7 @@ class PipeConditionSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,expression_template=self.jinja2_expression_template,expression=None,outcomes=self.outcomes,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_design.plx b/pipelex/builder/pipe/pipe_design.plxsimilarity index 96%rename from pipelex/libraries/pipelines/builder/pipe/pipe_design.plxrename to pipelex/builder/pipe/pipe_design.plxindex a0c0c601f..83754ff1a 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_design.plx+++ b/pipelex/builder/pipe/pipe_design.plx@@ -117,8 +117,8 @@ You will specifically generate the PipeLLM related to this signature:@pipe_signatureIf it's a structured generation, indicate it in the system_prompt to clarify the task.-If it's to generate free form text, the prompt_template should indicate to be concise.-If it's to generate an image generation, the prompt_template should indicate to be VERY concise and focus and apply the best practice for image generation.+If it's to generate free form text, the prompt should indicate to be concise.+If it's to generate an image generation prompt, the prompt should indicate to be VERY concise and focus and apply the best practice for image generation.Here are the concepts you can use for inputs/outputs:@concept_specsdiff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_extract_spec.py b/pipelex/builder/pipe/pipe_extract_spec.pysimilarity index 95%rename from pipelex/libraries/pipelines/builder/pipe/pipe_extract_spec.pyrename to pipelex/builder/pipe/pipe_extract_spec.pyindex 7dd4016f4..10e5edd39 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_extract_spec.py+++ b/pipelex/builder/pipe/pipe_extract_spec.py@@ -4,8 +4,8 @@ from pydantic import Field, field_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.exceptions import PipeDefinitionError-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprintfrom pipelex.types import StrEnum@@ -44,7 +44,7 @@ class PipeExtractSpec(PipeSpec):"""type: SkipJsonSchema[Literal["PipeExtract"]] = "PipeExtract"- category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"+ pipe_category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"extract_skill: ExtractSkill | str = Field(description="Select the most adequate extraction model skill according to the task to be performed.")page_images: bool | None = Field(default=None, description="Whether to include detected images in the Extract output.")page_image_captions: bool | None = Field(default=None, description="Whether to generate captions for detected images using AI.")@@ -83,7 +83,7 @@ class PipeExtractSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,model=extract_model_choice,page_images=self.page_images,page_image_captions=self.page_image_captions,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_func_spec.py b/pipelex/builder/pipe/pipe_func_spec.pysimilarity index 81%rename from pipelex/libraries/pipelines/builder/pipe/pipe_func_spec.pyrename to pipelex/builder/pipe/pipe_func_spec.pyindex 16e190ae0..20c8b8731 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_func_spec.py+++ b/pipelex/builder/pipe/pipe_func_spec.py@@ -4,7 +4,7 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint@@ -12,7 +12,7 @@ class PipeFuncSpec(PipeSpec):"""PipeFunc enables calling custom functions in the Pipelex framework."""type: SkipJsonSchema[Literal["PipeFunc"]] = "PipeFunc"- category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"+ pipe_category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"function_name: str = Field(description="The name of the function to call.")@override@@ -23,6 +23,6 @@ class PipeFuncSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,function_name=self.function_name,)diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_img_gen_spec.py b/pipelex/builder/pipe/pipe_img_gen_spec.pysimilarity index 93%rename from pipelex/libraries/pipelines/builder/pipe/pipe_img_gen_spec.pyrename to pipelex/builder/pipe/pipe_img_gen_spec.pyindex 365744eb5..55924a085 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_img_gen_spec.py+++ b/pipelex/builder/pipe/pipe_img_gen_spec.py@@ -3,7 +3,7 @@ from typing import Literalfrom pydantic import Field, field_validatorfrom typing_extensions import override-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprintfrom pipelex.types import StrEnum@@ -39,7 +39,7 @@ class PipeImgGenSpec(PipeSpec):"""type: Literal["PipeImgGen"] = "PipeImgGen"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"img_gen_skill: ImgGenSkill | None = Nonenb_output: int | None = Field(default=None, ge=1)@@ -60,7 +60,7 @@ class PipeImgGenSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,img_gen_prompt=None,img_gen_prompt_var_name=None,model=self.img_gen_skill.model_recommendation if self.img_gen_skill else None,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_llm_spec.py b/pipelex/builder/pipe/pipe_llm_spec.pysimilarity index 97%rename from pipelex/libraries/pipelines/builder/pipe/pipe_llm_spec.pyrename to pipelex/builder/pipe/pipe_llm_spec.pyindex c9aa5b442..600667baf 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_llm_spec.py+++ b/pipelex/builder/pipe/pipe_llm_spec.py@@ -4,9 +4,9 @@ from pydantic import Field, field_validator, model_validatorfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.exceptions import PipeDefinitionError-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpecfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintfrom pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_listfrom pipelex.types import Self, StrEnum@@ -82,7 +82,7 @@ class PipeLLMSpec(PipeSpec):"""type: SkipJsonSchema[Literal["PipeLLM"]] = "PipeLLM"- category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"+ pipe_category: SkipJsonSchema[Literal["PipeOperator"]] = "PipeOperator"llm: LLMSkill | str = Field(description="Select the most adequate LLM model skill according to the task to be performed.")temperature: float | None = Field(default=None, ge=0, le=1)system_prompt: str | None = Field(default=None, description="A system prompt to guide the LLM's behavior, style and skills. Can be a template.")@@ -143,7 +143,7 @@ class PipeLLMSpec(PipeSpec):return PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description=base_blueprint.description,inputs=base_blueprint.inputs,output=base_blueprint.output,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_parallel_spec.py b/pipelex/builder/pipe/pipe_parallel_spec.pysimilarity index 89%rename from pipelex/libraries/pipelines/builder/pipe/pipe_parallel_spec.pyrename to pipelex/builder/pipe/pipe_parallel_spec.pyindex c147fdec2..e468b6aad 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_parallel_spec.py+++ b/pipelex/builder/pipe/pipe_parallel_spec.py@@ -3,10 +3,10 @@ from typing import Literalfrom pydantic import Field, field_validator, model_validatorfrom typing_extensions import override+from pipelex.builder.concept.concept_spec import ConceptSpec+from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.exceptions import PipeDefinitionError-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintfrom pipelex.types import Self@@ -27,7 +27,7 @@ class PipeParallelSpec(PipeSpec):"""type: Literal["PipeParallel"] = "PipeParallel"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"parallels: list[SubPipeSpec] = Field(description="List of SubPipeSpec instances to execute concurrently.")add_each_output: bool = Field(description="Whether to include individual pipe outputs in the combined result.")combined_output: str | None = Field(default=None, description="Optional ConceptCode in PascalCasefor the combined output structure.")@@ -62,7 +62,7 @@ class PipeParallelSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,parallels=core_parallels,add_each_output=self.add_each_output,combined_output=self.combined_output,diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_sequence_spec.py b/pipelex/builder/pipe/pipe_sequence_spec.pysimilarity index 81%rename from pipelex/libraries/pipelines/builder/pipe/pipe_sequence_spec.pyrename to pipelex/builder/pipe/pipe_sequence_spec.pyindex 92856ad32..1ff721629 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_sequence_spec.py+++ b/pipelex/builder/pipe/pipe_sequence_spec.py@@ -4,8 +4,8 @@ from pydantic import Fieldfrom pydantic.json_schema import SkipJsonSchemafrom typing_extensions import override-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpec+from pipelex.builder.pipe.pipe_signature import PipeSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint@@ -16,7 +16,7 @@ class PipeSequenceSpec(PipeSpec):"""type: SkipJsonSchema[Literal["PipeSequence"]] = "PipeSequence"- category: SkipJsonSchema[Literal["PipeController"]] = "PipeController"+ pipe_category: SkipJsonSchema[Literal["PipeController"]] = "PipeController"steps: list[SubPipeSpec] = Field(description=("List of SubPipeSpec instances to execute sequentially. Each step runs after the previous one completes."))@@ -30,6 +30,6 @@ class PipeSequenceSpec(PipeSpec):inputs=base_blueprint.inputs,output=base_blueprint.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,steps=core_steps,)diff --git a/pipelex/libraries/pipelines/builder/pipe/pipe_signature.py b/pipelex/builder/pipe/pipe_signature.pysimilarity index 96%rename from pipelex/libraries/pipelines/builder/pipe/pipe_signature.pyrename to pipelex/builder/pipe/pipe_signature.pyindex 2ad9c9bbb..50c13d88c 100644--- a/pipelex/libraries/pipelines/builder/pipe/pipe_signature.py+++ b/pipelex/builder/pipe/pipe_signature.py@@ -3,11 +3,11 @@ from typing import Any, Literalfrom pydantic import Field, field_validatorfrom pipelex import log+from pipelex.builder.concept.concept_spec import ConceptSpecfrom pipelex.core.pipes.exceptions import PipeBlueprintErrorfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.pipe_blueprint import AllowedPipeCategories, AllowedPipeTypes, PipeBlueprintfrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpecfrom pipelex.tools.misc.string_utils import is_snake_case, normalize_to_ascii@@ -18,8 +18,8 @@ class PipeSignature(StructuredContent):"""code: str = Field(description="Pipe code identifying the pipe. Must be snake_case.")- category: Literal["PipeSignature"] = "PipeSignature"type: AllowedPipeTypes = Field(description="Pipe type.")+ pipe_category: Literal["PipeSignature"] = "PipeSignature"description: str = Field(description="What the pipe does")inputs: dict[str, str] = Field(description="Pipe inputs: keys are the input variable_names in snake_case, values are the ConceptCodes in PascalCase."@@ -39,7 +39,7 @@ class PipeSpec(StructuredContent):pipe_code: str = Field(description="Pipe code. Must be snake_case.")type: Any = Field(description=f"Pipe type. It is defined with type `Any` but validated at runtime and it must be one of: {AllowedPipeTypes}")- category: Any = Field(+ pipe_category: Any = Field(description=f"Pipe category. It is defined with type `Any` but validated at runtime and it must be one of: {AllowedPipeCategories}")description: str | None = Field(description="Natural language description of what the pipe does.")@@ -106,5 +106,5 @@ class PipeSpec(StructuredContent):inputs=converted_inputs,output=self.output,type=self.type,- category=self.category,+ pipe_category=self.pipe_category,)diff --git a/pipelex/libraries/pipelines/builder/pipe/sub_pipe_spec.py b/pipelex/builder/pipe/sub_pipe_spec.pysimilarity index 100%rename from pipelex/libraries/pipelines/builder/pipe/sub_pipe_spec.pyrename to pipelex/builder/pipe/sub_pipe_spec.pydiff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex 85b041cf0..6382ea9e4 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -5,6 +5,7 @@ from typing_extensions import overridefrom pipelex.cli.commands import init_app, show_app, validate_appfrom pipelex.cli.commands.build_cmd import build_app+from pipelex.cli.commands.kit_cmd import kit_appclass PipelexCLI(TyperGroup):@@ -35,3 +36,4 @@ app.add_typer(init_app, name="init", help="Initialization commands")app.add_typer(validate_app, name="validate", help="Validation and dry-run commands")app.add_typer(show_app, name="show", help="Show and list commands")app.add_typer(build_app, name="build", help="Build artifacts like pipeline blueprints")+app.add_typer(kit_app, name="kit", help="Manage kit assets")diff --git a/pipelex/cli/commands/__init__.py b/pipelex/cli/commands/__init__.pyindex fbec641e0..2a74b504f 100644--- a/pipelex/cli/commands/__init__.py+++ b/pipelex/cli/commands/__init__.py@@ -3,8 +3,8 @@This package organizes CLI commands into logical modules."""-from .init_cmd import init_app-from .show_cmd import show_app-from .validate_cmd import validate_app+from pipelex.cli.commands.init_cmd import init_app+from pipelex.cli.commands.show_cmd import show_app+from pipelex.cli.commands.validate_cmd import validate_app__all__ = ["init_app", "show_app", "validate_app"]diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.pyindex 403252540..5a398ddef 100644--- a/pipelex/cli/commands/build_cmd.py+++ b/pipelex/cli/commands/build_cmd.py@@ -5,17 +5,16 @@ from typing import Annotatedimport typerfrom pipelex import pretty_print+from pipelex.builder.builder import PipelexBundleSpec+from pipelex.builder.builder_loop import BuilderLoopfrom pipelex.hub import get_report_delegatefrom pipelex.language.plx_factory import PlxFactory-from pipelex.libraries.pipelines.builder.builder import PipelexBundleSpec-from pipelex.libraries.pipelines.builder.builder_loop import BuilderLoop-from pipelex.libraries.pipelines.builder.flow_factory import FlowFactoryfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipelinefrom pipelex.tools.misc.file_utils import ensure_directory_for_file_path, save_text_to_pathfrom pipelex.tools.misc.json_utils import save_as_json_to_path-build_app = typer.Typer(help="Build artifacts like pipelines", no_args_is_help=True)+build_app = typer.Typer(help="Build working pipelines from natural language requirements", no_args_is_help=True)"""Today's example:@@ -29,63 +28,11 @@ Other ideas:pipelex build pipe "Take a photo as input, and render the opposite of the photo, don't structure anything, use only text content, be super concise"pipelex build pipe "Take a photo as input, and render the opposite of the photo"pipelex build pipe "Given an RDFP PDF, build a compliance matrix"+pipelex build pipe "Given an theme, write a Haiku""""-@build_app.command("one-shot")-def build_one_shot_cmd(- brief: Annotated[- str,- typer.Argument(help="Brief description of what the pipeline should do"),- ],- output_path: Annotated[- str,- typer.Option("--output", "-o", help="Path to save the generated PLX file"),- ] = "./results/generated_pipeline.plx",- no_output: Annotated[- bool,- typer.Option("--no-output", help="Skip saving the pipeline to file"),- ] = False,-) -> None:- Pipelex.make(relative_config_folder_path="../../../pipelex/libraries", from_file=True)- typer.echo("=" * 70)- typer.echo(typer.style("🔥 Starting pipe builder... 🚀", fg=typer.colors.GREEN))- typer.echo("")-- async def run_pipeline():- if no_output:- typer.echo(typer.style("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW))- elif not output_path:- typer.echo(typer.style("\n🛑 Cannot save a pipeline to an empty file name", fg=typer.colors.RED))- raise typer.Exit(1)- else:- ensure_directory_for_file_path(file_path=output_path)-- pipe_output = await execute_pipeline(- pipe_code="pipe_builder",- input_memory={"brief": brief},- )- pretty_print(pipe_output, title="Pipe Output")-- # Save to file unless explicitly disabled with --no-output- if no_output:- typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))- return-- pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)- plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())- save_text_to_path(text=plx_content, path=output_path)- typer.echo(typer.style(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN))-- start_time = time.time()- asyncio.run(run_pipeline())- end_time = time.time()- typer.echo(typer.style(f"\n✅ Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN))-- get_report_delegate().generate_report()---@build_app.command("pipe")+@build_app.command("pipe", help="Generate a pipeline with one validation/fix loop corecting the deterministic issues")def build_pipe_cmd(brief: Annotated[str,@@ -100,7 +47,7 @@ def build_pipe_cmd(typer.Option("--no-output", help="Skip saving the pipeline to file"),] = False,) -> None:- Pipelex.make(relative_config_folder_path="../../../pipelex/libraries", from_file=True)+ Pipelex.make()typer.echo("=" * 70)typer.echo(typer.style("🔥 Starting pipe builder... 🚀", fg=typer.colors.GREEN))typer.echo("")@@ -133,8 +80,8 @@ def build_pipe_cmd(get_report_delegate().generate_report()-@build_app.command("partial")-def build_partial_cmd(+@build_app.command("one-shot", help="Generate a pipeline in one shot without validation loop (fast but may need manual fixes)")+def build_one_shot_cmd(brief: Annotated[str,typer.Argument(help="Brief description of what the pipeline should do"),@@ -148,7 +95,7 @@ def build_partial_cmd(typer.Option("--no-output", help="Skip saving the pipeline to file"),] = False,) -> None:- Pipelex.make(relative_config_folder_path="../../../pipelex/libraries", from_file=True)+ Pipelex.make()typer.echo("=" * 70)typer.echo(typer.style("🔥 Starting pipe builder... 🚀", fg=typer.colors.GREEN))typer.echo("")@@ -166,12 +113,16 @@ def build_partial_cmd(pipe_code="pipe_builder",input_memory={"brief": brief},)+ pretty_print(pipe_output, title="Pipe Output")+# Save to file unless explicitly disabled with --no-outputif no_output:typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))return- json_output = pipe_output.main_stuff.content.smart_dump()- save_as_json_to_path(object_to_save=json_output, path=output_path)++ pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)+ plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())+ save_text_to_path(text=plx_content, path=output_path)typer.echo(typer.style(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN))start_time = time.time()@@ -182,8 +133,8 @@ def build_partial_cmd(get_report_delegate().generate_report()-@build_app.command("flow")-def build_flow_cmd(+@build_app.command("partial", help="Generate a partial pipeline specification and save it as JSON (for debugging)")+def build_partial_cmd(brief: Annotated[str,typer.Argument(help="Brief description of what the pipeline should do"),@@ -197,7 +148,7 @@ def build_flow_cmd(typer.Option("--no-output", help="Skip saving the pipeline to file"),] = False,) -> None:- Pipelex.make(relative_config_folder_path="../../../pipelex/libraries", from_file=True)+ Pipelex.make()typer.echo("=" * 70)typer.echo(typer.style("🔥 Starting pipe builder... 🚀", fg=typer.colors.GREEN))typer.echo("")@@ -219,9 +170,7 @@ def build_flow_cmd(if no_output:typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))return- pipelex_bundle_spec = pipe_output.main_stuff_as(content_type=PipelexBundleSpec)- flow = FlowFactory.make_from_bundle_spec(bundle_spec=pipelex_bundle_spec)- json_output = flow.smart_dump()+ json_output = pipe_output.main_stuff.content.smart_dump()save_as_json_to_path(object_to_save=json_output, path=output_path)typer.echo(typer.style(f"\n✅ Pipeline saved to: {output_path}", fg=typer.colors.GREEN))diff --git a/pipelex/cli/commands/common.py b/pipelex/cli/commands/common.pydeleted file mode 100644index 8b567cf65..000000000--- a/pipelex/cli/commands/common.py+++ /dev/null@@ -1,21 +0,0 @@-from __future__ import annotations--import os-from typing import Final--REQUIRED_PIPELEX_SUBDIRS: Final[list[str]] = [- "pipelines",-]---def is_pipelex_libraries_folder(folder_path: str) -> bool:- """Check if the given folder path contains a valid pipelex libraries structure."""- if not os.path.exists(folder_path) or not os.path.isdir(folder_path):- return False-- for subdir in REQUIRED_PIPELEX_SUBDIRS:- subdir_path = os.path.join(folder_path, subdir)- if not os.path.exists(subdir_path) or not os.path.isdir(subdir_path):- return False-- return Truediff --git a/pipelex/cli/commands/init_cmd.py b/pipelex/cli/commands/init_cmd.pyindex 60861e604..75083607d 100644--- a/pipelex/cli/commands/init_cmd.py+++ b/pipelex/cli/commands/init_cmd.py@@ -6,33 +6,16 @@ from typing import Annotatedimport typerfrom pipelex.exceptions import PipelexCLIError-from pipelex.libraries.library_config import LibraryConfig-from pipelex.tools.config.manager import config_manager+from pipelex.kit.paths import get_configs_dir+from pipelex.system.configuration.config_loader import config_managerPACKAGE_NAME = __name__.split(".", maxsplit=1)[0]PACKAGE_VERSION = metadata(PACKAGE_NAME)["Version"]-def do_init_libraries(directory: str = ".", overwrite: bool = False) -> None:- try:- target_dir = os.path.join(directory, "pipelex_libraries")- os.makedirs(directory, exist_ok=True)-- library_config = LibraryConfig(config_dir_path=target_dir)- library_config.export_libraries(overwrite=overwrite)-- if overwrite:- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_dir}' (all files overwritten)")- else:- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_dir}' (only created non-existing files)")- except Exception as exc:- msg = f"Failed to initialize libraries at '{directory}': {exc}"- raise PipelexCLIError(msg) from exc--def do_init_config(reset: bool = False) -> None:"""Initialize pipelex configuration in the current directory."""- config_template_dir = os.path.join(config_manager.pipelex_root_dir, "config_template")+ config_template_dir = str(get_configs_dir())target_config_dir = config_manager.pipelex_config_diros.makedirs(target_config_dir, exist_ok=True)@@ -82,14 +65,6 @@ def do_init_config(reset: bool = False) -> None:init_app = typer.Typer(help="Initialization commands", no_args_is_help=True)-@init_app.command("libraries")-def init_libraries_cmd(- directory: Annotated[str, typer.Argument(help="Directory where to create the pipelex_libraries folder")] = ".",- overwrite: Annotated[bool, typer.Option("--overwrite", "-o", help="Warning: If set, existing files will be overwritten.")] = False,-) -> None:- do_init_libraries(directory=directory, overwrite=overwrite)--@init_app.command("config")def init_config_cmd(reset: Annotated[bool, typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten.")] = False,diff --git a/pipelex/cli/commands/kit_cmd.py b/pipelex/cli/commands/kit_cmd.pynew file mode 100644index 000000000..1f68b7c06--- /dev/null+++ b/pipelex/cli/commands/kit_cmd.py@@ -0,0 +1,132 @@+"""CLI commands for kit asset management."""++from pathlib import Path++import typer+from typing_extensions import Annotated++from pipelex.exceptions import PipelexCLIError+from pipelex.kit.cursor_export import export_cursor_rules, remove_cursor_rules+from pipelex.kit.index_loader import load_index+from pipelex.kit.migrations_export import export_migration_instructions+from pipelex.kit.targets_update import build_merged_rules, remove_from_targets, update_targets++kit_app = typer.Typer(help="Manage kit assets: export Cursor rules and merge agent docs", no_args_is_help=True)+++@kit_app.command("rules")+def agent_rules(+ repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,+ cursor: Annotated[bool, typer.Option("--cursor/--no-cursor", help="Export Cursor rules to .cursor/rules")] = True,+ single_files: Annotated[bool, typer.Option("--single-files/--no-single-files", help="Update single-file agent documentation targets")] = True,+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Show what would be done without making changes")] = False,+ diff: Annotated[bool, typer.Option("--diff", help="Show unified diff of changes")] = False,+ backup: Annotated[str | None, typer.Option("--backup", help="Backup suffix (e.g., '.bak')")] = None,+) -> None:+ """Sync kit assets: export Cursor rules and merge agent documentation.++ This command:+ 1. Exports agent markdown files to Cursor .mdc files with YAML front-matter+ 2. Builds merged agent documentation and updates target files using markers+ """+ try:+ if repo_root is None:+ repo_root = Path()++ idx = load_index()++ if cursor:+ typer.echo("📤 Exporting Cursor rules...")+ export_cursor_rules(repo_root, idx, dry_run=dry_run)++ if single_files:+ typer.echo("📝 Building merged agent documentation...")+ merged_md = build_merged_rules(idx)+ typer.echo("📝 Updating target files...")+ update_targets(repo_root, merged_md, idx.agent_rules.targets, dry_run=dry_run, diff=diff, backup=backup)++ if dry_run:+ typer.echo("✅ Dry run completed - no changes made")+ else:+ typer.echo("✅ Kit sync completed successfully")++ except Exception as exc:+ msg = f"Failed to sync kit assets for agent rules: {exc}"+ raise PipelexCLIError(msg) from exc+++@kit_app.command("remove-rules")+def remove_rules(+ repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,+ cursor: Annotated[bool, typer.Option("--cursor/--no-cursor", help="Remove Cursor rules from .cursor/rules")] = True,+ single_files: Annotated[bool, typer.Option("--single-files/--no-single-files", help="Remove agent documentation from target files")] = True,+ delete_files: Annotated[bool, typer.Option("--delete-files", help="Delete entire target files instead of just removing marked sections")] = False,+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Show what would be done without making changes")] = False,+ diff: Annotated[bool, typer.Option("--diff", help="Show unified diff of changes")] = False,+ backup: Annotated[str | None, typer.Option("--backup", help="Backup suffix (e.g., '.bak')")] = None,+) -> None:+ """Remove agent rules: delete Cursor rules and remove marked sections from target files.++ This command:+ 1. Deletes agent markdown files from Cursor .mdc files in .cursor/rules+ 2. Removes marked sections from target files (or deletes entire files with --delete-files)+ """+ try:+ if repo_root is None:+ repo_root = Path()++ idx = load_index()++ if cursor:+ typer.echo("🗑️ Removing Cursor rules...")+ remove_cursor_rules(repo_root, dry_run=dry_run)++ if single_files:+ if delete_files:+ typer.echo("🗑️ Deleting target files...")+ else:+ typer.echo("🗑️ Removing marked sections from target files...")+ remove_from_targets(+ repo_root,+ idx.agent_rules.targets,+ delete_files=delete_files,+ dry_run=dry_run,+ diff=diff,+ backup=backup,+ )++ if dry_run:+ typer.echo("✅ Dry run completed - no changes made")+ else:+ typer.echo("✅ Agent rules removal completed successfully")++ except Exception as exc:+ msg = f"Failed to remove agent rules: {exc}"+ raise PipelexCLIError(msg) from exc+++@kit_app.command("migrations")+def migration_instructions(+ repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Show what would be done without making changes")] = False,+) -> None:+ """Sync migration instructions from kit to .pipelex/migrations.++ This command copies migration documentation files from the pipelex.kit+ package to the user's .pipelex/migrations directory.+ """+ try:+ if repo_root is None:+ repo_root = Path()++ typer.echo("📄 Syncing migration instructions...")+ export_migration_instructions(repo_root, dry_run=dry_run)++ if dry_run:+ typer.echo("✅ Dry run completed - no changes made")+ else:+ typer.echo(f"✅ Migration instructions synced to {repo_root / '.pipelex' / 'migrations'}")++ except Exception as exc:+ msg = f"Failed to sync migration instructions: {exc}"+ raise PipelexCLIError(msg) from excdiff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.pyindex f7068cb11..9584761e0 100644--- a/pipelex/cli/commands/show_cmd.py+++ b/pipelex/cli/commands/show_cmd.py@@ -10,7 +10,7 @@ from pipelex.cogt.model_backends.model_lists import ModelListerfrom pipelex.exceptions import PipelexCLIError, PipelexConfigErrorfrom pipelex.hub import get_pipe_library, get_required_pipefrom pipelex.pipelex import Pipelex-from pipelex.tools.config.manager import config_manager+from pipelex.system.configuration.config_loader import config_managerdef do_show_config() -> None:@@ -26,9 +26,9 @@ def do_show_config() -> None:raise PipelexConfigError(msg) from exc-def do_list_pipes(relative_config_folder_path: str = "pipelex_libraries") -> None:+def do_list_pipes() -> None:"""List all available pipes."""- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ Pipelex.make()try:get_pipe_library().pretty_list_pipes()@@ -37,9 +37,9 @@ def do_list_pipes(relative_config_folder_path: str = "pipelex_libraries") -> Nonraise PipelexCLIError(msg) from exc-def do_show_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:+def do_show_pipe(pipe_code: str) -> None:"""Show a single pipe definition from the library."""- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ Pipelex.make()pipe = get_required_pipe(pipe_code=pipe_code)pretty_print(pipe, title=f"Pipe '{pipe_code}'")@@ -54,33 +54,20 @@ def show_config_cmd() -> None:@show_app.command("pipes")-def list_pipes_cmd(- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "pipelex_libraries",-) -> None:- do_list_pipes(relative_config_folder_path=relative_config_folder_path)+def list_pipes_cmd() -> None:+ do_list_pipes()@show_app.command("pipe")def show_pipe_cmd(pipe_code: Annotated[str, typer.Argument(help="Pipeline code to show definition for")],- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",) -> None:- do_show_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)+ do_show_pipe(pipe_code=pipe_code)@show_app.command("models")def show_models_cmd(backend_name: Annotated[str, typer.Argument(help="Backend name to list models for")],- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",flat: Annotated[bool,typer.Option("--flat", "-f", help="Output in flat CSV format for easy copy-pasting"),@@ -89,7 +76,6 @@ def show_models_cmd(asyncio.run(ModelLister.list_models(backend_name=backend_name,- relative_config_folder_path=relative_config_folder_path,flat=flat,))diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.pyindex 01c05af4d..6deb49c50 100644--- a/pipelex/cli/commands/validate_cmd.py+++ b/pipelex/cli/commands/validate_cmd.py@@ -6,33 +6,22 @@ from typing import Annotatedimport typerfrom pipelex import log-from pipelex.cli.commands.common import is_pipelex_libraries_folderfrom pipelex.hub import get_pipeline_tracker, get_pipes, get_required_pipefrom pipelex.pipe_run.dry_run import dry_run_pipe, dry_run_pipesfrom pipelex.pipelex import Pipelex-def do_validate_all_libraries_and_dry_run(relative_config_folder_path: str = "./pipelex_libraries") -> None:+def do_validate_all_libraries_and_dry_run() -> None:"""Validate libraries and dry-run all pipes."""- if not is_pipelex_libraries_folder(relative_config_folder_path):- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")- raise typer.Exit(1)-- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ pipelex_instance = Pipelex.make()pipelex_instance.validate_libraries()asyncio.run(dry_run_pipes(pipes=get_pipes(), raise_on_failure=True))log.info("Setup sequence passed OK, config and pipelines are validated.")-def do_dry_run_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:+def do_dry_run_pipe(pipe_code: str) -> None:"""Dry run a single pipe."""- if not is_pipelex_libraries_folder(relative_config_folder_path):- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")- raise typer.Exit(1)-- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ pipelex_instance = Pipelex.make()pipelex_instance.validate_libraries()asyncio.run(@@ -49,21 +38,12 @@ validate_app = typer.Typer(help="Validation and dry-run commands", no_args_is_he@validate_app.command("all")-def validate_all_cmd(- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",-) -> None:- do_validate_all_libraries_and_dry_run(relative_config_folder_path=relative_config_folder_path)+def validate_all_cmd() -> None:+ do_validate_all_libraries_and_dry_run()@validate_app.command("pipe")def dry_run_pipe_cmd(pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",) -> None:- do_dry_run_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)+ do_dry_run_pipe(pipe_code=pipe_code)diff --git a/pipelex/client/api_serializer.py b/pipelex/client/api_serializer.pyindex 9c5b60b05..64fbc017b 100644--- a/pipelex/client/api_serializer.py+++ b/pipelex/client/api_serializer.py@@ -36,7 +36,7 @@ class ApiSerializer:return compact_memoryfor stuff_name, stuff in working_memory.root.items():- if NativeConceptCode.is_text(concept_code=stuff.concept.code):+ if NativeConceptCode.is_text_concept(concept_code=stuff.concept.code):stuff_content = cast("TextContent", stuff.content)item_dict: dict[str, Any] = {"concept_code": stuff.concept.code,diff --git a/pipelex/client/client.py b/pipelex/client/client.pyindex a199776b6..3ec34063c 100644--- a/pipelex/client/client.py+++ b/pipelex/client/client.py@@ -10,7 +10,7 @@ from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.exceptions import ClientAuthenticationErrorfrom pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity-from pipelex.tools.environment import get_required_env+from pipelex.system.environment import get_required_envclass PipelexClient(PipelexProtocol):diff --git a/pipelex/cogt/config_cogt.py b/pipelex/cogt/config_cogt.pyindex eb6a2e8c9..c0a4229d4 100644--- a/pipelex/cogt/config_cogt.py+++ b/pipelex/cogt/config_cogt.py@@ -2,7 +2,7 @@ from pipelex.cogt.exceptions import LLMConfigErrorfrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams, ImgGenJobParamsDefaultsfrom pipelex.cogt.llm.llm_job_components import LLMJobConfigfrom pipelex.plugins.fal.fal_config import FalConfig-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.tools.misc.file_utils import find_files_in_dir@@ -65,7 +65,7 @@ class InferenceConfig(ConfigModel):return f"{self.inference_config_path}/backends/{backend_name}.toml"def get_model_deck_paths(self) -> list[str]:- """Get all LLM deck TOML file paths sorted alphabetically."""+ """Get all Model deck TOML file paths sorted alphabetically."""model_deck_paths = [str(path)for path in find_files_in_dir(diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.pyindex 1d5b87c8d..91d5b66b4 100644--- a/pipelex/cogt/content_generation/content_generator.py+++ b/pipelex/cogt/content_generation/content_generator.py@@ -37,7 +37,7 @@ from pipelex.tools.typing.pydantic_utils import BaseModelTypeVarclass ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_llm_text( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_llm_text(self,job_metadata: JobMetadata,llm_setting_main: LLMSetting,@@ -57,7 +57,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_object_direct( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_object_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -80,7 +80,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_text_then_object( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_text_then_object(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -127,7 +127,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_object_list_direct( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_object_list_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -150,7 +150,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_text_then_object_list( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_text_then_object_list(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -196,7 +196,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_single_image( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_single_image(self,job_metadata: JobMetadata,img_gen_handle: str,@@ -219,7 +219,7 @@ class ContentGenerator(ContentGeneratorProtocol):@override@update_job_metadata- async def make_image_list( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_image_list(self,job_metadata: JobMetadata,img_gen_handle: str,@@ -249,7 +249,6 @@ class ContentGenerator(ContentGeneratorProtocol):templating_style: TemplatingStyle | None = None,template_category: TemplateCategory | None = None,) -> str:- log.debug(f"context: {context}")templating_assignment = TemplatingAssignment(context=context,template=template,diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex 578fd30b3..f1ed386b7 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -21,6 +21,8 @@ from pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsingfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar+DRY_BASE_64_IMAGE = "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8z8BQz0AEYBxVSF+FABJADveWkH6oAAAAAElFTkSuQmCC"+class ContentGeneratorDry(ContentGeneratorProtocol):"""This class is used to generate mock content for testing purposes.@@ -33,7 +35,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_llm_text( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_llm_text(self,job_metadata: JobMetadata,llm_setting_main: LLMSetting,@@ -46,7 +48,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_object_direct( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_object_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -70,7 +72,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_text_then_object( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_text_then_object(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -91,7 +93,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_object_list_direct( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_object_list_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -114,7 +116,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_text_then_object_list( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_text_then_object_list(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -136,7 +138,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_single_image( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_single_image(self,job_metadata: JobMetadata,img_gen_handle: str,@@ -156,7 +158,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):@override@update_job_metadata- async def make_image_list( # pyright: ignore[reportIncompatibleMethodOverride]+ async def make_image_list(self,job_metadata: JobMetadata,img_gen_handle: str,@@ -222,7 +224,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):extracted_images=[],page_view=ExtractedImageFromPage(image_id=f"page_view_{page_index}",- base_64="",+ base_64=DRY_BASE_64_IMAGE,caption="DRY RUN: OCR text",),)diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.pyindex 4de591978..fe19411ae 100644--- a/pipelex/cogt/content_generation/content_generator_protocol.py+++ b/pipelex/cogt/content_generation/content_generator_protocol.py@@ -45,22 +45,22 @@ def update_job_metadata(func: Callable[P, Coroutine[Any, Any, R]]) -> Callable[Pclass ContentGeneratorProtocol(Protocol):- async def make_llm_text(+ def make_llm_text(self,job_metadata: JobMetadata,llm_setting_main: LLMSetting,llm_prompt_for_text: LLMPrompt,- ) -> str: ...+ ) -> Coroutine[Any, Any, str]: ...- async def make_object_direct(+ def make_object_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],llm_setting_for_object: LLMSetting,llm_prompt_for_object: LLMPrompt,- ) -> BaseModelTypeVar: ...+ ) -> Coroutine[Any, Any, BaseModelTypeVar]: ...- async def make_text_then_object(+ def make_text_then_object(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -68,18 +68,18 @@ class ContentGeneratorProtocol(Protocol):llm_setting_for_object: LLMSetting,llm_prompt_for_text: LLMPrompt,llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,- ) -> BaseModelTypeVar: ...+ ) -> Coroutine[Any, Any, BaseModelTypeVar]: ...- async def make_object_list_direct(+ def make_object_list_direct(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],llm_setting_for_object_list: LLMSetting,llm_prompt_for_object_list: LLMPrompt,nb_items: int | None = None,- ) -> list[BaseModelTypeVar]: ...+ ) -> Coroutine[Any, Any, list[BaseModelTypeVar]]: ...- async def make_text_then_object_list(+ def make_text_then_object_list(self,job_metadata: JobMetadata,object_class: type[BaseModelTypeVar],@@ -88,18 +88,18 @@ class ContentGeneratorProtocol(Protocol):llm_prompt_for_text: LLMPrompt,llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,nb_items: int | None = None,- ) -> list[BaseModelTypeVar]: ...+ ) -> Coroutine[Any, Any, list[BaseModelTypeVar]]: ...- async def make_single_image(+ def make_single_image(self,job_metadata: JobMetadata,img_gen_handle: str,img_gen_prompt: ImgGenPrompt,img_gen_job_params: ImgGenJobParams | None = None,img_gen_job_config: ImgGenJobConfig | None = None,- ) -> GeneratedImage: ...+ ) -> Coroutine[Any, Any, GeneratedImage]: ...- async def make_image_list(+ def make_image_list(self,job_metadata: JobMetadata,img_gen_handle: str,@@ -107,21 +107,21 @@ class ContentGeneratorProtocol(Protocol):nb_images: int,img_gen_job_params: ImgGenJobParams | None = None,img_gen_job_config: ImgGenJobConfig | None = None,- ) -> list[GeneratedImage]: ...+ ) -> Coroutine[Any, Any, list[GeneratedImage]]: ...- async def make_templated_text(+ def make_templated_text(self,context: dict[str, Any],template: str,templating_style: TemplatingStyle | None = None,template_category: TemplateCategory | None = None,- ) -> str: ...+ ) -> Coroutine[Any, Any, str]: ...- async def make_extract_pages(+ def make_extract_pages(self,job_metadata: JobMetadata,extract_input: ExtractInput,extract_handle: str,extract_job_params: ExtractJobParams,extract_job_config: ExtractJobConfig,- ) -> ExtractOutput: ...+ ) -> Coroutine[Any, Any, ExtractOutput]: ...diff --git a/pipelex/cogt/content_generation/templating_generate.py b/pipelex/cogt/content_generation/templating_generate.pyindex 8feaaa46b..ddec36a4e 100644--- a/pipelex/cogt/content_generation/templating_generate.py+++ b/pipelex/cogt/content_generation/templating_generate.py@@ -1,19 +1,12 @@from pipelex.cogt.content_generation.assignment_models import TemplatingAssignment-from pipelex.cogt.templating.template_preprocessor import preprocess_template-from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsing-from pipelex.tools.jinja2.jinja2_rendering import render_jinja2+from pipelex.cogt.templating.template_rendering import render_templateasync def templating_gen_text(templating_assignment: TemplatingAssignment) -> str:- # Intermediate call to preprocess the template with our syntax patterns (@, $, @?, etc.)- if templating_assignment.template:- templating_assignment.template = preprocess_template(template=templating_assignment.template)- check_jinja2_parsing(templating_assignment.template)-- templated_text: str = await render_jinja2(- template_category=templating_assignment.category,- temlating_context=templating_assignment.context,- template_source=templating_assignment.template,+ templated_text: str = await render_template(+ template=templating_assignment.template,+ category=templating_assignment.category,+ context=templating_assignment.context,templating_style=templating_assignment.templating_style,)diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.pyindex 22c7c4dfb..5deb982ff 100644--- a/pipelex/cogt/exceptions.py+++ b/pipelex/cogt/exceptions.py@@ -1,4 +1,4 @@-from pipelex.tools.exceptions import FatalError, RootException+from pipelex.system.exceptions import FatalError, RootExceptionfrom pipelex.types import StrEnum@@ -10,6 +10,10 @@ class LLMConfigError(CogtError):pass+class ImageContentError(CogtError):+ pass++class InferenceManagerWorkerSetupError(CogtError, FatalError):pass@@ -58,6 +62,10 @@ class ModelDeckValidatonError(CogtError):pass+class ModelNotFoundError(CogtError):+ pass++class LLMHandleNotFoundError(CogtError):pass@@ -106,6 +114,10 @@ class PromptImageFactoryError(CogtError):pass+class PromptImageDefinitionError(CogtError):+ pass++class PromptImageFormatError(CogtError):pass@@ -143,7 +155,7 @@ class MissingPluginError(CogtError):pass-class OcrCapabilityError(CogtError):+class ExtractCapabilityError(CogtError):pass@@ -151,6 +163,10 @@ class RoutingProfileLibraryNotFoundError(CogtError):pass+class RoutingProfileValidationError(CogtError):+ pass++class RoutingProfileLibraryError(CogtError):pass@@ -163,6 +179,14 @@ class InferenceBackendError(CogtError):pass+class InferenceBackendLibraryNotFoundError(CogtError):+ pass+++class InferenceBackendLibraryValidationError(CogtError):+ pass++class InferenceBackendCredentialsErrorType(StrEnum):VAR_NOT_FOUND = "var_not_found"UNKNOWN_VAR_PREFIX = "unknown_var_prefix"@@ -191,9 +215,13 @@ class RoutingProfileError(CogtError):pass-class ModelsManagerError(CogtError):+class ModelManagerError(CogtError):passclass ModelDeckNotFoundError(CogtError):pass+++class ModelDeckValidationError(CogtError):+ passdiff --git a/pipelex/cogt/extract/extract_job_components.py b/pipelex/cogt/extract/extract_job_components.pyindex 5bc9bc558..9371ea85f 100644--- a/pipelex/cogt/extract/extract_job_components.py+++ b/pipelex/cogt/extract/extract_job_components.py@@ -1,6 +1,6 @@from pydantic import BaseModel-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass ExtractJobParams(BaseModel):diff --git a/pipelex/cogt/extract/extract_output.py b/pipelex/cogt/extract/extract_output.pyindex de6ce3fd9..11daeafa6 100644--- a/pipelex/cogt/extract/extract_output.py+++ b/pipelex/cogt/extract/extract_output.py@@ -1,7 +1,7 @@from pydantic import Fieldfrom pipelex import log-from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file+from pipelex.tools.misc.base_64_utils import save_base_64_str_to_binary_filefrom pipelex.tools.misc.file_utils import ensure_directory_exists, save_text_to_pathfrom pipelex.tools.typing.pydantic_utils import CustomBaseModel, empty_list_factory_of@@ -17,7 +17,7 @@ class ExtractedImage(CustomBaseModel):if base_64 := self.base_64:filename = self.image_idfile_path = f"{directory}/{filename}"- save_base64_to_binary_file(b64=base_64, file_path=file_path)+ save_base_64_str_to_binary_file(base_64_str=base_64, file_path=file_path)class ExtractedImageFromPage(ExtractedImage):diff --git a/pipelex/cogt/extract/extract_setting.py b/pipelex/cogt/extract/extract_setting.pyindex 87a02f721..a6a7d77ab 100644--- a/pipelex/cogt/extract/extract_setting.py+++ b/pipelex/cogt/extract/extract_setting.py@@ -2,7 +2,7 @@ from typing import Unionfrom pydantic import Field-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass ExtractSetting(ConfigModel):diff --git a/pipelex/cogt/image/prompt_image.py b/pipelex/cogt/image/prompt_image.pyindex 433712d3b..269479de0 100644--- a/pipelex/cogt/image/prompt_image.py+++ b/pipelex/cogt/image/prompt_image.py@@ -6,7 +6,12 @@ from pydantic import BaseModelfrom typing_extensions import overridefrom pipelex.tools.misc.attribute_utils import AttributePolisher-from pipelex.tools.misc.filetype_utils import FileType, detect_file_type_from_base64, detect_file_type_from_bytes, detect_file_type_from_path+from pipelex.tools.misc.filetype_utils import (+ FileType,+ detect_file_type_from_base64,+ detect_file_type_from_bytes,+ detect_file_type_from_path,+)from pipelex.tools.typing.pydantic_utils import CustomBaseModeldiff --git a/pipelex/cogt/image/prompt_image_factory.py b/pipelex/cogt/image/prompt_image_factory.pyindex b0b608042..1790087a1 100644--- a/pipelex/cogt/image/prompt_image_factory.py+++ b/pipelex/cogt/image/prompt_image_factory.py@@ -1,12 +1,18 @@from pipelex.cogt.exceptions import PromptImageFactoryError-from pipelex.cogt.image.prompt_image import PromptImage, PromptImageBase64, PromptImageBinary, PromptImagePath, PromptImageUrl+from pipelex.cogt.image.prompt_image import (+ PromptImage,+ PromptImageBase64,+ PromptImageBinary,+ PromptImagePath,+ PromptImageUrl,+)from pipelex.tools.misc.base_64_utils import (encode_to_base64_async,load_binary_as_base64_async,load_binary_async,+ strip_base_64_str_if_needed,)from pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx_async-from pipelex.tools.misc.path_utils import clarify_path_or_urlclass PromptImageFactory:@@ -16,28 +22,21 @@ class PromptImageFactory:file_path: str | None = None,url: str | None = None,base_64: bytes | None = None,+ base_64_str: str | None = None,) -> PromptImage:- if file_path:+ if base_64:+ return PromptImageBase64(base_64=base_64)+ elif base_64_str:+ stripped_base_64_str = strip_base_64_str_if_needed(base_64_str)+ return PromptImageBase64(base_64=stripped_base_64_str.encode())+ elif file_path:return PromptImagePath(file_path=file_path)elif url:return PromptImageUrl(url=url)- elif base_64:- return PromptImageBase64(base_64=base_64)else:msg = "PromptImageFactory requires one of file_path, url, or image_bytes"raise PromptImageFactoryError(msg)- @classmethod- def make_prompt_image_from_uri(- cls,- uri: str,- ) -> PromptImage:- file_path, url = clarify_path_or_url(path_or_uri=uri)- return PromptImageFactory.make_prompt_image(- file_path=file_path,- url=url,- )-@classmethodasync def make_promptimagebase64_from_url_async(cls,diff --git a/pipelex/cogt/img_gen/img_gen_job_components.py b/pipelex/cogt/img_gen/img_gen_job_components.pyindex 53ed07cc6..5615a55d8 100644--- a/pipelex/cogt/img_gen/img_gen_job_components.py+++ b/pipelex/cogt/img_gen/img_gen_job_components.py@@ -2,7 +2,7 @@ from typing import Literalfrom pydantic import BaseModel, Field-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.types import StrEnumdiff --git a/pipelex/cogt/img_gen/img_gen_prompt.py b/pipelex/cogt/img_gen/img_gen_prompt.pyindex ea7859f65..e9b1b7216 100644--- a/pipelex/cogt/img_gen/img_gen_prompt.py+++ b/pipelex/cogt/img_gen/img_gen_prompt.py@@ -3,8 +3,8 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.exceptions import ImgGenPromptError+from pipelex.system.runtime import ProblemReaction, runtime_managerfrom pipelex.tools.misc.json_utils import json_str-from pipelex.tools.runtime_manager import ProblemReaction, runtime_managerclass ImgGenPrompt(BaseModel):diff --git a/pipelex/cogt/img_gen/img_gen_setting.py b/pipelex/cogt/img_gen/img_gen_setting.pyindex 446a6fe30..e2708914a 100644--- a/pipelex/cogt/img_gen/img_gen_setting.py+++ b/pipelex/cogt/img_gen/img_gen_setting.py@@ -4,7 +4,7 @@ from pydantic import Field, model_validatorfrom pipelex.cogt.exceptions import ImgGenSettingsValidationErrorfrom pipelex.cogt.img_gen.img_gen_job_components import Quality-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.types import Selfdiff --git a/pipelex/cogt/llm/llm_prompt.py b/pipelex/cogt/llm/llm_prompt.pyindex fbcbe1d56..37bb3e3a7 100644--- a/pipelex/cogt/llm/llm_prompt.py+++ b/pipelex/cogt/llm/llm_prompt.py@@ -4,8 +4,8 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.exceptions import LLMPromptParameterErrorfrom pipelex.cogt.image.prompt_image import PromptImage+from pipelex.system.runtime import ProblemReaction, runtime_managerfrom pipelex.tools.misc.string_utils import is_none_or_has_text, is_not_none_and_has_text-from pipelex.tools.runtime_manager import ProblemReaction, runtime_managerclass LLMPrompt(BaseModel):diff --git a/pipelex/cogt/llm/llm_prompt_template.py b/pipelex/cogt/llm/llm_prompt_template.pyindex c46c6bdc2..450b3fdf1 100644--- a/pipelex/cogt/llm/llm_prompt_template.py+++ b/pipelex/cogt/llm/llm_prompt_template.py@@ -3,12 +3,15 @@ from typing import Anyfrom typing_extensions import overridefrom pipelex import log-from pipelex.cogt.exceptions import LLMPromptFactoryError, LLMPromptTemplateInputsError+from pipelex.cogt.exceptions import LLMPromptTemplateInputsErrorfrom pipelex.cogt.image.prompt_image import PromptImagefrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstractfrom pipelex.cogt.llm.llm_prompt_template_inputs import LLMPromptTemplateInputs+from pipelex.cogt.templating.template_category import TemplateCategory+from pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormatfrom pipelex.config import get_config+from pipelex.hub import get_content_generatorfrom pipelex.tools.misc.string_utils import is_none_or_has_text@@ -24,10 +27,6 @@ class LLMPromptTemplate(LLMPromptFactoryAbstract):proto_prompt: LLMPrompt = make_empty_prompt()base_template_inputs: LLMPromptTemplateInputs = LLMPromptTemplateInputs()- # fields kept for reference and debugging only (they have no effect)- source_system_template_name: str | None = None- source_user_template_name: str | None = None-@overrideasync def make_llm_prompt_from_args(self,@@ -48,7 +47,7 @@ class LLMPromptTemplate(LLMPromptFactoryAbstract):user_images = [arguments_dict.pop("user_image")]is_user_images_append: bool | None = arguments_dict.pop("is_user_images_append", None)- return self._make_llm_prompt(+ return await self._make_llm_prompt(system_text=system_text,user_text=user_text,user_images=user_images,@@ -56,7 +55,7 @@ class LLMPromptTemplate(LLMPromptFactoryAbstract):template_inputs=LLMPromptTemplateInputs(root=arguments_dict),)- def _make_llm_prompt(+ async def _make_llm_prompt(self,system_text: str | None = None,user_text: str | None = None,@@ -91,25 +90,25 @@ class LLMPromptTemplate(LLMPromptFactoryAbstract):# input variables can be applied to prompt texts used as templatesif llm_prompt.system_text:- try:- # TODO: use jinja2 templating here- llm_prompt.system_text = llm_prompt.system_text.format(**all_template_inputs.root)- except KeyError as exc:- error_msg = f"Could not apply inputs to system_text. KeyError = {exc}. system_text = '{llm_prompt.system_text}'"- if template_name := self.source_system_template_name:- error_msg = f"Error using template named '{template_name}': {error_msg}.\n\n Available inputs: {all_template_inputs.list_keys()}"- log.error(error_msg)- raise LLMPromptFactoryError(message=error_msg) from exc+ llm_prompt.system_text = await get_content_generator().make_templated_text(+ context=all_template_inputs.root,+ template=llm_prompt.system_text,+ templating_style=TemplatingStyle(+ tag_style=TagStyle.XML,+ text_format=TextFormat.MARKDOWN,+ ),+ template_category=TemplateCategory.LLM_PROMPT,+ )if llm_prompt.user_text:- try:- # TODO: use jinja2 templating here- llm_prompt.user_text = llm_prompt.user_text.format(**all_template_inputs.root)- except KeyError as exc:- error_msg = f"Could not apply inputs to user_text. KeyError = {exc}. user_text = '{llm_prompt.user_text}'"- if template_name := self.source_user_template_name:- error_msg = f"Error using template named '{template_name}': {error_msg}.\n\n Available inputs: {all_template_inputs.list_keys()}"- log.error(error_msg)- raise LLMPromptFactoryError(message=error_msg) from exc+ llm_prompt.user_text = await get_content_generator().make_templated_text(+ context=all_template_inputs.root,+ template=llm_prompt.user_text,+ templating_style=TemplatingStyle(+ tag_style=TagStyle.XML,+ text_format=TextFormat.MARKDOWN,+ ),+ template_category=TemplateCategory.LLM_PROMPT,+ )return llm_promptdiff --git a/pipelex/cogt/llm/llm_prompt_template_inputs.py b/pipelex/cogt/llm/llm_prompt_template_inputs.pyindex ff0b735fe..dc8f62f37 100644--- a/pipelex/cogt/llm/llm_prompt_template_inputs.py+++ b/pipelex/cogt/llm/llm_prompt_template_inputs.py@@ -7,9 +7,9 @@ from typing_extensions import TYPE_CHECKING, overridefrom pipelex import logfrom pipelex.cogt.exceptions import LLMPromptTemplateInputsError+from pipelex.system.runtime import ProblemReaction, runtime_managerfrom pipelex.tools.misc.json_utils import json_strfrom pipelex.tools.misc.string_utils import can_inject_text-from pipelex.tools.runtime_manager import ProblemReaction, runtime_managerif TYPE_CHECKING:from collections.abc import ItemsViewdiff --git a/pipelex/cogt/llm/llm_setting.py b/pipelex/cogt/llm/llm_setting.pyindex e8c304152..d40d0d364 100644--- a/pipelex/cogt/llm/llm_setting.py+++ b/pipelex/cogt/llm/llm_setting.py@@ -4,7 +4,7 @@ from pydantic import Field, field_validatorfrom pipelex.cogt.llm.llm_job_components import LLMJobParamsfrom pipelex.cogt.model_backends.prompting_target import PromptingTarget-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.types import Selfdiff --git a/pipelex/cogt/model_backends/backend.py b/pipelex/cogt/model_backends/backend.pyindex 94ec65cd5..f3db3983d 100644--- a/pipelex/cogt/model_backends/backend.py+++ b/pipelex/cogt/model_backends/backend.py@@ -3,7 +3,7 @@ from typing import Anyfrom pydantic import Fieldfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass InferenceBackend(ConfigModel):diff --git a/pipelex/cogt/model_backends/backend_factory.py b/pipelex/cogt/model_backends/backend_factory.pyindex 352fe5b75..ba8e1150f 100644--- a/pipelex/cogt/model_backends/backend_factory.py+++ b/pipelex/cogt/model_backends/backend_factory.py@@ -5,7 +5,7 @@ from pydantic import Fieldfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.plugins.openai.vertexai_factory import VertexAIFactory-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass InferenceBackendBlueprint(ConfigModel):diff --git a/pipelex/cogt/model_backends/backend_library.py b/pipelex/cogt/model_backends/backend_library.pyindex 085dd599d..f481fb515 100644--- a/pipelex/cogt/model_backends/backend_library.py+++ b/pipelex/cogt/model_backends/backend_library.py@@ -6,15 +6,17 @@ from pipelex.cogt.exceptions import (InferenceBackendCredentialsError,InferenceBackendCredentialsErrorType,InferenceBackendLibraryError,+ InferenceBackendLibraryNotFoundError,+ InferenceBackendLibraryValidationError,InferenceModelSpecError,)from pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.backend_factory import InferenceBackendBlueprint, InferenceBackendFactoryfrom pipelex.cogt.model_backends.model_spec_factory import InferenceModelSpecBlueprint, InferenceModelSpecFactoryfrom pipelex.config import get_config+from pipelex.system.runtime import runtime_managerfrom pipelex.tools.misc.dict_utils import apply_to_strings_recursivefrom pipelex.tools.misc.toml_utils import load_toml_from_path-from pipelex.tools.runtime_manager import runtime_managerfrom pipelex.tools.secrets.secrets_utils import UnknownVarPrefixError, VarFallbackPatternError, VarNotFoundError, substitute_varsfrom pipelex.types import Self@@ -38,9 +40,12 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):backends_library_path = get_config().cogt.inference_config.backends_library_pathtry:backends_dict = load_toml_from_path(path=backends_library_path)- except (FileNotFoundError, InferenceBackendLibraryError) as exc:- msg = f"Failed to load inference backend library from file '{backends_library_path}': {exc}"- raise InferenceBackendLibraryError(msg) from exc+ except FileNotFoundError as file_not_found_exc:+ msg = f"Could not find inference backend library at '{backends_library_path}': {file_not_found_exc}"+ raise InferenceBackendLibraryNotFoundError(msg) from file_not_found_exc+ except ValidationError as exc:+ msg = f"Invalid inference backend library configuration in '{backends_library_path}': {exc}"+ raise InferenceBackendLibraryValidationError(msg) from excfor backend_name, backend_dict in backends_dict.items():# We'll split the read settings into standard fields and extra configstandard_fields = InferenceBackendBlueprint.model_fields.keys()@@ -63,7 +68,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):) from var_fallback_pattern_excexcept VarNotFoundError as var_not_found_exc:msg = (- f"Variable substitution failed due to a variable not found error in file '{backends_library_path}':"+ f"Variable substitution failed due to a 'variable not found' error in file '{backends_library_path}':"f"\n{var_not_found_exc}\nRun mode: '{runtime_manager.run_mode}'")raise InferenceBackendCredentialsError(diff --git a/pipelex/cogt/model_backends/model_lists.py b/pipelex/cogt/model_backends/model_lists.pyindex b7f7aec82..777571c18 100644--- a/pipelex/cogt/model_backends/model_lists.py+++ b/pipelex/cogt/model_backends/model_lists.py@@ -31,17 +31,15 @@ class ModelLister:async def list_models(cls,backend_name: str,- relative_config_folder_path: str = "./pipelex_libraries",flat: bool = False,) -> None:"""List available models for a specific backend.Args:backend_name: Name of the backend to list models for- relative_config_folder_path: Path to pipelex libraries configflat: Whether to output in flat CSV format"""- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ Pipelex.make()try:backend = get_models_manager().get_required_inference_backend(backend_name)diff --git a/pipelex/cogt/model_backends/model_spec.py b/pipelex/cogt/model_backends/model_spec.pyindex 30568f829..dc653b387 100644--- a/pipelex/cogt/model_backends/model_spec.py+++ b/pipelex/cogt/model_backends/model_spec.py@@ -4,7 +4,7 @@ from pipelex.cogt.model_backends.model_constraints import ModelConstraintsfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.model_backends.prompting_target import PromptingTargetfrom pipelex.cogt.usage.cost_category import CostsByCategoryDict-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.tools.typing.pydantic_utils import empty_list_factory_ofdiff --git a/pipelex/cogt/model_backends/model_spec_factory.py b/pipelex/cogt/model_backends/model_spec_factory.pyindex c251282d8..27bfb3157 100644--- a/pipelex/cogt/model_backends/model_spec_factory.py+++ b/pipelex/cogt/model_backends/model_spec_factory.py@@ -5,7 +5,7 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.model_backends.prompting_target import PromptingTargetfrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.tools.typing.pydantic_utils import empty_list_factory_ofdiff --git a/pipelex/cogt/model_routing/routing_profile.py b/pipelex/cogt/model_routing/routing_profile.pyindex 109dfbf7a..7a14f01e4 100644--- a/pipelex/cogt/model_routing/routing_profile.py+++ b/pipelex/cogt/model_routing/routing_profile.py@@ -1,7 +1,7 @@from pydantic import Fieldfrom pipelex.cogt.model_routing.routing_models import BackendMatchForModel, BackendMatchingMethod-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.tools.misc.string_utils import matches_wildcard_patterndiff --git a/pipelex/cogt/model_routing/routing_profile_factory.py b/pipelex/cogt/model_routing/routing_profile_factory.pyindex 42f225678..a3c6e71e7 100644--- a/pipelex/cogt/model_routing/routing_profile_factory.py+++ b/pipelex/cogt/model_routing/routing_profile_factory.py@@ -1,7 +1,7 @@from pydantic import Fieldfrom pipelex.cogt.model_routing.routing_profile import RoutingProfile-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass RoutingProfileBlueprint(ConfigModel):diff --git a/pipelex/cogt/model_routing/routing_profile_library.py b/pipelex/cogt/model_routing/routing_profile_library.pyindex 1fb265580..c2b27d257 100644--- a/pipelex/cogt/model_routing/routing_profile_library.py+++ b/pipelex/cogt/model_routing/routing_profile_library.py@@ -1,7 +1,7 @@from pydantic import Field, RootModel, ValidationErrorfrom pipelex import log-from pipelex.cogt.exceptions import RoutingProfileLibraryError, RoutingProfileLibraryNotFoundError+from pipelex.cogt.exceptions import RoutingProfileLibraryError, RoutingProfileLibraryNotFoundError, RoutingProfileValidationErrorfrom pipelex.cogt.model_routing.routing_models import BackendMatchForModelfrom pipelex.cogt.model_routing.routing_profile import RoutingProfilefrom pipelex.cogt.model_routing.routing_profile_factory import (@@ -45,14 +45,14 @@ class RoutingProfileLibrary(RootModel[RoutingProfileLibraryRoot]):try:catalog_dict = load_toml_from_path(path=routing_profile_library_path)except FileNotFoundError as not_found_exc:- msg = f"Failed to load routing profile library from file '{routing_profile_library_path}': {not_found_exc}"+ msg = f"Could not find routing profile library at '{routing_profile_library_path}': {not_found_exc}"raise RoutingProfileLibraryNotFoundError(msg) from not_found_exctry:catalog_blueprint = RoutingProfileLibraryBlueprint.model_validate(catalog_dict)except ValidationError as exc:msg = f"Invalid routing profile library configuration in '{routing_profile_library_path}': {exc}"- raise RoutingProfileLibraryError(msg) from exc+ raise RoutingProfileValidationError(msg) from exc# Validate that the active config existsif catalog_blueprint.active not in catalog_blueprint.profiles:diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.pyindex 8c2932bbb..6a068d439 100644--- a/pipelex/cogt/models/model_deck.py+++ b/pipelex/cogt/models/model_deck.py@@ -10,14 +10,20 @@ from pipelex.cogt.exceptions import (LLMHandleNotFoundError,LLMSettingsValidationError,ModelDeckValidatonError,+ ModelNotFoundError,)from pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSettingfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting-from pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSetting, LLMSettingChoices, LLMSettingChoicesDefaults+from pipelex.cogt.llm.llm_setting import (+ LLMModelChoice,+ LLMSetting,+ LLMSettingChoices,+ LLMSettingChoicesDefaults,+)from pipelex.cogt.model_backends.model_constraints import ModelConstraintsfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.exceptions import ConfigValidationError+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.exceptions import ConfigValidationErrorfrom pipelex.types import SelfLLM_PRESET_DISABLED = "disabled"@@ -69,7 +75,11 @@ class ModelDeck(ConfigModel):img_gen_presets: dict[str, ImgGenSetting] = Field(default_factory=dict)img_gen_choice_default: ImgGenModelChoice- def check_llm_setting(self, llm_setting_or_preset_id: LLMModelChoice, is_disabled_allowed: bool = False):+ def check_llm_setting(+ self,+ llm_setting_or_preset_id: LLMModelChoice,+ is_disabled_allowed: bool = False,+ ):if isinstance(llm_setting_or_preset_id, LLMSetting):returnpreset_id: str = llm_setting_or_preset_id@@ -114,7 +124,7 @@ class ModelDeck(ConfigModel):raise ImgGenChoiceNotFoundError(msg)@classmethod- def final_validate(cls, deck: Self): # pyright: ignore[reportIncompatibleMethodOverride]+ def final_validate(cls, deck: Self):for llm_preset_id, llm_setting in deck.llm_presets.items():inference_model = deck.get_required_inference_model(model_handle=llm_setting.model)try:@@ -191,7 +201,7 @@ class ModelDeck(ConfigModel):for alias in alias_list:if inference_model := self.get_optional_inference_model(model_handle=alias):return inference_model- log.warning(f"Skipping model handle '{model_handle}' because it's not found in deck")+ log.warning(f"Skipping model handle '{model_handle}' because it's not found in deck, it could be an external plugin.")return Nonedef is_handle_defined(self, model_handle: str) -> bool:@@ -201,7 +211,7 @@ class ModelDeck(ConfigModel):inference_model = self.get_optional_inference_model(model_handle=model_handle)if inference_model is None:msg = f"Model handle '{model_handle}' not found in deck"- raise LLMHandleNotFoundError(msg)+ raise ModelNotFoundError(msg)if model_handle not in self.inference_models:log.dev(f"Model handle '{model_handle}' is an alias which resolves to '{inference_model.name}'")return inference_modeldiff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.pyindex e237868e6..a8f4cff27 100644--- a/pipelex/cogt/models/model_manager.py+++ b/pipelex/cogt/models/model_manager.py@@ -1,10 +1,10 @@-import osfrom typing import Any+from pydantic import ValidationErrorfrom typing_extensions import overridefrom pipelex import log-from pipelex.cogt.exceptions import ModelDeckNotFoundError, ModelsManagerError+from pipelex.cogt.exceptions import ModelDeckNotFoundError, ModelDeckValidationError, ModelManagerErrorfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.backend_library import InferenceBackendLibraryfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec@@ -47,23 +47,22 @@ class ModelManager(ModelManagerAbstract):deck_paths = get_config().cogt.inference_config.get_model_deck_paths()full_deck_dict: dict[str, Any] = {}if not deck_paths:- msg = "No LLM deck paths found. Please run `pipelex init-libraries` to create it."+ msg = "No Model deck paths found. Please run `pipelex init config` to create the set up the base deck."raise ModelDeckNotFoundError(msg)for deck_path in deck_paths:- if not os.path.exists(deck_path):- msg = f"LLM deck path `{deck_path}` not found. Please run `pipelex init-libraries` to create it."- raise ModelDeckNotFoundError(msg)try:deck_dict = load_toml_from_path(path=deck_path)- log.debug(f"Loaded LLM deck from {deck_path}")- deep_update(full_deck_dict, deck_dict)- except Exception as exc:- msg = f"Failed to load LLM deck file '{deck_path}': {exc}"- log.error(msg)- raise+ except FileNotFoundError as not_found_exc:+ msg = f"Could not find Model Deck file at '{deck_path}': {not_found_exc}"+ raise ModelDeckNotFoundError(msg) from not_found_exc+ deep_update(full_deck_dict, deck_dict)- return ModelDeckBlueprint.model_validate(full_deck_dict)+ try:+ return ModelDeckBlueprint.model_validate(full_deck_dict)+ except ValidationError as exc:+ msg = f"Invalid Model Deck configuration in {deck_paths}: {exc}"+ raise ModelDeckValidationError(msg) from excdef build_deck(self, model_deck_blueprint: ModelDeckBlueprint) -> ModelDeck:all_models_and_possible_backends = self.inference_backend_library.get_all_models_and_possible_backends()@@ -80,7 +79,7 @@ class ModelManager(ModelManagerAbstract):backend = self.inference_backend_library.get_inference_backend(backend_name=matched_backend_name)if backend is None:msg = f"Backend '{matched_backend_name}', requested for model '{model_name}', could not be found"- raise ModelsManagerError(msg)+ raise ModelManagerError(msg)model_spec = backend.get_model_spec(model_name)if model_spec is None:# Not finding the model spec can be an error or not according to the matching method@@ -90,7 +89,7 @@ class ModelManager(ModelManagerAbstract):f"Model spec '{model_name}' not found in backend '{matched_backend_name}' "f"which was matched exactly in routing profile '{backend_match_for_model.routing_profile_name}'")- raise ModelsManagerError(msg)+ raise ModelManagerError(msg)case BackendMatchingMethod.PATTERN_MATCH:log.verbose(f"Model spec '{model_name}' not found in backend '{matched_backend_name}' but it's OK because "@@ -109,7 +108,7 @@ class ModelManager(ModelManagerAbstract):backend = self.inference_backend_library.get_inference_backend(backend_name=available_backend)if backend is None:msg = f"Backend '{available_backend}' not found for model '{model_name}'"- raise ModelsManagerError(msg)+ raise ModelManagerError(msg)model_spec = backend.get_model_spec(model_name)if model_spec is not None:break@@ -118,7 +117,7 @@ class ModelManager(ModelManagerAbstract):f"Model spec '{model_name}' not found in any of the available backends '{available_backends}' "f"which was set as default in routing profile '{backend_match_for_model.routing_profile_name}'")- raise ModelsManagerError(msg)+ raise ModelManagerError(msg)inference_models[model_name] = model_specreturn ModelDeck(@@ -145,5 +144,5 @@ class ModelManager(ModelManagerAbstract):backend = self.inference_backend_library.get_inference_backend(backend_name)if backend is None:msg = f"Inference backend '{backend_name}' not found"- raise ModelsManagerError(msg)+ raise ModelManagerError(msg)return backenddiff --git a/pipelex/cogt/templating/template_blueprint.py b/pipelex/cogt/templating/template_blueprint.pyindex 6c9c682aa..4afc27280 100644--- a/pipelex/cogt/templating/template_blueprint.py+++ b/pipelex/cogt/templating/template_blueprint.py@@ -8,7 +8,7 @@ from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsingclass TemplateBlueprint(BaseModel):- source: str = Field(description="Raw template source")+ template: str = Field(description="Raw template source")templating_style: TemplatingStyle | None = Field(default=None, description="Style of prompting to use (typically for different LLMs)")category: TemplateCategory = Field(description="Category of the template (could also be HTML, MARKDOWN, MERMAID, etc.), influences template rendering rules",@@ -17,5 +17,5 @@ class TemplateBlueprint(BaseModel):@model_validator(mode="after")def validate_template(self) -> "TemplateBlueprint":- check_jinja2_parsing(template_source=self.source, template_category=self.category)+ check_jinja2_parsing(template_source=self.template, template_category=self.category)return selfdiff --git a/pipelex/cogt/templating/template_rendering.py b/pipelex/cogt/templating/template_rendering.pynew file mode 100644index 000000000..0fc02f47e--- /dev/null+++ b/pipelex/cogt/templating/template_rendering.py@@ -0,0 +1,21 @@+from typing import Any++from pipelex.cogt.templating.template_category import TemplateCategory+from pipelex.cogt.templating.template_preprocessor import preprocess_template+from pipelex.cogt.templating.templating_style import TemplatingStyle+from pipelex.tools.jinja2.jinja2_rendering import render_jinja2+++async def render_template(+ template: str,+ category: TemplateCategory,+ context: dict[str, Any],+ templating_style: TemplatingStyle | None = None,+) -> str:+ template_source = preprocess_template(template)+ return await render_jinja2(+ template_source=template_source,+ template_category=category,+ temlating_context=context,+ templating_style=templating_style,+ )diff --git a/pipelex/config.py b/pipelex/config.pyindex fb6a63757..2f328d22b 100644--- a/pipelex/config.py+++ b/pipelex/config.py@@ -9,11 +9,10 @@ from pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.exceptions import PipelexConfigError, StaticValidationErrorTypefrom pipelex.hub import get_required_configfrom pipelex.language.plx_config import PlxConfig-from pipelex.libraries.library_config import LibraryConfigfrom pipelex.pipeline.track.tracker_config import TrackerConfig+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.configuration.config_root import ConfigRootfrom pipelex.tools.aws.aws_config import AwsConfig-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.config.config_root import ConfigRootfrom pipelex.tools.log.log_config import LogConfigfrom pipelex.types import StrEnum@@ -62,11 +61,6 @@ class DryRunConfig(ConfigModel):return value-class GenericTemplateNames(ConfigModel):- structure_from_preliminary_text_user: str- structure_from_preliminary_text_system: str--class StructureConfig(ConfigModel):is_default_text_then_structure: bool@@ -100,14 +94,23 @@ class ObserverConfig(ConfigModel):observer_dir: str+class ScanConfig(ConfigModel):+ excluded_dirs: frozenset[str]++ @field_validator("excluded_dirs", mode="before")+ @classmethod+ def validate_excluded_dirs(cls, value: list[str] | frozenset[str]) -> frozenset[str]:+ if isinstance(value, frozenset):+ return value+ return frozenset(value)++class Pipelex(ConfigModel):feature_config: FeatureConfiglog_config: LogConfigaws_config: AwsConfig- library_config: LibraryConfigstatic_validation_config: StaticValidationConfig- generic_template_names: GenericTemplateNamestracker_config: TrackerConfigstructure_config: StructureConfigprompting_config: PromptingConfig@@ -117,12 +120,30 @@ class Pipelex(ConfigModel):pipe_run_config: PipeRunConfigreporting_config: ReportingConfigobserver_config: ObserverConfig+ scan_config: ScanConfig+++class MigrationConfig(ConfigModel):+ migration_maps: dict[str, dict[str, str]]++ def text_in_renaming_keys(self, category: str, text: str) -> list[tuple[str, str]]:+ renaming_map = self.migration_maps.get(category)+ if not renaming_map:+ return []+ return [(key, value) for key, value in renaming_map.items() if text in key]++ def text_in_renaming_values(self, category: str, text: str) -> list[tuple[str, str]]:+ renaming_map = self.migration_maps.get(category)+ if not renaming_map:+ return []+ return [(key, value) for key, value in renaming_map.items() if text in value]class PipelexConfig(ConfigRoot):session_id: str = shortuuid.uuid()cogt: Cogtpipelex: Pipelex+ migration: MigrationConfigdef get_config() -> PipelexConfig:diff --git a/pipelex/config_template/pipelex.toml b/pipelex/config_template/pipelex.tomldeleted file mode 100644index dddcbf095..000000000--- a/pipelex/config_template/pipelex.toml+++ /dev/null@@ -1,25 +0,0 @@-[pipelex]-[pipelex.observer_config]-observer_dir = "results/observer"--[pipelex.aws_config]-api_key_method = "env"-# The possible values are "env" and "secret_provider".-# "env" means means that the env var are stored in your .env file.-# "secret_provider" means that the env var are stored in your Secret Manager (See the doc for injecting a secret provider).--[cogt]--####################################################################################################-# OCR config-####################################################################################################--[cogt.extract_config]-page_output_text_file_name = "page_text.md"--[pipelex.feature_config]-# WIP/Experimental feature flags-is_pipeline_tracking_enabled = false-is_activity_tracking_enabled = false-is_reporting_enabled = true-diff --git a/pipelex/core/concepts/concept.py b/pipelex/core/concepts/concept.pyindex 1720d8c1f..3b981555b 100644--- a/pipelex/core/concepts/concept.py+++ b/pipelex/core/concepts/concept.py@@ -9,8 +9,8 @@ from pipelex.core.domains.domain_blueprint import DomainBlueprintfrom pipelex.core.stuffs.image_field_search import search_for_nested_image_fieldsfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.exceptions import PipelexUnexpectedError-from pipelex.tools.class_registry_utils import ClassRegistryUtilsfrom pipelex.tools.misc.string_utils import pascal_case_to_sentence+from pipelex.tools.typing.class_utils import are_classes_equivalent, has_compatible_fieldclass Concept(BaseModel):@@ -74,7 +74,7 @@ class Concept(BaseModel):if strict:# Check if classes are equivalent (same fields, types, descriptions)- return ClassRegistryUtils.are_classes_equivalent(concept_1_class, concept_2_class)+ return are_classes_equivalent(concept_1_class, concept_2_class)# Check if concept_1 is a subclass of concept_2try:if issubclass(concept_1_class, concept_2_class):@@ -83,7 +83,7 @@ class Concept(BaseModel):pass# Check if concept_1 has compatible fields with concept_2- return ClassRegistryUtils.has_compatible_field(concept_1_class, concept_2_class)+ return has_compatible_field(concept_1_class, concept_2_class)return False@classmethoddiff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.pyindex 94639d2b9..c9817c97e 100644--- a/pipelex/core/concepts/concept_factory.py+++ b/pipelex/core/concepts/concept_factory.py@@ -119,7 +119,14 @@ class ConceptFactory:code=native_concept_code,domain=SpecialDomain.NATIVE,description="A prompt for an LLM",- structure_class_name=structure_class_name,+ structure_class_name=NativeConceptCode.TEXT.structure_class_name,+ )+ case NativeConceptCode.IMG_GEN_PROMPT:+ return Concept(+ code=native_concept_code,+ domain=SpecialDomain.NATIVE,+ description="A prompt for an image generator",+ structure_class_name=NativeConceptCode.TEXT.structure_class_name,)case NativeConceptCode.PAGE:return Concept(diff --git a/pipelex/core/concepts/concept_native.py b/pipelex/core/concepts/concept_native.pyindex 559976609..df4c44b7d 100644--- a/pipelex/core/concepts/concept_native.py+++ b/pipelex/core/concepts/concept_native.py@@ -14,6 +14,7 @@ class NativeConceptCode(StrEnum):TEXT_AND_IMAGES = "TextAndImages"NUMBER = "Number"LLM_PROMPT = "LlmPrompt"+ IMG_GEN_PROMPT = "ImgGenPrompt"PAGE = "Page"ANYTHING = "Anything"@@ -26,7 +27,7 @@ class NativeConceptCode(StrEnum):return f"{self.value}Content"@classmethod- def is_text(cls, concept_code: str) -> bool:+ def is_text_concept(cls, concept_code: str) -> bool:try:enum_value = NativeConceptCode(concept_code)except ValueError:@@ -42,6 +43,7 @@ class NativeConceptCode(StrEnum):| NativeConceptCode.TEXT_AND_IMAGES| NativeConceptCode.NUMBER| NativeConceptCode.LLM_PROMPT+ | NativeConceptCode.IMG_GEN_PROMPT| NativeConceptCode.PAGE| NativeConceptCode.ANYTHING):diff --git a/pipelex/core/domains/domain_factory.py b/pipelex/core/domains/domain_factory.pyindex 0875040a1..fa8f69353 100644--- a/pipelex/core/domains/domain_factory.py+++ b/pipelex/core/domains/domain_factory.py@@ -18,6 +18,6 @@ class DomainFactory:prompt_template_to_structure=blueprint.prompt_template_to_structure,)except ValidationError as exc:- formatted_error_msg = format_pydantic_validation_error(exc)- msg = f"Could not make domain from blueprint: {formatted_error_msg}"+ validation_error_msg = format_pydantic_validation_error(exc)+ msg = f"Could not make domain from blueprint: {validation_error_msg}"raise DomainDefinitionError(message=msg, domain_code=blueprint.code, description=blueprint.description) from excdiff --git a/pipelex/core/interpreter.py b/pipelex/core/interpreter.pyindex 65cd9321c..d85693850 100644--- a/pipelex/core/interpreter.py+++ b/pipelex/core/interpreter.py@@ -1,18 +1,16 @@from pathlib import Path-from typing import Any-import tomlfrom pydantic import BaseModel, model_validatorfrom pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.core.exceptions import (PipelexConfigurationError,)-from pipelex.tools.misc.file_utils import load_text_from_path+from pipelex.tools.misc.toml_utils import TomlError, load_toml_from_content, load_toml_from_pathfrom pipelex.types import Self-class PLXDecodeError(toml.TomlDecodeError):+class PLXDecodeError(TomlError):"""Raised when PLX decoding fails."""@@ -22,20 +20,6 @@ class PipelexInterpreter(BaseModel):file_path: Path | None = Nonefile_content: str | None = None- @staticmethod- def escape_plx_string(value: str | None) -> str:- """Escape a string for plx serialization."""- if value is None:- return ""- # Escape backslashes first (must be done first)- value = value.replace("\\", "\\\\")- # Escape quotes- value = value.replace('"', '\\"')- # Replace actual newlines with escaped newlines- value = value.replace("\n", "\\n")- value = value.replace("\r", "\\r")- return value.replace("\t", "\\t")-@model_validator(mode="after")def check_file_path_or_file_content(self) -> Self:"""Need to check if there is at least one of file_path or file_content"""@@ -44,16 +28,6 @@ class PipelexInterpreter(BaseModel):raise PipelexConfigurationError(msg)return self- def _get_content(self) -> str:- """Load PLX content from file_path or use file_content directly."""- if self.file_path:- return load_text_from_path(path=str(self.file_path))- elif self.file_content:- return self.file_content- else:- msg = "file_content must be provided if file_path is not provided"- raise PipelexConfigurationError(msg)-@staticmethoddef is_pipelex_file(file_path: Path) -> bool:"""Check if a file is a valid Pipelex PLX file.@@ -89,19 +63,18 @@ class PipelexInterpreter(BaseModel):# If we can't read the file, it's not a valid Pipelex filereturn False- def _parse_plx_content(self, content: str) -> dict[str, Any]:- """Parse PLX content and return the dictionary."""- try:- return toml.loads(content)- except toml.TomlDecodeError as exc:- file_path_str = str(self.file_path) if self.file_path else "content"- msg = f"PLX parsing error in '{file_path_str}': {exc}"- raise PLXDecodeError(msg, exc.doc, exc.pos) from exc-def make_pipelex_bundle_blueprint(self) -> PipelexBundleBlueprint:"""Make a PipelexBundleBlueprint from the file_path or file_content"""- file_content = self._get_content()- blueprint_dict = self._parse_plx_content(file_content)- if self.file_path:- blueprint_dict.update(source=str(self.file_path))+ # Load PLX content from file_path or use file_content directly.+ try:+ if self.file_path:+ blueprint_dict = load_toml_from_path(path=str(self.file_path))+ blueprint_dict.update(source=str(self.file_path))+ elif self.file_content:+ blueprint_dict = load_toml_from_content(content=self.file_content)+ else:+ msg = "Could not make PipelexBundleBlueprint: either file_path or file_content must be provided"+ raise PipelexConfigurationError(msg)+ except TomlError as exc:+ raise PLXDecodeError(message=exc.message, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from excreturn PipelexBundleBlueprint.model_validate(blueprint_dict)diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.pyindex 2cf9faa8e..75cd20819 100644--- a/pipelex/core/memory/working_memory.py+++ b/pipelex/core/memory/working_memory.py@@ -28,6 +28,7 @@ from pipelex.types import SelfMAIN_STUFF_NAME = "main_stuff"BATCH_ITEM_STUFF_NAME = "BATCH_ITEM"PRETTY_PRINT_MAX_LENGTH = 1000+TEST_DUMMY_NAME = "dummy_result"StuffDict = dict[str, Stuff]StuffArtefactDict = dict[str, StuffArtefact]@@ -125,12 +126,13 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):raise WorkingMemoryConsistencyError(msg)if name in self.root or name in self.aliases:existing_stuff = self.get_stuff(name=name)- if existing_stuff == stuff:+ if existing_stuff == stuff and name != TEST_DUMMY_NAME:log.warning(f"Key '{name}' already exists in WorkingMemory with the same stuff")return- log.warning(f"Key '{name}' already exists in WorkingMemory and will be replaced by something different")- log.verbose(f"Existing stuff: {existing_stuff}")- log.verbose(f"New stuff: {stuff}")+ elif name != TEST_DUMMY_NAME:+ log.warning(f"Key '{name}' already exists in WorkingMemory and will be replaced by something different")+ log.verbose(f"Existing stuff: {existing_stuff}")+ log.verbose(f"New stuff: {stuff}")# it's a new stuffself.set_stuff(name=name, stuff=stuff)@@ -157,7 +159,6 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):if target not in self.root:msg = f"Cannot create alias to non-existent target '{target}'"raise WorkingMemoryConsistencyError(msg)- log.debug(f"Setting alias '{alias}' pointing to target '{target}'")self.aliases[alias] = targetdef add_alias(self, alias: str, target: str) -> None:@@ -166,7 +167,6 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):msg = f"Cannot add alias '{alias}' as it already exists"raise WorkingMemoryConsistencyError(msg)self.set_alias(alias=alias, target=target)- log.debug(f"Added alias '{alias}' pointing to target '{target}'")def remove_alias(self, alias: str) -> None:"""Remove an alias if it exists."""@@ -193,19 +193,16 @@ class WorkingMemory(BaseModel, ContextProviderAbstract):################################################################################################@override- def generate_jinja2_context(self) -> dict[str, Any]:- # TODO: Add unit tests for this method+ def generate_context(self) -> dict[str, Any]:artefact_dict: StuffArtefactDict = {}for name, stuff in self.root.items():- a = stuff.make_artefact()- artefact_dict[name] = a+ artefact_dict[name] = stuff.make_artefact()for alias, target in self.aliases.items():artefact_dict[alias] = artefact_dict[target]return artefact_dict@overridedef get_typed_object_or_attribute(self, name: str, wanted_type: type[Any] | None = None) -> Any:- # TODO: Add unit tests for this method# TODO: Refactor this method. In the python paradigm, we should not have those ".", but arrays with field names.if "." in name:parts = name.split(".", 1) # Split only at the first dotdiff --git a/pipelex/core/memory/working_memory_factory.py b/pipelex/core/memory/working_memory_factory.pyindex f43f39518..bafe3ceb4 100644--- a/pipelex/core/memory/working_memory_factory.py+++ b/pipelex/core/memory/working_memory_factory.py@@ -193,11 +193,6 @@ class WorkingMemoryFactory(BaseModel):working_memory = cls.make_empty()for requirement in needed_inputs:- log.debug(- f"Creating dry run mock for '{requirement.variable_name}' with concept "- f"'{requirement.concept.code}' and class '{requirement.structure_class.__name__}'",- )-try:if not requirement.multiplicity:mock_content = cls.create_mock_content(requirement)diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.pyindex f062d4499..f93a60642 100644--- a/pipelex/core/pipes/pipe_abstract.py+++ b/pipelex/core/pipes/pipe_abstract.py@@ -16,7 +16,7 @@ from pipelex.pipeline.job_metadata import JobMetadataclass PipeAbstract(ABC, BaseModel):model_config = ConfigDict(strict=True, extra="forbid")- category: Any # Any so that subclasses can put a Literal+ pipe_category: Any # Any so that subclasses can put a Literaltype: Any # Any so that subclasses can put a Literalcode: strdomain: strdiff --git a/pipelex/core/pipes/pipe_blueprint.py b/pipelex/core/pipes/pipe_blueprint.pyindex a825fad06..2e803dafb 100644--- a/pipelex/core/pipes/pipe_blueprint.py+++ b/pipelex/core/pipes/pipe_blueprint.py@@ -1,6 +1,6 @@from typing import Any-from pydantic import BaseModel, ConfigDict, field_validator+from pydantic import BaseModel, ConfigDict, Field, field_validatorfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprintfrom pipelex.core.pipes.exceptions import PipeBlueprintError@@ -55,7 +55,7 @@ class AllowedPipeTypes(StrEnum):class PipeBlueprint(BaseModel):model_config = ConfigDict(extra="forbid")source: str | None = None- category: Any+ pipe_category: Any = Field(exclude=True) # Technical field for Union discrimination, not user-facingtype: Any # TODO: Find a better way to handle this.description: str | None = Noneinputs: dict[str, str | InputRequirementBlueprint] | None = None@@ -94,7 +94,7 @@ class PipeBlueprint(BaseModel):raise PipeBlueprintError(msg)return value- @field_validator("category", mode="after")+ @field_validator("pipe_category", mode="after")@staticmethoddef validate_pipe_category(value: Any) -> Any:"""Validate that the pipe category is one of the allowed values."""diff --git a/pipelex/core/registry_models.py b/pipelex/core/registry_models.pyindex 73a5eeb4f..4cebf39c9 100644--- a/pipelex/core/registry_models.py+++ b/pipelex/core/registry_models.py@@ -32,10 +32,10 @@ from pipelex.pipe_operators.img_gen.pipe_img_gen import PipeImgGenfrom pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactoryfrom pipelex.pipe_operators.llm.pipe_llm import PipeLLMfrom pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory-from pipelex.tools.registry_models import ModelType, RegistryModels+from pipelex.system.registries.registry_base import ModelType, RegistryModels-class PipelexRegistryModels(RegistryModels):+class CoreRegistryModels(RegistryModels):FIELD_EXTRACTION: ClassVar[list[ModelType]] = []PIPE_OPERATORS: ClassVar[list[PipeAbstractType]] = [diff --git a/pipelex/core/stuffs/image_content.py b/pipelex/core/stuffs/image_content.pyindex 288f187ba..2c08a98cd 100644--- a/pipelex/core/stuffs/image_content.py+++ b/pipelex/core/stuffs/image_content.py@@ -6,9 +6,10 @@ from PIL import Imagefrom typing_extensions import overridefrom yattag import Doc+from pipelex.cogt.exceptions import ImageContentErrorfrom pipelex.cogt.extract.extract_output import ExtractedImagefrom pipelex.core.stuffs.stuff_content import StuffContent-from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file+from pipelex.tools.misc.base_64_utils import prefixed_base64_str_from_base64_str, save_base_64_str_to_binary_filefrom pipelex.tools.misc.file_utils import ensure_directory_exists, get_incremental_file_path, save_text_to_pathfrom pipelex.tools.misc.filetype_utils import detect_file_type_from_base64from pipelex.tools.misc.path_utils import InterpretedPathOrUrl, interpret_path_or_url@@ -29,7 +30,7 @@ class ImageContent(StuffContent):@overridedef rendered_plain(self) -> str:- return self.url+ return self.url[:500]@overridedef rendered_html(self) -> str:@@ -40,7 +41,7 @@ class ImageContent(StuffContent):@overridedef rendered_markdown(self, level: int = 1, is_pretty: bool = False) -> str:- return f""+ return f"![{self.url[:100]}]({self.url})"@overridedef rendered_json(self) -> str:@@ -48,19 +49,25 @@ class ImageContent(StuffContent):@classmethoddef make_from_extracted_image(cls, extracted_image: ExtractedImage) -> Self:- return cls(- url=extracted_image.image_id,- base_64=extracted_image.base_64,- caption=extracted_image.caption,- )+ if base_64 := extracted_image.base_64:+ prefixed_base64_str = prefixed_base64_str_from_base64_str(b64_str=base_64)+ return cls(+ url=prefixed_base64_str,+ base_64=extracted_image.base_64,+ caption=extracted_image.caption,+ )+ else:+ msg = f"Base 64 is required for image content: {extracted_image}"+ raise ImageContentError(msg)@classmethoddef make_from_image(cls, image: Image.Image) -> Self:buffer = BytesIO()image.save(buffer, format="PNG")base_64 = base64.b64encode(buffer.getvalue()).decode("utf-8")+ prefixed_base64_str = prefixed_base64_str_from_base64_str(b64_str=base_64)return cls(- url=f"data:image/png;base64,{base_64}",+ url=prefixed_base64_str,base_64=base_64,)@@ -83,7 +90,7 @@ class ImageContent(StuffContent):extension=extension,avoid_suffix_if_possible=True,)- save_base64_to_binary_file(b64=base_64, file_path=file_path)+ save_base_64_str_to_binary_file(base_64_str=base_64, file_path=file_path)if caption := self.caption:caption_file_path = get_incremental_file_path(@@ -101,29 +108,3 @@ class ImageContent(StuffContent):avoid_suffix_if_possible=True,)save_text_to_path(text=source_prompt, path=source_prompt_file_path)---class PDFContent(StuffContent):- url: str-- @property- @override- def short_desc(self) -> str:- url_desc = interpret_path_or_url(path_or_uri=self.url).desc- return f"{url_desc} of a PDF document"-- @override- def rendered_plain(self) -> str:- return self.url-- @override- def rendered_html(self) -> str:- doc = Doc()- doc.stag("a", href=self.url, klass="msg-pdf")- doc.text(self.url)-- return doc.getvalue()-- @override- def rendered_markdown(self, level: int = 1, is_pretty: bool = False) -> str:- return f"[{self.url}]({self.url})"diff --git a/pipelex/core/validation.py b/pipelex/core/validation.pynew file mode 100644index 000000000..b43cf4a35--- /dev/null+++ b/pipelex/core/validation.py@@ -0,0 +1,64 @@+from pydantic import ValidationError++from pipelex import log+from pipelex.config import get_config+from pipelex.tools.typing.pydantic_utils import analyze_pydantic_validation_error+++def report_validation_error(category: str, validation_error: ValidationError) -> str:+ validation_error_analysis = analyze_pydantic_validation_error(validation_error)++ migration_config = get_config().migration++ migration_reports: list[str] = []++ # Build field-to-renamings mapping for missing fields+ log.debug(validation_error_analysis.missing_fields, title="Missing fields")+ missing_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}+ for missing_field in validation_error_analysis.missing_fields:+ text = missing_field.split(".")[-1]+ if renamings := migration_config.text_in_renaming_values(category=category, text=text):+ # Use tuple of renamings as key for grouping+ renamings_key = tuple(renamings)+ if renamings_key not in missing_field_renamings:+ missing_field_renamings[renamings_key] = []+ missing_field_renamings[renamings_key].append(missing_field)++ # Build field-to-renamings mapping for extra fields+ log.debug(validation_error_analysis.extra_fields, title="Extra fields")+ extra_field_renamings: dict[tuple[tuple[str, str], ...], list[str]] = {}+ for extra_field in validation_error_analysis.extra_fields:+ # Extract field path before the colon (extra fields include ": value")+ field_path = extra_field.split(":")[0].strip()+ text = field_path.split(".")[-1]+ if renamings := migration_config.text_in_renaming_keys(category=category, text=text):+ renamings_key = tuple(renamings)+ if renamings_key not in extra_field_renamings:+ extra_field_renamings[renamings_key] = []+ extra_field_renamings[renamings_key].append(extra_field)++ # Format grouped output for missing fields+ for renamings_tuple, fields in missing_field_renamings.items():+ renamings_str = "\n".join(f"• '{key}' -> '{value}'" for key, value in renamings_tuple)+ if len(fields) == 1:+ msg = f"Missing field '{fields[0]}' is possibly a new name related to one of these renamings:\n{renamings_str}"+ else:+ fields_str = ", ".join(f"'{f}'" for f in fields)+ msg = f"Missing fields [{fields_str}] are possibly new names related to one of these renamings:\n{renamings_str}"+ migration_reports.append(msg)++ # Format grouped output for extra fields+ for renamings_tuple, fields in extra_field_renamings.items():+ renamings_str = "\n".join(f"• '{key}' -> '{value}'" for key, value in renamings_tuple)+ if len(fields) == 1:+ msg = f"Extra field '{fields[0]}' is possibly an old deprecated name related to one of these renamings:\n{renamings_str}"+ else:+ fields_str = ", ".join(f"'{f}'" for f in fields)+ msg = f"Extra fields [{fields_str}] are possibly old deprecated names related to one of these renamings:\n{renamings_str}"+ migration_reports.append(msg)++ report_msg = validation_error_analysis.error_msg+ if migration_reports:+ migration_reports_str = "\n".join(migration_reports)+ report_msg += "\n\nNote that some fields have been renamed in the new version of Pipelex.\n\n" + migration_reports_str+ return report_msgdiff --git a/pipelex/exceptions.py b/pipelex/exceptions.pyindex 6e2972921..704fd178f 100644--- a/pipelex/exceptions.py+++ b/pipelex/exceptions.py@@ -1,7 +1,7 @@from click import ClickExceptionfrom typing_extensions import override-from pipelex.tools.exceptions import RootException+from pipelex.system.exceptions import RootExceptionfrom pipelex.tools.misc.context_provider_abstract import ContextProviderExceptionfrom pipelex.types import StrEnumdiff --git a/pipelex/hub.py b/pipelex/hub.pyindex 70b32ec16..53321d316 100644--- a/pipelex/hub.py+++ b/pipelex/hub.py@@ -28,8 +28,8 @@ from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstractfrom pipelex.pipeline.track.pipeline_tracker_protocol import PipelineTrackerProtocolfrom pipelex.plugins.plugin_manager import PluginManagerfrom pipelex.reporting.reporting_protocol import ReportingProtocol-from pipelex.tools.config.config_root import ConfigRoot-from pipelex.tools.config.manager import config_manager+from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.configuration.config_root import ConfigRootfrom pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstractfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstractdiff --git a/tests/e2e/pipelex/pipes/pipe_controller/__init__.py b/pipelex/kit/__init__.pysimilarity index 100%rename from tests/e2e/pipelex/pipes/pipe_controller/__init__.pyrename to pipelex/kit/__init__.pydiff --git a/pipelex/kit/agent_rules/commands.md b/pipelex/kit/agent_rules/commands.mdnew file mode 100644index 000000000..83ef8cd48--- /dev/null+++ b/pipelex/kit/agent_rules/commands.md@@ -0,0 +1,4 @@+# Commands++ - When you want to run commands such as `python`, `pytest` or any of our CLI such as `pipelex` or `cocode`, ALWAYS use the current obvious virtual env. If the installaton is standard, the venv is named `.venv` so always check that first.+\ No newline at end of filediff --git a/pipelex/kit/agent_rules/docs.md b/pipelex/kit/agent_rules/docs.mdnew file mode 100644index 000000000..67c5674b2--- /dev/null+++ b/pipelex/kit/agent_rules/docs.md@@ -0,0 +1,10 @@+# Writing Docs++Write docs and answer questions about writing docs.++We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.++## MkDocs Markdown Requirements++- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.+diff --git a/pipelex/kit/agent_rules/llms.md b/pipelex/kit/agent_rules/llms.mdnew file mode 100644index 000000000..f79c234bf--- /dev/null+++ b/pipelex/kit/agent_rules/llms.md@@ -0,0 +1,78 @@+# Rules to choose LLM models used in PipeLLMs.++## LLM Configuration System++In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:++- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`++## LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++## Using an LLM Handle in a PipeLLM++Here is an example of using an llm_handle to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""+```++As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").++## LLM Presets++Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:++```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:++The category of this invoice is: $invoice_details.category.++@invoice_text+"""+```++The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.+++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.+diff --git a/pipelex/kit/agent_rules/pytest_standards.md b/pipelex/kit/agent_rules/pytest_standards.mdnew file mode 100644index 000000000..8e06a0db8--- /dev/null+++ b/pipelex/kit/agent_rules/pytest_standards.md@@ -0,0 +1,101 @@+# Writing unit tests++## Unit test generalities++NEVER USE unittest.mock or MagicMock. YOU MUST USE pytest-mock instead.++### Test file structure++- Name test files with `test_` prefix+- Place test files in the appropriate test category directory:+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation+ - `tests/integration/` - for integration tests that test component interactions+ - `tests/e2e/` - for end-to-end tests that test complete workflows+ - `tests/test_pipelines/` - for test pipeline definitions (PLX files and their structuring python files)+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.+- Always put test inside Test classes.+- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`++### Markers++Apply the appropriate markers:+- "llm: uses an LLM to generate text or objects"+- "img_gen: uses an image generation AI"+- "extract: uses text/image extraction from documents"+- "inference: uses either an LLM or an image generation AI"+- "gha_disabled: will not be able to run properly on GitHub Actions"++Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.++### Important rules++- Never use the unittest.mock. Use pytest-mock.++### Test Class Structure++- Always group the tests of a module into a test class:++```python[email protected](loop_scope="class")+class TestFooBar:+ @pytest.mark.parametrize(+ "topic, test_case_blueprint",+ [+ TestCases.CASE_1,+ TestCases.CASE_2,+ ],+ )+ async def test_pipe_processing(+ self,+ request: FixtureRequest,+ topic: str,+ test_case_blueprint: StuffBlueprint,+ ):+ # Test implementation+```++- Never more than 1 class per test file.+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters+- Sometimes it can be convenient to access the test's name in its body, for instance to include into a job_id. To achieve that, add the argument `request: FixtureRequest` into the signature and then you can get the test name using `cast(str, request.node.originalname), # type: ignore`.++### Test Data Organization++- If it's not already there, create a `test_data.py` file in the proper test directory+- Define test cases using `StuffBlueprint`:++```python+class TestCases:+ CASE_BLUEPRINT_1 = StuffBlueprint(+ name="test_case_1",+ concept_code="domain.ConceptName1",+ value="test_value"+ )+ CASE_BLUEPRINT_2 = StuffBlueprint(+ name="test_case_2",+ concept_code="domain.ConceptName2",+ value="test_value"+ )++ CASE_BLUEPRINTS: ClassVar[list[tuple[str, str]]] = [ # topic, blueprint"+ ("topic1", CASE_BLUEPRINT_1),+ ("topic2", CASE_BLUEPRINT_2),+ ]+```++Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.+Also note that we provide a topic for the test case, which is purely for convenience.++## Best Practices for Testing++- Whenever possible, use strong asserts to test value, not just type and presence.+- Use parametrize for multiple test cases+- Test both success and failure cases+- Verify working memory state+- Check output structure and content+- Use meaningful test case names+- Include docstrings explaining test purpose but not on top of the file+- Log outputs for debugging+- Generate reports for cost trackingdiff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.mdnew file mode 100644index 000000000..9bd022ede--- /dev/null+++ b/pipelex/kit/agent_rules/python_standards.md@@ -0,0 +1,169 @@+# Coding Standards & Best Practices for Python Code++This document outlines the core coding standards, best practices, and quality control procedures for the codebase.++## Variables, loops and indexes++ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.++## Enums and tests++ - When defining enums related to string values, always inherit from `StrEnum`+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.++## Imports++### **Imports at the top of the file**++ - Import all necessary libraries at the top of the file+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting+ - Do not bother with ordering the imports, our Ruff linter will handle it for us. Same goes with removing unused imports.++- **Logging and Pretty Printing**:++ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:+ ```python+ from pipelex import log, pretty_print++ log.info("Hello, world!")+ ```+ - Both have a title arg which is handy when logging/printing objects:++ ```python+ log.debug("Hello, world!", title="Your first Pipelex log")+ pretty_print(output_object, title="Your first Pipelex output")+ ```+ - Both handle formatting json using Rich, pretty_print makes it prettier.++- **StrEnum and Self type**:++ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import Self, StrEnum+ ```++## Typing++### **Always Use Type Hints**++ - Every function parameter must be typed+ - Every function return must be typed+ - Use type hints for all variables where type is not obvious+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]+ - Use type hints for all fields+ - Use Field(default_factory=...) for mutable defaults+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.++### **BaseModel / Pydantic Standards**++ - Use `BaseModel` and respect Pydantic v2 standards+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`+ - Keep models focused and single-purpose+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:+ ```python+ from pydantic import BaseModel, Field+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++ class MyModel(BaseModel):+ names: list[str] = Field(default_factory=list) # OK for strings+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")+ ```++## Factory Pattern++ - Use Factory Pattern for object creation when dealing with multiple implementations+ - Our factory methods are named `make_from_...` and such++## Error Handling++ - Always catch exceptions at the place where you can add useful context to it.+ - Use try/except blocks with specific exceptions+ - Convert third-party exceptions to our custom ones+ - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands+ - NEVER raise generic exceptions like ValueError or TypeError, create new error classes and raise them instead+ - Always add `from exc` to the exception++ ```python+ try:+ self.models_manager.setup()+ except RoutingProfileLibraryNotFoundError as exc:+ msg = "The routing library could not be found, please call `pipelex init config` to create it"+ raise PipelexSetupError(msg) from exc+ ```++ **Note**: Following Ruff rules, we set the error message as a variable before raising it, for cleaner error traces.++## Documentation++1. **Docstring Format**+ ```python+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:+ """Process and resize an image.++ Args:+ image_path: Path to the source image+ size: Tuple of (width, height) for resizing++ Returns:+ Processed image as bytes+ """+ pass+ ```++2. **Class Documentation**+ ```python+ class ImageProcessor:+ """Handles image processing operations.++ Provides methods for resizing, converting, and optimizing images.+ """+ ```++## Code Quality Checks++### Linting and Type Checking++Before finalizing a task, run:+```bash+make fix-unused-imports+make check+```++This runs multiple code quality tools:+- Pyright: Static type checking+- Ruff: Fast Python linter+- Mypy: Static type checker++Always fix any issues reported by these tools before proceeding.++### Running Tests++1. **Quick Test Run** (no LLM/image generation):+ ```bash+ make tp+ ```+ Runs tests with markers: `(dry_runnable or not (inference or llm or img_gen or extract)) and not (needs_output or pipelex_api)`++2. **Specific Tests**:+ ```bash+ make tp TEST=TestClassName+ # or+ make tp TEST=test_function_name+ ```+ Note: Matches names starting with the provided string.++**Important**: Never run `make ti`, `make test-inference`, `make te`, `make test-extract`, `make tg`, or `make test-img-gen` - these use costly inference.++## Pipelines++- Always validate pipelines after creation/edit with `make validate`.+ Iterate if there are errors.++## Project Structure++- **Tests**: `tests/` directory+- **Documentation**: `docs/` directorydiff --git a/pipelex/kit/agent_rules/run_pipelex.md b/pipelex/kit/agent_rules/run_pipelex.mdnew file mode 100644index 000000000..21787573f--- /dev/null+++ b/pipelex/kit/agent_rules/run_pipelex.md@@ -0,0 +1,225 @@+# Guide to execute a pipeline and write example code++## Example to execute a pipeline with text output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+++async def hello_world() -> str:+ """+ This function demonstrates the use of a super simple Pipelex pipeline to generate text.+ """+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="hello_world",+ )++ return pipe_output.main_stuff_as_str+++# start Pipelex+Pipelex.make()+# run sample using asyncio+output_text = asyncio.run(hello_world())+pretty_print(output_text, title="Your first Pipelex output")+```++## Example to execute a pipeline with structured output++```python+import asyncio++from pipelex import pretty_print+from pipelex.pipelex import Pipelex+from pipelex.pipeline.execute import execute_pipeline+from pipelex.core.stuffs.image_content import ImageContent++from my_project.gantt.gantt_struct import GanttChart++SAMPLE_NAME = "extract_gantt"+IMAGE_URL = "assets/gantt/gantt_tree_house.png"+++async def extract_gantt(image_url: str) -> GanttChart:+ # Run the pipe+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )+ # Output the result+ return pipe_output.main_stuff_as(content_type=GanttChart)+++# start Pipelex+Pipelex.make()++# run sample using asyncio+gantt_chart = asyncio.run(extract_gantt(image_url=IMAGE_URL))+pretty_print(gantt_chart, title="Gantt Chart")+```++## Setting up the input memory++### Explanation of input memory++The input memory is a dictionary, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:+```python+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str+ImplicitMemory = dict[str, StuffContentOrData]+```+As you can seen, we made it so different ways can be used to define that stuff using structured content or data.++### Different ways to set up the input memory++So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:++```python+# Here we have a single input and it's a Text.+# If you assign a string, by default it will be considered as a TextContent.+ pipe_output = await execute_pipeline(+ pipe_code="master_advisory_orchestrator",+ input_memory={+ "user_input": problem_description,+ },+ )++# Here we have a single input and it's a PDF.+# Because PDFContent is a native concept, we can use it directly as a value,+# the system knows what content it corresponds to:+ pipe_output = await execute_pipeline(+ pipe_code="power_extractor_dpe",+ input_memory={+ "document": PDFContent(url=pdf_url),+ },+ )++# Here we have a single input and it's an Image.+# Because ImageContent is a native concept, we can use it directly as a value:+ pipe_output = await execute_pipeline(+ pipe_code="fashion_variation_pipeline",+ input_memory={+ "fashion_photo": ImageContent(url=image_url),+ },+ )++# Here we have a single input, it's an image but+# its actually a more specific concept gantt.GanttImage which refines Image,+# so we must provide it using a dict with the concept and the content:+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )++# Here is a more complex example with multiple inputs assigned using different ways:+ pipe_output = await execute_pipeline(+ pipe_code="retrieve_then_answer",+ dynamic_output_concept_code="contracts.Fees",+ input_memory={+ "text": load_text_from_path(path=text_path),+ "question": {+ "concept": "answer.Question",+ "content": question,+ },+ "client_instructions": client_instructions,+ },+ )+```++## Using the outputs of a pipeline++All pipe executions return a `PipeOutput` object.+It's a BaseModel which contains the resulting working memory at the end of the execution and the pipeline run id.+It also provides a bunch of accessor functions and properties to unwrap the main stuff, which is the last stuff added to the working memory:++```python++class PipeOutput(BaseModel):+ working_memory: WorkingMemory = Field(default_factory=WorkingMemory)+ pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)++ @property+ def main_stuff(self) -> Stuff:+ ...++ def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:+ ...++ def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]:+ ...++ def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType:+ ...++ @property+ def main_stuff_as_text(self) -> TextContent:+ ...++ @property+ def main_stuff_as_str(self) -> str:+ ...++ @property+ def main_stuff_as_image(self) -> ImageContent:+ ...++ @property+ def main_stuff_as_text_and_image(self) -> TextAndImagesContent:+ ...++ @property+ def main_stuff_as_number(self) -> NumberContent:+ ...++ @property+ def main_stuff_as_html(self) -> HtmlContent:+ ...++ @property+ def main_stuff_as_mermaid(self) -> MermaidContent:+ ...+```++As you can see, you can extract any variable from the output working memory.++### Getting the main stuff as a specific type++Simple text as a string:++```python+result = pipe_output.main_stuff_as_str+```+Structured object (BaseModel):++```python+result = pipe_output.main_stuff_as(content_type=GanttChart)+```++If it's a list, you can get a `ListContent` of the specific type.++```python+result_list_content = pipe_output.main_stuff_as_list(item_type=GanttChart)+```++or if you want, you can get the actual items as a regular python list:++```python+result_list = pipe_output.main_stuff_as_items(item_type=GanttChart)+```++---+diff --git a/pipelex/kit/agent_rules/tdd.md b/pipelex/kit/agent_rules/tdd.mdnew file mode 100644index 000000000..0f41c71e8--- /dev/null+++ b/pipelex/kit/agent_rules/tdd.md@@ -0,0 +1,24 @@+# Test-Driven Development Guide++This document outlines our test-driven development (TDD) process and the tools available for testing.++## TDD Cycle++1. **Write a Test First**+[pytest.mdc](pytest.mdc)++2. **Write the Code**+ - Implement the minimum amount of code needed to pass the test+ - Follow the project's coding standards+ - Keep it simple - don't write more than needed++3. **Run Linting and Type Checking**+[coding_standards.mdc](coding_standards.mdc)++4. **Refactor if needed**+If the code needs refactoring, with the best practices [coding_standards.mdc](coding_standards.mdc)++5. **Validate tests**++Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.+diff --git a/pipelex/kit/agent_rules/write_pipelex.md b/pipelex/kit/agent_rules/write_pipelex.mdnew file mode 100644index 000000000..4237b9c18--- /dev/null+++ b/pipelex/kit/agent_rules/write_pipelex.md@@ -0,0 +1,830 @@+# Guide to write or edit pipelines using the Pipelex language in .plx files++- Always first write your "plan" in natural language, then transcribe it in pipelex.+- You should ALWAYS RUN the terminal command `make validate` when you are writing or editing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.+- Please use POSIX standard for files. (empty lines, no trailing whitespaces, etc.)++## Pipeline File Naming+- Files must be `.plx` for pipelines (Always add an empty line at the end of the file, and do not add trailing whitespaces to PLX files at all)+- Files must be `.py` for code defining the data structures+- Use descriptive names in `snake_case`++## Pipeline File Outline+A pipeline file has three main sections:+1. Domain statement+2. Concept definitions+3. Pipe definitions++### Domain Statement+```plx+domain = "domain_name"+description = "Description of the domain" # Optional+```+Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.++### Concept Definitions++Concepts represent ideas and semantic entities in your pipeline. They define what something *is*, not how it's structured.++```plx+[concept]+ConceptName = "Description of the concept"+```++**Naming Rules:**+- Use PascalCase for concept names+- Never use plurals (no "Stories", use "Story") - lists are handled implicitly by Pipelex+- Avoid circumstantial adjectives (no "LargeText", use "Text") - focus on the essence of what the concept represents+- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)++**Native Concepts:**+Pipelex provides built-in native concepts: `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`. Use these directly or refine them when appropriate.++**Refining Native Concepts:**+To create a concept that specializes a native concept without adding fields:++```plx+[concept.Landscape]+description = "A scenic outdoor photograph"+refines = "Image"+```++For details on how to structure concepts with fields, see the "Structuring Models" section below.++### Pipe Definitions++## Pipe Base Definition++```plx+[pipe.your_pipe_name]+type = "PipeLLM"+description = "A description of what your pipe does"+inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }+output = "ConceptName"+```++The pipes will all have at least this base definition.+- `inputs`: Dictionary of key being the variable used in the prompts, and the value being the ConceptName. It should ALSO LIST THE INPUTS OF THE INTERMEDIATE STEPS (if PipeSequence) or of the conditional pipes (if PipeCondition).+So If you have this error:+`StaticValidationError: missing_input_variable • domain='expense_validator' • pipe='validate_expense' •+variable='['invoice']'``+That means that the pipe validate_expense is missing the input `invoice` because one of the subpipe is needing it.++NEVER WRITE THE INPUTS BY BREAKING THE LINE LIKE THIS:+<NEVER DO THIS>+```plx+inputs = {+ input_1 = "ConceptName1",+ input_2 = "ConceptName2"+}+```+</NEVER DO THIS>++- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:++## Structuring Models++Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.++### Three Ways to Structure Concepts++**1. No Structure Needed**++If a concept only refines a native concept without adding fields, use the TOML table syntax shown in "Concept Definitions" above. No structure section is needed.++**2. Inline Structure Definition (RECOMMENDED for most cases)**++For concepts with structured fields, define them inline using TOML syntax:++```plx+[concept.Invoice]+description = "A commercial document issued by a seller to a buyer"++[concept.Invoice.structure]+invoice_number = "The unique invoice identifier"+issue_date = { type = "date", description = "The date the invoice was issued", required = true }+total_amount = { type = "number", description = "The total invoice amount", required = true }+vendor_name = "The name of the vendor"+line_items = { type = "list", item_type = "text", description = "List of items", required = false }+```++**Supported inline field types:** `text`, `integer`, `boolean`, `number`, `date`, `list`, `dict`++**Field properties:** `type`, `description`, `required` (default: true), `default_value`, `choices`, `item_type` (for lists), `key_type` and `value_type` (for dicts)++**Simple syntax** (creates required text field):+```plx+field_name = "Field description"+```++**Detailed syntax** (with explicit properties):+```plx+field_name = { type = "text", description = "Field description", required = false, default_value = "default" }+```++**3. Python StructuredContent Class (For Advanced Features)**++Create a Python class when you need:+- Custom validation logic (@field_validator, @model_validator)+- Computed properties (@property methods)+- Custom methods or class methods+- Complex cross-field validation+- Reusable structures across multiple domains++```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pydantic import Field, field_validator++class Invoice(StructuredContent):+ """A commercial invoice with validation."""++ invoice_number: str = Field(description="The unique invoice identifier")+ total_amount: float = Field(ge=0, description="The total invoice amount")+ tax_amount: float = Field(ge=0, description="Tax amount")++ @field_validator('tax_amount')+ @classmethod+ def validate_tax(cls, v, info):+ """Ensure tax doesn't exceed total."""+ total = info.data.get('total_amount', 0)+ if v > total:+ raise ValueError('Tax amount cannot exceed total amount')+ return v+```++**Location:** Create models in `my_project/some_domain/some_domain_struct.py`. Classes inheriting from `StructuredContent` are automatically discovered.++### Decision Rules for Agents++**If concept already exists:**+- If it's already inline → KEEP IT INLINE unless user explicitly asks to convert or features require Python class+- If it's already a Python class → KEEP IT as Python class++**If creating new concept:**+1. Does it only refine a native concept without adding fields? → Use concept-only declaration+2. Does it need custom validation, computed properties, or methods? → Use Python class+3. Otherwise → Use inline structure (fastest and simplest)++**When to suggest conversion to Python class:**+- User needs validation logic beyond type checking+- User needs computed properties or custom methods+- Structure needs to be reused across multiple domains+- Complex type relationships or inheritance required++### Inline Structure Limitations++Inline structures:+- ✅ Support all common field types (text, number, date, list, dict, etc.)+- ✅ Support required/optional fields, defaults, choices+- ✅ Generate full Pydantic models with validation+- ❌ Cannot have custom validators or complex validation logic+- ❌ Cannot have computed properties or custom methods+- ❌ Cannot refine custom (non-native) concepts+- ❌ Limited IDE autocomplete compared to explicit Python classes+++## Pipe Controllers and Pipe Operators++Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:++1. **Controllers** - For flow control:+ - `PipeSequence` - For creating a sequence of multiple steps+ - `PipeCondition` - If the next pipe depends of the expression of a stuff in the working memory+ - `PipeParallel` - For parallelizing pipes++2. **Operators** - For specific tasks:+ - `PipeLLM` - Generate Text and Objects (include Vision LLM)+ - `PipeExtract` - Extract text and images from an image or a PDF+ - `PipeCompose` - For composing text using Jinja2 templates: supports html, markdown, mermaid, etc.+ - `PipeImgGen` - Generate Images+ - `PipeFunc` - For running classic python scripts++## PipeSequence controller++Purpose: PipeSequence executes multiple pipes in a defined order, where each step can use results from original inputs or from previous steps.++### Basic Definition+```plx+[pipe.your_sequence_name]+type = "PipeSequence"+description = "Description of what this sequence does"+inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps+output = "OutputType"+steps = [+ { pipe = "first_pipe", result = "first_result" },+ { pipe = "second_pipe", result = "second_result" },+ { pipe = "final_pipe", result = "final_result" }+]+```++### Key Components++1. **Steps Array**: List of pipes to execute in sequence+ - `pipe`: Name of the pipe to execute+ - `result`: Name to assign to the pipe's output that will be in the working memory++### Using PipeBatch in Steps++You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:++```plx+steps = [+ { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"+ }+]+```++1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.+ - Must be a `ListContent` type containing the items to process+ - Can reference inputs or results from previous steps++2. **batch_as**: Defines the name that will be used to reference the current item being processed+ - This name can be used in the pipe's input mappings+ - Makes each item from the batch available as a single element++The result of a batched step will be a `ListContent` containing the outputs from processing each item.++## PipeCondition controller++The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.++### Basic usage++```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression = "input_data.category"+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```+or+```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "A conditional pipe to decide whether..."+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression_template = "{{ input_data.category }}" # Jinja2 code+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```++### Key Parameters++- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)+- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)+- `outcomes`: Dictionary mapping expression results to pipe codes:+ 1. The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`+ 2. The value on the right (`process_small`, `process_medium`, etc.) is the name of the pipe to trigger+- `default_outcome`: **Required** - The pipe to execute if the expression doesn't match any key in outcomes. Use `"fail"` if you want the pipeline to fail when no match is found++Example with fail as default:+```plx+[pipe.strict_validation]+type = "PipeCondition"+description = "Validate with strict matching"+inputs = { status = "Status" }+output = "Text"+expression = "status.value"+default_outcome = "fail"++[pipe.strict_validation.outcomes]+approved = "process_approved"+rejected = "process_rejected"+```++## PipeLLM operator++PipeLLM is used to:+1. Generate text or objects with LLMs+2. Process images with Vision LLMs++### Basic Usage++Simple Text Generation:+```plx+[pipe.write_story]+type = "PipeLLM"+description = "Write a short story"+output = "Text"+prompt = """+Write a short story about a programmer.+"""+```++Structured Data Extraction:+```plx+[pipe.extract_info]+type = "PipeLLM"+description = "Extract information"+inputs = { text = "Text" }+output = "PersonInfo"+prompt = """+Extract person information from this text:+@text+"""+```++Supports system instructions:+```plx+[pipe.expert_analysis]+type = "PipeLLM"+description = "Expert analysis"+output = "Analysis"+system_prompt = "You are a data analysis expert"+prompt = "Analyze this data"+```++### Multiple Outputs++Generate multiple outputs (fixed number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+nb_output = 3 # Generate exactly 3 ideas+```++Generate multiple outputs (variable number):+```plx+[pipe.generate_ideas]+type = "PipeLLM"+description = "Generate ideas"+output = "Idea"+multiple_output = true # Let the LLM decide how many to generate+```++### Vision++Process images with VLMs (image inputs must be tagged in the prompt):+```plx+[pipe.analyze_image]+type = "PipeLLM"+description = "Analyze image"+inputs = { image = "Image" }+output = "ImageAnalysis"+prompt = """+Describe what you see in this image:++$image+"""+```++You can also reference images inline in meaningful sentences to guide the Visual LLM:+```plx+[pipe.compare_images]+type = "PipeLLM"+description = "Compare two images"+inputs = { photo = "Image", painting = "Image" }+output = "Analysis"+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++### Writing prompts for PipeLLM++**Insert stuff inside a tagged block**++If the inserted text is supposedly a long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.++Example template:+```plx+prompt = """+Match the expense with its corresponding invoice:++@expense++@invoices+"""+```+In the example above, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doesn't need to be explicitly written in the prompt.++DO NOT write things like "Here is the expense: @expense".+DO write simply "@expense" alone in an isolated line.++**Insert stuff inline**++If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.++Example template:+```plx+prompt = """+Your goal is to summarize everything related to $topic in the provided text:++@text++Please provide only the summary, with no additional text or explanations.+Your summary should not be longer than 2 sentences.+"""+```++In the example above, $topic will be inserted inline, whereas @text will be a a delimited block.+Be sure to make the proper choice of prefix for each insertion.++DO NOT write "$topic" alone in an isolated line.+DO write things like "Write an essay about $topic" to include text into an actual sentence.+++## PipeExtract operator++The PipeExtract operator is used to extract text and images from an image or a PDF++### Simple Text Extraction+```plx+[pipe.extract_info]+type = "PipeExtract"+description = "extract the information"+inputs = { document = "PDF" } # or { image = "Image" } if it's an image. This is the only input.+output = "Page"+```++Using Extract Model Settings:+```plx+[pipe.extract_with_model]+type = "PipeExtract"+description = "Extract with specific model"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral" # Use predefined extract preset or model alias+```++Only one input is allowed and it must either be an `Image` or a `PDF`. The input can be named anything.++The output concept `Page` is a native concept, with the structure `PageContent`:+It corresponds to 1 page. Therefore, the PipeExtract is outputing a `ListContent` of `Page`++```python+class TextAndImagesContent(StuffContent):+ text: TextContent | None+ images: list[ImageContent] | None++class PageContent(StructuredContent): # CONCEPT IS "Page"+ text_and_images: TextAndImagesContent+ page_view: ImageContent | None = None+```+- `text_and_images` are the text, and the related images found in the input image or PDF.+- `page_view` is the screenshot of the whole pdf page/image.++## PipeCompose operator++The PipeCompose operator is used to compose text using Jinja2 templates. It supports various output formats including HTML, Markdown, Mermaid diagrams, and more.++### Basic Usage++Simple Template Composition:+```plx+[pipe.compose_report]+type = "PipeCompose"+description = "Compose a report using template"+inputs = { data = "ReportData" }+output = "Text"+template = """+# Report Summary++Based on the analysis:+$data++Generated on: {{ current_date }}+"""+```++Using Named Templates:+```plx+[pipe.use_template]+type = "PipeCompose"+description = "Use a predefined template"+inputs = { content = "Text" }+output = "Text"+template_name = "standard_report_template"+```++Using Nested Template Section (for more control):+```plx+[pipe.advanced_template]+type = "PipeCompose"+description = "Use advanced template settings"+inputs = { data = "ReportData" }+output = "Text"++[pipe.advanced_template.template]+template = "Report: $data"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+```++CRM Email Template:+```plx+[pipe.compose_follow_up_email]+type = "PipeCompose"+description = "Compose a personalized follow-up email for CRM"+inputs = { customer = "Customer", deal = "Deal", sales_rep = "SalesRep" }+output = "Text"+template_category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+template = """+Subject: Following up on our $deal.product_name discussion++Hi $customer.first_name,++I hope this email finds you well! I wanted to follow up on our conversation about $deal.product_name from $deal.last_contact_date.++Based on our discussion, I understand that your key requirements are: $deal.customer_requirements++I'm excited to let you know that we can definitely help you achieve your goals. Here's what I'd like to propose:++**Next Steps:**+- Schedule a demo tailored to your specific needs+- Provide you with a customized quote based on your requirements+- Connect you with our implementation team++Would you be available for a 30-minute call this week? I have openings on:+{% for slot in available_slots %}+- {{ slot }}+{% endfor %}++Looking forward to moving this forward together!++Best regards,+$sales_rep.name+$sales_rep.title+$sales_rep.phone | $sales_rep.email+"""+```++### Key Parameters++- `template`: Inline template string (mutually exclusive with template_name)+- `template_name`: Name of a predefined template (mutually exclusive with template)+- `template_category`: Template type ("llm_prompt", "html", "markdown", "mermaid", etc.)+- `templating_style`: Styling options for template rendering+- `extra_context`: Additional context variables for template++For more control, you can use a nested `template` section instead of the `template` field:+- `template.template`: The template string+- `template.category`: Template type+- `template.templating_style`: Styling options++### Template Variables++Use the same variable insertion rules as PipeLLM:+- `@variable` for block insertion (multi-line content)+- `$variable` for inline insertion (short text)++## PipeImgGen operator++The PipeImgGen operator is used to generate images using AI image generation models.++### Basic Usage++Simple Image Generation:+```plx+[pipe.generate_image]+type = "PipeImgGen"+description = "Generate an image from prompt"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+```++Using Image Generation Settings:+```plx+[pipe.generate_photo]+type = "PipeImgGen"+description = "Generate a high-quality photo"+inputs = { prompt = "ImgGenPrompt" }+output = "Photo"+model = { model = "fast-img-gen" }+aspect_ratio = "16:9"+quality = "hd"+```++Multiple Image Generation:+```plx+[pipe.generate_variations]+type = "PipeImgGen"+description = "Generate multiple image variations"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+nb_output = 3+seed = "auto"+```++Advanced Configuration:+```plx+[pipe.generate_custom]+type = "PipeImgGen"+description = "Generate image with custom settings"+inputs = { prompt = "ImgGenPrompt" }+output = "Image"+model = "img_gen_preset_name" # Use predefined preset+aspect_ratio = "1:1"+quality = "hd"+background = "transparent"+output_format = "png"+is_raw = false+safety_tolerance = 3+```++### Key Parameters++**Image Generation Settings:**+- `model`: Model choice (preset name or inline settings with model name)+- `quality`: Image quality ("standard", "hd")++**Output Configuration:**+- `nb_output`: Number of images to generate+- `aspect_ratio`: Image dimensions ("1:1", "16:9", "9:16", etc.)+- `output_format`: File format ("png", "jpeg", "webp")+- `background`: Background type ("default", "transparent")++**Generation Control:**+- `seed`: Random seed (integer or "auto")+- `is_raw`: Whether to apply post-processing+- `is_moderated`: Enable content moderation+- `safety_tolerance`: Content safety level (1-6)++### Input Requirements++PipeImgGen requires exactly one input that must be either:+- An `ImgGenPrompt` concept+- A concept that refines `ImgGenPrompt`++The input can be named anything but must contain the prompt text for image generation.++## PipeFunc operator++The PipeFunc operator is used to run custom Python functions within a pipeline. This allows integration of classic Python scripts and custom logic.++### Basic Usage++Simple Function Call:+```plx+[pipe.process_data]+type = "PipeFunc"+description = "Process data using custom function"+inputs = { input_data = "DataType" }+output = "ProcessedData"+function_name = "process_data_function"+```++File Processing Example:+```plx+[pipe.read_file]+type = "PipeFunc"+description = "Read file content"+inputs = { file_path = "FilePath" }+output = "FileContent"+function_name = "read_file_content"+```++### Key Parameters++- `function_name`: Name of the Python function to call (must be registered in func_registry)++### Function Requirements++The Python function must:++1. **Be registered** in the `func_registry`+2. **Accept `working_memory`** as a parameter:+ ```python+ async def my_function(working_memory: WorkingMemory) -> StuffContent | list[StuffContent] | str:+ # Function implementation+ pass+ ```++3. **Return appropriate types**:+ - `StuffContent`: Single content object+ - `list[StuffContent]`: Multiple content objects (becomes ListContent)+ - `str`: Simple string (becomes TextContent)++### Function Registration++Functions must be registered in the function registry before use:++```python+from pipelex.system.registries.func_registry import func_registry++@func_registry.register("my_function_name")+async def my_custom_function(working_memory: WorkingMemory) -> StuffContent:+ # Access inputs from working memory+ input_data = working_memory.get_stuff("input_name")++ # Process data+ result = process_logic(input_data.content)++ # Return result+ return MyResultContent(data=result)+```++### Working Memory Access++Inside the function, access pipeline inputs through working memory:++```python+async def process_function(working_memory: WorkingMemory) -> TextContent:+ # Get input stuff by name+ input_stuff = working_memory.get_stuff("input_name")++ # Access the content+ input_content = input_stuff.content++ # Process and return+ processed_text = f"Processed: {input_content.text}"+ return TextContent(text=processed_text)+```++---++## Rules to choose LLM models used in PipeLLMs.++### LLM Configuration System++In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:++- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`+- **Routing**: `.pipelex/inference/routing_profiles.toml`++### LLM Handles++An llm_handle can be either:+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:++```toml+[aliases]+base-claude = "claude-4.5-sonnet"+base-gpt = "gpt-5"+base-gemini = "gemini-2.5-flash"+base-mistral = "mistral-medium"+```++The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.++### Using an LLM Handle in a PipeLLM++Here is an example of using a model to specify which LLM to use in a PipeLLM:++```plx+[pipe.hello_world]+type = "PipeLLM"+description = "Write text about Hello World."+output = "Text"+model = { model = "gpt-5", temperature = 0.9 }+prompt = """+Write a haiku about Hello World.+"""+```++As you can see, to use the LLM, you must also indicate the temperature (float between 0 and 1) and max_tokens (either an int or the string "auto").++### LLM Presets++Presets are meant to record the choice of an llm with its hyper parameters (temperature and max_tokens) if it's good for a particular task. LLM Presets are skill-oriented.++Examples:+```toml+llm_to_reason = { model = "base-claude", temperature = 1 }+llm_to_extract_invoice = { model = "claude-3-7-sonnet", temperature = 0.1, max_tokens = "auto" }+```++The interest is that these presets can be used to set the LLM choice in a PipeLLM, like this:++```plx+[pipe.extract_invoice]+type = "PipeLLM"+description = "Extract invoice information from an invoice text transcript"+inputs = { invoice_text = "InvoiceText" }+output = "Invoice"+model = "llm_to_extract_invoice"+prompt = """+Extract invoice information from this invoice:++The category of this invoice is: $invoice_details.category.++@invoice_text+"""+```++The setting here `model = "llm_to_extract_invoice"` works because "llm_to_extract_invoice" has been declared as an llm_preset in the deck.+You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.++You can override the predefined llm presets by setting them in `.pipelex/inference/deck/overrides.toml`.++---++ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+Then, create an example file to run the pipeline in the `examples` folder.+But don't write documentation unless asked explicitly to.diff --git a/pipelex/config_template/inference/backends.toml b/pipelex/kit/configs/inference/backends.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends.tomlrename to pipelex/kit/configs/inference/backends.tomldiff --git a/pipelex/config_template/inference/backends/anthropic.toml b/pipelex/kit/configs/inference/backends/anthropic.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/anthropic.tomlrename to pipelex/kit/configs/inference/backends/anthropic.tomldiff --git a/pipelex/config_template/inference/backends/azure_openai.toml b/pipelex/kit/configs/inference/backends/azure_openai.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/azure_openai.tomlrename to pipelex/kit/configs/inference/backends/azure_openai.tomldiff --git a/pipelex/config_template/inference/backends/bedrock.toml b/pipelex/kit/configs/inference/backends/bedrock.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/bedrock.tomlrename to pipelex/kit/configs/inference/backends/bedrock.tomldiff --git a/pipelex/config_template/inference/backends/blackboxai.toml b/pipelex/kit/configs/inference/backends/blackboxai.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/blackboxai.tomlrename to pipelex/kit/configs/inference/backends/blackboxai.tomldiff --git a/pipelex/config_template/inference/backends/fal.toml b/pipelex/kit/configs/inference/backends/fal.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/fal.tomlrename to pipelex/kit/configs/inference/backends/fal.tomldiff --git a/pipelex/config_template/inference/backends/google.toml b/pipelex/kit/configs/inference/backends/google.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/google.tomlrename to pipelex/kit/configs/inference/backends/google.tomldiff --git a/pipelex/config_template/inference/backends/internal.toml b/pipelex/kit/configs/inference/backends/internal.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/internal.tomlrename to pipelex/kit/configs/inference/backends/internal.tomldiff --git a/pipelex/config_template/inference/backends/mistral.toml b/pipelex/kit/configs/inference/backends/mistral.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/mistral.tomlrename to pipelex/kit/configs/inference/backends/mistral.tomldiff --git a/pipelex/config_template/inference/backends/ollama.toml b/pipelex/kit/configs/inference/backends/ollama.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/ollama.tomlrename to pipelex/kit/configs/inference/backends/ollama.tomldiff --git a/pipelex/config_template/inference/backends/openai.toml b/pipelex/kit/configs/inference/backends/openai.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/openai.tomlrename to pipelex/kit/configs/inference/backends/openai.tomldiff --git a/pipelex/config_template/inference/backends/perplexity.toml b/pipelex/kit/configs/inference/backends/perplexity.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/perplexity.tomlrename to pipelex/kit/configs/inference/backends/perplexity.tomldiff --git a/pipelex/config_template/inference/backends/pipelex_inference.toml b/pipelex/kit/configs/inference/backends/pipelex_inference.tomlsimilarity index 95%rename from pipelex/config_template/inference/backends/pipelex_inference.tomlrename to pipelex/kit/configs/inference/backends/pipelex_inference.tomlindex d850714e8..eb82d8c71 100644--- a/pipelex/config_template/inference/backends/pipelex_inference.toml+++ b/pipelex/kit/configs/inference/backends/pipelex_inference.toml@@ -101,6 +101,13 @@ inputs = ["text", "images"]outputs = ["text", "structured"]costs = { input = 0.10, output = 0.40 }+["gemini-2.5-pro"]+model_id = "gemini-2.5-pro"+inputs = ["text", "images"]+outputs = ["text", "structured"]+max_prompt_images = 3000+costs = { input = 1.25, output = 10.0 }+["gemini-2.5-flash"]model_id = "gemini/gemini-2.5-flash"inputs = ["text", "images"]diff --git a/pipelex/config_template/inference/backends/vertexai.toml b/pipelex/kit/configs/inference/backends/vertexai.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/vertexai.tomlrename to pipelex/kit/configs/inference/backends/vertexai.tomldiff --git a/pipelex/config_template/inference/backends/xai.toml b/pipelex/kit/configs/inference/backends/xai.tomlsimilarity index 100%rename from pipelex/config_template/inference/backends/xai.tomlrename to pipelex/kit/configs/inference/backends/xai.tomldiff --git a/pipelex/config_template/inference/deck/base_deck.toml b/pipelex/kit/configs/inference/deck/base_deck.tomlsimilarity index 100%rename from pipelex/config_template/inference/deck/base_deck.tomlrename to pipelex/kit/configs/inference/deck/base_deck.tomldiff --git a/pipelex/config_template/inference/deck/overrides.toml b/pipelex/kit/configs/inference/deck/overrides.tomlsimilarity index 100%rename from pipelex/config_template/inference/deck/overrides.tomlrename to pipelex/kit/configs/inference/deck/overrides.tomldiff --git a/pipelex/config_template/inference/routing_profiles.toml b/pipelex/kit/configs/inference/routing_profiles.tomlsimilarity index 100%rename from pipelex/config_template/inference/routing_profiles.tomlrename to pipelex/kit/configs/inference/routing_profiles.tomldiff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlnew file mode 100644index 000000000..31419274e--- /dev/null+++ b/pipelex/kit/configs/pipelex.toml@@ -0,0 +1,67 @@+####################################################################################################+# Pipelex Configuration File+####################################################################################################+#+# This configuration file is copied to your project's .pipelex/ directory when you run:+# pipelex init config+#+# Purpose:+# - This file allows you to override Pipelex's default settings for your specific project+# - Feel free to modify any settings below to suit your needs+# - You can add any configuration sections that exist in the main pipelex.toml+#+# Finding Available Settings:+# - See the full default configuration in: pipelex/pipelex.toml (in the Pipelex package)+# - See the configuration structure classes in: pipelex/config.py and pipelex/cogt/config_cogt.py+#+# Common Customizations:+# - Logging levels and behavior+# - Excluded directories for scanning+# - LLM prompt dumping for debugging+# - Feature flags for tracking and reporting+# - Observer and reporting output directories+#+####################################################################################################++[pipelex.scan_config]+excluded_dirs = [+ ".venv",+ "venv",+ "env",+ ".env",+ "virtualenv",+ ".virtualenv",+ ".git",+ "__pycache__",+ ".pytest_cache",+ ".mypy_cache",+ ".ruff_cache",+ "node_modules",+ "results",+]++[pipelex.log_config]+default_log_level = "INFO"++[pipelex.log_config.package_log_levels]+pipelex = "INFO"++[cogt.llm_config]+# Uncomment any of these to enable dumping the inputs or outputs of text-genration with an LLM+# is_dump_text_prompts_enabled = true+# is_dump_response_text_enabled = true++[cogt.llm_config.instructor_config]+# Uncomment any of these to enable dumping the kwargs, response or errors of the instructor+# is_dump_kwargs_enabled = true+# is_dump_response_enabled = true+# is_dump_error_enabled = true++[pipelex.observer_config]+observer_dir = "results/observer"++[pipelex.feature_config]+# WIP/Experimental feature flags+is_pipeline_tracking_enabled = false+is_activity_tracking_enabled = false+is_reporting_enabled = truediff --git a/pipelex/kit/cursor_export.py b/pipelex/kit/cursor_export.pynew file mode 100644index 000000000..ca207b8a9--- /dev/null+++ b/pipelex/kit/cursor_export.py@@ -0,0 +1,99 @@+from collections.abc import Iterable+from importlib.abc import Traversable+from pathlib import Path+from typing import Any++import typer+import yaml++from pipelex.kit.index_models import KitIndex+from pipelex.kit.paths import get_agents_dir+++def _iter_agent_files(agents_dir: Traversable) -> Iterable[tuple[str, str]]:+ """Iterate over agent markdown files.++ Args:+ agents_dir: Traversable pointing to agents directory++ Yields:+ Tuples of (filename, file_content)+ """+ for child in agents_dir.iterdir():+ if child.name.endswith(".md") and child.is_file():+ yield child.name, child.read_text(encoding="utf-8")+++def _front_matter_for(name: str, idx: KitIndex) -> dict[str, Any]:+ """Build front-matter for a specific file.++ Args:+ name: Filename (e.g., "pytest_standards.md")+ idx: Kit index configuration++ Returns:+ Merged front-matter dictionary+ """+ base = idx.agent_rules.cursor.front_matter.copy()+ key = name.removesuffix(".md")+ if key in idx.agent_rules.cursor.files:+ base |= idx.agent_rules.cursor.files[key].front_matter+ # Remove globs if it's an empty list+ if "globs" in base and base["globs"] == []:+ del base["globs"]+ return base+++def export_cursor_rules(repo_root: Path, idx: KitIndex, dry_run: bool = False) -> None:+ """Export agent markdown files to Cursor .mdc files with YAML front-matter.++ Args:+ repo_root: Repository root directory+ idx: Kit index configuration+ dry_run: If True, only print what would be done+ """+ agents_dir = get_agents_dir()+ out_dir = repo_root / ".cursor" / "rules"+ out_dir.mkdir(parents=True, exist_ok=True)++ for fname, body in _iter_agent_files(agents_dir):+ fm = _front_matter_for(fname, idx)+ yaml_block = "---\n" + yaml.safe_dump(fm, sort_keys=False).rstrip() + "\n---\n"+ mdc = yaml_block + body+ out_path = out_dir / (fname.removesuffix(".md") + ".mdc")++ if dry_run:+ typer.echo(f"[DRY] write {out_path}")+ else:+ out_path.write_text(mdc, encoding="utf-8")+ typer.echo(f"✅ Exported {out_path}")+++def remove_cursor_rules(repo_root: Path, dry_run: bool = False) -> None:+ """Remove Cursor .mdc files that correspond to agent markdown files.++ Args:+ repo_root: Repository root directory+ dry_run: If True, only print what would be done+ """+ agents_dir = get_agents_dir()+ out_dir = repo_root / ".cursor" / "rules"++ if not out_dir.exists():+ typer.echo(f"⚠️ Directory {out_dir} does not exist - nothing to remove")+ return++ removed_count = 0+ for fname, _ in _iter_agent_files(agents_dir):+ out_path = out_dir / (fname.removesuffix(".md") + ".mdc")++ if out_path.exists():+ if dry_run:+ typer.echo(f"[DRY] delete {out_path}")+ else:+ out_path.unlink()+ typer.echo(f"🗑️ Deleted {out_path}")+ removed_count += 1++ if removed_count == 0:+ typer.echo("⚠️ No Cursor rules found to remove")diff --git a/pipelex/kit/index.toml b/pipelex/kit/index.tomlnew file mode 100644index 000000000..148f5aa4f--- /dev/null+++ b/pipelex/kit/index.toml@@ -0,0 +1,97 @@+[meta]+version = "1.0.0"+description = "Pipelex kit configuration for agent rules"++[agent_rules]+demote = 1+default_set = "pipelex_language"++[agent_rules.sets]+coding_standards = [+ "commands.md",+ "python_standards.md",+ "pytest_standards.md",+ "docs.md",+ "tdd.md",+]+pipelex_language = ["write_pipelex.md", "run_pipelex.md", "llms.md"]+all = [+ "commands.md",+ "python_standards.md",+ "write_pipelex.md",+ "run_pipelex.md",+ "docs.md",+ "llms.md",+ "pytest_standards.md",+ "tdd.md",+]++[agent_rules.cursor.front_matter]+alwaysApply = false++[agent_rules.cursor.files.python_standards.front_matter]+description = "Python coding standards and best practices"+globs = ["**/*.py"]++[agent_rules.cursor.files.pytest_standards.front_matter]+description = "Guidelines for writing unit tests"+globs = ["tests/**/*.py"]++[agent_rules.cursor.files.docs.front_matter]+description = "Guidelines for writing documentation"+globs = ["docs/**/*.md"]++[agent_rules.cursor.files.commands.front_matter]+description = "Guidelines for running commands"+alwaysApply = true++[agent_rules.cursor.files.tdd.front_matter]+description = "Guidelines for writing test-driven development code"+globs = []++[agent_rules.cursor.files.write_pipelex.front_matter]+description = "Guidelines for writing Pipelex pipelines"+globs = ["**/*.plx", "**/pipelines/**/*.py"]++[agent_rules.cursor.files.run_pipelex.front_matter]+description = "Guidelines for running Pipelex pipelines"+globs = ["examples/**/*.py"]++[agent_rules.cursor.files.llms.front_matter]+description = "LLM configuration and usage guidelines"+globs = ["*.plx", "*.toml"]++[agent_rules.targets.agents]+path = "AGENTS.md"+strategy = "merge"+marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"+marker_end = "<!-- END_PIPELEX_RULES -->"+heading_1 = "# Pipelex Rules"++[agent_rules.targets.claude]+path = "CLAUDE.md"+strategy = "merge"+marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"+marker_end = "<!-- END_PIPELEX_RULES -->"+heading_1 = "# Pipelex Rules"++[agent_rules.targets.github_copilot]+path = ".github/copilot-instructions.md"+strategy = "merge"+marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"+marker_end = "<!-- END_PIPELEX_RULES -->"+heading_1 = "# Pipelex Rules"++[agent_rules.targets.windsurf]+path = ".windsurfrules.md"+strategy = "merge"+marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"+marker_end = "<!-- END_PIPELEX_RULES -->"+heading_1 = "# Pipelex Rules"++[agent_rules.targets.blackbox]+path = "BLACKBOX_RULES.md"+strategy = "merge"+marker_begin = "<!-- BEGIN_PIPELEX_RULES -->"+marker_end = "<!-- END_PIPELEX_RULES -->"+heading_1 = "# Pipelex Rules"diff --git a/pipelex/kit/index_loader.py b/pipelex/kit/index_loader.pynew file mode 100644index 000000000..adb8b5556--- /dev/null+++ b/pipelex/kit/index_loader.py@@ -0,0 +1,30 @@+from pydantic import ValidationError++from pipelex.exceptions import PipelexException+from pipelex.kit.index_models import KitIndex+from pipelex.kit.paths import get_kit_root+from pipelex.tools.misc.toml_utils import load_toml_from_path+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+++class KitIndexLoadingError(PipelexException):+ pass+++def load_index() -> KitIndex:+ """Load and validate the kit index.toml configuration.++ Returns:+ Validated KitIndex model++ Raises:+ TomlError: If TOML parsing fails+ KitIndexLoadingError: If validation fails+ """+ index_path = get_kit_root() / "index.toml"+ data = load_toml_from_path(str(index_path))+ try:+ return KitIndex.model_validate(data)+ except ValidationError as exc:+ msg = f"Validation error in kit index at '{index_path}': {format_pydantic_validation_error(exc)}"+ raise KitIndexLoadingError(message=msg) from excdiff --git a/pipelex/kit/index_models.py b/pipelex/kit/index_models.pynew file mode 100644index 000000000..a74881181--- /dev/null+++ b/pipelex/kit/index_models.py@@ -0,0 +1,45 @@+from typing import Any++from pydantic import Field++from pipelex.config import ConfigModel+++class CursorFileOverride(ConfigModel):+ """Per-file front-matter overrides for Cursor export."""++ front_matter: dict[str, Any] = Field(default_factory=dict, description="Front-matter to override for this file")+++class CursorSpec(ConfigModel):+ """Configuration for Cursor rules export."""++ front_matter: dict[str, Any] = Field(default_factory=dict, description="Default YAML front-matter for all Cursor files")+ files: dict[str, CursorFileOverride] = Field(default_factory=dict, description="Per-file front-matter overrides")+++class Target(ConfigModel):+ """Configuration for a single-file merge target."""++ path: str = Field(description="Path to the target file relative to repo root")+ strategy: str = Field(description="Merge strategy (currently only 'merge' supported)")+ marker_begin: str = Field(description="Beginning marker for content insertion")+ marker_end: str = Field(description="Ending marker for content insertion")+ heading_1: str | None = Field(default=None, description="Main title (H1) to add when inserting into empty file or file with no H1 headings")+++class AgentRules(ConfigModel):+ """Configuration for merging agent documentation files."""++ sets: dict[str, list[str]] = Field(description="Named sets of agent_rules files (e.g., coding_standards, pipelex_language, all)")+ default_set: str = Field(default="pipelex_language", description="Default set to use when syncing")+ demote: int = Field(default=1, description="Number of levels to demote headings when merging")+ cursor: CursorSpec = Field(description="Cursor rules export configuration")+ targets: dict[str, Target] = Field(description="Dictionary of single-file merge targets keyed by ID")+++class KitIndex(ConfigModel):+ """Root configuration model for kit index.toml."""++ meta: dict[str, Any] = Field(default_factory=dict, description="Metadata about the kit configuration")+ agent_rules: AgentRules = Field(description="Agent documentation merge configuration")diff --git a/pipelex/kit/markers.py b/pipelex/kit/markers.pynew file mode 100644index 000000000..26048fcb1--- /dev/null+++ b/pipelex/kit/markers.py@@ -0,0 +1,51 @@+def find_span(text: str, begin: str, end: str) -> tuple[int, int] | None:+ """Find the span of text between begin and end markers.++ Args:+ text: Text to search in+ begin: Beginning marker string+ end: Ending marker string++ Returns:+ Tuple of (start, end) indices if both markers found, None otherwise+ The end index includes the end marker itself+ """+ start = text.find(begin)+ if start == -1:+ return None++ end_pos = text.find(end, start)+ if end_pos == -1:+ return None++ end_pos += len(end)+ return (start, end_pos)+++def wrap(begin: str, end: str, content: str) -> str:+ """Wrap content with begin and end markers.++ Args:+ begin: Beginning marker+ end: Ending marker+ content: Content to wrap++ Returns:+ Wrapped content with markers and newlines+ """+ return f"{begin}\n{content.rstrip()}\n{end}"+++def replace_span(text: str, span: tuple[int, int], replacement: str) -> str:+ """Replace the text at the given span with replacement.++ Args:+ text: Original text+ span: Tuple of (start, end) indices+ replacement: Replacement text++ Returns:+ Text with span replaced by replacement+ """+ start, end = span+ return text[:start] + replacement + text[end:]diff --git a/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md b/pipelex/kit/migrations/migrate_0.11.0_0.12.0.mdnew file mode 100644index 000000000..3bbc88aed--- /dev/null+++ b/pipelex/kit/migrations/migrate_0.11.0_0.12.0.md@@ -0,0 +1,1161 @@+# Migration Guide - Breaking Changes++This guide will help you migrate your Pipelex pipelines and configurations to the latest version.++## Overview++This release introduces several breaking changes to make the Pipelex language more declarative, intuitive, and consistent. The changes affect:+- Project structure and organization+- Pipeline definitions (.plx files)+- Configuration files (.pipelex/ directory)+- Python code initialization+- Python import paths (module refactoring)+- Test markers++## Migration Checklist++- [ ] **Migrate from pipelex_libraries system (CRITICAL)**+- [ ] Move .plx files to appropriate locations in your project+- [ ] Update Pipelex.make() calls (remove config path parameters)+- [ ] Add @pipe_func() decorators to custom functions used in PipeFunc operators+- [ ] Update PipeCompose (formerly PipeJinja2)+- [ ] Update PipeExtract (formerly PipeOCR)+- [ ] Update PipeLLM prompts and fields+- [ ] Update PipeImgGen fields+- [ ] Update PipeCondition fields+- [ ] Update configuration files+- [ ] Update test markers+- [ ] **Update Python imports for StuffContent subclasses**+- [ ] **Update imports for core module relocations (pipe_works, pipe_input, etc.)**+- [ ] **Update all OCR-related imports to Extract equivalents**+- [ ] **Update Provider → Library class names and hub method calls**+- [ ] Run validation++## 1. Library System Removal (CRITICAL)++The centralized `pipelex_libraries` folder system has been removed in favor of automatic pipeline discovery throughout your project.++### Key Changes++1. **No more `pipelex init libraries` command**+2. **No centralized `pipelex_libraries` directory required**+3. **Pipelines are auto-discovered** from anywhere in your project+4. **No config path parameters** needed in commands or code+5. **Custom functions require `@pipe_func()` decorator**+6. **Structure classes are auto-discovered**++### Step 1: Move Pipeline Files (Flexible Organization)++**The key change:** `.plx` files can now live ANYWHERE in your project. No special directory required!++**Recommendation:** Put `.plx` files with related code. If you have topic-based organization, keep pipelines with their topics.++**Example Migration Patterns:**++**Pattern A: Topic-Based (Recommended if you have domain modules)**+```+Before:+pipelex_libraries/pipelines/+├── finance.plx+├── finance.py+├── legal.plx+└── legal.py++After - Keep with related code:+my_project/+├── finance/+│ ├── models.py+│ ├── services.py+│ ├── invoices.plx # Pipeline with finance code+│ └── invoices_struct.py # Structure classes+└── legal/+ ├── models.py+ ├── services.py+ ├── contracts.plx # Pipeline with legal code+ └── contracts_struct.py+```++**Pattern B: Centralized Pipelines (If you prefer grouping)**+```+After - Group pipelines together:+my_project/+├── pipelines/+│ ├── finance.plx+│ ├── finance_struct.py+│ ├── legal.plx+│ └── legal_struct.py+└── core/+ └── (your other code)+```++**Pattern C: Flat (Small projects)**+```+After - Just put them in your source directory:+my_project/+├── finance_pipeline.plx+├── finance_struct.py+└── main.py+```++**Action Items:**++1. **Choose your organization** (any of the above patterns work)++2. **Move .plx files** to where they make sense for YOUR project:+ ```bash+ # Example: Moving to topic-based structure+ mv pipelex_libraries/pipelines/finance.plx my_project/finance/+ ```++3. **Rename structure files** with `_struct.py` suffix:+ ```bash+ # Example+ mv my_project/finance/finance.py my_project/finance/finance_struct.py+ ```++4. **Clean up:**+ ```bash+ # After all files are moved+ rm -rf pipelex_libraries/++ # Remove from .gitignore if present+ sed -i '/^\/pipelex_libraries$/d' .gitignore+ ```++**Remember:** Configuration (`.pipelex/`) stays at repository root.++### Step 2: Update Python Code Initialization++**Before:**+```python+from pipelex.pipelex import Pipelex++pipelex_instance = Pipelex.make(+ relative_config_folder_path="../../../pipelex/libraries",+ from_file=True+)+```++**After:**+```python+from pipelex.pipelex import Pipelex++# No path needed - automatic discovery+pipelex_instance = Pipelex.make()+```++**Find and replace in all Python files:**+- Remove `relative_config_folder_path` parameter+- Remove `config_folder_path` parameter+- Remove `from_file` parameter++### Step 3: Update Custom Functions++All custom functions used in `PipeFunc` operators must now have the `@pipe_func()` decorator for auto-discovery.++**Before:**+```python+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.text_content import TextContent++async def my_custom_function(working_memory: WorkingMemory) -> TextContent:+ input_data = working_memory.get_stuff("input_name")+ return TextContent(text=f"Processed: {input_data.content.text}")+```++**After:**+```python+from pipelex.system.registries.func_registry import pipe_func+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.text_content import TextContent++@pipe_func() # Add this decorator+async def my_custom_function(working_memory: WorkingMemory) -> TextContent:+ input_data = working_memory.get_stuff("input_name")+ return TextContent(text=f"Processed: {input_data.content.text}")++# Optional: specify a custom name+@pipe_func(name="custom_processor")+async def another_function(working_memory: WorkingMemory) -> TextContent:+ # Implementation+ pass+```++### Step 4: Update CLI Commands++**Before:**+```bash+# You had to specify config folder path+pipelex validate all -c path/to/pipelex/libraries+pipelex build blueprint "..." -c your/path/to/pipelex/libraries+```++**After:**+```bash+# No config path needed - automatic discovery+pipelex validate all+pipelex build blueprint "..."+```++### Step 5: Update Imports in Python Code++Update imports from the old library structure:++**Before:**+```python+from pipelex.libraries.pipelines.finance import Invoice, InvoiceData+```++**After:**+```python+# Import from your own project structure+from my_project.pipelines.finance_struct import Invoice, InvoiceData+```++### Step 6: Update Concept References (Optional)++While domain-prefixed concept references still work, you can now use simpler references:++**Before:**+```plx+inputs = { prompt = "images.ImgGenPrompt" }+inputs = { wedding_photo = "images.Photo" }+```++**After:**+```plx+# Simpler references (domain prefix optional)+inputs = { prompt = "ImgGenPrompt" }+inputs = { wedding_photo = "Photo" }+```++### Auto-Discovery Explained++**The big change:** Pipelex now scans your entire project and finds:++- **`.plx` files** - Pipeline definitions (wherever they are!)+- **Structure classes** - Classes inheriting from `StructuredContent`+- **Custom functions** - Functions decorated with `@pipe_func()`++**This means:**+- No special `pipelex_libraries/pipelines/` folder needed+- Put `.plx` files where they logically belong in YOUR codebase+- Keep related things together (pipelines with their code)++**Excluded directories** (automatically skipped):+- `.venv`, `.git`, `__pycache__`+- `.pytest_cache`, `.mypy_cache`, `.ruff_cache`+- `node_modules`, `.env`, `results`++### Troubleshooting++**Issue: Pipelines not found**++Solution: Ensure `.plx` files are not in excluded directories and run:+```bash+pipelex show pipes # See what was discovered+```++**Issue: Structure classes not registered**++Solution:+1. Ensure classes inherit from `StructuredContent`+2. Check class names match concept names exactly+3. Use `_struct.py` suffix for structure files++**Issue: Custom functions not found**++Solution:+1. Add `@pipe_func()` decorator+2. Ensure function is `async` and accepts `working_memory`+3. Verify function is in a discoverable location++## 2. General Changes++### Rename `definition` to `description`++**Find:** `definition = "`+**Replace with:** `description = "`++This applies to all pipe types.++**Before:**+```plx+[pipe.example]+type = "PipeLLM"+definition = "Process data"+```++**After:**+```plx+[pipe.example]+type = "PipeLLM"+description = "Process data"+```++## 3. PipeCompose (formerly PipeJinja2)++### Rename pipe type++**Find:** `type = "PipeJinja2"`+**Replace with:** `type = "PipeCompose"`++### Rename template fields++**Find:** `jinja2 = `+**Replace with:** `template = `++**Find:** `jinja2_name = `+**Replace with:** `template_name = `++**Before:**+```plx+[pipe.compose_report]+type = "PipeJinja2"+description = "Compose a report"+inputs = { data = "ReportData" }+output = "Text"+jinja2 = """+Report: $data+"""+```++**After:**+```plx+[pipe.compose_report]+type = "PipeCompose"+description = "Compose a report"+inputs = { data = "ReportData" }+output = "Text"+template = """+Report: $data+"""+```++### Nested template section (optional)++If you need more control, you can now use a nested template section:++**Before:**+```plx+[pipe.example]+type = "PipeJinja2"+jinja2 = "Template content"+template_category = "html"+```++**After:**+```plx+[pipe.example]+type = "PipeCompose"++[pipe.example.template]+template = "Template content"+category = "html"+templating_style = { tag_style = "square_brackets", text_format = "html" }+```++## 4. PipeExtract (formerly PipeOCR)++### Rename pipe type++**Find:** `type = "PipeOCR"`+**Replace with:** `type = "PipeExtract"`++### Rename model field++**Find:** `ocr_model = `+**Replace with:** `model = `++### Input naming++The input no longer needs to be named `ocr_input`. You can name it anything as long as it's a single input that is either an `Image` or a `PDF`.++**Before:**+```plx+[pipe.extract_info]+type = "PipeOCR"+description = "Extract text from document"+inputs = { ocr_input = "PDF" }+output = "Page"+ocr_model = "mistral-ocr"+```++**After:**+```plx+[pipe.extract_info]+type = "PipeExtract"+description = "Extract text from document"+inputs = { document = "PDF" }+output = "Page"+model = "base_extract_mistral"+```++### Python function renames++If you're using these functions in Python code:++**Find:** `ocr_page_contents_from_pdf`+**Replace with:** `extract_page_contents_from_pdf`++**Find:** `ocr_page_contents_and_views_from_pdf`+**Replace with:** `extract_page_contents_and_views_from_pdf`++## 5. PipeLLM Changes++### Rename prompt field++**Find:** `prompt_template = `+**Replace with:** `prompt = `++### Rename model fields++**Find:** `llm = `+**Replace with:** `model = `++**Find:** `llm_to_structure = `+**Replace with:** `model_to_structure = `++### Tag image inputs in prompts++Image inputs must now be explicitly tagged in the prompt using `$image_name` or `@image_name`.++**Before:**+```plx+[pipe.analyze_image]+type = "PipeLLM"+description = "Analyze image"+inputs = { image = "Image" }+output = "ImageAnalysis"+prompt_template = "Describe what you see in this image"+```++**After:**+```plx+[pipe.analyze_image]+type = "PipeLLM"+description = "Analyze image"+inputs = { image = "Image" }+output = "ImageAnalysis"+prompt = """+Describe what you see in this image:++$image+"""+```++You can also reference images inline:+```plx+prompt = "Analyze the colors in $photo and the shapes in $painting."+```++**Complete example:**++**Before:**+```plx+[pipe.extract_info]+type = "PipeLLM"+definition = "Extract information"+inputs = { text = "Text" }+output = "PersonInfo"+llm = { llm_handle = "gpt-4o", temperature = 0.1 }+prompt_template = """+Extract person information from this text:+@text+"""+```++**After:**+```plx+[pipe.extract_info]+type = "PipeLLM"+description = "Extract information"+inputs = { text = "Text" }+output = "PersonInfo"+model = { model = "gpt-4o", temperature = 0.1 }+prompt = """+Extract person information from this text:+@text+"""+```++## 6. PipeImgGen Changes++### Rename model field++**Find:** `img_gen = `+**Replace with:** `model = `++### Remove technical settings from pipe level++Settings like `nb_steps` and `guidance_scale` should now be configured in model settings or presets, not at the pipe level.++**Before:**+```plx+[pipe.generate_photo]+type = "PipeImgGen"+description = "Generate a photo"+inputs = { prompt = "ImgGenPrompt" }+output = "Photo"+img_gen = { img_gen_handle = "fast-img-gen", quality = "hd" }+aspect_ratio = "16:9"+nb_steps = 8+```++**After:**+```plx+[pipe.generate_photo]+type = "PipeImgGen"+description = "Generate a photo"+inputs = { prompt = "ImgGenPrompt" }+output = "Photo"+model = { model = "fast-img-gen" }+aspect_ratio = "16:9"+quality = "hd"+```++Or use a preset:+```plx+model = "img_gen_preset_name"+```++## 7. PipeCondition Changes++### Rename outcome fields++**Find:** `[pipe.your_pipe.pipe_map]`+**Replace with:** `[pipe.your_pipe.outcomes]`++**Find:** `default_pipe_code = `+**Replace with:** `default_outcome = `++### Add required default_outcome++The `default_outcome` field is now **required**. If you don't want any default behavior, use `"fail"`.++**Before:**+```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "Decide which pipe to run"+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression = "input_data.category"++[pipe.conditional_operation.pipe_map]+small = "process_small"+medium = "process_medium"+large = "process_large"+```++**After:**+```plx+[pipe.conditional_operation]+type = "PipeCondition"+description = "Decide which pipe to run"+inputs = { input_data = "CategoryInput" }+output = "native.Text"+expression = "input_data.category"+default_outcome = "process_medium"++[pipe.conditional_operation.outcomes]+small = "process_small"+medium = "process_medium"+large = "process_large"+```++To fail when no match:+```plx+default_outcome = "fail"+```++## 8. Configuration Files (.pipelex/ directory)++### LLM presets in deck files++**Find:** `llm_handle = `+**Replace with:** `model = `++**Before (.pipelex/inference/deck/base_deck.toml):**+```toml+[presets.llm]+llm_to_reason = { llm_handle = "claude-3-5-sonnet", temperature = 1 }+```++**After:**+```toml+[presets.llm]+llm_to_reason = { model = "claude-3-5-sonnet", temperature = 1 }+```++### Image generation presets++**Find:** `img_gen_handle = `+**Replace with:** `model = `++**Before:**+```toml+[presets.img_gen]+fast_gen = { img_gen_handle = "fast-img-gen", quality = "standard" }+```++**After:**+```toml+[presets.img_gen]+fast_gen = { model = "fast-img-gen", quality = "standard" }+```++### Extract presets (formerly OCR)++**Find:** `ocr_handle = `+**Replace with:** `model = `++**Find:** `[presets.ocr]`+**Replace with:** `[presets.extract]`++**Find:** `base_ocr_pypdfium2`+**Replace with:** `base_extract_pypdfium2`++**Find:** `base_ocr_mistral`+**Replace with:** `base_extract_mistral`++**Before:**+```toml+[presets.ocr]+base_ocr_mistral = { ocr_handle = "mistral-ocr" }+```++**After:**+```toml+[presets.extract]+base_extract_mistral = { model = "mistral-ocr" }+```++### pipelex.toml++**Find:** `ocr_config`+**Replace with:** `extract_config`++**Find:** `is_auto_setup_preset_ocr`+**Replace with:** `is_auto_setup_preset_extract`++**Find:** `nb_ocr_pages`+**Replace with:** `nb_extract_pages`++**Before (.pipelex/pipelex.toml):**+```toml+[ocr_config]+is_auto_setup_preset_ocr = true+nb_ocr_pages = 10+```++**After:**+```toml+[extract_config]+is_auto_setup_preset_extract = true+nb_extract_pages = 10+```++## 9. Test Markers++### Update pytest markers++**Find:** `@pytest.mark.ocr`+**Replace with:** `@pytest.mark.extract`++**Before:**+```python+class TestOCRPipeline:+ async def test_extract(self):+ # test code+```++**After:**+```python+class TestExtractPipeline:+ async def test_extract(self):+ # test code+```++### Update test markers in pytest.ini or pyproject.toml++**Find:** `ocr: uses OCR`+**Replace with:** `extract: uses text/image extraction from documents`++### Update make commands++**Find:** `make test-ocr` or `make to`+**Replace with:** `make test-extract` or `make te`++## 10. Validation++After making changes, thoroughly test your migration:++### Activate Virtual Environment++```bash+# Activate your virtual environment first+source .venv/bin/activate # Unix/macOS+# or+.venv\Scripts\activate # Windows+```++### Validation Steps++1. **Validate pipeline syntax:**+ ```bash+ pipelex validate all+ ```++2. **Check specific pipes:**+ ```bash+ pipelex show pipes # List all discovered pipes+ pipelex show pipe YOUR_PIPE_CODE # Inspect specific pipe+ ```++3. **Run your test suite:**+ ```bash+ pytest tests/+ # or if using make:+ make test+ ```++4. **Test pipeline execution:**+ - Run your application+ - Execute example pipelines+ - Verify outputs are as expected++5. **Check for issues:**+ - Review any validation errors+ - Check imports are working+ - Verify structure classes are discovered+ - Confirm custom functions are registered++## 11. Python API Changes for Client Projects++These changes affect Python code that imports from or uses pipelex.++### Refactoring of `StuffContent` Subclasses++The monolithic `pipelex.core.stuffs.stuff_content` module has been split into individual files for each core content type. This improves modularity and clarity.++**You must update your imports for these classes:**++| Class Name | Old Import Path | New Import Path |+| :--- | :--- | :--- |+| `StructuredContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.structured_content` |+| `TextContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.text_content` |+| `ImageContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.image_content` |+| `ListContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.list_content` |+| `PDFContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.pdf_content` |+| `PageContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.page_content` |+| `NumberContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.number_content` |+| `HtmlContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.html_content` |+| `MermaidContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.mermaid_content` |+| `TextAndImagesContent` | `pipelex.core.stuffs.stuff_content` | `pipelex.core.stuffs.text_and_images_content` |++**Example Migration:**++**Before:**+```python+from pipelex.core.stuffs.stuff_content import StructuredContent, TextContent, ImageContent++class MyData(StructuredContent):+ text_field: TextContent+ image_field: ImageContent+```++**After:**+```python+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.core.stuffs.image_content import ImageContent++class MyData(StructuredContent):+ text_field: TextContent+ image_field: ImageContent+```++### Core Module Relocations++Several core modules related to pipe execution and input handling have been moved to more logical packages.++#### Pipe Execution (`pipe_works` → `pipe_run`)++The `pipelex.pipe_works` package has been renamed to `pipelex.pipe_run`. Additionally, `PipeRunParams` and its factory have been moved into this new package.++| Old Path | New Path | Description |+| :--- | :--- | :--- |+| `pipelex.core.pipes.pipe_run_params` | `pipelex.pipe_run.pipe_run_params` | Contains `PipeRunParams`, `PipeRunMode`, etc. |+| `pipelex.core.pipes.pipe_run_params_factory` | `pipelex.pipe_run.pipe_run_params_factory` | Factory for creating `PipeRunParams`. |+| `pipelex.pipe_works.pipe_router_protocol` | `pipelex.pipe_run.pipe_router_protocol` | The abstract protocol for the pipe router. |++**Example Migration:**++**Before:**+```python+from pipelex.core.pipes.pipe_run_params import PipeRunParams+from pipelex.pipe_works.pipe_router_protocol import PipeRouterProtocol+```++**After:**+```python+from pipelex.pipe_run.pipe_run_params import PipeRunParams+from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol+```++#### Pipe Input Specifications (`pipe_input` → `input_requirements`)++Modules for defining pipe inputs have been renamed for clarity. The main class `PipeInputSpec` is now `InputRequirements`.++| Old Path | New Path | Description |+| :--- | :--- | :--- |+| `pipelex.core.pipes.pipe_input` | `pipelex.core.pipes.input_requirements` | Contains `InputRequirements` (formerly `PipeInputSpec`). |+| `pipelex.core.pipes.pipe_input_blueprint` | `pipelex.core.pipes.input_requirement_blueprint` | Contains `InputRequirementBlueprint`. |+| `pipelex.core.pipes.pipe_input_factory` | `pipelex.core.pipes.input_requirements_factory` | Contains `InputRequirementsFactory`. |++**Example Migration:**++**Before:**+```python+from pipelex.core.pipes.pipe_input import PipeInputSpec+from pipelex.core.pipes.pipe_input_factory import PipeInputSpecFactory+```++**After:**+```python+from pipelex.core.pipes.input_requirements import InputRequirements+from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory+```++#### Comprehensive OCR → Extract Renaming++All modules, classes, and configurations related to `ocr` have been renamed to `extract` to better reflect capabilities beyond OCR.++| Old Path / Name | New Path / Name | Description |+| :--- | :--- | :--- |+| `pipelex.cogt.ocr` | `pipelex.cogt.extract` | Main package for extraction logic. |+| `pipelex.pipe_operators.ocr` | `pipelex.pipe_operators.extract` | Package for the `PipeExtract` operator. |+| `pipelex.cogt.ocr.ocr_input` | `pipelex.cogt.extract.extract_input` | Contains `ExtractInput` (formerly `OcrInput`). |+| `pipelex.cogt.ocr.ocr_job` | `pipelex.cogt.extract.extract_job` | Contains `ExtractJob` (formerly `OcrJob`). |+| `pipelex.cogt.ocr.ocr_worker_abstract` | `pipelex.cogt.extract.extract_worker_abstract` | Contains `ExtractWorkerAbstract`. |++**Example Migration:**++**Before:**+```python+from pipelex.cogt.ocr.ocr_input import OcrInput+from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract+```++**After:**+```python+from pipelex.cogt.extract.extract_input import ExtractInput+from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract+```++### Renamed Base Library Pipes++**Find:** `ocr_page_contents_from_pdf`+**Replace with:** `extract_page_contents_from_pdf`++**Find:** `ocr_page_contents_and_views_from_pdf`+**Replace with:** `extract_page_contents_and_views_from_pdf`++**Before:**+```python+pipe_output = await execute_pipeline(+ pipe_code="ocr_page_contents_from_pdf",+ input_memory={+ "ocr_input": PDFContent(url=pdf_url),+ },+)+```++**After:**+```python+pipe_output = await execute_pipeline(+ pipe_code="extract_page_contents_from_pdf",+ input_memory={+ "document": PDFContent(url=pdf_url),+ },+)+```++### Builder Module Relocation++The pipeline builder has been promoted to a top-level package:++**Old:** `pipelex.libraries.pipelines.builder`+**New:** `pipelex.builder`++**Note:** This change only affects internal Pipelex code. Most users won't need to update anything related to the builder module.++### Core Abstractions (`Provider` → `Library`)++The abstract base classes for core components have been renamed for consistency. Hub accessor methods have been updated accordingly.++| Old Class Name | New Class Name | Hub Accessor Method |+| :--- | :--- | :--- |+| `ConceptProviderAbstract` | `ConceptLibraryAbstract` | `get_concept_library()` |+| `DomainProviderAbstract` | `DomainLibraryAbstract` | `get_domain_library()` |+| `PipeProviderAbstract` | `PipeLibraryAbstract` | `get_pipe_library()` |++**Example Migration:**++**Before:**+```python+from pipelex.core.concepts.concept_provider_abstract import ConceptProviderAbstract+concept_provider = hub.get_concept_provider()+```++**After:**+```python+from pipelex.core.concepts.concept_library_abstract import ConceptLibraryAbstract+concept_library = hub.get_concept_library()+```++### Templating Refactoring++The `pipelex.tools.templating` package has been refactored:++- Core logic now resides in `pipelex.cogt.templating`+- Jinja2-specific utilities are in `pipelex.tools.jinja2`++**Example Migration:**++**Before:**+```python+from pipelex.tools.templating.template_processor import TemplateProcessor+```++**After:**+```python+from pipelex.cogt.templating.template_processor import TemplateProcessor+```++### Removed Methods and Classes++The following methods and classes have been removed. If your code uses them, you'll need to refactor:++- `PipeLibrary.add_or_update_pipe()` - Removed+- `PipelexHub.get_optional_library_manager()` - Removed+- Hub methods: `get_optional_domain_provider()` and `get_optional_concept_provider()` - Removed++### Hub Method Renames++If you use hub methods directly:++**Find:** `get_*_provider()`+**Replace with:** `get_*_library()`++**Find:** `set_*_provider()`+**Replace with:** `set_*_library()`++### External Plugin API Changes++If you're using external LLM plugins:++**Find:** `llm_handle` parameter+**Replace with:** `model` parameter++**Before:**+```python+get_inference_manager().set_llm_worker_from_external_plugin(+ llm_handle="my_custom_llm",+ llm_worker_class=MyLLMWorker,+)+```++**After:**+```python+get_inference_manager().set_llm_worker_from_external_plugin(+ model="my_custom_llm",+ llm_worker_class=MyLLMWorker,+)+```++## 12. File Cleanup++### Remove Deprecated Files++Remove the following files if they exist in your project:++```bash+# Remove old template file (moved to .pipelex/pipelex.toml)+rm -f pipelex_libraries/templates/base_templates.toml+rm -rf pipelex_libraries/templates/ # If empty after removal+```++### Update Documentation Files++If your project has `AGENTS.md` or `CLAUDE.md` files with Pipelex examples:++1. Update all PLX syntax examples following sections 1-8 of this guide+2. Update Python code examples following section 10+3. Search for and update:+ - `ocr_page_contents_from_pdf` → `extract_page_contents_from_pdf`+ - `type = "PipeOcr"` → `type = "PipeExtract"`+ - `ocr_model` → `model`+ - `llm = ` → `model = `+ - `prompt_template = ` → `prompt = `++## 13. Common Issues++### Issue: Pipeline validation fails with "unknown field"++**Cause:** You may have used an old field name (e.g., `prompt_template`, `jinja2`, `llm`, `ocr_model`).++**Solution:** Search your .plx files for the old field names and replace them according to this guide.++### Issue: Tests fail with marker errors++**Cause:** Test markers haven't been updated from `ocr` to `extract`.++**Solution:** Update all `@pytest.mark.ocr` to `@pytest.mark.extract`.++### Issue: Configuration not loading++**Cause:** Configuration files still use old section names (e.g., `[presets.ocr]`).++**Solution:** Rename sections and fields in your .pipelex/ configuration files.++### Issue: Import errors for renamed classes++**Cause:** Code imports classes that were renamed (e.g., `ConceptProviderAbstract`).++**Solution:** Update imports to use new names (`ConceptLibraryAbstract`, etc.) or refactor to avoid using internal classes.++### Issue: base_templates.toml not found++**Cause:** The `base_templates.toml` file has been removed. Generic prompts moved to `.pipelex/pipelex.toml`.++**Solution:** Remove references to this file. The templates are now auto-loaded from the config.++### Issue: ImportError for StuffContent subclasses++**Cause:** Imports still use the old monolithic `pipelex.core.stuffs.stuff_content` module.++**Solution:** Update imports to use individual modules. See Section 11 for the complete mapping table.++**Example:**+```python+# Old (will fail)+from pipelex.core.stuffs.stuff_content import StructuredContent, TextContent++# New (correct)+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.text_content import TextContent+```++### Issue: ImportError for pipe execution classes++**Cause:** Code imports from old `pipelex.pipe_works` package or old pipe input modules.++**Solution:** Update to use new package names:+- `pipelex.pipe_works` → `pipelex.pipe_run`+- `pipelex.core.pipes.pipe_input` → `pipelex.core.pipes.input_requirements`+- `PipeInputSpec` → `InputRequirements`++### Issue: ImportError for OCR-related classes++**Cause:** Code still imports from `pipelex.cogt.ocr` or `pipelex.pipe_operators.ocr`.++**Solution:** Update all OCR imports to Extract:+- `pipelex.cogt.ocr` → `pipelex.cogt.extract`+- `pipelex.pipe_operators.ocr` → `pipelex.pipe_operators.extract`+- All class names: `Ocr*` → `Extract*`++## 14. Automation Tools++You can automate many of these text replacements using standard tools available on your platform:++### Available Tools by Platform++**Unix/Linux/macOS:**+- `sed` - Stream editor for find/replace in files+- `find` - Locate files and execute commands on them+- `grep` - Search for patterns in files++**Windows:**+- PowerShell's `Get-Content` and `-replace` operator+- Git Bash (includes Unix tools)+- WSL (Windows Subsystem for Linux)++### What Can Be Automated++The following replacements can be done with find/replace tools:++**In `.plx` files:**+- `definition = "` → `description = "`+- `type = "PipeJinja2"` → `type = "PipeCompose"`+- `type = "PipeOCR"` → `type = "PipeExtract"`+- `prompt_template = ` → `prompt = `+- `jinja2 = ` → `template = `+- `ocr_model = ` → `model = `+- `[pipe.X.pipe_map]` → `[pipe.X.outcomes]`+- `default_pipe_code = ` → `default_outcome = `++**In `.py` files:**+- `ocr_page_contents_from_pdf` → `extract_page_contents_from_pdf`+- Remove `relative_config_folder_path` parameters from `Pipelex.make()`+- Remove `config_folder_path` parameters from `Pipelex.make()`+- `from pipelex.core.stuffs.stuff_content import` → Update to specific module imports+- `from pipelex.pipe_works` → `from pipelex.pipe_run`+- `from pipelex.core.pipes.pipe_input` → `from pipelex.core.pipes.input_requirements`+- `from pipelex.cogt.ocr` → `from pipelex.cogt.extract`+- `PipeInputSpec` → `InputRequirements`+- `ConceptProviderAbstract` → `ConceptLibraryAbstract`+- `DomainProviderAbstract` → `DomainLibraryAbstract`+- `PipeProviderAbstract` → `PipeLibraryAbstract`+- `.get_*_provider()` → `.get_*_library()`+- `.set_*_provider()` → `.set_*_library()`++**In `.toml` files:**+- `llm_handle = ` → `model = `+- `img_gen_handle = ` → `model = `+- `ocr_handle = ` → `model = `+- `[presets.ocr]` → `[presets.extract]`+- `base_ocr_*` → `base_extract_*`++**In test files:**+- `@pytest.mark.ocr` → `@pytest.mark.extract`++### What CANNOT Be Automated++These require manual intervention:++1. Moving `.plx` files to appropriate locations (project-specific)+2. Renaming structure files to `*_struct.py` suffix+3. Adding `@pipe_func()` decorator to custom functions+4. Updating imports to match your new structure+5. Splitting `StuffContent` imports into individual module imports (requires analyzing which classes are used)+6. Adding `default_outcome` to `PipeCondition` pipes+7. Tagging image inputs in `PipeLLM` prompts with `$` or `@`+8. Reviewing and testing all changes++### Recommendation++1. **Test incrementally:** Apply changes to one file type at a time+2. **Use version control:** Commit before migrating so you can revert if needed+3. **Activate your virtual environment** before running Pipelex commands+4. **Validate after each change** (see Validation section)++## 15. Additional Resources++- See AGENTS.md for complete documentation of the current syntax+- Run `make validate` frequently to catch issues early+- Check the test files in `tests/test_pipelines/` for examples of the new syntax++## Support++If you encounter issues during migration:+1. Check that all old field names have been replaced+2. Run `make validate` to see specific error messages+3. Review the examples in AGENTS.md+4. Check that required fields like `default_outcome` are present+diff --git a/pipelex/kit/migrations_export.py b/pipelex/kit/migrations_export.pynew file mode 100644index 000000000..792124310--- /dev/null+++ b/pipelex/kit/migrations_export.py@@ -0,0 +1,28 @@+from pathlib import Path++import typer++from pipelex.kit.paths import get_migrations_dir+++def export_migration_instructions(repo_root: Path, dry_run: bool = False) -> None:+ """Export migration documentation files to user's .pipelex/migrations directory.++ Args:+ repo_root: Repository root directory+ dry_run: If True, only print what would be done+ """+ migrations_dir = get_migrations_dir()+ out_dir = repo_root / ".pipelex" / "migrations"+ out_dir.mkdir(parents=True, exist_ok=True)++ for child in migrations_dir.iterdir():+ if child.name.endswith(".md") and child.is_file():+ content = child.read_text(encoding="utf-8")+ out_path = out_dir / child.name++ if dry_run:+ typer.echo(f"[DRY] write {out_path}")+ else:+ out_path.write_text(content, encoding="utf-8")+ typer.echo(f"✅ Copied {child.name}")diff --git a/pipelex/kit/paths.py b/pipelex/kit/paths.pynew file mode 100644index 000000000..db49b0dbc--- /dev/null+++ b/pipelex/kit/paths.py@@ -0,0 +1,38 @@+from importlib.abc import Traversable+from importlib.resources import files+++def get_kit_root() -> Traversable:+ """Get the root directory of the kit package.++ Returns:+ Traversable object pointing to pipelex.kit package+ """+ return files("pipelex.kit")+++def get_agents_dir() -> Traversable:+ """Get the agents directory within the kit package.++ Returns:+ Traversable object pointing to pipelex.kit/agent_rules+ """+ return get_kit_root() / "agent_rules"+++def get_configs_dir() -> Traversable:+ """Get the configs directory within the kit package.++ Returns:+ Traversable object pointing to pipelex.kit/configs+ """+ return get_kit_root() / "configs"+++def get_migrations_dir() -> Traversable:+ """Get the migrations directory within the kit package.++ Returns:+ Traversable object pointing to pipelex.kit/migrations+ """+ return get_kit_root() / "migrations"diff --git a/pipelex/kit/targets_update.py b/pipelex/kit/targets_update.pynew file mode 100644index 000000000..3f80f530f--- /dev/null+++ b/pipelex/kit/targets_update.py@@ -0,0 +1,282 @@+import difflib+import re+from importlib.abc import Traversable+from pathlib import Path++import typer++from pipelex.kit.index_models import KitIndex, Target+from pipelex.kit.markers import find_span, replace_span, wrap+from pipelex.kit.paths import get_agents_dir+++def _read_agent_file(agents_dir: Traversable, name: str) -> str:+ """Read an agent markdown file.++ Args:+ agents_dir: Traversable pointing to agents directory+ name: Filename to read++ Returns:+ File content as string+ """+ return (agents_dir / name).read_text(encoding="utf-8")+++def _demote_headings(md_content: str, levels: int) -> str:+ """Demote all headings in markdown content by specified levels.++ Args:+ md_content: Markdown content+ levels: Number of levels to demote++ Returns:+ Markdown with demoted headings+ """+ if levels == 0:+ return md_content++ # Use regex to add extra # to ATX-style headings+ def demote_match(match: re.Match[str]) -> str:+ hashes = match.group(1)+ rest = match.group(2)+ return f"{'#' * levels}{hashes}{rest}"++ # Match lines starting with # (ATX-style headings)+ pattern = r"^(#{1,6})(.*)$"+ return re.sub(pattern, demote_match, md_content, flags=re.MULTILINE)+++# TODO: fix bug which makes it not idempotent (because heading 1 gets deleted)+def build_merged_rules(idx: KitIndex, agent_set: str | None = None) -> str:+ """Build merged agent documentation from ordered files.++ Args:+ idx: Kit index configuration+ agent_set: Name of the agent set to use (defaults to idx.agents.default_set)++ Returns:+ Merged markdown content with demoted headings+ """+ agents_dir = get_agents_dir()++ if agent_set is None:+ agent_set = idx.agent_rules.default_set++ if agent_set not in idx.agent_rules.sets:+ msg = f"Agent set '{agent_set}' not found in index.toml. Available sets: {list(idx.agent_rules.sets.keys())}"+ raise ValueError(msg)++ parts: list[str] = []++ for name in idx.agent_rules.sets[agent_set]:+ md = _read_agent_file(agents_dir, name)+ demoted = _demote_headings(md, idx.agent_rules.demote)+ parts.append(demoted.rstrip())++ return ("\n\n".join(parts)).strip() + "\n"+++def _insert_block_with_markers(target_md: str, block_md: str, main_title: str | None, markers: tuple[str, str]) -> str:+ """Insert block into target markdown using marker-based logic.++ Args:+ target_md: Existing target markdown content+ block_md: Block to insert+ main_title: Main title (H1) to add inside markers when inserting into empty file or file with no H1 headings+ markers: Tuple of (begin_marker, end_marker)++ Returns:+ Updated markdown with block inserted and markers added+ """+ marker_begin, marker_end = markers++ # Check if file is empty or has no H1 heading+ is_empty = not target_md or not target_md.strip()+ h1_pattern = r"^#\s+.+$"+ has_h1 = bool(target_md) and bool(re.search(h1_pattern, target_md, flags=re.MULTILINE))++ # If empty or no H1 heading, add main_title INSIDE the markers+ if (is_empty or not has_h1) and main_title:+ content_with_heading = f"{main_title}\n\n{block_md}"+ wrapped_block = wrap(marker_begin, marker_end, content_with_heading)+ else:+ # File already has H1 heading, don't add another one+ wrapped_block = wrap(marker_begin, marker_end, block_md)++ # Append at the end+ if is_empty:+ return wrapped_block + "\n"+ return target_md.rstrip() + "\n\n" + wrapped_block + "\n"+++def _diff(before: str, after: str, path: str) -> str:+ """Generate unified diff between before and after.++ Args:+ before: Original content+ after: Modified content+ path: File path for diff header++ Returns:+ Unified diff string+ """+ return "".join(+ difflib.unified_diff(+ before.splitlines(keepends=True),+ after.splitlines(keepends=True),+ fromfile=path,+ tofile=path,+ )+ )+++def update_targets(+ repo_root: Path,+ merged_rules: str,+ targets: dict[str, Target],+ dry_run: bool,+ diff: bool,+ backup: str | None,+) -> None:+ """Update target files with merged agent documentation.++ Args:+ repo_root: Repository root directory+ merged_rules: Merged markdown content to insert+ targets: Dictionary of target file configurations keyed by ID+ dry_run: If True, only print what would be done+ diff: If True, show unified diff+ backup: Backup suffix (e.g., ".bak"), or None for no backup+ """+ for target in targets.values():+ target_path = repo_root / target.path+ before = target_path.read_text(encoding="utf-8") if target_path.exists() else ""++ span = find_span(before, target.marker_begin, target.marker_end)++ if span:+ # Markers exist - replace content between them+ wrapped_block = wrap(target.marker_begin, target.marker_end, merged_rules)+ after = replace_span(before, span, wrapped_block)+ else:+ # No markers - insert with markers+ after = _insert_block_with_markers(+ before,+ merged_rules,+ target.heading_1,+ (target.marker_begin, target.marker_end),+ )++ if dry_run:+ typer.echo(f"[DRY] update {target_path}")+ if diff:+ diff_output = _diff(before, after, str(target_path))+ if diff_output:+ typer.echo(diff_output)+ else:+ if backup and target_path.exists():+ backup_path = target_path.with_suffix(target_path.suffix + backup)+ backup_path.write_text(before, encoding="utf-8")+ typer.echo(f"📦 Backup saved to {backup_path}")++ target_path.parent.mkdir(parents=True, exist_ok=True)+ target_path.write_text(after, encoding="utf-8")+ typer.echo(f"✅ Updated {target_path}")++ if diff:+ diff_output = _diff(before, after, str(target_path))+ if diff_output:+ typer.echo(diff_output)+++def remove_from_targets(+ repo_root: Path,+ targets: dict[str, Target],+ delete_files: bool,+ dry_run: bool,+ diff: bool,+ backup: str | None,+) -> None:+ """Remove agent documentation from target files.++ Args:+ repo_root: Repository root directory+ targets: Dictionary of target file configurations keyed by ID+ delete_files: If True, delete entire files; if False, only remove marked sections+ dry_run: If True, only print what would be done+ diff: If True, show unified diff+ backup: Backup suffix (e.g., ".bak"), or None for no backup+ """+ for target in targets.values():+ target_path = repo_root / target.path++ if not target_path.exists():+ typer.echo(f"⚠️ File {target_path} does not exist - skipping")+ continue++ if delete_files:+ # Delete the entire file+ if dry_run:+ typer.echo(f"[DRY] delete {target_path}")+ else:+ if backup:+ backup_path = target_path.with_suffix(target_path.suffix + backup)+ target_path.rename(backup_path)+ typer.echo(f"📦 Backup saved to {backup_path}")+ else:+ target_path.unlink()+ typer.echo(f"🗑️ Deleted {target_path}")+ else:+ # Remove only the marked section+ before = target_path.read_text(encoding="utf-8")+ span = find_span(before, target.marker_begin, target.marker_end)++ if not span:+ typer.echo(f"⚠️ No marked section found in {target_path} - skipping")+ continue++ # Remove the marked section entirely+ before_section = before[: span[0]].rstrip()+ after_section = before[span[1] :].lstrip()++ # If there's content before or after, join them+ if before_section and after_section:+ after = before_section + "\n\n" + after_section+ elif before_section:+ after = before_section + "\n"+ elif after_section:+ after = after_section+ else:+ # File only contained the marked section - delete the file+ if dry_run:+ typer.echo(f"[DRY] delete {target_path} (file only contained marked section)")+ else:+ if backup:+ backup_path = target_path.with_suffix(target_path.suffix + backup)+ target_path.rename(backup_path)+ typer.echo(f"📦 Backup saved to {backup_path}")+ else:+ target_path.unlink()+ typer.echo(f"🗑️ Deleted {target_path} (file only contained marked section)")+ continue++ if dry_run:+ typer.echo(f"[DRY] remove marked section from {target_path}")+ if diff:+ diff_output = _diff(before, after, str(target_path))+ if diff_output:+ typer.echo(diff_output)+ else:+ if backup:+ backup_path = target_path.with_suffix(target_path.suffix + backup)+ backup_path.write_text(before, encoding="utf-8")+ typer.echo(f"📦 Backup saved to {backup_path}")++ target_path.write_text(after, encoding="utf-8")+ typer.echo(f"✅ Removed marked section from {target_path}")++ if diff:+ diff_output = _diff(before, after, str(target_path))+ if diff_output:+ typer.echo(diff_output)diff --git a/pipelex/language/plx_config.py b/pipelex/language/plx_config.pyindex f5a8f65ea..639fb40cc 100644--- a/pipelex/language/plx_config.py+++ b/pipelex/language/plx_config.py@@ -1,4 +1,4 @@-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass PlxConfigStrings(ConfigModel):diff --git a/pipelex/language/plx_factory.py b/pipelex/language/plx_factory.pyindex 2d9fd666e..6494a979d 100644--- a/pipelex/language/plx_factory.py+++ b/pipelex/language/plx_factory.py@@ -22,6 +22,7 @@ class SectionKey(StrEnum):CONCEPT_STRUCTURE_FIELD_KEY = "structure"PIPE_TEMPLATE_FIELD_KEY = "template"+PIPE_CATEGORY_FIELD_KEY = "pipe_category"class PlxFactory:@@ -102,7 +103,7 @@ class PlxFactory:# If field ordering is provided, add fields in the specified order firstif field_ordering:for field_key in field_ordering:- if field_key in mapping and field_key != "category": # Skip category field (pipe metadata)+ if field_key in mapping and field_key != PIPE_CATEGORY_FIELD_KEY: # Skip category field (pipe metadata)field_value = mapping[field_key]if isinstance(field_value, Mapping):# Special handling for template field - create nested table instead of inline@@ -117,7 +118,7 @@ class PlxFactory:# Add any remaining fields not in the orderingfor field_key, field_value in mapping.items():- if field_key not in field_ordering and field_key != "category":+ if field_key not in field_ordering and field_key != PIPE_CATEGORY_FIELD_KEY:if isinstance(field_value, Mapping):# Special handling for template field - create nested table instead of inlineif field_key == PIPE_TEMPLATE_FIELD_KEY:@@ -132,7 +133,7 @@ class PlxFactory:# No field ordering provided, use original logicfor field_key, field_value in mapping.items():# Skip the category field as it's not needed in PLX output (pipe metadata)- if field_key == "category":+ if field_key == PIPE_CATEGORY_FIELD_KEY:continueif isinstance(field_value, Mapping):@@ -331,4 +332,5 @@ class PlxFactory:@classmethoddef make_plx_content(cls, blueprint: PipelexBundleBlueprint) -> str:blueprint_dict = blueprint.model_dump(serialize_as_any=True)+ # blueprint_dict = cls._remove_pipe_category_from_pipes(blueprint_dict)return cls.dict_to_plx_styled_toml(data=blueprint_dict)diff --git a/pipelex/libraries/library_config.py b/pipelex/libraries/library_config.pydeleted file mode 100644index e91e624e3..000000000--- a/pipelex/libraries/library_config.py+++ /dev/null@@ -1,51 +0,0 @@-from pathlib import Path-from typing import ClassVar--from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.misc.file_utils import copy_file_from_package, copy_folder_from_package--PIPELEX_LIBRARIES_PATH = "libraries"---class LibraryConfig(ConfigModel):- package_name: ClassVar[str] = "pipelex"- config_dir_path: str = "pipelex_libraries"-- @property- def pipelines_dir_path(self) -> str:- return f"{self.config_dir_path}/pipelines"-- @property- def base_pipelines_dir_path(self) -> str:- return f"{self.config_dir_path}/pipelines/base_library"-- @property- def test_pipelines_dir_path(self) -> str:- return "tests/test_pipelines"-- @property- def failing_pipelines_file_paths(self) -> set[Path]:- return {Path("tests/test_pipelines/failing_pipelines.plx")}-- def export_libraries(self, overwrite: bool = False) -> None:- """Duplicate pipelex libraries files in the client project, preserving directory structure."""- copy_file_from_package(- package_name=self.package_name,- file_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/__init__.py",- target_path=f"{self.config_dir_path}/__init__.py",- overwrite=overwrite,- )-- # pipelines- copy_folder_from_package(- package_name=self.package_name,- folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines",- target_dir=self.base_pipelines_dir_path,- overwrite=overwrite,- )- copy_file_from_package(- package_name=self.package_name,- file_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines/__init__.py",- target_path=f"{self.pipelines_dir_path}/__init__.py",- overwrite=overwrite,- )diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 174ac4e53..8a55f93f1 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -5,7 +5,6 @@ from pydantic import ValidationErrorfrom typing_extensions import overridefrom pipelex import log-from pipelex.config import get_configfrom pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.concepts.concept_factory import ConceptFactory@@ -18,6 +17,8 @@ from pipelex.core.interpreter import PipelexInterpreterfrom pipelex.core.pipes.pipe_abstract import PipeAbstractfrom pipelex.core.pipes.pipe_factory import PipeFactoryfrom pipelex.core.pipes.pipe_library import PipeLibrary+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.validation import report_validation_errorfrom pipelex.exceptions import (ConceptDefinitionError,ConceptLibraryError,@@ -30,13 +31,15 @@ from pipelex.exceptions import (PipeLibraryError,PipeLoadingError,)-from pipelex.libraries.library_config import LibraryConfigfrom pipelex.libraries.library_manager_abstract import LibraryManagerAbstract-from pipelex.tools.class_registry_utils import ClassRegistryUtils-from pipelex.tools.func_registry_utils import FuncRegistryUtils-from pipelex.tools.misc.file_utils import find_files_in_dir-from pipelex.tools.runtime_manager import runtime_manager-from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+from pipelex.libraries.library_utils import (+ find_plx_files_in_dir,+ get_pipelex_package_dir_for_imports,+ get_pipelex_plx_files_from_package,+)+from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.registries.class_registry_utils import ClassRegistryUtils+from pipelex.system.registries.func_registry_utils import FuncRegistryUtilsfrom pipelex.types import StrEnum@@ -67,12 +70,10 @@ class LibraryManager(LibraryManagerAbstract):domain_library: DomainLibrary,concept_library: ConceptLibrary,pipe_library: PipeLibrary,- library_config: LibraryConfig,):self.domain_library = domain_libraryself.concept_library = concept_libraryself.pipe_library = pipe_library- self.library_config = library_config@overridedef validate_libraries(self):@@ -97,23 +98,18 @@ class LibraryManager(LibraryManagerAbstract):self.teardown()self.setup()- def _get_pipeline_library_dirs(self) -> list[Path]:- library_dirs = [Path(self.library_config.pipelines_dir_path)]- if runtime_manager.is_unit_testing:- log.debug("Registering test pipeline structures for unit testing")- library_dirs += [Path(self.library_config.test_pipelines_dir_path)]- return library_dirs-- def _get_pipelex_plx_files_from_dirs(self, dirs: list[Path]) -> list[Path]:+ def _get_pipelex_plx_files_from_dirs(self, dirs: set[Path]) -> list[Path]:"""Get all valid Pipelex PLX files from the given directories."""all_plx_paths: list[Path] = []+ seen_files: set[str] = set() # Track by absolute path to avoid duplicates+for dir_path in dirs:if not dir_path.exists():- msg = f"Directory does not exist: {dir_path}"- raise LibraryError(msg)+ log.debug(f"Directory does not exist, skipping: {dir_path}")+ continue- # Find all TOML files in the directory- plx_files = find_files_in_dir(+ # Find all .plx files in the directory, excluding problematic directories+ plx_files = find_plx_files_in_dir(dir_path=str(dir_path),pattern="*.plx",is_recursive=True,@@ -121,8 +117,16 @@ class LibraryManager(LibraryManagerAbstract):# Filter to only include valid Pipelex filesfor plx_file in plx_files:+ absolute_path = str(plx_file.resolve())++ # Skip if already seen+ if absolute_path in seen_files:+ log.debug(f"Skipping duplicate PLX file: {plx_file}")+ continue+if PipelexInterpreter.is_pipelex_file(plx_file):all_plx_paths.append(plx_file)+ seen_files.add(absolute_path)else:log.debug(f"Skipping non-Pipelex PLX file: {plx_file}")@@ -216,43 +220,116 @@ class LibraryManager(LibraryManagerAbstract):pipes.append(pipe)return pipes+ def _import_pipelex_modules_directly(self) -> None:+ """Import pipelex modules to register @pipe_func decorated functions.++ This ensures critical pipelex functions are registered regardless of how pipelex+ is installed (wheel, source, relative path, etc.).+ """+ import pipelex.builder # noqa: PLC0415 - intentional local import++ log.verbose("Registering @pipe_func functions from pipelex.builder")+ functions_count = FuncRegistryUtils.register_pipe_funcs_from_package("pipelex.builder", pipelex.builder)+ log.verbose(f"Registered {functions_count} @pipe_func functions from pipelex.builder")+@overridedef load_libraries(self,library_dirs: list[Path] | None = None,library_file_paths: list[Path] | None = None,) -> None:- dirs_to_use = library_dirs or self._get_pipeline_library_dirs()+ # Collect directories to scan (user project directories)+ user_dirs: set[Path] = set()+ if library_dirs:+ user_dirs.update(library_dirs)+ else:+ user_dirs.add(Path(config_manager.local_root_dir))valid_plx_paths: list[Path]if library_file_paths:valid_plx_paths = library_file_pathselse:- all_plx_paths: list[Path] = self._get_pipelex_plx_files_from_dirs(dirs_to_use)- # Remove failing pipelines from the list- failing_pipelines_file_paths = get_config().pipelex.library_config.failing_pipelines_file_paths- valid_plx_paths = [path for path in all_plx_paths if path not in failing_pipelines_file_paths]+ # Get PLX files from user directories+ user_plx_paths: list[Path] = self._get_pipelex_plx_files_from_dirs(user_dirs)++ # Get PLX files from pipelex package using importlib.resources+ # This works reliably in all installation modes (wheel, source, relative)+ pipelex_plx_paths: list[Path] = get_pipelex_plx_files_from_package()++ # Combine and deduplicate+ all_plx_paths = user_plx_paths + pipelex_plx_paths+ seen_absolute_paths: set[str] = set()+ valid_plx_paths = []+ for plx_path in all_plx_paths:+ try:+ absolute_path = str(plx_path.resolve())+ except (OSError, RuntimeError):+ # For paths that can't be resolved (e.g., in zipped packages), use string representation+ absolute_path = str(plx_path)++ if absolute_path not in seen_absolute_paths:+ valid_plx_paths.append(plx_path)+ seen_absolute_paths.add(absolute_path)++ # Import modules to load them into sys.modules (but don't register classes yet)+ # Import from user directories+ for library_dir in user_dirs:+ # Only import files that contain StructuredContent subclasses (uses AST pre-check)+ ClassRegistryUtils.import_modules_in_folder(+ folder_path=str(library_dir),+ base_class_names=[StructuredContent.__name__],+ )+ # Only import files that contain @pipe_func decorated functions (uses AST pre-check)+ FuncRegistryUtils.register_funcs_in_folder(+ folder_path=str(library_dir),+ )- # Register classes in the directories- for library_dir in dirs_to_use:- ClassRegistryUtils.register_classes_in_folder(folder_path=str(library_dir))- FuncRegistryUtils.register_funcs_in_folder(folder_path=str(library_dir))+ # Import from pipelex package+ # Always directly import critical builder modules first (works in all installation modes)+ log.verbose("About to import pipelex.builder modules for @pipe_func registration")+ self._import_pipelex_modules_directly()++ # Verify critical functions were registered+ from pipelex.system.registries.func_registry import func_registry # noqa: PLC0415 - intentional local import++ critical_functions = ["create_concept_spec", "assemble_pipelex_bundle_spec"]+ for func_name in critical_functions:+ if func_registry.has_function(func_name):+ log.verbose(f"✓ Function '{func_name}' successfully registered")+ else:+ log.error(f"✗ Function '{func_name}' NOT registered - this will cause errors!")++ # Then try filesystem-based scanning if package is accessible (for completeness)+ pipelex_pkg_dir = get_pipelex_package_dir_for_imports()+ if pipelex_pkg_dir:+ log.debug(f"Additionally scanning pipelex package filesystem: {pipelex_pkg_dir}")+ ClassRegistryUtils.import_modules_in_folder(+ folder_path=str(pipelex_pkg_dir),+ base_class_names=[StructuredContent.__name__],+ )+ FuncRegistryUtils.register_funcs_in_folder(+ folder_path=str(pipelex_pkg_dir),+ )++ # Auto-discover and register all StructuredContent classes from sys.modules+ num_registered = ClassRegistryUtils.auto_register_all_subclasses(base_class=StructuredContent)+ log.debug(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")# Parse all blueprints firstblueprints: list[PipelexBundleBlueprint] = []for plx_file_path in valid_plx_paths:try:blueprint = PipelexInterpreter(file_path=plx_file_path).make_pipelex_bundle_blueprint()- except FileNotFoundError as domain_def_error:+ except FileNotFoundError as file_not_found_error:msg = f"Could not find PLX blueprint at '{plx_file_path}'"- raise LibraryLoadingError(msg) from domain_def_error- except PipeDefinitionError as domain_def_error:- msg = f"Could not load PLX blueprint from '{plx_file_path}': {domain_def_error}"- raise LibraryLoadingError(msg) from domain_def_error- except ValidationError as domain_def_error:- formatted_error_msg = format_pydantic_validation_error(domain_def_error)- msg = f"Could not load PLX blueprint from '{plx_file_path}' because of: {formatted_error_msg}"- raise LibraryLoadingError(msg) from domain_def_error+ raise LibraryLoadingError(msg) from file_not_found_error+ except PipeDefinitionError as pipe_def_error:+ msg = f"Could not load PLX blueprint from '{plx_file_path}': {pipe_def_error}"+ raise LibraryLoadingError(msg) from pipe_def_error+ except ValidationError as validation_error:+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)+ msg = f"Could not load PLX blueprint from '{plx_file_path}' because of: {validation_error_msg}"+ raise LibraryLoadingError(msg) from validation_errorblueprint.source = str(plx_file_path)blueprints.append(blueprint)@@ -265,8 +342,8 @@ class LibraryManager(LibraryManagerAbstract):msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {domain_def_error}"raise LibraryLoadingError(msg) from domain_def_errorexcept ValidationError as validation_error:- formatted_error_msg = format_pydantic_validation_error(validation_error)- msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {formatted_error_msg}"+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)+ msg = f"Could not load domain from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"raise LibraryLoadingError(msg) from validation_errorall_domains.append(domain)self.domain_library.add_domains(domains=all_domains)@@ -280,8 +357,8 @@ class LibraryManager(LibraryManagerAbstract):msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {concept_def_error}"raise LibraryLoadingError(msg) from concept_def_errorexcept ValidationError as validation_error:- formatted_error_msg = format_pydantic_validation_error(validation_error)- msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {formatted_error_msg}"+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)+ msg = f"Could not load concepts from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"raise LibraryLoadingError(msg) from validation_errorall_concepts.extend(concepts)self.concept_library.add_concepts(concepts=all_concepts)@@ -295,8 +372,8 @@ class LibraryManager(LibraryManagerAbstract):msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {pipe_def_error}"raise LibraryLoadingError(msg) from pipe_def_errorexcept ValidationError as validation_error:- formatted_error_msg = format_pydantic_validation_error(validation_error)- msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {formatted_error_msg}"+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)+ msg = f"Could not load pipes from PLX blueprint at '{blueprint.source}', domain code: '{blueprint.domain}': {validation_error_msg}"raise LibraryLoadingError(msg) from validation_errorall_pipes.extend(pipes)self.pipe_library.add_pipes(pipes=all_pipes)diff --git a/pipelex/libraries/library_manager_factory.py b/pipelex/libraries/library_manager_factory.pyindex c63ff4412..47631d1df 100644--- a/pipelex/libraries/library_manager_factory.py+++ b/pipelex/libraries/library_manager_factory.py@@ -1,7 +1,6 @@from pipelex.core.concepts.concept_library import ConceptLibraryfrom pipelex.core.domains.domain_library import DomainLibraryfrom pipelex.core.pipes.pipe_library import PipeLibrary-from pipelex.libraries.library_config import LibraryConfigfrom pipelex.libraries.library_manager import LibraryManager@@ -9,17 +8,15 @@ class LibraryManagerFactory:"""Factory for creating LibraryManager instances."""@classmethod- def make_empty(cls, config_dir_path: str) -> "LibraryManager":+ def make_empty(cls) -> "LibraryManager":domain_library = DomainLibrary.make_empty()concept_library = ConceptLibrary.make_empty()pipe_library = PipeLibrary.make_empty()- library_config = LibraryConfig(config_dir_path=config_dir_path)return LibraryManager(domain_library=domain_library,concept_library=concept_library,pipe_library=pipe_library,- library_config=library_config,)@classmethod@@ -28,14 +25,10 @@ class LibraryManagerFactory:domain_library: DomainLibrary,concept_library: ConceptLibrary,pipe_library: PipeLibrary,- config_dir_path: str,) -> "LibraryManager":"""Create a LibraryManager with provided libraries."""- library_config = LibraryConfig(config_dir_path=config_dir_path)-return LibraryManager(domain_library=domain_library,concept_library=concept_library,pipe_library=pipe_library,- library_config=library_config,)diff --git a/pipelex/libraries/library_utils.py b/pipelex/libraries/library_utils.pynew file mode 100644index 000000000..40c58a6c2--- /dev/null+++ b/pipelex/libraries/library_utils.py@@ -0,0 +1,91 @@+"""Utility functions for library management."""++from importlib.abc import Traversable+from importlib.resources import files+from pathlib import Path++from pipelex import log+from pipelex.config import get_config+from pipelex.core.interpreter import PipelexInterpreter+from pipelex.tools.misc.file_utils import find_files_in_dir+++def get_pipelex_plx_files_from_package() -> list[Path]:+ """Get all PLX files from the pipelex package using importlib.resources.++ This works reliably whether pipelex is installed as a wheel, from source,+ or as a relative path import.++ Returns:+ List of Path objects to PLX files in pipelex package+ """+ plx_files: list[Path] = []+ pipelex_package = files("pipelex")++ def _find_plx_in_traversable(traversable: Traversable, collected: list[Path]) -> None:+ """Recursively find .plx files in a Traversable."""+ excluded_dirs = get_config().pipelex.scan_config.excluded_dirs+ try:+ if not traversable.is_dir():+ return++ for child in traversable.iterdir():+ if child.is_file() and child.name.endswith(".plx"):+ # Convert to path string for validation+ plx_path_str = str(child)+ if PipelexInterpreter.is_pipelex_file(Path(plx_path_str)):+ collected.append(Path(plx_path_str))+ log.verbose(f"Found pipelex package PLX file: {plx_path_str}")+ elif child.is_dir():+ # Skip excluded directories+ if child.name not in excluded_dirs:+ _find_plx_in_traversable(child, collected)+ except (PermissionError, OSError) as exc:+ log.warning(f"Could not access {traversable}: {exc}")++ _find_plx_in_traversable(pipelex_package, plx_files)+ log.verbose(f"Found {len(plx_files)} PLX files in pipelex package")+ return plx_files+++def get_pipelex_package_dir_for_imports() -> Path | None:+ """Get the pipelex package directory as a Path for importing Python modules.++ Returns:+ Path to the pipelex package directory, or None if not accessible as filesystem+ """+ pipelex_package = files("pipelex")+ try:+ # Try to convert to Path (works for filesystem paths)+ pkg_path = Path(str(pipelex_package))+ if pkg_path.exists() and pkg_path.is_dir():+ return pkg_path+ except (TypeError, ValueError, OSError) as exc:+ log.warning(f"Could not convert importlib.resources Traversable to filesystem Path: {exc}")+ return None+++def find_plx_files_in_dir(dir_path: str, pattern: str, is_recursive: bool) -> list[Path]:+ """Find PLX files matching a pattern in a directory, excluding problematic directories.++ Args:+ dir_path: Directory path to search in+ pattern: File pattern to match (e.g. "*.plx")+ is_recursive: Whether to search recursively in subdirectories++ Returns:+ List of matching Path objects, filtered to exclude problematic directories+ """+ # Get all files using the base utility+ all_files = find_files_in_dir(dir_path, pattern, is_recursive)++ # Filter out files in excluded directories+ filtered_files: list[Path] = []+ excluded_dirs = get_config().pipelex.scan_config.excluded_dirs+ for file_path in all_files:+ # Check if any parent directory is in the exclude list+ should_exclude = any(part in excluded_dirs for part in file_path.parts)+ if not should_exclude:+ filtered_files.append(file_path)++ return filtered_filesdiff --git a/pipelex/libraries/pipelines/documents.plx b/pipelex/libraries/pipelines/documents.plxdeleted file mode 100644index 8fee29f27..000000000--- a/pipelex/libraries/pipelines/documents.plx+++ /dev/null@@ -1,28 +0,0 @@---domain = "documents"-description = "The domain of documents that can comprise pages, text, images, etc. in PDF or other formats"--[concept]-TextAndImagesContent = "A content that comprises text and images where the text can include local links to the images"--[pipe]-# PipeExtract requires to have a single input-# It can be named however you want-# but it must be either an image or a pdf or a concept which refines one of them-[pipe.extract_page_contents_from_pdf]-type = "PipeExtract"-description = "Extract page contents from a PDF document"-inputs = { document = "PDF" }-output = "Page"-page_images = true-page_views = false--[pipe.extract_page_contents_and_views_from_pdf]-type = "PipeExtract"-description = "Extract page contents from a PDF document as well as full page views"-inputs = { document = "PDF" }-output = "Page"-page_images = true-page_views = true-diff --git a/pipelex/libraries/pipelines/images.plx b/pipelex/libraries/pipelines/images.plxdeleted file mode 100644index dd3e1ae6d..000000000--- a/pipelex/libraries/pipelines/images.plx+++ /dev/null@@ -1,36 +0,0 @@-domain = "images"-description = "Generic image-related domain"--[concept]-VisualDescription = "Visual description of something"--[concept.ImgGenPrompt]-description = "Prompt to generate an image"-refines = "Text"--[concept.Photo]-description = "Photo"-refines = "Image"--[pipe]-#################################################################-# Image generation: PipeImgGen generating images as output-#################################################################---# PipeImgGen requires to have a single input-# It can be named however you want,-# but it must be either an ImgGenPrompt or a concept which refines ImgGenPrompt-[pipe.generate_image]-type = "PipeImgGen"-description = "Generate an image"-inputs = { prompt = "ImgGenPrompt" }-output = "Image"---[pipe.generate_photo]-type = "PipeImgGen"-description = "Generate a photo"-inputs = { prompt = "ImgGenPrompt" }-output = "images.Photo"-diff --git a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.pyindex 7fefdb2cf..e65090aac 100644--- a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py+++ b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py@@ -7,7 +7,7 @@ from pipelex.core.pipes.pipe_blueprint import PipeBlueprintclass PipeBatchBlueprint(PipeBlueprint):type: Literal["PipeBatch"] = "PipeBatch"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"branch_pipe_code: strinput_list_name: str | None = Noneinput_item_name: str | None = Nonediff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.pyindex 7d03c4bf1..554eaccb4 100644--- a/pipelex/pipe_controllers/condition/pipe_condition.py+++ b/pipelex/pipe_controllers/condition/pipe_condition.py@@ -249,7 +249,7 @@ class PipeCondition(PipeController):# Evaluate the expression using templatingevaluated_expression = await content_generator.make_templated_text(- context=working_memory.generate_jinja2_context(),+ context=working_memory.generate_context(),template=self.applied_expression_template,template_category=TemplateCategory.EXPRESSION,)@@ -311,7 +311,6 @@ class PipeCondition(PipeController):# Get required variables and validate they exist in working memoryrequired_variables = chosen_pipe.required_variables()- log.debug(required_variables, title=f"Required variables for PipeCondition '{self.code}'")required_stuff_names = {required_variable for required_variable in required_variables if not required_variable.startswith("_")}try:required_stuffs = working_memory.get_stuffs(names=required_stuff_names)diff --git a/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py b/pipelex/pipe_controllers/condition/pipe_condition_blueprint.pyindex e03136cc8..e8940d992 100644--- a/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py+++ b/pipelex/pipe_controllers/condition/pipe_condition_blueprint.py@@ -11,7 +11,7 @@ OutcomeMap = dict[str, str]class PipeConditionBlueprint(PipeBlueprint):type: Literal["PipeCondition"] = "PipeCondition"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"expression_template: str | None = Noneexpression: str | None = Noneoutcomes: OutcomeMap = Field(default_factory=OutcomeMap)diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.pyindex 8fba98166..5c498cf0a 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py@@ -12,7 +12,7 @@ from pipelex.types import Selfclass PipeParallelBlueprint(PipeBlueprint):type: Literal["PipeParallel"] = "PipeParallel"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"parallels: list[SubPipeBlueprint]add_each_output: bool = Falsecombined_output: str | None = Nonediff --git a/pipelex/pipe_controllers/pipe_controller.py b/pipelex/pipe_controllers/pipe_controller.pyindex e186eeef6..d4d94430a 100644--- a/pipelex/pipe_controllers/pipe_controller.py+++ b/pipelex/pipe_controllers/pipe_controller.py@@ -12,7 +12,7 @@ from pipelex.pipeline.job_metadata import JobMetadataclass PipeController(PipeAbstract):- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"@propertydef class_name(self) -> str:diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.pyindex 2b0b3c64b..a3ef87cdb 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py@@ -19,7 +19,7 @@ from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallelfrom pipelex.pipe_controllers.pipe_controller import PipeControllerfrom pipelex.pipe_controllers.sequence.exceptions import PipeSequenceErrorfrom pipelex.pipe_controllers.sub_pipe import SubPipe-from pipelex.pipe_run.pipe_run_params import PipeRunMode, PipeRunParams+from pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.types import Self@@ -203,10 +203,9 @@ class PipeSequence(PipeController):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeOutput:- if pipe_run_params.run_mode != PipeRunMode.DRY:+ if not pipe_run_params.run_mode.is_dry:msg = f"PipeSequence._dry_run_controller_pipe() called with run_mode = {pipe_run_params.run_mode} in pipe {self.code}"raise PipeRunParamsError(msg)- log.debug(f"PipeSequence._dry_run_controller_pipe() called with {self.code=} {pipe_run_params=}")# Verify the output of this pipe is matching the output of the last step.concept_string_of_last_step = get_required_pipe(pipe_code=self.sequential_sub_pipes[-1].pipe_code).output.concept_stringif self.output.concept_string != concept_string_of_last_step:diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py b/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.pyindex 58a498e6c..4df7f4344 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence_blueprint.py@@ -8,7 +8,7 @@ from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprintclass PipeSequenceBlueprint(PipeBlueprint):type: Literal["PipeSequence"] = "PipeSequence"- category: Literal["PipeController"] = "PipeController"+ pipe_category: Literal["PipeController"] = "PipeController"steps: list[SubPipeBlueprint]@propertydiff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.pyindex 31d28fd90..79d0c9c37 100644--- a/pipelex/pipe_controllers/sub_pipe.py+++ b/pipelex/pipe_controllers/sub_pipe.py@@ -30,7 +30,6 @@ class SubPipe(BaseModel):sub_pipe_run_params: PipeRunParams,) -> PipeOutput:"""Run or dry run a single operation self."""- log.debug(f"SubPipe {self.pipe_code} to generate {self.output_name}")if self.output_multiplicity:sub_pipe_run_params.output_multiplicity = self.output_multiplicitysub_pipe_run_params.batch_params = self.batch_params@@ -110,7 +109,6 @@ class SubPipe(BaseModel):else:# Case 3: Normal processingrequired_variables = sub_pipe.required_variables()- log.debug(required_variables, title=f"Required variables for {self.pipe_code}")required_stuff_names = {rv for rv in required_variables if not rv.startswith("_")}try:required_stuffs = working_memory.get_stuffs(names=required_stuff_names)diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.pyindex 99def447e..6f2ac546c 100644--- a/pipelex/pipe_operators/compose/pipe_compose.py+++ b/pipelex/pipe_operators/compose/pipe_compose.py@@ -44,13 +44,13 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):template: strtemplating_style: TemplatingStyle | None = None- template_category: TemplateCategory = TemplateCategory.BASIC+ category: TemplateCategory = TemplateCategory.BASICextra_context: dict[str, Any] | None = None@model_validator(mode="after")def validate_template(self) -> Self:try:- check_jinja2_parsing(template_source=self.template, template_category=self.template_category)+ check_jinja2_parsing(template_source=self.template, template_category=self.category)except Jinja2TemplateSyntaxError as exc:msg = f"Could not parse template for PipeCompose '{self.code}: {exc}"raise PipeDefinitionError(msg) from exc@@ -92,7 +92,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):@overridedef required_variables(self) -> set[str]:required_variables = detect_jinja2_required_variables(- template_category=self.template_category,+ template_category=self.category,template_source=self.template,)return {@@ -115,7 +115,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):msg = f"PipeCompose does not suppport multiple outputs, got output_multiplicity = {pipe_run_params.output_multiplicity}"raise PipeRunParamsError(msg)- context: dict[str, Any] = working_memory.generate_jinja2_context()+ context: dict[str, Any] = working_memory.generate_context()if pipe_run_params:context.update(**pipe_run_params.params)if self.extra_context:@@ -125,7 +125,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):context=context,template=self.template,templating_style=self.templating_style,- template_category=self.template_category,+ template_category=self.category,)log.verbose(f"Jinja2 rendered text:\n{jinja2_text}")assert isinstance(jinja2_text, str)diff --git a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py b/pipelex/pipe_operators/compose/pipe_compose_blueprint.pyindex 22af36c1b..28927c757 100644--- a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py+++ b/pipelex/pipe_operators/compose/pipe_compose_blueprint.py@@ -8,13 +8,13 @@ from pipelex.core.pipes.pipe_blueprint import PipeBlueprintclass PipeComposeBlueprint(PipeBlueprint):type: Literal["PipeCompose"] = "PipeCompose"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"template: str | TemplateBlueprint@propertydef template_source(self) -> str:if isinstance(self.template, TemplateBlueprint):- return self.template.source+ return self.template.templatereturn self.template@propertydiff --git a/pipelex/pipe_operators/compose/pipe_compose_factory.py b/pipelex/pipe_operators/compose/pipe_compose_factory.pyindex c63e6f558..5a54dcac4 100644--- a/pipelex/pipe_operators/compose/pipe_compose_factory.py+++ b/pipelex/pipe_operators/compose/pipe_compose_factory.py@@ -50,7 +50,7 @@ class PipeComposeFactory(PipeFactoryProtocol[PipeComposeBlueprint, PipeCompose])),template=preprocessed_template,templating_style=blueprint.templating_style,- template_category=blueprint.template_category,+ category=blueprint.template_category,extra_context=blueprint.extra_context,)diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.pyindex 55a23c72d..518b56296 100644--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py@@ -6,7 +6,7 @@ from pipelex.core.pipes.pipe_blueprint import PipeBlueprintclass PipeExtractBlueprint(PipeBlueprint):type: Literal["PipeExtract"] = "PipeExtract"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"model: ExtractModelChoice | None = Nonepage_images: bool | None = Nonepage_image_captions: bool | None = Nonediff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.pyindex 31aec4b5d..601d92762 100644--- a/pipelex/pipe_operators/func/pipe_func.py+++ b/pipelex/pipe_operators/func/pipe_func.py@@ -17,7 +17,7 @@ from pipelex.exceptions import DryRunErrorfrom pipelex.pipe_operators.pipe_operator import PipeOperatorfrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadata-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registryclass PipeFuncOutput(PipeOutput):diff --git a/pipelex/pipe_operators/func/pipe_func_blueprint.py b/pipelex/pipe_operators/func/pipe_func_blueprint.pyindex 5d9af5be2..723d9a657 100644--- a/pipelex/pipe_operators/func/pipe_func_blueprint.py+++ b/pipelex/pipe_operators/func/pipe_func_blueprint.py@@ -7,5 +7,5 @@ from pipelex.core.pipes.pipe_blueprint import PipeBlueprintclass PipeFuncBlueprint(PipeBlueprint):type: Literal["PipeFunc"] = "PipeFunc"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"function_name: str = Field(description="The name of the function to call.")diff --git a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.pyindex 6d88e2357..734e45bd4 100644--- a/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py+++ b/pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py@@ -9,7 +9,7 @@ from pipelex.core.pipes.pipe_blueprint import PipeBlueprintclass PipeImgGenBlueprint(PipeBlueprint):type: Literal["PipeImgGen"] = "PipeImgGen"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"img_gen_prompt: str | None = Noneimg_gen_prompt_var_name: str | None = Nonediff --git a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py b/pipelex/pipe_operators/llm/llm_prompt_blueprint.pyindex 2df0dc2da..6d5a031ed 100644--- a/pipelex/pipe_operators/llm/llm_prompt_blueprint.py+++ b/pipelex/pipe_operators/llm/llm_prompt_blueprint.py@@ -13,6 +13,7 @@ from pipelex.core.stuffs.image_content import ImageContentfrom pipelex.hub import get_content_generatorfrom pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variablesfrom pipelex.tools.misc.context_provider_abstract import ContextProviderAbstract, ContextProviderException+from pipelex.tools.misc.dict_utils import substitute_nested_in_contextif TYPE_CHECKING:from pipelex.cogt.image.prompt_image import PromptImage@@ -34,7 +35,7 @@ class LLMPromptBlueprint(BaseModel):required_variables.update(user_images_top_object_name)if self.prompt_blueprint:- template_source = preprocess_template(self.prompt_blueprint.source)+ template_source = preprocess_template(self.prompt_blueprint.template)required_variables.update(detect_jinja2_required_variables(template_category=self.prompt_blueprint.category,@@ -43,7 +44,7 @@ class LLMPromptBlueprint(BaseModel):)if self.system_prompt_blueprint:- system_prompt_template_source = preprocess_template(self.system_prompt_blueprint.source)+ system_prompt_template_source = preprocess_template(self.system_prompt_blueprint.template)required_variables.update(detect_jinja2_required_variables(template_category=self.system_prompt_blueprint.category,@@ -76,12 +77,8 @@ class LLMPromptBlueprint(BaseModel):# Try to get as a single ImageContent firsttry:prompt_image_content = context_provider.get_typed_object_or_attribute(name=user_image_name, wanted_type=ImageContent)- if prompt_image_content is not None: # An ImageContent can be optional- if base_64 := prompt_image_content.base_64:- user_image = PromptImageFactory.make_prompt_image(base_64=base_64)- else:- image_uri = prompt_image_content.url- user_image = PromptImageFactory.make_prompt_image_from_uri(uri=image_uri)+ if isinstance(prompt_image_content, ImageContent):+ user_image = PromptImageFactory.make_prompt_image(url=prompt_image_content.url, base_64_str=prompt_image_content.base_64)prompt_user_images[user_image_name] = user_imageexcept ContextProviderException:# If single image failed, try to get as a collection (list or tuple)@@ -91,12 +88,7 @@ class LLMPromptBlueprint(BaseModel):if isinstance(image_collection, (list, tuple)):for image_item in image_collection: # type: ignore[assignment]if isinstance(image_item, ImageContent):- item_base_64 = image_item.base_64- if item_base_64:- user_image = PromptImageFactory.make_prompt_image(base_64=item_base_64) # type: ignore[arg-type]- else:- image_uri = image_item.url- user_image = PromptImageFactory.make_prompt_image_from_uri(uri=image_uri)+ user_image = PromptImageFactory.make_prompt_image(url=image_item.url, base_64_str=image_item.base_64)prompt_user_images[user_image_name] = user_imageelse:msg = (@@ -114,9 +106,10 @@ class LLMPromptBlueprint(BaseModel):if prompt_user_images:if not extra_params:extra_params = {}- for image_index, image_name in enumerate(prompt_user_images.keys()):+ image_names = list(prompt_user_images.keys())+ for image_index, image_name in enumerate(image_names):+ # Replacing image variable '{image_name}' with numbered tag '[Image {image_index + 1}]'extra_params[image_name] = f"[Image {image_index + 1}]"- log.warning(f"Replacing image variable '{image_name}' with numbered tag '[Image {image_index + 1}]'")user_text: str | None = Noneif self.prompt_blueprint:user_text = await self._unravel_text(@@ -163,18 +156,15 @@ class LLMPromptBlueprint(BaseModel):jinja2_blueprint.templating_style = templating_stylelog.verbose(f"Setting prompting style to {templating_style}")- log.info(f"extra_params: {extra_params}")- log.info(f"jinja2_blueprint.extra_context: {jinja2_blueprint.extra_context}")-- context: dict[str, Any] = context_provider.generate_jinja2_context()+ context: dict[str, Any] = context_provider.generate_context()if extra_params:- context.update(**extra_params)+ context = substitute_nested_in_context(context=context, extra_params=extra_params)if jinja2_blueprint.extra_context:context.update(**jinja2_blueprint.extra_context)return await get_content_generator().make_templated_text(context=context,- template=jinja2_blueprint.source,+ template=jinja2_blueprint.template,templating_style=self.templating_style,template_category=jinja2_blueprint.category,)diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.pyindex b1393fb57..2a5280f5f 100644--- a/pipelex/pipe_operators/llm/pipe_llm.py+++ b/pipelex/pipe_operators/llm/pipe_llm.py@@ -256,11 +256,7 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):# we acknowledge the code here with llm_prompt_1 and llm_prompt_2 is overly complex and should be refactored.the_content: StuffContent- log.debug(f"output_concept.structure_class_name: {output_concept.structure_class_name}")- log.debug(f"TextContent.__class__.__name__: {TextContent.__class__.__name__}")- log.debug(f"is_multiple_output: {is_multiple_output}")if output_concept.structure_class_name == "TextContent" and not is_multiple_output:- log.info(f"PipeLLM generating a single text output: {self.__class__.__name__}_gen_text")llm_prompt_1_for_text = await self.llm_prompt_spec.make_llm_prompt(output_concept_string=output_concept.concept_string,context_provider=working_memory,diff --git a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py b/pipelex/pipe_operators/llm/pipe_llm_blueprint.pyindex 19ce6aa13..d438e73f4 100644--- a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py+++ b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py@@ -16,7 +16,7 @@ class StructuringMethod(StrEnum):class PipeLLMBlueprint(PipeBlueprint):type: Literal["PipeLLM"] = "PipeLLM"- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"model: LLMModelChoice | None = Nonemodel_to_structure: LLMModelChoice | None = Nonediff --git a/pipelex/pipe_operators/llm/pipe_llm_factory.py b/pipelex/pipe_operators/llm/pipe_llm_factory.pyindex e9c73e751..c8b277704 100644--- a/pipelex/pipe_operators/llm/pipe_llm_factory.py+++ b/pipelex/pipe_operators/llm/pipe_llm_factory.py@@ -36,7 +36,7 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):if system_prompt:try:system_prompt_jinja2_blueprint = TemplateBlueprint(- source=system_prompt,+ template=system_prompt,category=TemplateCategory.LLM_PROMPT,)except Jinja2TemplateSyntaxError as exc:@@ -50,7 +50,7 @@ class PipeLLMFactory(PipeFactoryProtocol[PipeLLMBlueprint, PipeLLM]):if blueprint.prompt:try:user_text_jinja2_blueprint = TemplateBlueprint(- source=blueprint.prompt,+ template=blueprint.prompt,category=TemplateCategory.LLM_PROMPT,)except Jinja2TemplateSyntaxError as exc:diff --git a/pipelex/pipe_operators/pipe_operator.py b/pipelex/pipe_operators/pipe_operator.pyindex e4f1629be..8db54169b 100644--- a/pipelex/pipe_operators/pipe_operator.py+++ b/pipelex/pipe_operators/pipe_operator.py@@ -17,7 +17,7 @@ PipeOperatorOutputType = TypeVar("PipeOperatorOutputType", bound=PipeOutput)class PipeOperator(PipeAbstract, Generic[PipeOperatorOutputType]):- category: Literal["PipeOperator"] = "PipeOperator"+ pipe_category: Literal["PipeOperator"] = "PipeOperator"@propertydef class_name(self) -> str:diff --git a/pipelex/pipe_run/dry_run.py b/pipelex/pipe_run/dry_run.pyindex fa9d4e5c1..c2ca25716 100644--- a/pipelex/pipe_run/dry_run.py+++ b/pipelex/pipe_run/dry_run.py@@ -12,6 +12,7 @@ from pipelex.core.pipes.input_requirements import InputRequirements, TypedNamedIfrom pipelex.core.pipes.pipe_abstract import PipeAbstractfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.exceptions import PipeStackOverflowErrorfrom pipelex.hub import get_class_registryfrom pipelex.pipe_run.pipe_run_params import PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory@@ -26,7 +27,6 @@ class DryRunError(Exception):class DryRunStatus(StrEnum):SUCCESS = "SUCCESS"FAILURE = "FAILURE"- WARNING = "WARNING"@propertydef is_failure(self) -> bool:@@ -35,15 +35,12 @@ class DryRunStatus(StrEnum):return Truecase DryRunStatus.SUCCESS:return False- case DryRunStatus.WARNING:- return Falseclass DryRunOutput(BaseModel):pipe_code: strstatus: DryRunStatuserror_message: str | None = None- warning_message: str | None = Noneasync def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> DryRunOutput:@@ -60,17 +57,14 @@ async def dry_run_pipe(pipe: PipeAbstract, raise_on_failure: bool = False) -> Drworking_memory=working_memory,pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY),)- except Exception as exc:+ except PipeStackOverflowError as exc:if pipe.code in allowed_to_fail_pipes:- warning_message = f"Allowed to fail dry run for pipe '{pipe.code}': {exc}"- log.warning(warning_message)- return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.WARNING, warning_message=warning_message)-- if raise_on_failure:+ error_message = f"Allowed to fail dry run for pipe '{pipe.code}': {exc}"+ return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.FAILURE, error_message=error_message)+ elif raise_on_failure:raiseerror_message = f"Dry run failed for pipe '{pipe.code}': {exc}"- log.error(error_message)return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.FAILURE, error_message=error_message)log.info(f"Pipe '{pipe.code}' dry run completed successfully")return DryRunOutput(pipe_code=pipe.code, status=DryRunStatus.SUCCESS)@@ -117,8 +111,14 @@ async def dry_run_pipes(pipes: list[PipeAbstract], run_in_parallel: bool = True,for pipe in pipes:results[pipe.code] = await dry_run_pipe(pipe, raise_on_failure=raise_on_failure)- successful_pipes = [pipe_code for pipe_code, status in results.items() if status.status == DryRunStatus.SUCCESS]- failed_pipes = [pipe_code for pipe_code, status in results.items() if status.status != DryRunStatus.SUCCESS]+ successful_pipes: list[str] = []+ failed_pipes: list[str] = []+ for pipe_code, dry_run_output in results.items():+ match dry_run_output.status:+ case DryRunStatus.SUCCESS:+ successful_pipes.append(pipe_code)+ case DryRunStatus.FAILURE:+ failed_pipes.append(pipe_code)unexpected_failures = {pipe_code: results[pipe_code] for pipe_code in failed_pipes if pipe_code not in allowed_to_fail_pipes}diff --git a/pipelex/pipe_run/pipe_run_params.py b/pipelex/pipe_run/pipe_run_params.pyindex d14a14a83..d487c83f6 100644--- a/pipelex/pipe_run/pipe_run_params.py+++ b/pipelex/pipe_run/pipe_run_params.py@@ -19,6 +19,14 @@ class PipeRunMode(StrEnum):LIVE = "live"DRY = "dry"+ @property+ def is_dry(self) -> bool:+ match self:+ case PipeRunMode.DRY:+ return True+ case PipeRunMode.LIVE:+ return False+FORCE_DRY_RUN_MODE_ENV_KEY = "PIPELEX_FORCE_DRY_RUN_MODE"diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex f6aa4d883..a38b9f575 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -1,5 +1,3 @@-import inspect-import osfrom importlib.metadata import metadatafrom typing import cast@@ -14,7 +12,15 @@ from pipelex.cogt.content_generation.content_generator import ContentGeneratorfrom pipelex.cogt.content_generation.content_generator_protocol import (ContentGeneratorProtocol,)-from pipelex.cogt.exceptions import InferenceBackendCredentialsError, RoutingProfileLibraryNotFoundError+from pipelex.cogt.exceptions import (+ InferenceBackendCredentialsError,+ InferenceBackendLibraryNotFoundError,+ InferenceBackendLibraryValidationError,+ ModelDeckNotFoundError,+ ModelDeckValidationError,+ RoutingProfileLibraryNotFoundError,+ RoutingProfileValidationError,+)from pipelex.cogt.inference.inference_manager import InferenceManagerfrom pipelex.cogt.models.model_manager import ModelManagerfrom pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract@@ -22,7 +28,8 @@ from pipelex.config import PipelexConfig, get_configfrom pipelex.core.concepts.concept_library import ConceptLibraryfrom pipelex.core.domains.domain_library import DomainLibraryfrom pipelex.core.pipes.pipe_library import PipeLibrary-from pipelex.core.registry_models import PipelexRegistryModels+from pipelex.core.registry_models import CoreRegistryModels+from pipelex.core.validation import report_validation_errorfrom pipelex.exceptions import PipelexConfigError, PipelexSetupErrorfrom pipelex.hub import PipelexHub, set_pipelex_hubfrom pipelex.libraries.library_manager_factory import LibraryManagerFactory@@ -44,15 +51,15 @@ from pipelex.pipeline.track.pipeline_tracker_protocol import (from pipelex.plugins.plugin_manager import PluginManagerfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocol-from pipelex.test_extras.registry_test_models import PipelexTestModels-from pipelex.tools.config.config_root import ConfigRoot-from pipelex.tools.func_registry import func_registry-from pipelex.tools.runtime_manager import runtime_manager+from pipelex.system.configuration.config_root import ConfigRoot+from pipelex.system.registries.func_registry import func_registry+from pipelex.system.runtime import runtime_manager+from pipelex.test_extras.registry_test_models import TestRegistryModelsfrom pipelex.tools.secrets.env_secrets_provider import EnvSecretsProviderfrom pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstractfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract-from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorfrom pipelex.types import Self+from pipelex.urls import URLsPACKAGE_NAME = __name__.split(".", maxsplit=1)[0]PACKAGE_VERSION = metadata(PACKAGE_NAME)["Version"]@@ -61,7 +68,7 @@ PACKAGE_VERSION = metadata(PACKAGE_NAME)["Version"]class Pipelex(metaclass=MetaSingleton):def __init__(self,- config_dir_path: str,+ config_dir_path: str = "./pipelex",# Dependency injectionpipelex_hub: PipelexHub | None = None,config_cls: type[ConfigRoot] | None = None,@@ -80,10 +87,10 @@ class Pipelex(metaclass=MetaSingleton):# toolstry:self.pipelex_hub.setup_config(config_cls=config_cls or PipelexConfig)- except ValidationError as exc:- formatted_error_msg = format_pydantic_validation_error(exc)- msg = f"Could not setup config because of: {formatted_error_msg}"- raise PipelexConfigError(msg) from exc+ except ValidationError as validation_error:+ validation_error_msg = report_validation_error(category="config", validation_error=validation_error)+ msg = f"Could not setup config because of: {validation_error_msg}"+ raise PipelexConfigError(msg) from validation_errorlog.configure(project_name=get_config().project_name or "unknown_project",@@ -108,7 +115,7 @@ class Pipelex(metaclass=MetaSingleton):self.reporting_delegate: ReportingProtocolif get_config().pipelex.feature_config.is_reporting_enabled:- self.reporting_delegate = reporting_delegate or ReportingManager(reporting_config=get_config().pipelex.reporting_config)+ self.reporting_delegate = reporting_delegate or ReportingManager()else:self.reporting_delegate = ReportingNoOp()self.pipelex_hub.set_report_delegate(self.reporting_delegate)@@ -125,7 +132,6 @@ class Pipelex(metaclass=MetaSingleton):domain_library=domain_library,concept_library=concept_library,pipe_library=pipe_library,- config_dir_path=config_dir_path,)self.pipelex_hub.set_library_manager(library_manager=self.library_manager)@@ -152,6 +158,32 @@ class Pipelex(metaclass=MetaSingleton):log.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} init done")+ @staticmethod+ def _get_config_not_found_error_msg(component_name: str) -> str:+ """Generate error message for missing config files."""+ return f"Config files are missing for the {component_name}. Run `pipelex init config` to generate the missing files."++ @staticmethod+ def _get_validation_error_msg(component_name: str, validation_exc: Exception) -> str:+ """Generate error message for invalid config files."""+ msg = ""+ cause_exc = validation_exc.__cause__+ if cause_exc is None:+ msg += f"\nUnxpexted cause:{cause_exc}"+ raise PipelexSetupError(msg) from cause_exc+ if not isinstance(cause_exc, ValidationError):+ msg += f"\nUnxpexted cause:{cause_exc}"+ raise PipelexSetupError(msg) from cause_exc+ report = report_validation_error(category="config", validation_error=cause_exc)+ return f"""{msg}+{report}++Config files are invalid for the {component_name}.+You can fix them manually, or run `pipelex init config --reset` to regenerate them.+Note that this command resets all config files to their default values.+If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.+"""+def setup(self,secrets_provider: SecretsProviderAbstract | None = None,@@ -167,21 +199,37 @@ class Pipelex(metaclass=MetaSingleton):self.plugin_manager.setup()try:self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as routing_profile_library_exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from routing_profile_library_exc+ except RoutingProfileLibraryNotFoundError as routing_not_found_exc:+ msg = self._get_config_not_found_error_msg("routing profile library")+ raise PipelexSetupError(msg) from routing_not_found_exc+ except InferenceBackendLibraryNotFoundError as backend_not_found_exc:+ msg = self._get_config_not_found_error_msg("inference backend library")+ raise PipelexSetupError(msg) from backend_not_found_exc+ except ModelDeckNotFoundError as deck_not_found_exc:+ msg = self._get_config_not_found_error_msg("model deck")+ raise PipelexSetupError(msg) from deck_not_found_exc+ except RoutingProfileValidationError as routing_validation_exc:+ msg = self._get_validation_error_msg("routing profile library", routing_validation_exc)+ raise PipelexSetupError(msg) from routing_validation_exc+ except InferenceBackendLibraryValidationError as backend_validation_exc:+ msg = self._get_validation_error_msg("inference backend library", backend_validation_exc)+ raise PipelexSetupError(msg) from backend_validation_exc+ except ModelDeckValidationError as deck_validation_exc:+ msg = self._get_validation_error_msg("model deck", deck_validation_exc)+ msg += "\n\nIf you added your own config files to the model deck then you'll have to change them manually."+ raise PipelexSetupError(msg) from deck_validation_excexcept InferenceBackendCredentialsError as credentials_exc:backend_name = credentials_exc.backend_namevar_name = credentials_exc.key_nameerror_msg: strif secrets_provider:error_msg = (- f"Could not get credentials for inference backend {backend_name}:\n{credentials_exc},"+ f"Could not get credentials for inference backend '{backend_name}':\n{credentials_exc},"f"\ncheck that secret '{var_name}' is available from your secrets provider.")else:error_msg = (- f"Could not get credentials for inference backend {backend_name}:\n{credentials_exc},\n"+ f"Could not get credentials for inference backend '{backend_name}':\n{credentials_exc},\n"f"you need to add '{var_name}' to your environment variables or to your .env file.")if credentials_exc.backend_name == "pipelex_inference":@@ -189,15 +237,15 @@ class Pipelex(metaclass=MetaSingleton):"\nYou can check the project's README about getting a Pipelex Inference API key,\n\n""or you can bring your own 'OPENAI_API_KEY', ""'AZURE_OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'MISTRAL_API_KEY' etc.\n"- "--> choose which inference backends to enable in .pipelex/inference/backends.toml\n"+ "--> choose which inference backends to enable in '.pipelex/inference/backends.toml'\n")raise PipelexSetupError(error_msg) from credentials_excself.pipelex_hub.set_content_generator(content_generator or ContentGenerator())self.reporting_delegate.setup()- self.class_registry.register_classes(PipelexRegistryModels.get_all_models())+ self.class_registry.register_classes(CoreRegistryModels.get_all_models())if runtime_manager.is_unit_testing:log.debug("Registering test models for unit testing")- self.class_registry.register_classes(PipelexTestModels.get_all_models())+ self.class_registry.register_classes(TestRegistryModels.get_all_models())self.activity_manager.setup()observer_provider = observer_provider or LocalObserver()@@ -219,10 +267,10 @@ class Pipelex(metaclass=MetaSingleton):def validate_libraries(self):try:self.library_manager.validate_libraries()- except ValidationError as exc:- formatted_error_msg = format_pydantic_validation_error(exc)- msg = f"Could not validate libraries because of: {formatted_error_msg}"- raise PipelexSetupError(msg) from exc+ except ValidationError as validation_error:+ validation_error_msg = report_validation_error(category="plx", validation_error=validation_error)+ msg = f"Could not validate libraries because of: {validation_error_msg}"+ raise PipelexSetupError(msg) from validation_errorlog.debug(f"{PACKAGE_NAME} version {PACKAGE_VERSION} validate libraries done for {get_config().project_name}")def teardown(self):@@ -250,58 +298,21 @@ class Pipelex(metaclass=MetaSingleton):# TODO: add kwargs to make() so that subclasses can employ specific parameters@classmethod- def make(- cls,- relative_config_folder_path: str | None = None,- absolute_config_folder_path: str | None = None,- from_file: bool | None = True,- ) -> Self:+ def make(cls) -> Self:"""Create and initialize a Pipelex instance.- Args:- relative_config_folder_path: Path to config folder relative to either the caller file or current working directory.- Cannot be used together with absolute_config_folder_path.- absolute_config_folder_path: Absolute path to config folder.- Cannot be used together with relative_config_folder_path.- from_file: Only used when relative_config_folder_path is provided.- If True (default), the relative path is resolved relative to the file where make() was called.- If False, the relative path is resolved relative to the current working directory (useful for CLI scenarios).-Returns:Initialized Pipelex instance.Raises:- PipelexSetupError: If both relative_config_folder_path and absolute_config_folder_path are provided.- Or if frame inspection fails when using relative paths with from_file=True.-- Note:- If neither path is provided, defaults to "./pipelex_libraries".+ if setup fails"""- if relative_config_folder_path is not None and absolute_config_folder_path is not None:- msg = "Cannot specify both relative_config_folder_path and absolute_config_folder_path"+ if cls.get_optional_instance() is not None:+ msg = "Pipelex is already initialized"raise PipelexSetupError(msg)- if relative_config_folder_path is not None:- if from_file:- current_frame = inspect.currentframe()- if current_frame is None:- msg = "Could not find relative config folder path because of: Failed to get current frame"- raise PipelexSetupError(msg)- if current_frame.f_back is None:- msg = "Could not find relative config folder path because of: Failed to get caller frame"- raise PipelexSetupError(msg)- caller_file = current_frame.f_back.f_code.co_filename- caller_dir = os.path.dirname(os.path.abspath(caller_file))- config_dir_path = os.path.abspath(os.path.join(caller_dir, relative_config_folder_path))- else:- config_dir_path = os.path.abspath(os.path.join(os.getcwd(), relative_config_folder_path))- elif absolute_config_folder_path is not None:- config_dir_path = absolute_config_folder_path- else:- config_dir_path = "./pipelex_libraries"-- pipelex_instance = cls(config_dir_path=config_dir_path)+ pipelex_instance = cls()pipelex_instance.setup()pipelex_instance.setup_libraries()return pipelex_instancediff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 8d86c747b..127664346 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -4,6 +4,23 @@[pipelex.observer_config]observer_dir = "results/observer"+[pipelex.scan_config]+excluded_dirs = [+ ".venv",+ "venv",+ "env",+ ".env",+ "virtualenv",+ ".virtualenv",+ ".git",+ "__pycache__",+ ".pytest_cache",+ ".mypy_cache",+ ".ruff_cache",+ "node_modules",+ "results",+]+[pipelex.feature_config]# WIP/Experimental feature flagsis_pipeline_tracking_enabled = false@@ -107,10 +124,9 @@ You are a data modeling expert specialized in extracting structure from text."""structure_from_preliminary_text_user = """-Your job is to extract and structure information from a text.-Here is the text:+Extract and structure information from this text:-{preliminary_text}+{{ preliminary_text | tag }}Now generate the JSON in the required format.Do not create information that is not in the text.@@ -118,17 +134,20 @@ Do not create information that is not in the text.output_structure_prompt = """+---The instance we want to generate will be for the following class:{{ class_structure_str }}Don't bother with JSON formatting, we'll do that as a second step.For now, just output markdown with the details of the instance.-DO NOT create information.-If some information is not present for an attribute, output the default value or None according to the attribute definition.+DO NOT create new information.+If some information is not present for an attribute, output the default value or None according to the field definition."""output_structure_prompt_no_preliminary_text = """++---The instance we want to generate will be for the following class:{{ class_structure_str }}@@ -184,17 +203,6 @@ gemini = { tag_style = "xml" }[pipelex.structure_config]is_default_text_then_structure = false # turn this to true to get better results: generates text before structuring--####################################################################################################-# Pipelex libraries config-####################################################################################################--[pipelex.library_config]--[pipelex.generic_template_names]-structure_from_preliminary_text_system = "structure_from_preliminary_text_system"-structure_from_preliminary_text_user = "structure_from_preliminary_text_user"-##################################################################################################### Static validation config####################################################################################################@@ -246,11 +254,8 @@ text_gen_truncate_length = 256nb_list_items = 3nb_extract_pages = 4allowed_to_fail_pipes = [- "infinite_loop_1", # Loop but only for testing purposes- "fix_failing_pipes_once", # Loop but its normal- "validate_pipelex_bundle_spec", # Loop but its normal- "handle_validation_result", # Complex case of a sub pipe needed a ListContent because its batching on, but the dry_run cannot know.- "pipe_builder", # because of the 'fix_failing_pipes_once', 'validate_pipelex_bundle_spec', 'handle_validation_result'+ "infinite_loop_1", # Loop but only for testing purposes+ "pipe_builder", # Still not fully proofed]image_urls = ["https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg",@@ -276,3 +281,23 @@ structure_field_ordering = ["type", "description", "choices", "required"][pipelex.plx_config.pipes]field_ordering = ["type", "description", "inputs", "output"]++####################################################################################################+# Migration config+####################################################################################################++[migration.migration_maps.config]+definition = "description"+ocr_config = "extract_config"+ocr = "extract"+img_gen = "model"+llm_handle = "model"+llm = "model"+llm_to_structure = "model_to_structure"++[migration.migration_maps.plx]+img_gen = "model"+ocr = "model"+llm_handle = "model"+llm = "model"+llm_to_structure = "model_to_structure"diff --git a/pipelex/pipeline/activity/activity_manager.py b/pipelex/pipeline/activity/activity_manager.pyindex 662b74deb..494d7a188 100644--- a/pipelex/pipeline/activity/activity_manager.py+++ b/pipelex/pipeline/activity/activity_manager.py@@ -4,7 +4,7 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.pipeline.activity.activity_manager_protocol import ActivityManagerProtocolfrom pipelex.pipeline.activity.activity_models import ActivityCallback, ActivityReport-from pipelex.tools.exceptions import RootException+from pipelex.system.exceptions import RootExceptionclass ActivityManagerError(RootException):diff --git a/pipelex/pipeline/execute.py b/pipelex/pipeline/execute.pyindex f3e813849..ed0d789a7 100644--- a/pipelex/pipeline/execute.py+++ b/pipelex/pipeline/execute.py@@ -17,7 +17,7 @@ from pipelex.pipe_run.pipe_run_params import ()from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactoryfrom pipelex.pipeline.job_metadata import JobMetadata-from pipelex.tools.environment import get_optional_env+from pipelex.system.environment import get_optional_envasync def execute_pipeline(diff --git a/pipelex/pipeline/track/tracker_config.py b/pipelex/pipeline/track/tracker_config.pyindex 39abd0f2a..b208fe5c3 100644--- a/pipelex/pipeline/track/tracker_config.py+++ b/pipelex/pipeline/track/tracker_config.py@@ -1,6 +1,6 @@from typing import Literal-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelclass TrackerConfig(ConfigModel):diff --git a/pipelex/plugins/anthropic/anthropic_llm_worker.py b/pipelex/plugins/anthropic/anthropic_llm_worker.pyindex 88beb29b0..760bda121 100644--- a/pipelex/plugins/anthropic/anthropic_llm_worker.py+++ b/pipelex/plugins/anthropic/anthropic_llm_worker.py@@ -7,13 +7,22 @@ from typing_extensions import overridefrom pipelex import logfrom pipelex.cogt.exceptions import LLMCompletionError, SdkTypeErrorfrom pipelex.cogt.llm.llm_job import LLMJob-from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen+from pipelex.cogt.llm.llm_utils import (+ dump_error,+ dump_kwargs,+ dump_response_from_structured_gen,+)from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstractfrom pipelex.cogt.llm.structured_output import StructureMethodfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.config import get_config-from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicWorkerConfigurationError-from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory, AnthropicSdkVariant+from pipelex.plugins.anthropic.anthropic_exceptions import (+ AnthropicWorkerConfigurationError,+)+from pipelex.plugins.anthropic.anthropic_factory import (+ AnthropicFactory,+ AnthropicSdkVariant,+)from pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVarfrom pipelex.types import StrEnum@@ -81,7 +90,10 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):if instructor_config.is_dump_kwargs_enabled:self.instructor_for_objects.on(hook_name="completion:kwargs", handler=dump_kwargs)if instructor_config.is_dump_response_enabled:- self.instructor_for_objects.on(hook_name="completion:response", handler=dump_response_from_structured_gen)+ self.instructor_for_objects.on(+ hook_name="completion:response",+ handler=dump_response_from_structured_gen,+ )if instructor_config.is_dump_error_enabled:self.instructor_for_objects.on(hook_name="completion:error", handler=dump_error)@@ -141,7 +153,10 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):) -> BaseModelTypeVar:messages = await AnthropicFactory.make_simple_messages(llm_job=llm_job)max_tokens = self._adapt_max_tokens(max_tokens=llm_job.job_params.max_tokens)- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(+ (+ result_object,+ completion,+ ) = await self.instructor_for_objects.chat.completions.create_with_completion(messages=messages,response_model=schema,max_retries=llm_job.job_config.max_retries,diff --git a/pipelex/plugins/fal/fal_config.py b/pipelex/plugins/fal/fal_config.pyindex da52401e6..7dfc5fb09 100644--- a/pipelex/plugins/fal/fal_config.py+++ b/pipelex/plugins/fal/fal_config.py@@ -1,8 +1,8 @@from pydantic import field_validatorfrom pipelex.cogt.img_gen.img_gen_job_components import Quality-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.exceptions import ConfigValidationError+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.exceptions import ConfigValidationErrorclass FalConfig(ConfigModel):diff --git a/pipelex/plugins/mistral/mistral_extract_worker.py b/pipelex/plugins/mistral/mistral_extract_worker.pyindex e7d88e751..f3bf088f2 100644--- a/pipelex/plugins/mistral/mistral_extract_worker.py+++ b/pipelex/plugins/mistral/mistral_extract_worker.py@@ -4,7 +4,7 @@ from mistralai import Mistralfrom typing_extensions import overridefrom pipelex import log-from pipelex.cogt.exceptions import OcrCapabilityError, SdkTypeError+from pipelex.cogt.exceptions import ExtractCapabilityError, SdkTypeErrorfrom pipelex.cogt.extract.extract_input import ExtractInputErrorfrom pipelex.cogt.extract.extract_job import ExtractJobfrom pipelex.cogt.extract.extract_output import ExtractOutput@@ -88,7 +88,7 @@ class MistralExtractWorker(ExtractWorkerAbstract):) -> ExtractOutput:if should_caption_images:msg = "Captioning is not implemented for Mistral OCR."- raise OcrCapabilityError(msg)+ raise ExtractCapabilityError(msg)if should_include_page_views:log.debug("Page views are not implemented for Mistral OCR.")# TODO: use a model capability flag to check possibility before asking for itdiff --git a/pipelex/plugins/openai/openai_img_gen_worker.py b/pipelex/plugins/openai/openai_img_gen_worker.pyindex c87d949e6..66c7f627d 100644--- a/pipelex/plugins/openai/openai_img_gen_worker.py+++ b/pipelex/plugins/openai/openai_img_gen_worker.py@@ -13,7 +13,7 @@ from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.plugins.openai.openai_img_gen_factory import OpenAIImgGenFactoryfrom pipelex.reporting.reporting_protocol import ReportingProtocol-from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file+from pipelex.tools.misc.base_64_utils import save_base_64_str_to_binary_filefrom pipelex.tools.misc.file_utils import ensure_pathTEMP_OUTPUTS_DIR = "temp/img_gen_by_gpt_image"@@ -80,7 +80,7 @@ class OpenAIImgGenWorker(ImgGenWorkerAbstract):folder_path = TEMP_OUTPUTS_DIRensure_path(folder_path)img_path = f"{folder_path}/{image_id}_{image_index}.png"- save_base64_to_binary_file(b64=image_base64, file_path=img_path)+ save_base_64_str_to_binary_file(base_64_str=image_base64, file_path=img_path)log.debug(f"Saved image to {img_path}")generated_image_list.append(GeneratedImage(diff --git a/pipelex/plugins/openai/vertexai_factory.py b/pipelex/plugins/openai/vertexai_factory.pyindex 3adb49e5a..d0954c493 100644--- a/pipelex/plugins/openai/vertexai_factory.py+++ b/pipelex/plugins/openai/vertexai_factory.py@@ -1,8 +1,8 @@from typing import Anyfrom pipelex.cogt.exceptions import CogtError, MissingDependencyError-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.exceptions import CredentialsError+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.exceptions import CredentialsErrorfrom pipelex.tools.misc.json_utils import load_json_dict_from_pathfrom pipelex.tools.misc.placeholder import value_is_placeholderfrom pipelex.types import StrEnumdiff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 8169a7865..35eab2a47 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -7,7 +7,7 @@ from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokensUsagefrom pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.config import ReportingConfig+from pipelex.config import get_configfrom pipelex.pipeline.pipeline_models import SpecialPipelineIdfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path@@ -27,9 +27,9 @@ class UsageRegistry(RootModel[LLMUsageRegistryRoot]):class ReportingManager(ReportingProtocol):- def __init__(self, reporting_config: ReportingConfig):+ def __init__(self):+ self._reporting_config = get_config().pipelex.reporting_configself._usage_registries: dict[str, UsageRegistry] = {}- self._reporting_config = reporting_config############################################################# Manager lifecyclediff --git a/pipelex/libraries/pipelines/__init__.py b/pipelex/system/__init__.pysimilarity index 100%rename from pipelex/libraries/pipelines/__init__.pyrename to pipelex/system/__init__.pydiff --git a/pipelex/tools/config/__init__.py b/pipelex/system/configuration/__init__.pysimilarity index 100%rename from pipelex/tools/config/__init__.pyrename to pipelex/system/configuration/__init__.pydiff --git a/pipelex/tools/config/manager.py b/pipelex/system/configuration/config_loader.pysimilarity index 76%rename from pipelex/tools/config/manager.pyrename to pipelex/system/configuration/config_loader.pyindex 578356e84..85380ef09 100644--- a/pipelex/tools/config/manager.py+++ b/pipelex/system/configuration/config_loader.py@@ -3,30 +3,26 @@ import osfrom configparser import ConfigParserfrom typing import Any-import toml--from pipelex.tools.config.config_root import (+from pipelex.system.configuration.config_root import (CONFIG_BASE_OVERRIDES_AFTER_ENV,CONFIG_BASE_OVERRIDES_BEFORE_ENV,)+from pipelex.system.runtime import runtime_managerfrom pipelex.tools.misc.json_utils import deep_updatefrom pipelex.tools.misc.toml_utils import load_toml_from_path, load_toml_from_path_if_exists-from pipelex.tools.runtime_manager import runtime_managerCONFIG_DIR_NAME = ".pipelex"CONFIG_NAME = "pipelex.toml"-CONFIG_TEMPLATE_SUBPATH = "config_template"-INFERENCE_CONFIG_SUBPATH = "inference"class ConfigError(Exception):pass-class ConfigManager:+class ConfigLoader:@propertydef is_in_pipelex_config(self) -> bool:- return os.path.basename(self.local_root_dir) == "pipelex"+ return os.path.basename(os.getcwd()) == "pipelex"@propertydef pipelex_root_dir(self) -> str:@@ -47,11 +43,11 @@ class ConfigManager:"""Get the root directory of the project using pipelex.This is the directory from where the command is being run."""- return os.path.abspath(os.getcwd())+ return os.getcwd()@propertydef pipelex_config_dir(self) -> str:- return os.path.join(self.local_root_dir, CONFIG_DIR_NAME)+ return os.path.join(os.getcwd(), CONFIG_DIR_NAME)@propertydef pipelex_specific_config_file_path(self) -> str:@@ -82,9 +78,9 @@ class ConfigManager:This will be removed in the future.Requires to have a pyproject.toml file in the project root."""- pyproject_path = os.path.join(self.local_root_dir, "pyproject.toml")+ pyproject_path = os.path.join(os.getcwd(), "pyproject.toml")if not os.path.exists(pyproject_path):- print(f"pyproject.toml not found in {self.local_root_dir}")+ print(f"pyproject.toml not found in {os.getcwd()}")returndef _find_package_path(package_name: str) -> str | None:@@ -98,7 +94,7 @@ class ConfigManager:return None- pyproject = toml.load(pyproject_path)+ pyproject = load_toml_from_path(path=pyproject_path)if "tool" in pyproject and "pipelex" in pyproject["tool"] and "config_inheritance" in pyproject["tool"]["pipelex"]:for config_name in pyproject["tool"]["pipelex"]["config_inheritance"]:package_path = _find_package_path(config_name)@@ -147,9 +143,9 @@ class ConfigManager:for override in list_of_overrides:if override:if override == runtime_manager.run_mode.UNIT_TEST:- override_path = os.path.join(self.local_root_dir, "tests", f"pipelex_{override}.toml")+ override_path = os.path.join(os.getcwd(), "tests", f"pipelex_{override}.toml")else:- override_path = os.path.join(self.local_root_dir, "pipelex" if self.is_in_pipelex_config else "", f"pipelex_{override}.toml")+ override_path = os.path.join(os.getcwd(), "pipelex" if self.is_in_pipelex_config else "", f"pipelex_{override}.toml")if override_dict := load_toml_from_path_if_exists(override_path):deep_update(pipelex_config, override_dict)@@ -178,34 +174,20 @@ class ConfigManager:"""# First check pipelex's pyproject.toml- pipelex_pyproject_path = os.path.join(os.path.dirname(self.local_root_dir), "pyproject.toml")- try:- pyproject = toml.load(pipelex_pyproject_path)- if (project_name := pyproject.get("project", {}).get("name")) and isinstance(project_name, str):+ pipelex_pyproject_path = os.path.join(os.path.dirname(os.getcwd()), "pyproject.toml")+ if pipelex_pyproject := load_toml_from_path_if_exists(path=pipelex_pyproject_path):+ if (project_name := pipelex_pyproject.get("project", {}).get("name")) and isinstance(project_name, str):return str(project_name)- except FileNotFoundError:- pass- except toml.TomlDecodeError as exc:- print(f"Failed to parse pipelex pyproject.toml at {pipelex_pyproject_path}: {exc}")- except (KeyError, TypeError, AttributeError) as exc:- print(f"Invalid structure in pipelex pyproject.toml at {pipelex_pyproject_path}: {exc}")# Check local pyproject.toml- pyproject_path = os.path.join(self.local_root_dir, "pyproject.toml")- try:- pyproject = toml.load(pyproject_path)- name_obj: object = pyproject.get("project", {}).get("name") or pyproject.get("tool", {}).get("poetry", {}).get("name")+ local_pyproject_path = os.path.join(os.getcwd(), "pyproject.toml")+ if local_pyproject := load_toml_from_path_if_exists(local_pyproject_path):+ name_obj: object = local_pyproject.get("project", {}).get("name") or local_pyproject.get("tool", {}).get("poetry", {}).get("name")if isinstance(name_obj, str):return name_obj- except FileNotFoundError as exc:- print(f"Local pyproject.toml not found at {pyproject_path}: {exc}")- except toml.TomlDecodeError as exc:- print(f"Failed to parse local pyproject.toml at {pyproject_path}: {exc}")- except (KeyError, TypeError, AttributeError) as exc:- print(f"Invalid structure in local pyproject.toml at {pyproject_path}: {exc}")# Check setup.cfg- setup_cfg_path = os.path.join(self.local_root_dir, "setup.cfg")+ setup_cfg_path = os.path.join(os.getcwd(), "setup.cfg")try:config = ConfigParser()config.read(setup_cfg_path)@@ -217,7 +199,7 @@ class ConfigManager:print(f"Failed to parse setup.cfg at {setup_cfg_path}: {exc}")# Check setup.py as last resort- setup_py_path = os.path.join(self.local_root_dir, "setup.py")+ setup_py_path = os.path.join(os.getcwd(), "setup.py")try:with open(setup_py_path) as f:content = f.read()@@ -240,4 +222,4 @@ class ConfigManager:return None-config_manager = ConfigManager()+config_manager = ConfigLoader()diff --git a/pipelex/tools/config/config_model.py b/pipelex/system/configuration/config_model.pysimilarity index 97%rename from pipelex/tools/config/config_model.pyrename to pipelex/system/configuration/config_model.pyindex f92ae20b9..da0de9a62 100644--- a/pipelex/tools/config/config_model.py+++ b/pipelex/system/configuration/config_model.py@@ -2,7 +2,7 @@ from typing import TypeVarfrom pydantic import BaseModel, ConfigDict-from pipelex.tools.exceptions import ConfigModelError+from pipelex.system.exceptions import ConfigModelErrorfrom pipelex.types import StrEnumStrEnumType = TypeVar("StrEnumType", bound=StrEnum)diff --git a/pipelex/tools/config/config_root.py b/pipelex/system/configuration/config_root.pysimilarity index 90%rename from pipelex/tools/config/config_root.pyrename to pipelex/system/configuration/config_root.pyindex 65096c4cf..8fdcafc1e 100644--- a/pipelex/tools/config/config_root.py+++ b/pipelex/system/configuration/config_root.py@@ -2,8 +2,8 @@ from typing import Anyfrom pydantic import ValidationError-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.exceptions import ConfigValidationError+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.exceptions import ConfigValidationErrorfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorfrom pipelex.types import StrEnumdiff --git a/pipelex/tools/environment.py b/pipelex/system/environment.pysimilarity index 94%rename from pipelex/tools/environment.pyrename to pipelex/system/environment.pyindex ec2cef78f..926daa4fe 100644--- a/pipelex/tools/environment.py+++ b/pipelex/system/environment.py@@ -2,7 +2,7 @@ import osfrom dotenv import load_dotenv-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionfrom pipelex.tools.misc.placeholder import value_is_placeholderload_dotenv(dotenv_path=".env", override=True)diff --git a/pipelex/tools/exceptions.py b/pipelex/system/exceptions.pysimilarity index 90%rename from pipelex/tools/exceptions.pyrename to pipelex/system/exceptions.pyindex 20eef6a10..bd3547502 100644--- a/pipelex/tools/exceptions.py+++ b/pipelex/system/exceptions.py@@ -14,6 +14,10 @@ class ToolException(RootException):pass+class NestedKeyConflictError(ToolException):+ """Raised when attempting to create nested keys under a non-dict value."""++class CredentialsError(RootException):passdiff --git a/pipelex/system/registries/__init__.py b/pipelex/system/registries/__init__.pynew file mode 100644index 000000000..8b1378917--- /dev/null+++ b/pipelex/system/registries/__init__.py@@ -0,0 +1 @@+diff --git a/pipelex/system/registries/class_registry_utils.py b/pipelex/system/registries/class_registry_utils.pynew file mode 100644index 000000000..c81a118a5--- /dev/null+++ b/pipelex/system/registries/class_registry_utils.py@@ -0,0 +1,197 @@+import inspect+import sys+import warnings+from pathlib import Path+from typing import Any++from kajson.kajson_manager import KajsonManager++from pipelex import log+from pipelex.config import get_config+from pipelex.tools.misc.file_utils import find_files_in_dir as base_find_files_in_dir+from pipelex.tools.typing.module_inspector import (+ ModuleFileError,+ find_classes_in_module,+ import_module_from_file,+ import_module_from_file_if_has_classes,+)+++class ClassRegistryUtils:+ @classmethod+ def register_classes_in_file(+ cls,+ file_path: str,+ base_class: type[Any] | None,+ is_include_imported: bool,+ ) -> None:+ """Processes a Python file to find and register classes."""+ module = import_module_from_file(file_path)++ # Find classes that match criteria+ classes_to_register = find_classes_in_module(+ module=module,+ base_class=base_class,+ include_imported=is_include_imported,+ )++ KajsonManager.get_class_registry().register_classes(classes=classes_to_register)++ @classmethod+ def register_classes_in_folder(+ cls,+ folder_path: str,+ base_class: type[Any] | None = None,+ is_recursive: bool = True,+ is_include_imported: bool = False,+ ) -> None:+ """Registers all classes in Python files within folders that are subclasses of base_class.+ If base_class is None, registers all classes.++ Args:+ folder_path: Path to folder containing Python files+ base_class: Optional base class to filter registerable classes+ is_recursive: Whether to search recursively in subdirectories+ include_imported: Whether to include classes imported from other modules+ is_include_imported: Whether to include classes imported from other modules++ """+ python_files = cls.find_files_in_dir(+ dir_path=folder_path,+ pattern="*.py",+ is_recursive=is_recursive,+ )++ for python_file in python_files:+ cls.register_classes_in_file(+ file_path=str(python_file),+ base_class=base_class,+ is_include_imported=is_include_imported,+ )++ @classmethod+ def find_files_in_dir(cls, dir_path: str, pattern: str, is_recursive: bool) -> list[Path]:+ """Find files matching a pattern in a directory, excluding common build/cache directories.++ Args:+ dir_path: Directory path to search in+ pattern: File pattern to match (e.g. "*.py")+ is_recursive: Whether to search recursively in subdirectories++ Returns:+ List of matching Path objects, filtered to exclude problematic directories++ """+ # Get all files using the base utility+ all_files = base_find_files_in_dir(dir_path, pattern, is_recursive)++ # Filter out files in excluded directories+ filtered_files: list[Path] = []+ excluded_dirs = get_config().pipelex.scan_config.excluded_dirs+ for file_path in all_files:+ # Check if any parent directory is in the exclude list+ should_exclude = any(part in excluded_dirs for part in file_path.parts)+ if not should_exclude:+ filtered_files.append(file_path)++ return filtered_files++ @classmethod+ def import_modules_in_folder(+ cls,+ folder_path: str,+ is_recursive: bool = True,+ base_class_names: list[str] | None = None,+ ) -> None:+ """Import Python modules without registering their classes.++ This loads modules into sys.modules so their classes are available+ for discovery by auto_register_all_subclasses().++ If base_class_names is provided, uses AST parsing to first check if files+ contain relevant classes before importing them. This avoids executing module-level+ code in files that don't contain the classes you're looking for.++ Args:+ folder_path: Path to folder containing Python files+ is_recursive: Whether to search recursively in subdirectories+ base_class_names: Optional list of base class names (e.g. ["StructuredContent"]).+ If provided, only imports files that contain classes inheriting+ from these base classes. If None, imports all Python files.++ """+ python_files = cls.find_files_in_dir(+ dir_path=folder_path,+ pattern="*.py",+ is_recursive=is_recursive,+ )++ for python_file in python_files:+ try:+ if base_class_names is not None:+ # Use AST-based import to avoid executing modules without relevant classes+ import_module_from_file_if_has_classes(+ str(python_file),+ base_class_names=base_class_names,+ )+ else:+ # Import all modules regardless of content+ import_module_from_file(str(python_file))+ except ModuleFileError:+ # Expected: file validation issues (directories with .py extension, etc.)+ # log.debug(f"Skipping file {python_file}: {e}")+ pass+ except ImportError:+ # Common: missing dependencies, circular imports, relative imports+ # log.debug(f"Could not import {python_file}: {e}"+ pass+ except SyntaxError as exc:+ # Potentially problematic: invalid Python syntax may indicate broken code+ log.warning(f"Syntax error in {python_file}: {exc}")++ @classmethod+ def auto_register_all_subclasses(+ cls,+ base_class: type[Any],+ ) -> int:+ """Scan all loaded modules in sys.modules and register all subclasses of base_class.++ This enables auto-discovery of classes that are already in memory,+ making them available to concepts without explicit registration.++ Args:+ base_class: Base class to filter by (e.g., StructuredContent)++ Returns:+ Number of classes registered++ """+ registered_count = 0+ class_registry = KajsonManager.get_class_registry()++ # Create a snapshot of modules to avoid "dictionary changed size during iteration" error+ # (inspect.getmembers can trigger imports which modify sys.modules)+ modules_snapshot = list(sys.modules.values())++ # Iterate through all loaded modules+ for module in modules_snapshot:+ try:+ # Suppress all warnings during inspection (deprecation warnings from dependencies)+ with warnings.catch_warnings():+ warnings.simplefilter("ignore")+ # Find all classes in this module+ for _, obj in inspect.getmembers(module, inspect.isclass):+ # Check if it's a subclass of base_class (but not the base_class itself)+ if obj is not base_class and issubclass(obj, base_class):+ # Register if not already registered+ if not class_registry.has_class(name=obj.__name__):+ class_registry.register_class(obj)+ registered_count += 1+ except (AttributeError, ImportError, TypeError):+ # Expected: some modules in sys.modules can't be inspected+ # - Built-in/native modules (ImportError)+ # - Modules without expected attributes (AttributeError)+ # - Non-module objects (TypeError)+ pass++ return registered_countdiff --git a/pipelex/tools/func_registry.py b/pipelex/system/registries/func_registry.pysimilarity index 67%rename from pipelex/tools/func_registry.pyrename to pipelex/system/registries/func_registry.pyindex 7b7532417..e9abaff1a 100644--- a/pipelex/tools/func_registry.py+++ b/pipelex/system/registries/func_registry.py@@ -5,7 +5,8 @@ from typing import Any, TypeVar, cast, get_type_hintsfrom pydantic import Field, PrivateAttr, RootModel-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolException+from pipelex.urls import URLsFUNC_REGISTRY_LOGGER_CHANNEL_NAME = "func_registry"@@ -13,11 +14,51 @@ FUNC_REGISTRY_LOGGER_CHANNEL_NAME = "func_registry"T = TypeVar("T")FuncRegistryDict = dict[str, Callable[..., Any]]+# Attribute name used by the decorator to mark functions for registration+PIPE_FUNC_MARKER = "_is_pipe_func"+class FuncRegistryError(ToolException):pass+def pipe_func(name: str | None = None) -> Callable[[T], T]:+ """Decorator to mark a function for automatic registration in the func_registry.++ This decorator marks functions to be discovered and registered for use in PipeFunc operators.+ Functions marked with this decorator must follow the PipeFunc signature:+ - Accept exactly one parameter named "working_memory" of type WorkingMemory+ - Return a StuffContent or subclass++ Args:+ name: Optional custom name for registration. If not provided, uses function's __name__++ Returns:+ The decorated function unchanged, but marked for registration++ Example:+ @pipe_func()+ async def my_custom_function(working_memory: WorkingMemory) -> TextContent:+ result = working_memory.get_stuff("input")+ return TextContent(text=f"Processed: {result}")++ @pipe_func(name="custom_name")+ async def another_function(working_memory: WorkingMemory) -> MyContent:+ return MyContent(data="example")++ """++ def decorator(func: T) -> T:+ # Mark the function with the attribute+ setattr(func, PIPE_FUNC_MARKER, True)+ # Store custom name if provided+ if name is not None:+ func._pipe_func_name = name # type: ignore[attr-defined] # noqa: SLF001+ return func++ return decorator++class FuncRegistry(RootModel[FuncRegistryDict]):root: FuncRegistryDict = Field(default_factory=dict)_logger: logging.Logger = PrivateAttr(logging.getLogger(FUNC_REGISTRY_LOGGER_CHANNEL_NAME))@@ -36,7 +77,6 @@ class FuncRegistry(RootModel[FuncRegistryDict]):self,func: Callable[..., Any],name: str | None = None,- should_warn_if_already_registered: bool = True,) -> None:"""Registers a function in the registry with a name if it meets eligibility criteria."""if not self.is_eligible_function(func):@@ -44,11 +84,7 @@ class FuncRegistry(RootModel[FuncRegistryDict]):key = name or func.__name__if key in self.root:- if should_warn_if_already_registered:- self.log(f"Function '{key}' already exists in registry")- else:- msg = f"Function '{key}' already exists in registry"- raise FuncRegistryError(msg)+ self.log(f"Function '{key}' already exists in registry")else:self.log(f"Registered new single function '{key}' in registry")self.root[key] = func@@ -72,12 +108,12 @@ class FuncRegistry(RootModel[FuncRegistryDict]):def register_functions_dict(self, functions: dict[str, Callable[..., Any]]) -> None:"""Registers multiple functions in the registry with names if they meet eligibility criteria."""for name, func in functions.items():- self.register_function(func=func, name=name, should_warn_if_already_registered=False)+ self.register_function(func=func, name=name)def register_functions(self, functions: list[Callable[..., Any]]) -> None:"""Registers multiple functions in the registry with names if they meet eligibility criteria."""for func in functions:- self.register_function(func=func, should_warn_if_already_registered=False)+ self.register_function(func=func)def get_function(self, name: str) -> Callable[..., Any] | None:"""Retrieves a function from the registry by its name. Returns None if not found."""@@ -87,8 +123,10 @@ class FuncRegistry(RootModel[FuncRegistryDict]):"""Retrieves a function from the registry by its name. Raises an error if not found."""if name not in self.root:msg = (- f"Function '{name}' not found in registry:"- "See how to register a function here: https://docs.pipelex.com/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc"+ f"Function '{name}' not found in registry. "+ f"Since v0.12.0, custom functions require the @pipe_func() decorator for auto-discovery. "+ f"Add @pipe_func() above your function definition. "+ f"See: {URLs.pipe_func_docs}")raise FuncRegistryError(msg)return self.root[name]@@ -113,15 +151,40 @@ class FuncRegistry(RootModel[FuncRegistryDict]):"""Checks if a function is in the registry by its name."""return name in self.root- def is_eligible_function(self, func: Any) -> bool:+ def is_marked_pipe_func(self, func: Any) -> bool:+ """Checks if a function is marked with the @pipe_func decorator.++ Args:+ func: The function to check++ Returns:+ True if the function has the pipe_func marker attribute++ """+ return hasattr(func, PIPE_FUNC_MARKER) and getattr(func, PIPE_FUNC_MARKER) is True++ def is_eligible_function(self, func: Any, require_decorator: bool = False) -> bool:"""Checks if a function matches the criteria for PipeFunc registration:- Must be callable- Exactly 1 parameter named "working_memory" with type WorkingMemory- Return type that is a subclass of StuffContent+ - Optionally must be marked with @pipe_func decorator if require_decorator=True++ Args:+ func: The function to check+ require_decorator: If True, only functions marked with @pipe_func are eligible++ Returns:+ True if the function meets all eligibility criteria+"""if not callable(func):return False+ # If decorator is required, check for it first (fast check)+ if require_decorator and not self.is_marked_pipe_func(func):+ return False+the_function = cast("Callable[..., Any]", func)try:diff --git a/pipelex/system/registries/func_registry_utils.py b/pipelex/system/registries/func_registry_utils.pynew file mode 100644index 000000000..6f6458131--- /dev/null+++ b/pipelex/system/registries/func_registry_utils.py@@ -0,0 +1,217 @@+import importlib+import inspect+import pkgutil+from collections.abc import Callable+from pathlib import Path+from typing import Any++from pipelex import log+from pipelex.config import get_config+from pipelex.system.registries.func_registry import func_registry, pipe_func+from pipelex.tools.misc.file_utils import find_files_in_dir as base_find_files_in_dir+from pipelex.tools.typing.module_inspector import (+ ModuleFileError,+ import_module_from_file_if_has_decorated_functions,+)+++class FuncRegistryUtils:+ @classmethod+ def register_pipe_funcs_from_package(cls, package_name: str, package: Any) -> int:+ """Register all @pipe_func decorated functions from a package.++ Args:+ package_name: Full name of the package (e.g. "pipelex.builder")+ package: The imported package object++ Returns:+ Number of functions registered++ """+ functions_registered = 0++ if not hasattr(package, "__path__"):+ log.warning(f"Package {package_name} has no __path__ attribute, cannot walk modules")+ return 0++ log.verbose(f"Walking package {package_name} at {package.__path__}")++ for _importer, modname, _ispkg in pkgutil.walk_packages(+ path=package.__path__,+ prefix=f"{package_name}.",+ onerror=lambda _: None,+ ):+ # Import the module+ module = importlib.import_module(modname)+ log.verbose(f"Imported {modname}")++ # Find @pipe_func decorated functions in this module+ functions_to_register = cls._find_functions_in_module(module)++ for func in functions_to_register:+ func_name = cls._get_function_registration_name(func)+ func_registry.register_function(+ func=func,+ name=func_name,+ )+ functions_registered += 1+ log.verbose(f"Registered @pipe_func: {func_name} from {modname}")++ return functions_registered++ @classmethod+ def register_funcs_in_folder(+ cls,+ folder_path: str,+ is_recursive: bool = True,+ ) -> None:+ """Discovers and attempts to register all functions in Python files within a folder.+ Only functions that meet the eligibility criteria will be registered:+ - Must be an async function+ - Exactly 1 parameter named "working_memory" with type WorkingMemory+ - Return type that is a subclass of StuffContent+ - Must be marked with the @pipe_func decorator++ Uses AST parsing to first check if files contain @pipe_func decorated functions+ before importing them. This avoids executing module-level code in files that+ don't contain the functions you're looking for.++ The function name is used as the registry key (or custom name if provided to decorator).++ Args:+ folder_path: Path to folder containing Python files+ is_recursive: Whether to search recursively in subdirectories++ """+ python_files = cls._find_files_in_dir(+ dir_path=folder_path,+ pattern="*.py",+ is_recursive=is_recursive,+ )++ for python_file in python_files:+ cls._register_funcs_in_file(file_path=str(python_file))++ @classmethod+ def _register_funcs_in_file(+ cls,+ file_path: str,+ ) -> None:+ """Processes a Python file to find and register eligible @pipe_func decorated functions.++ Uses AST parsing to check if the file contains @pipe_func decorated functions before+ importing. Only functions marked with @pipe_func decorator are registered.++ Args:+ file_path: Path to the Python file++ """+ try:+ # Import the module only if it has @pipe_func decorated functions+ module = import_module_from_file_if_has_decorated_functions(+ file_path,+ decorator_names=[pipe_func.__name__],+ )+ # If no decorated functions found, module will be None+ if module is None:+ return++ # Find functions that match criteria+ functions_to_register = cls._find_functions_in_module(module)++ for func in functions_to_register:+ func_name = cls._get_function_registration_name(func)+ func_registry.register_function(+ func=func,+ name=func_name,+ )+ except ModuleFileError:+ # Expected: file validation issues (directories with .py extension, etc.)+ # log.verbose(f"Skipping file {file_path}: {e}")+ pass+ except ImportError:+ # Common: missing dependencies, circular imports, relative imports+ # log.verbose(f"Could not import {file_path}: {e}")+ pass+ except SyntaxError as exc:+ # Potentially problematic: invalid Python syntax may indicate broken code+ log.warning(f"Syntax error in {file_path}: {exc}")++ @classmethod+ def _find_functions_in_module(+ cls,+ module: Any,+ ) -> list[Callable[..., Any]]:+ """Finds all @pipe_func decorated functions in a module.++ Only functions marked with @pipe_func decorator are included.+ Full eligibility (signature, return type) will be checked during registration.++ Args:+ module: The module to search for functions++ Returns:+ List of @pipe_func decorated functions found in the module++ """+ functions: list[Callable[..., Any]] = []+ module_name = module.__name__++ # Find all functions in the module (not imported ones)+ for _, obj in inspect.getmembers(module, inspect.isfunction):+ # Skip functions imported from other modules+ if obj.__module__ != module_name:+ continue++ # Only include functions marked with @pipe_func+ if not func_registry.is_marked_pipe_func(obj):+ continue++ # Add function - full eligibility will be checked by func_registry.register_function+ functions.append(obj)++ return functions++ @classmethod+ def _get_function_registration_name(cls, func: Callable[..., Any]) -> str:+ """Extract the registration name for a function.++ If the function has a custom name from the @pipe_func decorator, use that.+ Otherwise, use the function's __name__.++ Args:+ func: The function to get the registration name for++ Returns:+ The name to use when registering the function++ """+ custom_name = getattr(func, "_pipe_func_name", None)+ return custom_name if custom_name is not None else func.__name__++ @classmethod+ def _find_files_in_dir(cls, dir_path: str, pattern: str, is_recursive: bool) -> list[Path]:+ """Find files matching a pattern in a directory, excluding common build/cache directories.++ Args:+ dir_path: Directory path to search in+ pattern: File pattern to match (e.g. "*.py")+ is_recursive: Whether to search recursively in subdirectories++ Returns:+ List of matching Path objects, filtered to exclude problematic directories++ """+ # Get all files using the base utility+ all_files = base_find_files_in_dir(dir_path, pattern, is_recursive)++ # Filter out files in excluded directories+ filtered_files: list[Path] = []+ excluded_dirs = get_config().pipelex.scan_config.excluded_dirs+ for file_path in all_files:+ # Check if any parent directory is in the exclude list+ should_exclude = any(part in excluded_dirs for part in file_path.parts)+ if not should_exclude:+ filtered_files.append(file_path)++ return filtered_filesdiff --git a/pipelex/tools/registry_models.py b/pipelex/system/registries/registry_base.pysimilarity index 100%rename from pipelex/tools/registry_models.pyrename to pipelex/system/registries/registry_base.pydiff --git a/pipelex/tools/runtime_manager.py b/pipelex/system/runtime.pysimilarity index 97%rename from pipelex/tools/runtime_manager.pyrename to pipelex/system/runtime.pyindex 6a3fb9f02..f23613e07 100644--- a/pipelex/tools/runtime_manager.py+++ b/pipelex/system/runtime.py@@ -1,6 +1,6 @@from pydantic import BaseModel-from pipelex.tools.environment import get_optional_env+from pipelex.system.environment import get_optional_envfrom pipelex.types import StrEnumRUN_MODE_ENV_VAR_KEY = "RUN_MODE"diff --git a/pipelex/test_extras/registry_test_models.py b/pipelex/test_extras/registry_test_models.pyindex 1071fea5d..20556ece7 100644--- a/pipelex/test_extras/registry_test_models.py+++ b/pipelex/test_extras/registry_test_models.py@@ -1,7 +1,7 @@from typing import ClassVarfrom pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.tools.registry_models import ModelType, RegistryModels+from pipelex.system.registries.registry_base import ModelType, RegistryModelsclass FictionCharacter(StructuredContent):@@ -11,5 +11,5 @@ class FictionCharacter(StructuredContent):backstory: str-class PipelexTestModels(RegistryModels):+class TestRegistryModels(RegistryModels):TEST_MODELS: ClassVar[list[ModelType]] = [FictionCharacter]diff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.pyindex 086bc5ea9..38f36c39f 100644--- a/pipelex/test_extras/shared_pytest_plugins.py+++ b/pipelex/test_extras/shared_pytest_plugins.py@@ -5,9 +5,9 @@ from pytest import FixtureRequest, Parserfrom rich import print # noqa: A004from pipelex.pipe_run.pipe_run_params import PipeRunMode-from pipelex.tools.environment import is_env_var_set, set_env+from pipelex.system.environment import is_env_var_set, set_env+from pipelex.system.runtime import RunMode, runtime_managerfrom pipelex.tools.misc.placeholder import make_placeholder_value, value_is_placeholder-from pipelex.tools.runtime_manager import RunMode, runtime_manager# List of environment variables that may need placeholders in CIENV_VAR_KEYS_WHICH_MAY_NEED_PLACEHOLDERS_IN_CI = [diff --git a/pipelex/tools/aws/aws_config.py b/pipelex/tools/aws/aws_config.pyindex cace6d869..cf591cc12 100644--- a/pipelex/tools/aws/aws_config.py+++ b/pipelex/tools/aws/aws_config.py@@ -2,9 +2,9 @@ from pydantic import Fieldfrom pipelex import logfrom pipelex.hub import get_secret-from pipelex.tools.config.config_model import ConfigModel-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env-from pipelex.tools.exceptions import CredentialsError+from pipelex.system.configuration.config_model import ConfigModel+from pipelex.system.environment import EnvVarNotFoundError, get_required_env+from pipelex.system.exceptions import CredentialsErrorfrom pipelex.tools.secrets.secrets_errors import SecretNotFoundErrorfrom pipelex.types import StrEnumdiff --git a/pipelex/tools/class_registry_utils.py b/pipelex/tools/class_registry_utils.pydeleted file mode 100644index 18e7384b8..000000000--- a/pipelex/tools/class_registry_utils.py+++ /dev/null@@ -1,154 +0,0 @@-import types-from pathlib import Path-from typing import TYPE_CHECKING, Annotated, Any, Union, get_args, get_origin--from kajson.kajson_manager import KajsonManager--if TYPE_CHECKING:- from pydantic.fields import FieldInfo--from pipelex.tools.typing.module_inspector import find_classes_in_module, import_module_from_file--_NoneType = type(None)-_UnionType = getattr(types, "UnionType", None) # Py3.10+: types.UnionType---class ClassRegistryUtils:- @classmethod- def register_classes_in_file(- cls,- file_path: str,- base_class: type[Any] | None,- is_include_imported: bool,- ) -> None:- """Processes a Python file to find and register classes."""- module = import_module_from_file(file_path)-- # Find classes that match criteria- classes_to_register = find_classes_in_module(- module=module,- base_class=base_class,- include_imported=is_include_imported,- )-- KajsonManager.get_class_registry().register_classes(classes=classes_to_register)-- @classmethod- def register_classes_in_folder(- cls,- folder_path: str,- base_class: type[Any] | None = None,- is_recursive: bool = True,- is_include_imported: bool = False,- ) -> None:- """Registers all classes in Python files within folders that are subclasses of base_class.- If base_class is None, registers all classes.-- Args:- folder_path: Path to folder containing Python files- base_class: Optional base class to filter registerable classes- is_recursive: Whether to search recursively in subdirectories- include_imported: Whether to include classes imported from other modules- is_include_imported: Whether to include classes imported from other modules-- """- python_files = cls.find_files_in_dir(- dir_path=folder_path,- pattern="*.py",- is_recursive=is_recursive,- )-- for python_file in python_files:- cls.register_classes_in_file(- file_path=str(python_file),- base_class=base_class,- is_include_imported=is_include_imported,- )-- @classmethod- def find_files_in_dir(cls, dir_path: str, pattern: str, is_recursive: bool) -> list[Path]:- """Find files matching a pattern in a directory.-- Args:- dir_path: Directory path to search in- pattern: File pattern to match (e.g. "*.py")- is_recursive: Whether to search recursively in subdirectories-- Returns:- List of matching Path objects-- """- path = Path(dir_path)- if is_recursive:- return list(path.rglob(pattern))- return list(path.glob(pattern))-- @staticmethod- def are_classes_equivalent(class_1: type[Any], class_2: type[Any]) -> bool:- """Check if two Pydantic classes are equivalent (same fields, types, descriptions)."""- if not (hasattr(class_1, "model_fields") and hasattr(class_2, "model_fields")):- return class_1 == class_2-- # Compare model schemas using Pydantic's built-in capabilities- try:- schema_1: dict[str, Any] = class_1.model_json_schema()- schema_2: dict[str, Any] = class_2.model_json_schema()- return schema_1 == schema_2- except Exception:- # Fallback to manual field comparison if schema comparison fails- fields_1: dict[str, FieldInfo] = class_1.model_fields- fields_2: dict[str, FieldInfo] = class_2.model_fields-- if set(fields_1.keys()) != set(fields_2.keys()):- return False-- for field_1_name, field_1_info in fields_1.items():- field_1: FieldInfo = field_1_info- field_2: FieldInfo = fields_2[field_1_name]-- # Compare field types- if field_1.annotation != field_2.annotation:- return False-- # Compare field descriptions if they exist- if getattr(field_1, "description", None) != getattr(field_2, "description", None):- return False-- # Compare default values- if field_1.default != field_2.default:- return False-- return True-- @staticmethod- def has_compatible_field(class_1: type[Any], class_2: type[Any]) -> bool:- """Check if class_1 has a field whose (possibly wrapped) type matches/subclasses class_2."""- if not hasattr(class_1, "model_fields"):- return False-- fields: dict[str, FieldInfo] = class_1.model_fields # type: ignore[attr-defined]-- def _is_compatible(t: Any) -> bool:- # Unwrap Annotated[T, ...]- if get_origin(t) is Annotated:- t = get_args(t)[0]-- origin = get_origin(t)-- # Handle unions, including PEP 604 (T | None)- if origin in (Union, _UnionType):- for arg in get_args(t):- if arg is _NoneType:- continue- if _is_compatible(arg):- return True- return False-- # Base case: direct match / subclass- try:- return t is class_2 or (isinstance(t, type) and issubclass(t, class_2))- except TypeError:- # Not a class type (e.g., typing constructs you don't care about)- return False-- return any(_is_compatible(field.annotation) for field in fields.values())diff --git a/pipelex/tools/func_registry_utils.py b/pipelex/tools/func_registry_utils.pydeleted file mode 100644index 469f8646d..000000000--- a/pipelex/tools/func_registry_utils.py+++ /dev/null@@ -1,91 +0,0 @@-import inspect-from collections.abc import Callable-from pathlib import Path-from typing import Any--from pipelex.tools.func_registry import func_registry-from pipelex.tools.typing.module_inspector import import_module_from_file---class FuncRegistryUtils:- @classmethod- def register_funcs_in_folder(- cls,- folder_path: str,- is_recursive: bool = True,- ) -> None:- """Discovers and attempts to register all functions in Python files within a folder.- Only functions that meet the eligibility criteria will be registered:- - Must be an async function- - Exactly 1 parameter named "working_memory" with type WorkingMemory- - Return type that is a subclass of StuffContent-- The function name is used as the registry key.-- Args:- folder_path: Path to folder containing Python files- is_recursive: Whether to search recursively in subdirectories-- """- python_files = cls._find_files_in_dir(- dir_path=folder_path,- pattern="*.py",- is_recursive=is_recursive,- )-- for python_file in python_files:- cls._register_funcs_in_file(file_path=str(python_file))-- @classmethod- def _register_funcs_in_file(cls, file_path: str) -> None:- """Processes a Python file to find and register eligible functions."""- try:- module = import_module_from_file(file_path)-- # Find functions that match criteria- functions_to_register = cls._find_functions_in_module(module)-- for func in functions_to_register:- func_registry.register_function(- func=func,- name=func.__name__,- should_warn_if_already_registered=True,- )- except Exception as e:- # Log error but continue processing other files- print(f"Error processing file {file_path}: {e}")-- @classmethod- def _find_functions_in_module(cls, module: Any) -> list[Callable[..., Any]]:- """Finds all functions in a module (eligibility will be checked during registration)."""- functions: list[Callable[..., Any]] = []- module_name = module.__name__-- # Find all functions in the module (not imported ones)- for _, obj in inspect.getmembers(module, inspect.isfunction):- # Skip functions imported from other modules- if obj.__module__ != module_name:- continue-- # Add all functions - eligibility will be checked by func_registry.register_function- functions.append(obj)-- return functions-- @classmethod- def _find_files_in_dir(cls, dir_path: str, pattern: str, is_recursive: bool) -> list[Path]:- """Find files matching a pattern in a directory.-- Args:- dir_path: Directory path to search in- pattern: File pattern to match (e.g. "*.py")- is_recursive: Whether to search recursively in subdirectories-- Returns:- List of matching Path objects-- """- path = Path(dir_path)- if is_recursive:- return list(path.rglob(pattern))- return list(path.glob(pattern))diff --git a/pipelex/tools/jinja2/jinja2_errors.py b/pipelex/tools/jinja2/jinja2_errors.pyindex 27f918f7d..c24ad62c9 100644--- a/pipelex/tools/jinja2/jinja2_errors.py+++ b/pipelex/tools/jinja2/jinja2_errors.py@@ -1,4 +1,4 @@-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionclass Jinja2TemplateSyntaxError(ToolException):diff --git a/pipelex/tools/jinja2/jinja2_rendering.py b/pipelex/tools/jinja2/jinja2_rendering.pyindex ad6fbb9a2..5d984150f 100644--- a/pipelex/tools/jinja2/jinja2_rendering.py+++ b/pipelex/tools/jinja2/jinja2_rendering.py@@ -1,15 +1,12 @@from typing import Any-from jinja2 import Template, metafrom jinja2.exceptions import (TemplateAssertionError,TemplateSyntaxError,UndefinedError,)-from pipelex import logfrom pipelex.cogt.templating.template_category import TemplateCategory-from pipelex.cogt.templating.template_preprocessor import preprocess_templatefrom pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.tools.jinja2.jinja2_environment import make_jinja2_env_without_loaderfrom pipelex.tools.jinja2.jinja2_errors import (@@ -18,7 +15,6 @@ from pipelex.tools.jinja2.jinja2_errors import (Jinja2TemplateRenderError,)from pipelex.tools.jinja2.jinja2_models import Jinja2ContextKey-from pipelex.tools.jinja2.jinja2_parsing import check_jinja2_parsingdef _add_to_templating_context(temlating_context: dict[str, Any], jinja2_context_key: Jinja2ContextKey, value: Any) -> None:@@ -38,23 +34,13 @@ async def render_jinja2(template_category=template_category,)- template: Templatetry:template = jinja2_env.from_string(template_source)except TemplateAssertionError as exc:msg = f"Jinja2 render error: '{exc}', template_source:\n{template_source}"raise Jinja2TemplateRenderError(msg) from exc- template_source = preprocess_template(template_source)- check_jinja2_parsing(- template_source=template_source,- template_category=template_category,- )- parsed_ast = jinja2_env.parse(template_source)- if undeclared_variables := meta.find_undeclared_variables(parsed_ast):- undeclared_variables.discard("preliminary_text")- if undeclared_variables:- log.verbose(undeclared_variables, "Jinja2 undeclared_variables")+ # Create a copy to avoid mutating the caller's original dictionarytemlating_context = temlating_context.copy()if templating_style:_add_to_templating_context(diff --git a/pipelex/tools/log/log_config.py b/pipelex/tools/log/log_config.pyindex c2ae2e5ac..bd8b56d4b 100644--- a/pipelex/tools/log/log_config.py+++ b/pipelex/tools/log/log_config.py@@ -5,7 +5,7 @@ from pydantic import Field, field_validatorfrom rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighterfrom rich.logging import RichHandler-from pipelex.tools.config.config_model import ConfigModel+from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.tools.log.log_levels import LogLevelfrom pipelex.types import StrEnumdiff --git a/pipelex/tools/misc/base_64_utils.py b/pipelex/tools/misc/base_64_utils.pyindex 14cda8867..fe7cab3ce 100644--- a/pipelex/tools/misc/base_64_utils.py+++ b/pipelex/tools/misc/base_64_utils.py@@ -4,6 +4,7 @@ import base64import aiofilesfrom pipelex.tools.misc.file_utils import save_bytes_to_binary_file+from pipelex.tools.misc.filetype_utils import detect_file_type_from_base64def load_binary_as_base64(path: str) -> bytes:@@ -31,19 +32,39 @@ async def encode_to_base64_async(data_bytes: bytes) -> bytes:return await asyncio.to_thread(base64.b64encode, data_bytes)-def save_base64_to_binary_file(- b64: str,- file_path: str,-):- # Ensure we're getting clean base64 data without any prefixes- base64_str = b64- # Remove potential data URL prefix if present+def strip_base_64_str_if_needed(base64_str: str) -> str:if "," in base64_str:- base64_str = base64_str.split(",", 1)[1]+ return base64_str.split(",", 1)[1]if "data:" in base64_str and ";base64," in base64_str:- base64_str = base64_str.split(";base64,", 1)[1]+ return base64_str.split(";base64,", 1)[1]+ return base64_str+++def prefixed_base64_str_from_base64_bytes(b64_bytes: bytes) -> str:+ file_type = detect_file_type_from_base64(b64_bytes)+ return f"data:{file_type.mime};base64,{base64.b64encode(b64_bytes).decode('utf-8')}"+++def prefixed_base64_str_from_base64_str(b64_str: str) -> str:+ """Create a data URL from an already base64-encoded string.++ Args:+ b64_str: Base64-encoded string (without data URL prefix)++ Returns:+ Data URL string: data:{mime};base64,{b64_str}+ """+ file_type = detect_file_type_from_base64(b64_str)+ return f"data:{file_type.mime};base64,{b64_str}"+++def save_base_64_str_to_binary_file(+ base_64_str: str,+ file_path: str,+):+ stripped_base_64_str = strip_base_64_str_if_needed(base_64_str)# Decode base64- byte_data = base64.b64decode(base64_str)+ byte_data = base64.b64decode(stripped_base_64_str)save_bytes_to_binary_file(file_path=file_path, byte_data=byte_data)diff --git a/pipelex/tools/misc/context_provider_abstract.py b/pipelex/tools/misc/context_provider_abstract.pyindex 23ba2c56e..84e8a8608 100644--- a/pipelex/tools/misc/context_provider_abstract.py+++ b/pipelex/tools/misc/context_provider_abstract.py@@ -1,7 +1,7 @@from abc import ABC, abstractmethodfrom typing import Any-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionclass ContextProviderException(ToolException):@@ -20,5 +20,5 @@ class ContextProviderAbstract(ABC):pass@abstractmethod- def generate_jinja2_context(self) -> dict[str, Any]:+ def generate_context(self) -> dict[str, Any]:passdiff --git a/pipelex/tools/misc/dict_utils.py b/pipelex/tools/misc/dict_utils.pyindex fe8c88a83..8db4d088f 100644--- a/pipelex/tools/misc/dict_utils.py+++ b/pipelex/tools/misc/dict_utils.py@@ -3,6 +3,9 @@from collections.abc import Callablefrom typing import Any, TypeVar, cast+from pipelex import log+from pipelex.system.exceptions import NestedKeyConflictError+K = TypeVar("K")V = TypeVar("V")@@ -93,3 +96,64 @@ def apply_to_strings_in_list(data: list[Any], transform_func: Callable[[str], stelse:result.append(item)return result+++def substitute_nested_in_context(context: dict[str, Any], extra_params: dict[str, Any] | None = None) -> dict[str, Any]:+ """Substitute nested values in context dict using dotted key notation.++ This function processes keys from extra_params that contain dots (e.g., "foo.bar.blip")+ and creates nested dictionary structures in the context dict. Keys without dots are+ added directly to the context.++ Args:+ context: The context dictionary to mutate+ extra_params: Dictionary with potentially dotted keys to process++ Returns:+ The mutated context dictionary++ Raises:+ NestedKeyConflictError: When attempting to create nested keys under a non-dict value++ Example:+ >>> context = {}+ >>> extra_params = {"foo.bar.blip": "hello"}+ >>> substitute_nested_in_context(context, extra_params)+ >>> context+ {'foo': {'bar': {'blip': 'hello'}}}++ """+ if not extra_params:+ return context++ original_context = context.copy()++ for key, value in extra_params.items():+ if "." not in key:+ # Simple key without dots - add directly to context+ context[key] = value+ else:+ # Dotted key - create nested structure+ segments = key.split(".")+ current = context++ # Navigate/create nested dicts for all segments except the last+ for segment in segments[:-1]:+ if segment not in current:+ # Create new nested dict+ current[segment] = {}+ elif not (hasattr(current[segment], "__getitem__") and hasattr(current[segment], "__setitem__")):+ # Conflict: trying to nest under a non-dict-like value+ # Must support both __getitem__ and __setitem__ to be dict-like (e.g., dict, StuffArtefact)+ error_message = f"Cannot set nested key '{key}': '{segment}' is not a dict-like object"+ log.error(original_context, title="original_context")+ log.error(extra_params, title="extra_params")+ raise NestedKeyConflictError(error_message)+ # Navigate into the nested dict+ current = current[segment]++ # Set the final value+ last_segment = segments[-1]+ current[last_segment] = value++ return contextdiff --git a/pipelex/tools/misc/filetype_utils.py b/pipelex/tools/misc/filetype_utils.pyindex af274d770..a1659e4b9 100644--- a/pipelex/tools/misc/filetype_utils.py+++ b/pipelex/tools/misc/filetype_utils.py@@ -5,7 +5,8 @@ from pathlib import Pathimport filetypefrom pydantic import BaseModel-from pipelex.tools.exceptions import ToolException+from pipelex import log+from pipelex.system.exceptions import ToolExceptionclass FileTypeException(ToolException):@@ -76,16 +77,19 @@ def detect_file_type_from_base64(b64: str | bytes) -> FileType:"""# Normalise to bytes holding only the Base-64 alphabetif isinstance(b64, bytes):+ log.debug(f"b64 is already bytes: {b64[:100]!r}")b64_bytes = b64else: # str → handle optional data-URL header+ log.debug(f"b64 is a string: {b64[:100]!r}")if b64.lstrip().startswith("data:") and "," in b64:b64 = b64.split(",", 1)[1]+ log.debug(f"b64 after split: {b64[:100]!r}")b64_bytes = b64.encode("ascii") # Base-64 is pure ASCIItry:raw = base64.b64decode(b64_bytes, validate=True)except binascii.Error as exc: # malformed Base-64- msg = f"Could not identify file type of given bytes because input is not valid Base-64: {exc}"+ msg = f"Could not identify file type of given bytes because input is not valid Base-64: {exc}\n{b64_bytes[:100]!r}"raise FileTypeException(msg) from excreturn detect_file_type_from_bytes(buf=raw)diff --git a/pipelex/tools/misc/json_utils.py b/pipelex/tools/misc/json_utils.pyindex 8a230f122..7950dfe84 100644--- a/pipelex/tools/misc/json_utils.py+++ b/pipelex/tools/misc/json_utils.py@@ -5,7 +5,7 @@ from typing import Any, Union, castfrom kajson import kajsonfrom pydantic import BaseModel-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionfrom pipelex.tools.misc.file_utils import save_text_to_pathfrom pipelex.tools.typing.pydantic_utils import CustomBaseModeldiff --git a/pipelex/tools/misc/toml_utils.py b/pipelex/tools/misc/toml_utils.pyindex b01ea4515..df67e28de 100644--- a/pipelex/tools/misc/toml_utils.py+++ b/pipelex/tools/misc/toml_utils.py@@ -2,11 +2,33 @@ from __future__ import annotationsfrom typing import Any-import toml+import tomli+from pipelex.system.exceptions import ToolExceptionfrom pipelex.tools.misc.file_utils import path_exists+class TomlError(ToolException):+ def __init__(self, message: str, doc: str, pos: int, lineno: int, colno: int):+ super().__init__(message)+ self.doc = doc+ self.pos = pos+ self.lineno = lineno+ self.colno = colno++ @classmethod+ def from_tomli_error(cls, exc: tomli.TOMLDecodeError) -> TomlError:+ return cls(message=exc.msg, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno)+++def load_toml_from_content(content: str) -> dict[str, Any]:+ """Load TOML from content."""+ try:+ return tomli.loads(content)+ except tomli.TOMLDecodeError as exc:+ raise TomlError.from_tomli_error(exc) from exc++def load_toml_from_path(path: str) -> dict[str, Any]:"""Load TOML from path.@@ -21,14 +43,11 @@ def load_toml_from_path(path: str) -> dict[str, Any]:"""try:- with open(path, encoding="utf-8") as file:- content = file.read()-- # Parse TOML first- return toml.loads(content)- except toml.TomlDecodeError as exc:- msg = f"TOML parsing error in file '{path}': {exc}"- raise toml.TomlDecodeError(msg, exc.doc, exc.pos) from exc+ with open(path, "rb") as f:+ return tomli.load(f)+ except tomli.TOMLDecodeError as exc:+ msg = f"TOML parsing error in file '{path}': {exc.msg}"+ raise TomlError(message=msg, doc=exc.doc, pos=exc.pos, lineno=exc.lineno, colno=exc.colno) from excdef load_toml_from_path_if_exists(path: str) -> dict[str, Any] | None:diff --git a/pipelex/tools/pdf/pypdfium2_renderer.py b/pipelex/tools/pdf/pypdfium2_renderer.pyindex 35a996a97..97fdb9f34 100644--- a/pipelex/tools/pdf/pypdfium2_renderer.py+++ b/pipelex/tools/pdf/pypdfium2_renderer.py@@ -7,7 +7,7 @@ from typing import TYPE_CHECKINGimport pypdfium2 as pdfiumfrom pypdfium2.raw import FPDFBitmap_BGRA-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionfrom pipelex.tools.misc.file_fetch_utils import fetch_file_from_url_httpx_asyncfrom pipelex.tools.misc.path_utils import clarify_path_or_urldiff --git a/pipelex/tools/secrets/env_secrets_provider.py b/pipelex/tools/secrets/env_secrets_provider.pyindex 14ccf218e..0976e49e2 100644--- a/pipelex/tools/secrets/env_secrets_provider.py+++ b/pipelex/tools/secrets/env_secrets_provider.py@@ -1,6 +1,6 @@from typing_extensions import override-from pipelex.tools.environment import EnvVarNotFoundError, get_optional_env, get_required_env+from pipelex.system.environment import EnvVarNotFoundError, get_optional_env, get_required_envfrom pipelex.tools.secrets.secrets_errors import SecretNotFoundErrorfrom pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstractdiff --git a/pipelex/tools/secrets/secrets_errors.py b/pipelex/tools/secrets/secrets_errors.pyindex 0f39641ab..9ad716962 100644--- a/pipelex/tools/secrets/secrets_errors.py+++ b/pipelex/tools/secrets/secrets_errors.py@@ -1,4 +1,4 @@-from pipelex.tools.exceptions import ToolException+from pipelex.system.exceptions import ToolExceptionclass SecretNotFoundError(ToolException):diff --git a/pipelex/tools/secrets/secrets_utils.py b/pipelex/tools/secrets/secrets_utils.pyindex 30d952d41..11d9ea8c8 100644--- a/pipelex/tools/secrets/secrets_utils.py+++ b/pipelex/tools/secrets/secrets_utils.py@@ -1,8 +1,8 @@import refrom pipelex.hub import get_secrets_provider-from pipelex.tools.environment import EnvVarNotFoundError, get_optional_env, get_required_env-from pipelex.tools.exceptions import ToolException+from pipelex.system.environment import EnvVarNotFoundError, get_optional_env, get_required_env+from pipelex.system.exceptions import ToolExceptionfrom pipelex.tools.secrets.secrets_errors import SecretNotFoundErrorfrom pipelex.types import StrEnumdiff --git a/pipelex/tools/typing/class_utils.py b/pipelex/tools/typing/class_utils.pynew file mode 100644index 000000000..c61024901--- /dev/null+++ b/pipelex/tools/typing/class_utils.py@@ -0,0 +1,79 @@+import types+from typing import TYPE_CHECKING, Annotated, Any, Union, get_args, get_origin++if TYPE_CHECKING:+ from pydantic.fields import FieldInfo+++_NoneType = type(None)+_UnionType = getattr(types, "UnionType", None) # Py3.10+: types.UnionType+++def are_classes_equivalent(class_1: type[Any], class_2: type[Any]) -> bool:+ """Check if two Pydantic classes are equivalent (same fields, types, descriptions)."""+ if not (hasattr(class_1, "model_fields") and hasattr(class_2, "model_fields")):+ return class_1 == class_2++ # Compare model schemas using Pydantic's built-in capabilities+ try:+ schema_1: dict[str, Any] = class_1.model_json_schema()+ schema_2: dict[str, Any] = class_2.model_json_schema()+ return schema_1 == schema_2+ except Exception:+ # Fallback to manual field comparison if schema comparison fails+ fields_1: dict[str, FieldInfo] = class_1.model_fields+ fields_2: dict[str, FieldInfo] = class_2.model_fields++ if set(fields_1.keys()) != set(fields_2.keys()):+ return False++ for field_1_name, field_1_info in fields_1.items():+ field_1: FieldInfo = field_1_info+ field_2: FieldInfo = fields_2[field_1_name]++ # Compare field types+ if field_1.annotation != field_2.annotation:+ return False++ # Compare field descriptions if they exist+ if getattr(field_1, "description", None) != getattr(field_2, "description", None):+ return False++ # Compare default values+ if field_1.default != field_2.default:+ return False++ return True+++def has_compatible_field(class_1: type[Any], class_2: type[Any]) -> bool:+ """Check if class_1 has a field whose (possibly wrapped) type matches/subclasses class_2."""+ if not hasattr(class_1, "model_fields"):+ return False++ fields: dict[str, FieldInfo] = class_1.model_fields # type: ignore[attr-defined]++ def _is_compatible(t: Any) -> bool:+ # Unwrap Annotated[T, ...]+ if get_origin(t) is Annotated:+ t = get_args(t)[0]++ origin = get_origin(t)++ # Handle unions, including PEP 604 (T | None)+ if origin in (Union, _UnionType):+ for arg in get_args(t):+ if arg is _NoneType:+ continue+ if _is_compatible(arg):+ return True+ return False++ # Base case: direct match / subclass+ try:+ return t is class_2 or (isinstance(t, type) and issubclass(t, class_2))+ except TypeError:+ # Not a class type (e.g., typing constructs you don't care about)+ return False++ return any(_is_compatible(field.annotation) for field in fields.values())diff --git a/pipelex/tools/typing/module_inspector.py b/pipelex/tools/typing/module_inspector.pyindex e86a143bc..dd5fce6bb 100644--- a/pipelex/tools/typing/module_inspector.py+++ b/pipelex/tools/typing/module_inspector.py@@ -1,7 +1,9 @@+import astimport importlib.utilimport inspectimport osimport sys+from pathlib import Pathfrom typing import Any@@ -27,8 +29,14 @@ def import_module_from_file(file_path: str) -> Any:msg = f"File {file_path} is not a Python file (must end with .py)"raise ModuleFileError(msg)+ # Validate that the path exists and is a file, not a directory+ path = Path(file_path)+ if path.exists() and not path.is_file():+ msg = f"Path {file_path} exists but is not a file (it may be a directory)"+ raise ModuleFileError(msg)+# Convert file path to module-style path to use as the actual module name- module_name = _convert_file_path_to_module_path(file_path)+ module_name = convert_file_path_to_module_path(file_path)# Check if module is already loaded to avoid duplicate loadingif module_name in sys.modules:@@ -51,16 +59,241 @@ def import_module_from_file(file_path: str) -> Any:return module-def _convert_file_path_to_module_path(file_path: str) -> str:- """Convert a file path to a module-style path."""+def convert_file_path_to_module_path(file_path: str) -> str:+ """Convert a file path to a valid module identifier.++ The module name doesn't need to match the actual package structure since+ we're using spec_from_file_location - it just needs to be a unique, valid+ Python identifier for registration in sys.modules.++ Args:+ file_path: Path to the Python file++ Returns:+ A unique, valid Python module name derived from the absolute file path+ """+ # Convert to absolute path for uniqueness and consistency+ abs_path = os.path.abspath(file_path)+# Remove .py extension- module_path = file_path.removesuffix(".py")+ module_path = abs_path.removesuffix(".py")++ # Replace all non-alphanumeric characters with underscores to create a valid identifier+ # This handles path separators, dots, hyphens, spaces, etc.+ valid_chars: list[str] = []+ for char in module_path:+ if char.isalnum():+ valid_chars.append(char)+ else:+ valid_chars.append("_")++ result = "".join(valid_chars)++ # Ensure it doesn't start with a number (Python requirement)+ if result and result[0].isdigit():+ result = "_" + result++ # Handle edge case of empty result+ if not result:+ msg = f"Cannot create valid module name from file path: {file_path}"+ raise ModuleFileError(msg)++ return result+++def find_class_names_in_file(file_path: str, base_class_names: list[str] | None = None) -> list[str]:+ """Find class names in a Python file without executing it using AST parsing.++ This is useful when you want to discover classes without running module-level code.++ Args:+ file_path: Path to the Python file to analyze+ base_class_names: Optional list of base class names to filter by.+ Only returns classes that inherit from these bases.+ If None, returns all class definitions.++ Returns:+ List of class names found in the file++ Raises:+ ModuleFileError: If the file cannot be read or parsed++ """+ # Validate that the file is a Python file+ if not file_path.endswith(".py"):+ msg = f"File {file_path} is not a Python file (must end with .py)"+ raise ModuleFileError(msg)++ # Validate that the path exists and is a file+ path = Path(file_path)+ if not path.exists() or not path.is_file():+ msg = f"Path {file_path} does not exist or is not a file"+ raise ModuleFileError(msg)++ try:+ # Read and parse the file+ with open(file_path, encoding="utf-8") as f:+ source = f.read()+ tree = ast.parse(source, filename=file_path)+ except Exception as e:+ msg = f"Failed to parse {file_path}: {e}"+ raise ModuleFileError(msg) from e++ class_names: list[str] = []++ # Walk through the AST to find class definitions+ for node in ast.walk(tree):+ if isinstance(node, ast.ClassDef):+ # If no base class filter, add all classes+ if base_class_names is None:+ class_names.append(node.name)+ continue++ # Check if this class inherits from any of the specified base classes+ for base in node.bases:+ # Handle simple names like "StructuredContent"+ if isinstance(base, ast.Name) and base.id in base_class_names:+ class_names.append(node.name)+ break+ # Handle attribute access like "pipelex.StructuredContent"+ if isinstance(base, ast.Attribute):+ if base.attr in base_class_names:+ class_names.append(node.name)+ break++ return class_names+++def find_decorated_function_names_in_file(+ file_path: str,+ decorator_names: list[str],+) -> list[str]:+ """Find function names decorated with specific decorators without executing the file.++ This uses AST parsing to find functions with specific decorators.++ Args:+ file_path: Path to the Python file to analyze+ decorator_names: List of decorator names to look for (e.g. ["pipe_func", "register_func"])++ Returns:+ List of function names that have the specified decorators++ Raises:+ ModuleFileError: If the file cannot be read or parsed++ """+ # Validate that the file is a Python file+ if not file_path.endswith(".py"):+ msg = f"File {file_path} is not a Python file (must end with .py)"+ raise ModuleFileError(msg)++ # Validate that the path exists and is a file+ path = Path(file_path)+ if not path.exists() or not path.is_file():+ msg = f"Path {file_path} does not exist or is not a file"+ raise ModuleFileError(msg)++ try:+ # Read and parse the file+ with open(file_path, encoding="utf-8") as f:+ source = f.read()+ tree = ast.parse(source, filename=file_path)+ except Exception as e:+ msg = f"Failed to parse {file_path}: {e}"+ raise ModuleFileError(msg) from e++ function_names: list[str] = []++ # Walk through the AST to find function definitions with decorators+ for node in ast.walk(tree):+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):+ # Check if function has any of the specified decorators+ for decorator in node.decorator_list:+ decorator_name = None++ # Handle simple decorator names like @pipe_func+ if isinstance(decorator, ast.Name):+ decorator_name = decorator.id+ # Handle decorator calls like @pipe_func() or @pipe_func(name="foo")+ elif isinstance(decorator, ast.Call):+ if isinstance(decorator.func, ast.Name):+ decorator_name = decorator.func.id+ # Handle qualified names like @registry.pipe_func()+ elif isinstance(decorator, ast.Attribute):+ decorator_name = decorator.attr++ if decorator_name in decorator_names:+ function_names.append(node.name)+ break # Found a matching decorator, no need to check others++ return function_names+++def import_module_from_file_if_has_decorated_functions(+ file_path: str,+ decorator_names: list[str],+) -> Any | None:+ """Import a module only if it contains functions with specific decorators.++ This function uses AST parsing to check if the file contains functions decorated+ with specific decorators before importing, avoiding execution of modules that don't+ have the functions you're looking for.++ Args:+ file_path: Path to the Python file to potentially import+ decorator_names: List of decorator names to look for (e.g. ["pipe_func"])++ Returns:+ The imported module if it contains decorated functions, None otherwise++ Raises:+ ModuleFileError: If the file is not a Python file or cannot be loaded++ """+ # First, use AST to check if file has decorated functions+ function_names = find_decorated_function_names_in_file(file_path, decorator_names)++ # If no decorated functions found, skip import+ if not function_names:+ return None++ # File has decorated functions, import it+ return import_module_from_file(file_path)+++def import_module_from_file_if_has_classes(+ file_path: str,+ base_class_names: list[str] | None = None,+) -> Any | None:+ """Import a module only if it contains classes (optionally filtered by base class).++ This function uses AST parsing to check if the file contains relevant classes+ before importing, avoiding execution of modules that don't have the classes+ you're looking for.++ Args:+ file_path: Path to the Python file to potentially import+ base_class_names: Optional list of base class names to filter by.+ Only imports if file contains classes inheriting from these.+ If None, imports if file contains any class definitions.++ Returns:+ The imported module if it contains relevant classes, None otherwise++ Raises:+ ModuleFileError: If the file is not a Python file or cannot be loaded++ """+ # First, use AST to check if file has relevant classes+ class_names = find_class_names_in_file(file_path, base_class_names)- # Replace path separators with dots- module_path = module_path.replace(os.sep, ".")+ # If no relevant classes found, skip import+ if not class_names:+ return None- # Handle __init__.py files by removing the __init__ part- return module_path.removesuffix(".__init__")+ # File has relevant classes, import it+ return import_module_from_file(file_path)def find_classes_in_module(diff --git a/pipelex/tools/typing/pydantic_utils.py b/pipelex/tools/typing/pydantic_utils.pyindex 411f40150..9cc4f9705 100644--- a/pipelex/tools/typing/pydantic_utils.py+++ b/pipelex/tools/typing/pydantic_utils.py@@ -20,14 +20,26 @@ def empty_list_factory_of(_: type[T]) -> Callable[[], list[T]]:return _factory-def format_pydantic_validation_error(exc: ValidationError) -> str:- """Format a Pydantic ValidationError into a readable string with detailed error information.+class PydanticValidationErrorAnalysis(BaseModel):+ error_msg: str++ missing_fields: list[str]+ extra_fields: list[str]+ type_errors: list[str]+ value_errors: list[str]+ enum_errors: list[str]+ union_tag_errors: list[str]+ model_type_errors: list[str]+++def analyze_pydantic_validation_error(exc: ValidationError) -> PydanticValidationErrorAnalysis:+ """Analyze a Pydantic ValidationError into a readable string with detailed error information.Args:exc: The Pydantic ValidationError exceptionReturns:- A formatted string containing categorized validation errors+ A PydanticValidationErrorAnalysis object containing categorized validation errors"""error_msg = "Validation error(s):"@@ -60,27 +72,49 @@ def format_pydantic_validation_error(exc: ValidationError) -> str:# Add each type of error to the message if presentif missing_fields:- error_msg += f"\nMissing required fields: {missing_fields}"+ error_msg += f"\n\nMissing required fields: {missing_fields}"if extra_fields:- error_msg += f"\nExtra forbidden fields: {extra_fields}"+ error_msg += f"\n\nExtra forbidden fields: {extra_fields}"if type_errors:- error_msg += f"\nType errors: {type_errors}"+ error_msg += f"\n\nType errors: {type_errors}"if value_errors:- error_msg += f"\nValue errors: {value_errors}"+ error_msg += f"\n\nValue errors: {value_errors}"if enum_errors:- error_msg += f"\nEnum errors: {enum_errors}"+ error_msg += f"\n\nEnum errors: {enum_errors}"if union_tag_errors:- error_msg += f"\nUnion discriminator errors: {union_tag_errors}"+ error_msg += f"\n\nUnion discriminator errors: {union_tag_errors}"if model_type_errors:- error_msg += f"\nModel type errors: {model_type_errors}"+ error_msg += f"\n\nModel type errors: {model_type_errors}"# If none of the specific error types were found, add the raw error messagesif not any([missing_fields, extra_fields, type_errors, value_errors, enum_errors, union_tag_errors, model_type_errors]):- error_msg += "\nOther validation errors:"+ error_msg += "\n\nOther validation errors:"for err in exc.errors():error_msg += f"\n{'.'.join(map(str, err['loc']))}: {err['type']}: {err['msg']}"- return error_msg+ return PydanticValidationErrorAnalysis(+ error_msg=error_msg,+ missing_fields=missing_fields,+ extra_fields=extra_fields,+ type_errors=type_errors,+ value_errors=value_errors,+ enum_errors=enum_errors,+ union_tag_errors=union_tag_errors,+ model_type_errors=model_type_errors,+ )+++def format_pydantic_validation_error(exc: ValidationError) -> str:+ """Format a Pydantic ValidationError into a readable string with detailed error information.++ Args:+ exc: The Pydantic ValidationError exception++ Returns:+ A formatted string containing categorized validation errors++ """+ return analyze_pydantic_validation_error(exc).error_msgdef convert_strenum_to_str(diff --git a/pipelex/urls.py b/pipelex/urls.pynew file mode 100644index 000000000..adec05ebf--- /dev/null+++ b/pipelex/urls.py@@ -0,0 +1,3 @@+class URLs:+ discord = "https://go.pipelex.com/discord"+ pipe_func_docs = "https://docs.pipelex.com/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc"diff --git a/pyproject.toml b/pyproject.tomlindex b54926cb7..4b1b6c5a1 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -39,7 +39,7 @@ dependencies = ["PyYAML>=6.0.2","rich>=13.8.1","shortuuid>=1.0.13",- "toml>=0.10.2",+ "tomli>=2.3.0","tomlkit>=0.13.2","typer>=0.16.0","typing-extensions>=4.13.2",@@ -53,7 +53,7 @@ Documentation = "https://docs.pipelex.com/"Changelog = "https://docs.pipelex.com/changelog/"[project.optional-dependencies]-anthropic = ["anthropic>=0.49.0"]+anthropic = ["anthropic>=0.60.0"]bedrock = ["boto3>=1.34.131", "aioboto3>=13.4.0"]fal = ["fal-client>=0.4.1"]google = ["google-auth-oauthlib>=1.2.1"]@@ -82,13 +82,10 @@ dev = ["ruff>=0.6.8","types-aioboto3[bedrock,bedrock-runtime]>=13.4.0","types-aiofiles>=24.1.0.20240626",- "types-beautifulsoup4>=4.12.0.20240907","types-markdown>=3.6.0.20240316","types-networkx>=3.3.0.20241020","types-openpyxl>=3.1.5.20250306",- "types-requests>=2.32.0.2024091","types-PyYAML>=6.0.12.20250326",- "types-toml>=0.10.8.20240310",][project.scripts]@@ -214,6 +211,11 @@ addopts = "--import-mode=importlib -ra -m 'not (inference or llm or img_gen or easyncio_default_fixture_loop_scope = "session"filterwarnings = ["ignore:Support for class-based `config` is deprecated:DeprecationWarning",+ "ignore:websockets.*is deprecated:DeprecationWarning",+ "ignore:typing\\.io is deprecated:DeprecationWarning",+ "ignore:typing\\.re is deprecated:DeprecationWarning",+ "ignore:.*has been moved to cryptography.*",+ "ignore:Use.*Types instead",]markers = ["needs_output: tests that need output to be displayed",@@ -359,7 +361,7 @@ convention = "google"[tool.ruff.lint.per-file-ignores]"tests/**/*.py" = [- "T201", # Allow print statements in test files+ "INP001", # Allow test files to not have __init__.py in their directories (avoids namespace collisions)][tool.uv]diff --git a/tests/cases/__init__.py b/tests/cases/__init__.pyindex 135dbd61f..7b1c6b63e 100644--- a/tests/cases/__init__.py+++ b/tests/cases/__init__.py@@ -4,11 +4,11 @@ This package contains pure-Python test-case definitions without test logic.Each module exposes only data constants that can be imported cleanly."""-from .documents import PDFTestCases-from .images import ImageTestCases-from .registry import ClassRegistryTestCases, FileHelperTestCases, Fruit-from .templates import JINJA2TestCases-from .urls import TestURLs+from tests.cases.documents import PDFTestCases+from tests.cases.images import ImageTestCases+from tests.cases.jinja2_templates import JINJA2TestCases+from tests.cases.registry import ClassRegistryTestCases, FileHelperTestCases, Fruit+from tests.cases.urls import TestURLs__all__ = ["ClassRegistryTestCases",diff --git a/tests/cases/templates.py b/tests/cases/jinja2_templates.pysimilarity index 98%rename from tests/cases/templates.pyrename to tests/cases/jinja2_templates.pyindex fd70714b6..a1e9169de 100644--- a/tests/cases/templates.py+++ b/tests/cases/jinja2_templates.py@@ -3,8 +3,7 @@from typing import Any, ClassVarfrom pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormat--from .registry import Fruit+from tests.cases.registry import Fruitclass JINJA2TestCases:diff --git a/tests/conftest.py b/tests/conftest.pyindex e765fdfc5..108c4e6a3 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -20,7 +20,7 @@ def reset_pipelex_config_fixture():# Code to run before each testConsole().print("[magenta]pipelex setup[/magenta]")try:- pipelex_instance = pipelex.pipelex.Pipelex.make(relative_config_folder_path="../pipelex/libraries")+ pipelex_instance = pipelex.pipelex.Pipelex.make()config = get_config()log.verbose(config, title="Test config")assert isinstance(config, pipelex.config.PipelexConfig)diff --git a/tests/data/test_migrate_v0_1_0_to_v0_2_0.toml b/tests/data/test_migrate_v0_1_0_to_v0_2_0.tomldeleted file mode 100644index 29640d8bc..000000000--- a/tests/data/test_migrate_v0_1_0_to_v0_2_0.toml+++ /dev/null@@ -1,31 +0,0 @@-domain = "test"-description = "Test domain"--[concept]-SimpleText = "A simple text concept"-SimpleDoc = "A simple document concept"--# Complex concepts with old syntax-[concept.ComplexConcept]-Concept = "A complex concept with old syntax"-refines = "Text"--[concept.AnotherConcept]-Concept = "Another concept to test migration"-structure = "CustomStructure"-refines = "Text"--[concept.AlreadyMigrated]-description = "This one is already migrated"-refines = "Text"--[pipe.test_pipe]-PipeLLM = "A test pipe that generates simple text"-inputs = { input_text = "SimpleText" }-output = "SimpleDoc"-prompt_template = """-Transform this text:-@input_text--Concept = "This is not a definition of concept, it's just text in the prompt template, so it should not be migrated"-"""diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/__init__.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_condition/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/__init__.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/__init__.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/e2e/pipelex/pipes/pipe_operators/__init__.py b/tests/e2e/pipelex/pipes/pipe_operators/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/__init__.py b/tests/integration/pipelex/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/libraries/pipelines/builder/concept/integration_test_data.py b/tests/integration/pipelex/builder/concept/integration_test_data.pysimilarity index 100%rename from tests/integration/pipelex/libraries/pipelines/builder/concept/integration_test_data.pyrename to tests/integration/pipelex/builder/concept/integration_test_data.pydiff --git a/tests/integration/pipelex/libraries/pipelines/builder/concept/test_concept_spec_generation.py b/tests/integration/pipelex/builder/concept/test_concept_spec_generation.pysimilarity index 92%rename from tests/integration/pipelex/libraries/pipelines/builder/concept/test_concept_spec_generation.pyrename to tests/integration/pipelex/builder/concept/test_concept_spec_generation.pyindex d2c538e4e..524b1b2df 100644--- a/tests/integration/pipelex/libraries/pipelines/builder/concept/test_concept_spec_generation.py+++ b/tests/integration/pipelex/builder/concept/test_concept_spec_generation.py@@ -1,12 +1,12 @@import pytestfrom pipelex import log, pretty_print+from pipelex.builder.concept.concept_spec import ConceptSpecfrom pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParamsfrom pipelex.cogt.llm.llm_job_factory import LLMJobFactoryfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.hub import get_llm_worker-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec-from tests.integration.pipelex.libraries.pipelines.builder.concept.integration_test_data import ConceptSpecGenerationTestCases+from tests.integration.pipelex.builder.concept.integration_test_data import ConceptSpecGenerationTestCases@pytest.mark.llmdiff --git a/tests/integration/pipelex/client/__init__.py b/tests/integration/pipelex/client/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/cogt/__init__.py b/tests/integration/pipelex/cogt/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/cogt/test_llm_vision.py b/tests/integration/pipelex/cogt/test_llm_vision.pyindex d4c8ea51c..80923acff 100644--- a/tests/integration/pipelex/cogt/test_llm_vision.py+++ b/tests/integration/pipelex/cogt/test_llm_vision.py@@ -18,7 +18,7 @@ from tests.integration.pipelex.cogt.test_data import LLMVisionTestCasesclass TestLLMVision:@pytest.mark.parametrize(("topic", "image_uri"), LLMVisionTestCases.IMAGE_URLS)async def test_gen_text_from_vision_by_url(self, llm_handle_for_vision: str, topic: str, image_uri: str):- prompt_image = PromptImageFactory.make_prompt_image_from_uri(uri=image_uri)+ prompt_image = PromptImageFactory.make_prompt_image(url=image_uri)llm_worker = get_llm_worker(llm_handle=llm_handle_for_vision)log.info(f"Using llm_worker: {llm_worker.desc}")llm_job = LLMJobFactory.make_llm_job(diff --git a/tests/integration/pipelex/core/memory/__init__.py b/tests/integration/pipelex/core/memory/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/kit/test_cursor_export.py b/tests/integration/pipelex/kit/test_cursor_export.pynew file mode 100644index 000000000..2a74c6243--- /dev/null+++ b/tests/integration/pipelex/kit/test_cursor_export.py@@ -0,0 +1,46 @@+from pathlib import Path++from pipelex.kit.cursor_export import export_cursor_rules+from pipelex.kit.index_loader import load_index+++class TestCursorExport:+ """Test Cursor rules export functionality."""++ def test_export_cursor_rules_dry_run(self, tmp_path: Path):+ """Test Cursor export in dry-run mode."""+ idx = load_index()+ repo_root = tmp_path++ # Dry run should not create files+ export_cursor_rules(repo_root, idx, dry_run=True)+ cursor_rules_dir = repo_root / ".cursor" / "rules"+ assert not cursor_rules_dir.exists() or len(list(cursor_rules_dir.iterdir())) == 0++ def test_export_cursor_rules_creates_mdc_files(self, tmp_path: Path):+ """Test that Cursor export creates .mdc files."""+ idx = load_index()+ repo_root = tmp_path++ export_cursor_rules(repo_root, idx, dry_run=False)++ # Verify output directory exists and contains .mdc files+ cursor_rules_dir = repo_root / ".cursor" / "rules"+ assert cursor_rules_dir.exists()+ mdc_files = list(cursor_rules_dir.glob("*.mdc"))+ assert len(mdc_files) > 0, "Expected .mdc files to be created"++ def test_export_cursor_rules_have_front_matter(self, tmp_path: Path):+ """Test that exported .mdc files have YAML front-matter."""+ idx = load_index()+ repo_root = tmp_path++ export_cursor_rules(repo_root, idx, dry_run=False)++ # Check first .mdc file for front-matter+ cursor_rules_dir = repo_root / ".cursor" / "rules"+ mdc_files = list(cursor_rules_dir.glob("*.mdc"))+ if mdc_files:+ content = mdc_files[0].read_text(encoding="utf-8")+ assert content.startswith("---\n"), "Expected YAML front-matter to start with ---"+ assert "---\n" in content[4:], "Expected YAML front-matter to end with ---"diff --git a/tests/integration/pipelex/kit/test_merged_rules.py b/tests/integration/pipelex/kit/test_merged_rules.pynew file mode 100644index 000000000..fdad74a6b--- /dev/null+++ b/tests/integration/pipelex/kit/test_merged_rules.py@@ -0,0 +1,45 @@+"""Integration tests for building merged agent documentation."""++import pytest++from pipelex.kit.index_loader import load_index+from pipelex.kit.targets_update import build_merged_rules+++class TestMergedRules:+ """Test building merged agent documentation."""++ def test_build_merged_rules_default_set(self):+ """Test building merged rules with default set."""+ idx = load_index()++ merged = build_merged_rules(idx)+ assert merged is not None+ assert len(merged) > 0+ assert merged.endswith("\n")++ def test_build_merged_rules_specific_set(self):+ """Test building merged rules with specific set."""+ idx = load_index()++ # Test with 'all' set+ merged = build_merged_rules(idx, agent_set="all")+ assert merged is not None+ assert len(merged) > 0++ def test_build_merged_rules_invalid_set(self):+ """Test building merged rules with invalid set name."""+ idx = load_index()++ with pytest.raises(ValueError, match="Agent set 'nonexistent' not found"):+ build_merged_rules(idx, agent_set="nonexistent")++ def test_merged_rules_contain_demoted_headings(self):+ """Test that merged rules have demoted headings."""+ idx = load_index()++ merged = build_merged_rules(idx)++ # If demote is 1, check that we have ## headings (demoted from #)+ if idx.agent_rules.demote > 0:+ assert "##" in merged, "Expected demoted headings in merged content"diff --git a/tests/integration/pipelex/kit/test_targets_update.py b/tests/integration/pipelex/kit/test_targets_update.pynew file mode 100644index 000000000..36a23917d--- /dev/null+++ b/tests/integration/pipelex/kit/test_targets_update.py@@ -0,0 +1,115 @@+from pathlib import Path++from pipelex.kit.index_loader import load_index+from pipelex.kit.markers import find_span+from pipelex.kit.targets_update import build_merged_rules, update_targets+++class TestTargetsUpdate:+ """Test target file updating functionality."""++ def test_update_targets_dry_run(self, tmp_path: Path):+ """Test updating targets in dry-run mode."""+ idx = load_index()++ # Create a temporary repo root with a target file+ repo_root = tmp_path / "repo"+ repo_root.mkdir()+ target_file = repo_root / "test_target.md"+ target_file.write_text("# Test\n\nOriginal content\n", encoding="utf-8")++ merged_rules = build_merged_rules(idx)++ # Create a test target+ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}++ original_content = target_file.read_text(encoding="utf-8")++ # Dry run should not modify file+ update_targets(repo_root, merged_rules, test_targets, dry_run=True, diff=False, backup=None)++ assert target_file.read_text(encoding="utf-8") == original_content++ def test_update_targets_inserts_with_markers(self, tmp_path: Path):+ """Test that update_targets inserts content with markers."""+ idx = load_index()++ repo_root = tmp_path / "repo"+ repo_root.mkdir()+ target_file = repo_root / "test_target.md"+ target_file.write_text("# Test\n\nOriginal content\n", encoding="utf-8")++ merged_rules = build_merged_rules(idx)++ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}++ update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=None)++ updated_content = target_file.read_text(encoding="utf-8")+ target = test_targets["test"]++ # Verify markers are present+ assert target.marker_begin in updated_content+ assert target.marker_end in updated_content++ # Verify content is between markers+ span = find_span(updated_content, target.marker_begin, target.marker_end)+ assert span is not None++ def test_update_targets_replaces_existing_markers(self, tmp_path: Path):+ """Test that update_targets replaces content between existing markers."""+ idx = load_index()++ repo_root = tmp_path / "repo"+ repo_root.mkdir()+ target_file = repo_root / "test_target.md"++ # Create file with existing markers+ marker_begin = "<!-- BEGIN_TEST -->"+ marker_end = "<!-- END_TEST -->"+ initial_content = f"# Test\n\n{marker_begin}\nOld content\n{marker_end}\n"+ target_file.write_text(initial_content, encoding="utf-8")++ merged_rules = build_merged_rules(idx)++ test_targets = {+ "test": idx.agent_rules.targets["agents"].model_copy(+ update={+ "path": "test_target.md",+ "marker_begin": marker_begin,+ "marker_end": marker_end,+ }+ )+ }++ update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=None)++ updated_content = target_file.read_text(encoding="utf-8")++ # Verify markers still exist+ assert marker_begin in updated_content+ assert marker_end in updated_content++ # Verify old content is replaced+ assert "Old content" not in updated_content++ def test_update_targets_creates_backup(self, tmp_path: Path):+ """Test that update_targets creates backup files when requested."""+ idx = load_index()++ repo_root = tmp_path / "repo"+ repo_root.mkdir()+ target_file = repo_root / "test_target.md"+ original_content = "# Test\n\nOriginal content\n"+ target_file.write_text(original_content, encoding="utf-8")++ merged_rules = build_merged_rules(idx)++ test_targets = {"test": idx.agent_rules.targets["agents"].model_copy(update={"path": "test_target.md"})}++ update_targets(repo_root, merged_rules, test_targets, dry_run=False, diff=False, backup=".bak")++ # Verify backup exists+ backup_file = target_file.with_suffix(target_file.suffix + ".bak")+ assert backup_file.exists()+ assert backup_file.read_text(encoding="utf-8") == original_contentdiff --git a/tests/integration/pipelex/language/__init__.py b/tests/integration/pipelex/language/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/libraries/pipelines/builder/__init__.py b/tests/integration/pipelex/libraries/pipelines/builder/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/libraries/pipelines/builder/concept/__init__.py b/tests/integration/pipelex/libraries/pipelines/builder/concept/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/__init__.py b/tests/integration/pipelex/pipes/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch/__init__.py b/tests/integration/pipelex/pipes/controller/pipe_batch/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/__init__.py b/tests/integration/pipelex/pipes/controller/pipe_condition/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/__init__.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/__init__.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/operator/__init__.py b/tests/integration/pipelex/pipes/operator/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/operator/pipe_extract/__init__.py b/tests/integration/pipelex/pipes/operator/pipe_extract/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/operator/pipe_func/__init__.py b/tests/integration/pipelex/pipes/operator/pipe_func/__init__.pydeleted file mode 100644index d610235db..000000000--- a/tests/integration/pipelex/pipes/operator/pipe_func/__init__.py+++ /dev/null@@ -1 +0,0 @@-# Empty file for Python package structurediff --git a/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py b/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.pyindex f92b3e72f..3400299f6 100644--- a/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py+++ b/tests/integration/pipelex/pipes/operator/pipe_func/test_pipe_func.py@@ -12,7 +12,7 @@ from pipelex.pipe_operators.func.pipe_func_factory import PipeFuncFactoryfrom pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.tools.func_registry import func_registry+from pipelex.system.registries.func_registry import func_registryfrom tests.cases.source_code import wrap_linesdiff --git a/tests/integration/pipelex/pipes/operator/pipe_img_gen/__init__.py b/tests/integration/pipelex/pipes/operator/pipe_img_gen/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/operator/pipe_llm/__init__.py b/tests/integration/pipelex/pipes/operator/pipe_llm/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/pipes/test_flow_factory.py b/tests/integration/pipelex/pipes/test_flow_factory.pyindex c4ce678f0..3710bd27e 100644--- a/tests/integration/pipelex/pipes/test_flow_factory.py+++ b/tests/integration/pipelex/pipes/test_flow_factory.py@@ -4,7 +4,7 @@ from pathlib import Pathimport pytestfrom pipelex import log, pretty_print-from pipelex.libraries.pipelines.builder.flow_factory import FlowFactory+from pipelex.builder.flow_factory import FlowFactoryfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprintfrom pipelex.tools.misc.file_utils import get_incremental_directory_path, remove_folderfrom pipelex.tools.misc.json_utils import save_as_json_to_path@@ -70,8 +70,8 @@ class TestFlowFactory:assert len(flow.flow_elements) > 0# Log some details about what we found- controller_count = sum(1 for pipe in flow.flow_elements.values() if pipe.category == "PipeController")- operator_count = sum(1 for pipe in flow.flow_elements.values() if pipe.category == "PipeSignature")+ controller_count = sum(1 for pipe in flow.flow_elements.values() if pipe.pipe_category == "PipeController")+ operator_count = sum(1 for pipe in flow.flow_elements.values() if pipe.pipe_category == "PipeSignature")log.info(f"flow contains {len(flow.flow_elements)} pipes: {controller_count} controllers, {operator_count} operators (as signatures)")@@ -111,7 +111,7 @@ class TestFlowFactory:# Find an operator pipe (LLM pipe) - converted to signatureoperator_pipe = flow.flow_elements.get("summarize_discord_channel_update_for_new_members")assert operator_pipe is not None- assert operator_pipe.category == "PipeSignature"+ assert operator_pipe.pipe_category == "PipeSignature"assert operator_pipe.type == "PipeLLM"# Verify it has the signature propertiesdiff --git a/tests/integration/pipelex/pipes/test_image_inputs.py b/tests/integration/pipelex/pipes/test_image_inputs.pyindex 569e3eeb9..657f0b9a1 100644--- a/tests/integration/pipelex/pipes/test_image_inputs.py+++ b/tests/integration/pipelex/pipes/test_image_inputs.py@@ -61,8 +61,9 @@ class TestImageInputs:But also accepts basic objects"""# Create the page content- image_content = ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG)- text_and_images = TextAndImagesContent(text=TextContent(text="This is the description of the page blablabla"), images=[])+ # image_content = ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG)+ image_content = ImageContent(url=f"file://{ImageTestCases.IMAGE_FILE_PATH_PNG}")+ text_and_images = TextAndImagesContent(text=TextContent(text="It was designed by Slartibartfast, a famous designer"), images=[])page_content = PageContent(text_and_images=text_and_images, page_view=image_content)# Create stuff from page contentdiff --git a/tests/integration/pipelex/pipes/test_pipe_compose.py b/tests/integration/pipelex/pipes/test_pipe_compose.pyindex 91b90621b..efe306684 100644--- a/tests/integration/pipelex/pipes/test_pipe_compose.py+++ b/tests/integration/pipelex/pipes/test_pipe_compose.py@@ -32,7 +32,7 @@ class TestPipeCompose:pipe_compose_blueprint = PipeComposeBlueprint(description="Jinja2 test for any context",template=TemplateBlueprint(- source=template_source,+ template=template_source,templating_style=TemplatingStyle(tag_style=TagStyle.TICKS, text_format=TextFormat.MARKDOWN),category=TemplateCategory.MARKDOWN,extra_context={"place_holder": "[some text from test_pipe_compose_for_any]"},@@ -63,7 +63,7 @@ class TestPipeCompose:pipe_compose_blueprint = PipeComposeBlueprint(description="Jinja2 test for stuff context",template=TemplateBlueprint(- source=template_source,+ template=template_source,templating_style=TemplatingStyle(tag_style=TagStyle.TICKS, text_format=TextFormat.MARKDOWN),category=TemplateCategory.MARKDOWN,extra_context={"place_holder": "[some text from test_pipe_compose_for_stuff]"},diff --git a/tests/integration/pipelex/pipes/test_pipe_running_variants.py b/tests/integration/pipelex/pipes/test_pipe_running_variants.pyindex d42458d3f..b97a9e599 100644--- a/tests/integration/pipelex/pipes/test_pipe_running_variants.py+++ b/tests/integration/pipelex/pipes/test_pipe_running_variants.py@@ -1,14 +1,12 @@-from pathlib import Pathfrom typing import Anyimport pytestfrom pytest import FixtureRequestfrom pipelex import log, pretty_print-from pipelex.config import get_configfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.stuffs.stuff import Stuff-from pipelex.hub import get_library_manager, get_pipe_router, get_required_pipe+from pipelex.hub import get_pipe_router, get_required_pipefrom pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import PipeOutputMultiplicity, PipeRunModefrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory@@ -120,11 +118,13 @@ class TestPipeRunningVariants:exception: type[Exception],expected_error_message: str,):- failing_pipelines_file_paths = get_config().pipelex.library_config.failing_pipelines_file_paths- library_manager = get_library_manager()- library_manager.load_libraries(- library_file_paths=[Path(failing_pipeline_file_path) for failing_pipeline_file_path in failing_pipelines_file_paths],- )+ # failing_pipelines_file_paths = get_config().pipelex.library_config.failing_pipelines_file_paths+ # library_manager = get_library_manager()+ # Reset library to avoid pipe name collisions from previous test runs+ # library_manager.reset()+ # library_manager.load_libraries(+ # library_file_paths=[Path(failing_pipeline_file_path) for failing_pipeline_file_path in failing_pipelines_file_paths],+ # )log.verbose(f"This pipe '{pipe_code}' is supposed to cause an error of type: {exception.__name__}")with pytest.raises(exception) as exc:diff --git a/tests/integration/pipelex/plugins/__init__.py b/tests/integration/pipelex/plugins/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/plugins/test_anthropic.py b/tests/integration/pipelex/plugins/test_anthropic.pyindex d4dd92c4c..dc53af96a 100644--- a/tests/integration/pipelex/plugins/test_anthropic.py+++ b/tests/integration/pipelex/plugins/test_anthropic.py@@ -8,7 +8,7 @@ from pipelex.hub import get_models_managerfrom pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedErrorfrom pipelex.plugins.anthropic.anthropic_llms import anthropic_list_available_modelsfrom pipelex.plugins.plugin_sdk_registry import Plugin-from pipelex.tools.environment import all_env_vars_are_set, any_env_var_is_placeholder+from pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholderREQUIRED_ENV_VARS = ["ANTHROPIC_API_KEY"]diff --git a/tests/integration/pipelex/plugins/test_bedrock.py b/tests/integration/pipelex/plugins/test_bedrock.pyindex 8c0dad4b3..816a4a395 100644--- a/tests/integration/pipelex/plugins/test_bedrock.py+++ b/tests/integration/pipelex/plugins/test_bedrock.py@@ -7,7 +7,7 @@ from rich import boxfrom rich.console import Consolefrom rich.table import Table-from pipelex.tools.environment import all_env_vars_are_set, any_env_var_is_placeholder+from pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholderwarnings.filterwarnings("ignore",diff --git a/tests/integration/pipelex/plugins/test_mistral.py b/tests/integration/pipelex/plugins/test_mistral.pyindex 674330309..23dca8dab 100644--- a/tests/integration/pipelex/plugins/test_mistral.py+++ b/tests/integration/pipelex/plugins/test_mistral.py@@ -5,7 +5,7 @@ from rich.table import Tablefrom pipelex import pretty_printfrom pipelex.plugins.mistral.mistral_llms import mistral_list_available_models-from pipelex.tools.environment import all_env_vars_are_set, any_env_var_is_placeholder+from pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholderREQUIRED_ENV_VARS = ["MISTRAL_API_KEY"]diff --git a/tests/integration/pipelex/plugins/test_openai.py b/tests/integration/pipelex/plugins/test_openai.pyindex c34d4728b..fdd012a34 100644--- a/tests/integration/pipelex/plugins/test_openai.py+++ b/tests/integration/pipelex/plugins/test_openai.py@@ -4,7 +4,7 @@ from pipelex import pretty_printfrom pipelex.hub import get_models_managerfrom pipelex.plugins.openai.openai_llms import openai_list_available_modelsfrom pipelex.plugins.plugin_sdk_registry import Plugin-from pipelex.tools.environment import all_env_vars_are_set, any_env_var_is_placeholder+from pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholder# make t VERBOSE=2 TEST=TestOpenAIdiff --git a/tests/integration/pipelex/plugins/test_openai_image_gen.py b/tests/integration/pipelex/plugins/test_openai_image_gen.pyindex 86a41fb1d..7043b3c86 100644--- a/tests/integration/pipelex/plugins/test_openai_image_gen.py+++ b/tests/integration/pipelex/plugins/test_openai_image_gen.py@@ -4,7 +4,7 @@ from pipelex.cogt.exceptions import ImgGenGenerationErrorfrom pipelex.hub import get_models_managerfrom pipelex.plugins.openai.openai_factory import OpenAIFactoryfrom pipelex.plugins.plugin_sdk_registry import Plugin-from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file+from pipelex.tools.misc.base_64_utils import save_base_64_str_to_binary_filefrom pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_pathfrom tests.conftest import TEST_OUTPUTS_DIRfrom tests.integration.pipelex.test_data import ImageGenTestCases@@ -51,4 +51,4 @@ class TestImgGenByOpenAIGpt:extension="png",avoid_suffix_if_possible=True,)- save_base64_to_binary_file(b64=image_base64, file_path=img_path)+ save_base_64_str_to_binary_file(base_64_str=image_base64, file_path=img_path)diff --git a/tests/integration/pipelex/test_data.py b/tests/integration/pipelex/test_data.pyindex 4af15eb79..3f36a8b8b 100644--- a/tests/integration/pipelex/test_data.py+++ b/tests/integration/pipelex/test_data.py@@ -151,11 +151,6 @@ class PipeTestCases:SIMPLE_STUFF_IMAGE,"simple_llm_test_from_image",),- (- "Extract page contents from PDF",- SIMPLE_STUFF_PDF,- "extract_page_contents_from_pdf",- ),]FAILURE_PIPES: ClassVar[list[tuple[str, type[Exception], str]]] = [(@@ -167,6 +162,7 @@ class PipeTestCases:class LibraryTestCases:+ TEST_PIPELINES_DIR_PATH = "tests/test_pipelines"KNOWN_CONCEPTS_AND_PIPES: ClassVar[list[tuple[str, str]]] = [ # concept, pipe("cars.CarDescription", "generate_car_description"),("animals.AnimalDescription", "generate_animal_description"),diff --git a/tests/integration/pipelex/test_fundamentals.py b/tests/integration/pipelex/test_fundamentals.pyindex 8933ccc52..fc9f5f1fc 100644--- a/tests/integration/pipelex/test_fundamentals.py+++ b/tests/integration/pipelex/test_fundamentals.py@@ -1,7 +1,8 @@import pytest+from pipelex.config import get_configfrom pipelex.hub import get_pipes-from pipelex.pipe_run.dry_run import DryRunStatus, dry_run_pipes+from pipelex.pipe_run.dry_run import dry_run_pipesfrom pipelex.pipelex import Pipelex@@ -21,8 +22,11 @@ class TestFundamentals:results = await dry_run_pipes(pipes=get_pipes(), raise_on_failure=False)# Check if there were any failures+ allowed_to_fail_pipes = get_config().pipelex.dry_run_config.allowed_to_fail_pipes- failed_pipes = {pipe_code: output for pipe_code, output in results.items() if output.status == DryRunStatus.FAILURE}+ failed_pipes = {+ pipe_code: output for pipe_code, output in results.items() if output.status.is_failure and pipe_code not in allowed_to_fail_pipes+ }if failed_pipes:failure_details = "\n".join([f" - {pipe_code}: {output.error_message}" for pipe_code, output in failed_pipes.items()])diff --git a/tests/integration/pipelex/test_libraries.py b/tests/integration/pipelex/test_libraries.pyindex 928be73c2..42fca143f 100644--- a/tests/integration/pipelex/test_libraries.py+++ b/tests/integration/pipelex/test_libraries.py@@ -6,7 +6,6 @@ from rich.console import Consolefrom rich.table import Tablefrom pipelex import pretty_print-from pipelex.config import get_configfrom pipelex.core.concepts.concept_library import ConceptLibraryfrom pipelex.core.pipes.pipe_library import PipeLibraryfrom pipelex.libraries.library_manager_factory import LibraryManagerFactory@@ -86,8 +85,8 @@ class TestLibraries:known_concept: str,known_pipe: str,):- library_manager = LibraryManagerFactory.make_empty(config_dir_path="pipelex/libraries")- test_pipelines_dir = [Path(get_config().pipelex.library_config.test_pipelines_dir_path)]+ library_manager = LibraryManagerFactory.make_empty()+ test_pipelines_dir = [Path(LibraryTestCases.TEST_PIPELINES_DIR_PATH)]library_manager.load_libraries(library_dirs=test_pipelines_dir)# Verify that libraries were loadedassert len(library_manager.concept_library.root) > 0, "No concepts were loaded"diff --git a/tests/integration/pipelex/tools/aws/__init__.py b/tests/integration/pipelex/tools/aws/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/tools/class_registry/__init__.py b/tests/integration/pipelex/tools/class_registry/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/tools/class_registry/test_class_registry_utils.py b/tests/integration/pipelex/tools/class_registry/test_class_registry_utils.pyindex 3f6cdbe0f..5d307e4f8 100644--- a/tests/integration/pipelex/tools/class_registry/test_class_registry_utils.py+++ b/tests/integration/pipelex/tools/class_registry/test_class_registry_utils.py@@ -4,7 +4,7 @@ import pytestfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.hub import get_class_registry-from pipelex.tools.class_registry_utils import ClassRegistryUtils+from pipelex.system.registries.class_registry_utils import ClassRegistryUtilsfrom tests.cases import ClassRegistryTestCasesdiff --git a/tests/integration/pipelex/tools/httpx/__init__.py b/tests/integration/pipelex/tools/httpx/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/integration/pipelex/tools/pdf/__init__.py b/tests/integration/pipelex/tools/pdf/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/test_pipelines/discord_newsletter.plx b/tests/test_pipelines/discord_newsletter.plxindex 8081a23cf..885ab08f6 100644--- a/tests/test_pipelines/discord_newsletter.plx+++ b/tests/test_pipelines/discord_newsletter.plx@@ -84,7 +84,7 @@ output = "HtmlNewsletter"[pipe.format_html_newsletter.template]category = "html"-source = """+template = """<!-- Weekly Summary --><h2>☀️ Weekly Summary</h2><p>diff --git a/tests/test_pipelines/misc_tests/multiplicity.plx b/tests/test_pipelines/misc_tests/multiplicity.plxindex f3e842522..3d0576808 100644--- a/tests/test_pipelines/misc_tests/multiplicity.plx+++ b/tests/test_pipelines/misc_tests/multiplicity.plx@@ -42,7 +42,7 @@ Just state what it is in a single sentence.type = "PipeLLM"description = "Imagine a fantasy scene including products of nature"inputs = { product_of_nature = "ProductOfNature" }-output = "images.ImgGenPrompt"+output = "ImgGenPrompt"prompt = """Imagine a fantasy scene including the following products of nature:@@ -54,7 +54,7 @@ Keep it short and concise, just one sentence.[pipe.imagine_nature_scene_of_original_power_rangers_colors]type = "PipeSequence"description = "Imagine nature scenes of Power Rangers colors"-output = "images.ImgGenPrompt"+output = "ImgGenPrompt"steps = [{ pipe = "original_power_ranger_colors", result = "color" },{ pipe = "imagine_nature_product", result = "product_of_nature" },@@ -64,7 +64,7 @@ steps = [[pipe.imagine_nature_scene_of_alltime_power_rangers_colors]type = "PipeSequence"description = "Imagine nature scenes of Power Rangers colors"-output = "images.ImgGenPrompt"+output = "ImgGenPrompt"steps = [{ pipe = "alltime_power_ranger_colors", result = "color", multiple_output = true },{ pipe = "imagine_nature_product", result = "product_of_nature" },diff --git a/tests/test_pipelines/misc_tests/test_jinja2.plx b/tests/test_pipelines/misc_tests/test_compose.plxsimilarity index 96%rename from tests/test_pipelines/misc_tests/test_jinja2.plxrename to tests/test_pipelines/misc_tests/test_compose.plxindex ca9ef194e..d54f33add 100644--- a/tests/test_pipelines/misc_tests/test_jinja2.plx+++ b/tests/test_pipelines/misc_tests/test_compose.plx@@ -8,7 +8,7 @@ inputs = { text = "Text" }output = "Text"[pipe.jinja2_test_1.template]-source = """+template = """This is a simple test prompt:@text"""diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.plx b/tests/test_pipelines/misc_tests/test_image_inputs.plxindex 78f454bf2..07a881919 100644--- a/tests/test_pipelines/misc_tests/test_image_inputs.plx+++ b/tests/test_pipelines/misc_tests/test_image_inputs.plx@@ -27,11 +27,11 @@ system_prompt = """You are an expert at describing page contents."""prompt = """-Extract the date and title of the article.+Extract the date and title.@page.page_view-Also, add this as the description of the article:+Also, add this to the description:$page.text_and_images.text.text"""diff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/text_list.py b/tests/test_pipelines/pipe_controllers/pipe_batch/text_list.pydeleted file mode 100644index 47672c1e5..000000000--- a/tests/test_pipelines/pipe_controllers/pipe_batch/text_list.py+++ /dev/null@@ -1,6 +0,0 @@-from pipelex.core.stuffs.list_content import ListContent-from pipelex.core.stuffs.text_content import TextContent---class TextList(ListContent[TextContent]):- passdiff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plxindex c3e82f1fb..dbbfd8445 100644--- a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx+++ b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.plx@@ -3,17 +3,8 @@ description = "Simple pipes for testing PipeBatch integration"[concept]UppercaseText = "Text that has been transformed to uppercase"-TextList = "List of Text"[pipe]-[pipe.test_pipe_batch]-type = "PipeBatch"-description = "Test pipe batch"-inputs = { text_item = "Text", text_list = "TextList" }-output = "UppercaseText"-branch_pipe_code = "uppercase_transformer"-input_list_name = "text_list"-input_item_name = "text_item"[pipe.uppercase_transformer]type = "PipeLLM"diff --git a/tests/test_pipelines/pipe_operators/pipe_llm_vision.plx b/tests/test_pipelines/pipe_operators/pipe_llm_vision.plxindex 140ab6ff8..a3e3645a8 100644--- a/tests/test_pipelines/pipe_operators/pipe_llm_vision.plx+++ b/tests/test_pipelines/pipe_operators/pipe_llm_vision.plx@@ -5,6 +5,10 @@ description = "Test PipeLLM with vision capabilities"VisionAnalysis = "Some analysis based on the image"BasicDescription = "Basic description of the image"+[concept.Photo]+description = "A photo"+refines = "Image"+[pipe.describe_image]type = "PipeLLM"description = "Describe what is in the image"@@ -42,7 +46,7 @@ Completely ignore $imageA.[pipe.vision_analysis]type = "PipeLLM"description = "Provide detailed analysis of the image"-inputs = { image = "images.Photo" }+inputs = { image = "Photo" }output = "VisionAnalysis"model = "llm_to_extract_diagram"system_prompt = "You are an expert image analyst. Provide detailed, accurate descriptions."diff --git a/tests/test_pipelines/test_file_func_registry.py b/tests/test_pipelines/test_file_func_registry.pyindex 714678249..84cdb5a8b 100644--- a/tests/test_pipelines/test_file_func_registry.py+++ b/tests/test_pipelines/test_file_func_registry.py@@ -3,6 +3,7 @@ from pydantic import Fieldfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.system.registries.func_registry import pipe_funcclass FilePath(StructuredContent):@@ -18,6 +19,7 @@ class CodebaseFileContent(StructuredContent):file_content: str = Field(description="Content of the codebase file")+@pipe_func()def read_file_content(working_memory: WorkingMemory) -> ListContent[CodebaseFileContent]:"""Read the content of related codebase files.diff --git a/tests/unit/pipelex/libraries/pipelines/builder/concept/test_concept_blueprint.py b/tests/unit/pipelex/builder/concept/test_concept_blueprint.pysimilarity index 86%rename from tests/unit/pipelex/libraries/pipelines/builder/concept/test_concept_blueprint.pyrename to tests/unit/pipelex/builder/concept/test_concept_blueprint.pyindex 3beb63904..4f3323bbd 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/concept/test_concept_blueprint.py+++ b/tests/unit/pipelex/builder/concept/test_concept_blueprint.py@@ -1,9 +1,9 @@import pytestfrom pipelex import log+from pipelex.builder.concept.concept_spec import ConceptSpecfrom pipelex.core.concepts.concept_blueprint import ConceptBlueprint-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec-from tests.unit.pipelex.libraries.pipelines.builder.concept.test_data import ConceptBlueprintTestCases, ConceptCodeValidationTestCases+from tests.unit.pipelex.builder.concept.test_data import ConceptBlueprintTestCases, ConceptCodeValidationTestCasesclass TestConceptBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/concept/test_data.py b/tests/unit/pipelex/builder/concept/test_data.pysimilarity index 98%rename from tests/unit/pipelex/libraries/pipelines/builder/concept/test_data.pyrename to tests/unit/pipelex/builder/concept/test_data.pyindex 9ea82f688..5fd196f8d 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/concept/test_data.py+++ b/tests/unit/pipelex/builder/concept/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.concept.concept_spec import ConceptSpec, ConceptStructureSpec, ConceptStructureSpecFieldTypefrom pipelex.core.concepts.concept_blueprint import ConceptBlueprint, ConceptStructureBlueprint, ConceptStructureBlueprintFieldType-from pipelex.libraries.pipelines.builder.concept.concept_spec import ConceptSpec, ConceptStructureSpec, ConceptStructureSpecFieldTypeclass ConceptBlueprintTestCases:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.pysimilarity index 91%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.pyindex 05d8a9b64..e51f941e2 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_batch_spec import PipeBatchSpecfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint@@ -20,7 +20,7 @@ class PipeBatchTestCases:inputs={"items": InputRequirementBlueprint(concept="ItemList")},output="ProcessedItems",type="PipeBatch",- category="PipeController",+ pipe_category="PipeController",branch_pipe_code="process_item",input_list_name=None,input_item_name=None,@@ -43,7 +43,7 @@ class PipeBatchTestCases:inputs={"data": InputRequirementBlueprint(concept="DataList")},output="Results",type="PipeBatch",- category="PipeController",+ pipe_category="PipeController",branch_pipe_code="transform_data",input_list_name="data_list",input_item_name="current_data",diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.pysimilarity index 72%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.pyindex d24b48b48..804a78db4 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_batch/test_pipe_batch.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_batch_spec import PipeBatchSpec+from pipelex.builder.pipe.pipe_batch_spec import PipeBatchSpecfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_controller.pipe_batch.test_data import PipeBatchTestCases+from tests.unit.pipelex.builder.pipe.pipe_controller.pipe_batch.test_data import PipeBatchTestCasesclass TestPipeBatchBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.pysimilarity index 91%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.pyindex bfecba1d9..b9394b22c 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_condition_spec import PipeConditionSpecfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint@@ -26,7 +26,7 @@ class PipeConditionTestCases:inputs={"item": InputRequirementBlueprint(concept="Item")},output="ProcessedItem",type="PipeCondition",- category="PipeController",+ pipe_category="PipeController",expression=None,expression_template="{{ item.category }}",outcomes={diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.pysimilarity index 72%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.pyindex 3ec34ceb7..4fece994d 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_condition/test_pipe_condition.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_condition_spec import PipeConditionSpec+from pipelex.builder.pipe.pipe_condition_spec import PipeConditionSpecfrom pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_controller.pipe_condition.test_data import PipeConditionTestCases+from tests.unit.pipelex.builder.pipe.pipe_controller.pipe_condition.test_data import PipeConditionTestCasesclass TestPipeConditionBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.pysimilarity index 93%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.pyindex 287352b1d..8d260a49e 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py@@ -1,8 +1,8 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_parallel_spec import PipeParallelSpec-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint@@ -27,7 +27,7 @@ class PipeParallelTestCases:inputs={"data": InputRequirementBlueprint(concept="Data")},output="Results",type="PipeParallel",- category="PipeController",+ pipe_category="PipeController",parallels=[SubPipeBlueprint(pipe="analyze_data", result="analysis"),SubPipeBlueprint(pipe="transform_data", result="transformed"),@@ -58,7 +58,7 @@ class PipeParallelTestCases:inputs={"input": InputRequirementBlueprint(concept="Input")},output="CombinedResult",type="PipeParallel",- category="PipeController",+ pipe_category="PipeController",parallels=[SubPipeBlueprint(pipe="pipe1", result="result1"),SubPipeBlueprint(pipe="pipe2", result="result2"),@@ -88,7 +88,7 @@ class PipeParallelTestCases:inputs={"input": InputRequirementBlueprint(concept="Input")},output="CombinedResult",type="PipeParallel",- category="PipeController",+ pipe_category="PipeController",parallels=[SubPipeBlueprint(pipe="pipe1", result="result1"),SubPipeBlueprint(pipe="pipe2", result="result2"),diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.pysimilarity index 75%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.pyindex 64db9b946..23750cb39 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_pipe_parallel.py@@ -1,9 +1,9 @@import pytestfrom pipelex import log-from pipelex.libraries.pipelines.builder.pipe.pipe_parallel_spec import PipeParallelSpec+from pipelex.builder.pipe.pipe_parallel_spec import PipeParallelSpecfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_controller.pipe_parallel.test_data import PipeParallelTestCases+from tests.unit.pipelex.builder.pipe.pipe_controller.pipe_parallel.test_data import PipeParallelTestCasesclass TestPipeParallelBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.pysimilarity index 91%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.pyindex 522994251..321999083 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_data.py@@ -1,8 +1,8 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_sequence_spec import PipeSequenceSpec-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprintfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint@@ -26,7 +26,7 @@ class PipeSequenceTestCases:inputs={"input_data": InputRequirementBlueprint(concept="Text")},output="ProcessedData",type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",steps=[SubPipeBlueprint(pipe="step1", result="result1"),SubPipeBlueprint(pipe="step2", result="result2"),@@ -57,7 +57,7 @@ class PipeSequenceTestCases:inputs={"items": InputRequirementBlueprint(concept="ItemList")},output="ProcessedItems",type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",steps=[SubPipeBlueprint(pipe="prepare", result="prepared_items"),SubPipeBlueprint(diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.pysimilarity index 72%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.pyrename to tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.pyindex 3acda38a6..6cd1fee7e 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_sequence/test_pipe_sequence.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_sequence_spec import PipeSequenceSpec+from pipelex.builder.pipe.pipe_sequence_spec import PipeSequenceSpecfrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_controller.pipe_sequence.test_data import PipeSequenceTestCases+from tests.unit.pipelex.builder.pipe.pipe_controller.pipe_sequence.test_data import PipeSequenceTestCasesclass TestPipeSequenceBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.pysimilarity index 89%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.pyindex d519826f3..18d72ec75 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_data.py@@ -1,10 +1,10 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpecfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprintfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle, TextFormatfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_compose_spec import PipeComposeSpecfrom pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint@@ -24,9 +24,9 @@ class PipeComposeTestCases:inputs={"data": InputRequirementBlueprint(concept="Data")},output="RenderedText",type="PipeCompose",- category="PipeOperator",+ pipe_category="PipeOperator",template=TemplateBlueprint(- source="Hello {{ data.name }}!",+ template="Hello {{ data.name }}!",category=TemplateCategory.MARKDOWN,templating_style=TemplatingStyle(tag_style=TagStyle.TICKS,diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.pysimilarity index 72%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.pyindex 0327a6aae..8af557694 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_compose/test_pipe_compose.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_compose_spec import PipeComposeSpec+from pipelex.builder.pipe.pipe_compose_spec import PipeComposeSpecfrom pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_operator.pipe_compose.test_data import PipeComposeTestCases+from tests.unit.pipelex.builder.pipe.pipe_operator.pipe_compose.test_data import PipeComposeTestCasesclass TestPipeComposeBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.pysimilarity index 91%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.pyindex b299d46aa..0de5da9a6 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_extract_spec import PipeExtractSpecfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint@@ -21,7 +21,7 @@ class PipeExtractTestCases:inputs={"image": InputRequirementBlueprint(concept="Image")},output="ExtractedText",type="PipeExtract",- category="PipeOperator",+ pipe_category="PipeOperator",model="base_ocr_mistral",),)@@ -44,7 +44,7 @@ class PipeExtractTestCases:inputs={"document": InputRequirementBlueprint(concept="PDF")},output="PageContent",type="PipeExtract",- category="PipeOperator",+ pipe_category="PipeOperator",model="base_ocr_mistral",page_images=True,page_image_captions=True,diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.pysimilarity index 74%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.pyindex 49ef48d46..3b7b96198 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_extract/test_pipe_extract.py@@ -1,9 +1,9 @@import pytestfrom pipelex import log-from pipelex.libraries.pipelines.builder.pipe.pipe_extract_spec import PipeExtractSpec+from pipelex.builder.pipe.pipe_extract_spec import PipeExtractSpecfrom pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_operator.pipe_extract.test_data import PipeExtractTestCases+from tests.unit.pipelex.builder.pipe.pipe_operator.pipe_extract.test_data import PipeExtractTestCasesclass TestPipeExtractBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.pysimilarity index 89%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.pyindex f6f796b26..1a0af343b 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_func_spec import PipeFuncSpecfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint@@ -20,7 +20,7 @@ class PipeFuncTestCases:inputs={"data": InputRequirementBlueprint(concept="Data")},output="ProcessedData",type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",function_name="process_data",),)@@ -39,7 +39,7 @@ class PipeFuncTestCases:inputs=None,output="GeneratedData",type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",function_name="generate_data",),)diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_pipe_func.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_pipe_func.pysimilarity index 73%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_pipe_func.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_pipe_func.pyindex 063c4ef0c..59496b93f 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/test_pipe_func.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_func/test_pipe_func.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_func_spec import PipeFuncSpec+from pipelex.builder.pipe.pipe_func_spec import PipeFuncSpecfrom pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_operator.pipe_func.test_data import PipeFuncTestCases+from tests.unit.pipelex.builder.pipe.pipe_operator.pipe_func.test_data import PipeFuncTestCasesclass TestPipeFuncBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.pysimilarity index 91%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.pyindex 473e393b5..93802c05a 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_data.py@@ -1,7 +1,7 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_img_gen_spec import ImgGenSkill, PipeImgGenSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_img_gen_spec import ImgGenSkill, PipeImgGenSpecfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint@@ -19,7 +19,7 @@ class PipeImgGenTestCases:inputs=None,output="GeneratedImage",type="PipeImgGen",- category="PipeOperator",+ pipe_category="PipeOperator",img_gen_prompt=None,img_gen_prompt_var_name=None,model=None,@@ -47,7 +47,7 @@ class PipeImgGenTestCases:inputs={"description": InputRequirementBlueprint(concept="Text")},output="Image",type="PipeImgGen",- category="PipeOperator",+ pipe_category="PipeOperator",img_gen_prompt=None,model=ImgGenSkill.GEN_IMAGE.model_recommendation,aspect_ratio=None,diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.pysimilarity index 72%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.pyindex c30257cf2..e29dbf9d2 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_img_gen/test_pipe_img_gen.py@@ -1,8 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.pipe_img_gen_spec import PipeImgGenSpec+from pipelex.builder.pipe.pipe_img_gen_spec import PipeImgGenSpecfrom pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_operator.pipe_img_gen.test_data import PipeImgGenTestCases+from tests.unit.pipelex.builder.pipe.pipe_operator.pipe_img_gen.test_data import PipeImgGenTestCasesclass TestPipeImgGenBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.pysimilarity index 98%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_data.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.pyindex 8821adff8..523bd3361 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_data.py@@ -1,8 +1,8 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpecfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_llm_spec import PipeLLMSpecfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintdiff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.pysimilarity index 81%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.pyrename to tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.pyindex 6609fa589..fa02b5156 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py+++ b/tests/unit/pipelex/builder/pipe/pipe_operator/pipe_llm/test_pipe_llm.py@@ -1,10 +1,10 @@import pytestfrom pipelex import pretty_print-from pipelex.libraries.pipelines.builder.pipe.pipe_llm_spec import PipeLLMSpec+from pipelex.builder.pipe.pipe_llm_spec import PipeLLMSpecfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintfrom pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory-from tests.unit.pipelex.libraries.pipelines.builder.pipe.pipe_operator.pipe_llm.test_data import PipeLLMTestCases+from tests.unit.pipelex.builder.pipe.pipe_operator.pipe_llm.test_data import PipeLLMTestCasesclass TestPipeLLMBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_inputs.py b/tests/unit/pipelex/builder/pipe/test_data_inputs.pysimilarity index 100%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_inputs.pyrename to tests/unit/pipelex/builder/pipe/test_data_inputs.pydiff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_pipe.py b/tests/unit/pipelex/builder/pipe/test_data_pipe.pysimilarity index 86%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_pipe.pyrename to tests/unit/pipelex/builder/pipe/test_data_pipe.pyindex 62a199fe8..2c0be069c 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_pipe.py+++ b/tests/unit/pipelex/builder/pipe/test_data_pipe.py@@ -1,8 +1,8 @@from typing import ClassVar+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprintfrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpecclass PipeBlueprintTestCases:@@ -11,14 +11,14 @@ class PipeBlueprintTestCases:PipeSpec(pipe_code="simple_pipe",type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="A simple pipe",inputs={"input": "Text"},output="ProcessedText",),PipeBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="A simple pipe",inputs={"input": InputRequirementBlueprint(concept="Text")},output="ProcessedText",@@ -30,7 +30,7 @@ class PipeBlueprintTestCases:PipeSpec(pipe_code="pipe_with_requirements",type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",description="Pipe with input requirements",inputs={"data": "Data",@@ -40,7 +40,7 @@ class PipeBlueprintTestCases:),PipeBlueprint(type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",description="Pipe with input requirements",inputs={"data": InputRequirementBlueprint(concept="Data"),@@ -55,14 +55,14 @@ class PipeBlueprintTestCases:PipeSpec(pipe_code="pipe_no_inputs",type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",description="Pipe without inputs",inputs={},output="GeneratedData",),PipeBlueprint(type="PipeFunc",- category="PipeOperator",+ pipe_category="PipeOperator",description="Pipe without inputs",inputs=None,output="GeneratedData",diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_sub_pipe.py b/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.pysimilarity index 94%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_sub_pipe.pyrename to tests/unit/pipelex/builder/pipe/test_data_sub_pipe.pyindex 23ae59a9f..be01916c3 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_data_sub_pipe.py+++ b/tests/unit/pipelex/builder/pipe/test_data_sub_pipe.py@@ -1,6 +1,6 @@from typing import ClassVar-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprintdiff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_inputs_blueprint.py b/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.pysimilarity index 87%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_inputs_blueprint.pyrename to tests/unit/pipelex/builder/pipe/test_inputs_blueprint.pyindex 524052bd6..521e9e7e2 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_inputs_blueprint.py+++ b/tests/unit/pipelex/builder/pipe/test_inputs_blueprint.py@@ -1,8 +1,7 @@import pytestfrom pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint--from .test_data_inputs import InputRequirementTestCases+from tests.unit.pipelex.builder.pipe.test_data_inputs import InputRequirementTestCasesclass TestInputRequirementBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_pipe_blueprint.py b/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.pysimilarity index 78%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_pipe_blueprint.pyrename to tests/unit/pipelex/builder/pipe/test_pipe_blueprint.pyindex 4f11b6b87..24748240a 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_pipe_blueprint.py+++ b/tests/unit/pipelex/builder/pipe/test_pipe_blueprint.py@@ -1,9 +1,8 @@import pytest+from pipelex.builder.pipe.pipe_signature import PipeSpecfrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint-from pipelex.libraries.pipelines.builder.pipe.pipe_signature import PipeSpec--from .test_data_pipe import PipeBlueprintTestCases+from tests.unit.pipelex.builder.pipe.test_data_pipe import PipeBlueprintTestCasesclass TestPipeBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_sub_pipe_blueprint.py b/tests/unit/pipelex/builder/pipe/test_sub_pipe_blueprint.pysimilarity index 77%rename from tests/unit/pipelex/libraries/pipelines/builder/pipe/test_sub_pipe_blueprint.pyrename to tests/unit/pipelex/builder/pipe/test_sub_pipe_blueprint.pyindex 19f83965d..5b21da5fb 100644--- a/tests/unit/pipelex/libraries/pipelines/builder/pipe/test_sub_pipe_blueprint.py+++ b/tests/unit/pipelex/builder/pipe/test_sub_pipe_blueprint.py@@ -1,9 +1,8 @@import pytest-from pipelex.libraries.pipelines.builder.pipe.sub_pipe_spec import SubPipeSpec+from pipelex.builder.pipe.sub_pipe_spec import SubPipeSpecfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint--from .test_data_sub_pipe import SubPipeTestCases+from tests.unit.pipelex.builder.pipe.test_data_sub_pipe import SubPipeTestCasesclass TestSubPipeBlueprintConversion:diff --git a/tests/unit/pipelex/libraries/pipelines/builder/test_pipelex_bundle_blueprint.py b/tests/unit/pipelex/builder/test_pipelex_bundle_blueprint.pysimilarity index 100%rename from tests/unit/pipelex/libraries/pipelines/builder/test_pipelex_bundle_blueprint.pyrename to tests/unit/pipelex/builder/test_pipelex_bundle_blueprint.pydiff --git a/tests/unit/pipelex/cli/commands/__init__.py b/tests/unit/pipelex/cli/commands/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/cli/commands/test_init_cmd.py b/tests/unit/pipelex/cli/commands/test_init_cmd.pyindex a160e7025..fecd8fade 100644--- a/tests/unit/pipelex/cli/commands/test_init_cmd.py+++ b/tests/unit/pipelex/cli/commands/test_init_cmd.py@@ -17,16 +17,16 @@ from pipelex.exceptions import PipelexCLIErrorclass TestInitCmd:def test_do_init_config_copies_all_files(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup directories- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")target_dir = tmp_path / ".pipelex"target_dir.mkdir()- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)@@ -41,18 +41,18 @@ class TestInitCmd:def test_do_init_config_skips_existing_files(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup directories with existing file- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")- (template_dir / "new_file.toml").write_text("[new]\nconfig = 'value'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ (kit_configs_dir / "new_file.toml").write_text("[new]\nconfig = 'value'")target_dir = tmp_path / ".pipelex"target_dir.mkdir()(target_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '0.9'")- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)@@ -75,17 +75,17 @@ class TestInitCmd:def test_do_init_config_reset_overwrites_files(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup directories with existing file- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")target_dir = tmp_path / ".pipelex"target_dir.mkdir()(target_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '0.9'")- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)@@ -105,11 +105,11 @@ class TestInitCmd:def test_do_init_config_nested_directory_structure(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup complex nested structure- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")- inference_dir = template_dir / "inference"+ inference_dir = kit_configs_dir / "inference"inference_dir.mkdir()(inference_dir / "backends.toml").write_text("[backends]\nconfig = 'value'")@@ -120,9 +120,9 @@ class TestInitCmd:target_dir = tmp_path / ".pipelex"target_dir.mkdir()- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)@@ -144,16 +144,16 @@ class TestInitCmd:def test_do_init_config_handles_permission_error(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup directories- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")target_dir = tmp_path / ".pipelex"target_dir.mkdir()- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)@@ -167,15 +167,15 @@ class TestInitCmd:def test_do_init_config_creates_target_directory(self, tmp_path: Path, mocker: MockerFixture) -> None:# Setup template directory only- template_dir = tmp_path / "config_template"- template_dir.mkdir()- (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")+ kit_configs_dir = tmp_path / "kit" / "configs"+ kit_configs_dir.mkdir(parents=True)+ (kit_configs_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")target_dir = tmp_path / ".pipelex" # Don't create this- # Mock config manager+ # Mock get_configs_dir and config manager+ mocker.patch("pipelex.cli.commands.init_cmd.get_configs_dir", return_value=kit_configs_dir)mock_config_manager = mocker.MagicMock()- mock_config_manager.pipelex_root_dir = str(tmp_path)mock_config_manager.pipelex_config_dir = str(target_dir)mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)diff --git a/tests/unit/pipelex/client/__init__.py b/tests/unit/pipelex/client/__init__.pydeleted file mode 100644index 713f4401d..000000000--- a/tests/unit/pipelex/client/__init__.py+++ /dev/null@@ -1 +0,0 @@-# Client testsdiff --git a/tests/unit/pipelex/cogt/models/__init__.py b/tests/unit/pipelex/cogt/models/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/cogt/models/test_model_deck.py b/tests/unit/pipelex/cogt/models/test_model_deck.pyindex 85b159f18..06d044333 100644--- a/tests/unit/pipelex/cogt/models/test_model_deck.py+++ b/tests/unit/pipelex/cogt/models/test_model_deck.py@@ -61,7 +61,9 @@ class TestModelDeckGetOptionalInferenceModel:# Assertassert result is None- mock_log.warning.assert_called_once_with("Skipping model handle 'nonexistent-model' because it's not found in deck")+ mock_log.warning.assert_called_once_with(+ "Skipping model handle 'nonexistent-model' because it's not found in deck, it could be an external plugin."+ )def test_simple_string_alias_resolution_success(self, mocker: MockerFixture):# Arrange@@ -88,7 +90,7 @@ class TestModelDeckGetOptionalInferenceModel:assert result is Nonemock_log.debug.assert_called_once_with("Redirection for 'best-gpt': nonexistent-model")# The final warning is about the original alias, not the target- mock_log.warning.assert_called_with("Skipping model handle 'best-gpt' because it's not found in deck")+ mock_log.warning.assert_called_with("Skipping model handle 'best-gpt' because it's not found in deck, it could be an external plugin.")def test_list_alias_resolution_first_success(self, mocker: MockerFixture):# Arrange@@ -176,7 +178,9 @@ class TestModelDeckGetOptionalInferenceModel:assert result is None# Empty list evaluates to False, so no debug log is calledmock_log.debug.assert_not_called()- mock_log.warning.assert_called_once_with("Skipping model handle 'empty-alias' because it's not found in deck")+ mock_log.warning.assert_called_once_with(+ "Skipping model handle 'empty-alias' because it's not found in deck, it could be an external plugin."+ )def test_circular_alias_prevention(self, mocker: MockerFixture):# Note: The current implementation doesn't have explicit circular reference detection,@@ -253,7 +257,9 @@ class TestModelDeckGetOptionalInferenceModel:# Assertassert result is None- mock_log.warning.assert_called_once_with(f"Skipping model handle '{llm_handle}' because it's not found in deck")+ mock_log.warning.assert_called_once_with(+ f"Skipping model handle '{llm_handle}' because it's not found in deck, it could be an external plugin."+ )def test_logging_behavior(self, mocker: MockerFixture):# Arrange@@ -268,4 +274,4 @@ class TestModelDeckGetOptionalInferenceModel:mock_log.debug.assert_called_with("Redirection for 'test-alias': target-model")# The final warning is about the original alias, not the target- mock_log.warning.assert_called_with("Skipping model handle 'test-alias' because it's not found in deck")+ mock_log.warning.assert_called_with("Skipping model handle 'test-alias' because it's not found in deck, it could be an external plugin.")diff --git a/tests/unit/pipelex/tools/templating/test_template_preprocessor.py b/tests/unit/pipelex/cogt/templating/test_template_preprocessor.pysimilarity index 100%rename from tests/unit/pipelex/tools/templating/test_template_preprocessor.pyrename to tests/unit/pipelex/cogt/templating/test_template_preprocessor.pydiff --git a/tests/unit/pipelex/core/bundles/__init__.py b/tests/unit/pipelex/core/bundles/__init__.pydeleted file mode 100644index e978cd940..000000000--- a/tests/unit/pipelex/core/bundles/__init__.py+++ /dev/null@@ -1 +0,0 @@-"""Unit tests for core bundles module."""diff --git a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.pyindex 100f32552..3adeea351 100644--- a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py+++ b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py@@ -18,9 +18,9 @@ class PipeSorterTestCases:# Test case 1: No dependencies - all operatorsNO_DEPENDENCIES_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {- "pipe_c": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="C", inputs={}, output="Text"),- "pipe_a": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="A", inputs={}, output="Text"),- "pipe_b": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="B", inputs={}, output="Text"),+ "pipe_c": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="C", inputs={}, output="Text"),+ "pipe_a": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="A", inputs={}, output="Text"),+ "pipe_b": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="B", inputs={}, output="Text"),}NO_DEPENDENCIES_EXPECTED: ClassVar[list[str]] = ["pipe_a", "pipe_b", "pipe_c"] # Alphabetical order (all are roots)@@ -28,16 +28,16 @@ class PipeSorterTestCases:SIMPLE_CHAIN_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"pipe_c": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="C depends on B",inputs={},output="Text",steps=[SubPipeBlueprint(pipe="pipe_b", result="result_b")],),- "pipe_a": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="A no deps", inputs={}, output="Text"),+ "pipe_a": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="A no deps", inputs={}, output="Text"),"pipe_b": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="B depends on A",inputs={},output="Text",@@ -55,7 +55,7 @@ class PipeSorterTestCases:DIAMOND_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"pipe_d": PipeParallelBlueprint(type="PipeParallel",- category="PipeController",+ pipe_category="PipeController",description="D depends on B and C",inputs={},output="Text",@@ -65,10 +65,10 @@ class PipeSorterTestCases:],add_each_output=True,),- "pipe_a": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="A", inputs={}, output="Text"),+ "pipe_a": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="A", inputs={}, output="Text"),"pipe_c": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="C depends on A",inputs={},output="Text",@@ -76,7 +76,7 @@ class PipeSorterTestCases:),"pipe_b": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="B depends on A",inputs={},output="Text",@@ -92,23 +92,23 @@ class PipeSorterTestCases:# Chain 1: A -> B"pipe_b": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="B depends on A",inputs={},output="Text",steps=[SubPipeBlueprint(pipe="pipe_a", result="result_a")],),- "pipe_a": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="A", inputs={}, output="Text"),+ "pipe_a": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="A", inputs={}, output="Text"),# Chain 2: X -> Y"pipe_y": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="Y depends on X",inputs={},output="Text",steps=[SubPipeBlueprint(pipe="pipe_x", result="result_x")],),- "pipe_x": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="X", inputs={}, output="Text"),+ "pipe_x": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="X", inputs={}, output="Text"),}# Depth-first from roots (pipe_b, pipe_y alphabetically): pipe_b -> pipe_a, then pipe_y -> pipe_xMULTIPLE_CHAINS_EXPECTED: ClassVar[list[str]] = ["pipe_b", "pipe_a", "pipe_y", "pipe_x"]@@ -117,13 +117,13 @@ class PipeSorterTestCases:PIPE_BATCH_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"batch_pipe": PipeBatchBlueprint(type="PipeBatch",- category="PipeController",+ pipe_category="PipeController",description="Batch depends on process",inputs={},output="Text",branch_pipe_code="process_item",),- "process_item": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="Process", inputs={}, output="Text"),+ "process_item": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="Process", inputs={}, output="Text"),}PIPE_BATCH_EXPECTED: ClassVar[list[str]] = ["batch_pipe", "process_item"]@@ -131,7 +131,7 @@ class PipeSorterTestCases:PIPE_CONDITION_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"router": PipeConditionBlueprint(type="PipeCondition",- category="PipeController",+ pipe_category="PipeController",description="Routes to different pipes",inputs={},output="Text",@@ -142,9 +142,9 @@ class PipeSorterTestCases:},default_outcome="process_default",),- "process_large": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="Large", inputs={}, output="Text"),- "process_small": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="Small", inputs={}, output="Text"),- "process_default": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="Default", inputs={}, output="Text"),+ "process_large": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="Large", inputs={}, output="Text"),+ "process_small": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="Small", inputs={}, output="Text"),+ "process_default": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="Default", inputs={}, output="Text"),}# Router first, then its dependencies in alphabetical orderPIPE_CONDITION_EXPECTED: ClassVar[list[str]] = ["router", "process_default", "process_large", "process_small"]@@ -153,7 +153,7 @@ class PipeSorterTestCases:CIRCULAR_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"pipe_a": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="A depends on C (circular!)",inputs={},output="Text",@@ -161,7 +161,7 @@ class PipeSorterTestCases:),"pipe_b": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="B depends on A",inputs={},output="Text",@@ -169,7 +169,7 @@ class PipeSorterTestCases:),"pipe_c": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="C depends on B",inputs={},output="Text",@@ -181,7 +181,7 @@ class PipeSorterTestCases:MISSING_DEPENDENCY_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"pipe_b": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="B depends on A and Z (Z doesn't exist)",inputs={},output="Text",@@ -190,7 +190,7 @@ class PipeSorterTestCases:SubPipeBlueprint(pipe="pipe_z", result="result_z"), # Z doesn't exist in this bundle],),- "pipe_a": PipeLLMBlueprint(type="PipeLLM", category="PipeOperator", description="A", inputs={}, output="Text"),+ "pipe_a": PipeLLMBlueprint(type="PipeLLM", pipe_category="PipeOperator", description="A", inputs={}, output="Text"),}MISSING_DEPENDENCY_EXPECTED: ClassVar[list[str]] = ["pipe_b", "pipe_a"] # Z is ignored as it's not in the bundle@@ -198,35 +198,35 @@ class PipeSorterTestCases:IMAGE_INVERSION_PIPES: ClassVar[dict[str, PipeBlueprintUnion]] = {"analyze_image_content": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Analyzes the input photo to understand visual elements, composition, mood, etc.",inputs={"input_photo": "Image"},output="ImageAnalysis",),"define_opposite_concept": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Determines what constitutes the opposite for the analyzed image",inputs={"image_analysis": "ImageAnalysis"},output="OppositeDefinition",),"generate_image_prompt": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Crafts a detailed image generation prompt from the opposite definition",inputs={"opposite_definition": "OppositeDefinition"},output="ImagePrompt",),"generate_opposite_image": PipeImgGenBlueprint(type="PipeImgGen",- category="PipeOperator",+ pipe_category="PipeOperator",description="Generates the opposite image using AI",inputs={"generation_prompt": "ImagePrompt"},output="Image",),"photo_opposite_pipeline": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="Main pipeline that generates the opposite of an input photo",inputs={"input_photo": "Image"},output="Image",@@ -252,7 +252,7 @@ class PipeSorterTestCases:# Main sequence with nested controllers"main_pipeline": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="Main pipeline with nested controllers",inputs={"input": "Text"},output="Text",@@ -265,7 +265,7 @@ class PipeSorterTestCases:# Nested sequence"nested_sequence": PipeSequenceBlueprint(type="PipeSequence",- category="PipeController",+ pipe_category="PipeController",description="Nested preparation sequence",inputs={"input": "Text"},output="Text",@@ -277,7 +277,7 @@ class PipeSorterTestCases:# Condition router"router": PipeConditionBlueprint(type="PipeCondition",- category="PipeController",+ pipe_category="PipeController",description="Routes based on size",inputs={"prepared": "Text"},output="Text",@@ -291,42 +291,42 @@ class PipeSorterTestCases:# LLM operators"prepare_data": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Prepare the data",inputs={"input": "Text"},output="Text",),"validate_data": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Validate the prepared data",inputs={"prepared_data": "Text"},output="Text",),"process_small": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Process small items",inputs={"item": "Text"},output="Text",),"process_large": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Process large items",inputs={"item": "Text"},output="Text",),"process_default": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Default processing",inputs={"item": "Text"},output="Text",),"final_process": PipeLLMBlueprint(type="PipeLLM",- category="PipeOperator",+ pipe_category="PipeOperator",description="Final processing step",inputs={"processed": "Text"},output="Text",diff --git a/tests/unit/pipelex/core/bundles/test_pipe_sorter.py b/tests/unit/pipelex/core/bundles/test_pipe_sorter.pyindex 66e0a1868..9f8718aaf 100644--- a/tests/unit/pipelex/core/bundles/test_pipe_sorter.py+++ b/tests/unit/pipelex/core/bundles/test_pipe_sorter.py@@ -4,8 +4,7 @@ import pytestfrom pipelex.core.bundles.pipe_sorter import sort_pipes_by_dependenciesfrom pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion--from .test_data_pipe_sorter import PipeSorterTestCases+from tests.unit.pipelex.core.bundles.test_data_pipe_sorter import PipeSorterTestCasesclass TestSortPipesByDependencies:diff --git a/tests/unit/pipelex/core/concepts/concept/__init__.py b/tests/unit/pipelex/core/concepts/concept/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/concepts/concept_factory/__init__.py b/tests/unit/pipelex/core/concepts/concept_factory/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py b/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.pyindex 2abbd115f..f64ffd299 100644--- a/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py+++ b/tests/unit/pipelex/core/concepts/concept_factory/test_concept_factory.py@@ -8,8 +8,7 @@ from pipelex.core.concepts.concept_blueprint import ()from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.exceptions import StructureClassError--from .data import TestCases+from tests.unit.pipelex.core.concepts.concept_factory.data import TestCasesclass TestConceptFactory:diff --git a/tests/unit/pipelex/core/concepts/concept_library/__init__.py b/tests/unit/pipelex/core/concepts/concept_library/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py b/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.pyindex 13ee6d3a1..ce04ace92 100644--- a/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py+++ b/tests/unit/pipelex/core/concepts/test_concept_find_image_field_paths.py@@ -6,7 +6,7 @@ from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.concepts.concept_native import NativeConceptCodefrom pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.hub import get_concept_library, get_native_concept, get_required_concept-from pipelex.tools.class_registry_utils import ClassRegistryUtils+from pipelex.system.registries.class_registry_utils import ClassRegistryUtilsfrom tests.unit.pipelex.core.concepts import datafrom tests.unit.pipelex.core.concepts.data import TestDatadiff --git a/tests/unit/pipelex/core/interpreter/__init__.py b/tests/unit/pipelex/core/interpreter/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/memory/__init__.py b/tests/unit/pipelex/core/memory/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/memory/conftest.py b/tests/unit/pipelex/core/memory/conftest.pynew file mode 100644index 000000000..faf9c248a--- /dev/null+++ b/tests/unit/pipelex/core/memory/conftest.py@@ -0,0 +1,206 @@+"""Shared fixtures and test data for WorkingMemory tests."""++from typing import ClassVar++import pytest++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.concept_native import NativeConceptCode+from pipelex.core.domains.domain import SpecialDomain+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.html_content import HtmlContent+from pipelex.core.stuffs.image_content import ImageContent+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent+from pipelex.core.stuffs.text_content import TextContent+++# Test data constants+class TestWorkingMemoryData:+ """Test data and constants for WorkingMemory tests."""++ # Sample text content+ SAMPLE_TEXT = """+ The Dawn of Ultra-Rapid Transit: NextGen High-Speed Trains Redefine Travel+ By Eliza Montgomery, Transportation Technology Reporter++ In an era where time is increasingly precious, a revolution in rail transportation is quietly+ transforming how we connect cities and regions. The emergence of ultra-high-speed train+ networks, capable of speeds exceeding 350 mph, promises to render certain short-haul+ flights obsolete while dramatically reducing carbon emissions.+ """++ # Sample PDF and image URLs+ SAMPLE_PDF_URL = "assets/extract_dpe/dpe_single_page.pdf"+ SAMPLE_IMAGE_URL = "assets/gantt_charts/sample_gantt.png"++ # Test cases for different content types+ SINGLE_TEXT_CASE = "single_text"+ SINGLE_IMAGE_CASE = "single_image"+ SINGLE_PDF_CASE = "single_pdf"+ MULTIPLE_STUFF_CASE = "multiple_stuff"+ WITH_ALIASES_CASE = "with_aliases"+ COMPLEX_LIST_CASE = "complex_list"+ TEXT_AND_IMAGES_CASE = "text_and_images"+ HTML_CONTENT_CASE = "html_content"+ NUMBER_CONTENT_CASE = "number_content"++ TEST_CASES: ClassVar[list[tuple[str, str]]] = [+ ("Single text content", SINGLE_TEXT_CASE),+ ("Single image content", SINGLE_IMAGE_CASE),+ ("Single PDF content", SINGLE_PDF_CASE),+ ("Multiple stuff items", MULTIPLE_STUFF_CASE),+ ("WorkingMemory with aliases", WITH_ALIASES_CASE),+ ("Complex list content", COMPLEX_LIST_CASE),+ ("Text and images content", TEXT_AND_IMAGES_CASE),+ ("HTML content", HTML_CONTENT_CASE),+ ("Number content", NUMBER_CONTENT_CASE),+ ]+++# Shared fixtures for all WorkingMemory tests+++def single_text_memory() -> WorkingMemory:+ """Create WorkingMemory with single text content."""+ return WorkingMemoryFactory.make_from_text(+ text=TestWorkingMemoryData.SAMPLE_TEXT, concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.TEXT, name="sample_text"+ )+++def single_image_memory() -> WorkingMemory:+ """Create WorkingMemory with single image content."""+ return WorkingMemoryFactory.make_from_image(+ image_url=TestWorkingMemoryData.SAMPLE_IMAGE_URL,+ concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.IMAGE,+ name="sample_image",+ )+++def single_pdf_memory() -> WorkingMemory:+ """Create WorkingMemory with single PDF content."""+ return WorkingMemoryFactory.make_from_pdf(+ pdf_url=TestWorkingMemoryData.SAMPLE_PDF_URL,+ concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.PDF,+ name="pdf_document",+ )+++def multiple_stuff_memory() -> WorkingMemory:+ """Create WorkingMemory with multiple stuff items."""+ text_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ name="question",+ content=TextContent(text="What are the aerodynamic features?"),+ )++ document_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ name="document",+ content=TextContent(text=TestWorkingMemoryData.SAMPLE_TEXT),+ )++ image_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),+ name="diagram",+ content=ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL),+ )++ return WorkingMemoryFactory.make_from_multiple_stuffs(stuff_list=[text_stuff, document_stuff, image_stuff], main_name="document")+++def memory_with_aliases() -> WorkingMemory:+ """Create WorkingMemory with aliases."""+ text_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ name="primary_text",+ content=TextContent(text="Primary content"),+ )++ secondary_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ name="secondary_text",+ content=TextContent(text="Secondary content"),+ )++ memory = WorkingMemoryFactory.make_from_multiple_stuffs(stuff_list=[text_stuff, secondary_stuff], main_name="primary_text")+ memory.set_alias(alias="main_text", target="primary_text")+ memory.set_alias(alias="backup_text", target="secondary_text")++ return memory+++def complex_list_memory() -> WorkingMemory:+ """Create WorkingMemory with complex list content containing mixed types."""+ complex_content: ListContent[TextContent | ImageContent | NumberContent] = ListContent(+ items=[+ TextContent(text="The quick brown fox jumps over the lazy dog"),+ ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL),+ NumberContent(number=42.5),+ ],+ )++ complex_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make(concept_code="List", domain=SpecialDomain.NATIVE, description="Lorem Ipsum", structure_class_name="ListContent"),+ name="mixed_list",+ content=complex_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=complex_stuff)+++def text_and_images_memory() -> WorkingMemory:+ """Create WorkingMemory with text and images content."""+ text_and_images_content = TextAndImagesContent(+ text=TextContent(text="Project overview with diagrams"),+ images=[ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL), ImageContent(url="assets/diagrams/architecture.png")],+ )++ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT_AND_IMAGES),+ name="project_overview",+ content=text_and_images_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+++def html_content_memory() -> WorkingMemory:+ """Create WorkingMemory with HTML content."""+ html_content = HtmlContent(+ inner_html="<h1>Test Report</h1><p>This is a <strong>test</strong> report.</p><ul><li>Item 1</li><li>Item 2</li></ul>",+ css_class="report-content",+ )++ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make(concept_code="Html", domain=SpecialDomain.NATIVE, description="Lorem Ipsum", structure_class_name="HtmlContent"),+ name="test_report",+ content=html_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+++def number_content_memory() -> WorkingMemory:+ """Create WorkingMemory with number content."""+ number_content = NumberContent(number=3.14159)++ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.NUMBER),+ name="pi_value",+ content=number_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)diff --git a/tests/unit/pipelex/core/memory/test_working_memory.py b/tests/unit/pipelex/core/memory/test_working_memory.pydeleted file mode 100644index d62bd4aa7..000000000--- a/tests/unit/pipelex/core/memory/test_working_memory.py+++ /dev/null@@ -1,227 +0,0 @@-from typing import ClassVar--import pytest--from pipelex.core.concepts.concept_factory import ConceptFactory-from pipelex.core.concepts.concept_native import NativeConceptCode-from pipelex.core.domains.domain import SpecialDomain-from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.stuffs.html_content import HtmlContent-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.list_content import ListContent-from pipelex.core.stuffs.number_content import NumberContent-from pipelex.core.stuffs.stuff_factory import StuffFactory-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.core.stuffs.text_content import TextContent---class TestWorkingMemoryData:- # Sample text content- SAMPLE_TEXT = """- The Dawn of Ultra-Rapid Transit: NextGen High-Speed Trains Redefine Travel- By Eliza Montgomery, Transportation Technology Reporter-- In an era where time is increasingly precious, a revolution in rail transportation is quietly- transforming how we connect cities and regions. The emergence of ultra-high-speed train- networks, capable of speeds exceeding 350 mph, promises to render certain short-haul- flights obsolete while dramatically reducing carbon emissions.- """-- # Sample PDF and image URLs- SAMPLE_PDF_URL = "assets/extract_dpe/dpe_single_page.pdf"- SAMPLE_IMAGE_URL = "assets/gantt_charts/sample_gantt.png"-- # Test cases for different content types- SINGLE_TEXT_CASE = "single_text"- SINGLE_IMAGE_CASE = "single_image"- SINGLE_PDF_CASE = "single_pdf"- MULTIPLE_STUFF_CASE = "multiple_stuff"- WITH_ALIASES_CASE = "with_aliases"- COMPLEX_LIST_CASE = "complex_list"- TEXT_AND_IMAGES_CASE = "text_and_images"- HTML_CONTENT_CASE = "html_content"- NUMBER_CONTENT_CASE = "number_content"-- TEST_CASES: ClassVar[list[tuple[str, str]]] = [- ("Single text content", SINGLE_TEXT_CASE),- ("Single image content", SINGLE_IMAGE_CASE),- ("Single PDF content", SINGLE_PDF_CASE),- ("Multiple stuff items", MULTIPLE_STUFF_CASE),- ("WorkingMemory with aliases", WITH_ALIASES_CASE),- ("Complex list content", COMPLEX_LIST_CASE),- ("Text and images content", TEXT_AND_IMAGES_CASE),- ("HTML content", HTML_CONTENT_CASE),- ("Number content", NUMBER_CONTENT_CASE),- ]---class TestWorkingMemory:- """Unit tests for WorkingMemory class - focus on core functionality."""-- @pytest.fixture- def single_text_memory(self) -> WorkingMemory:- """Create WorkingMemory with single text content."""- return WorkingMemoryFactory.make_from_text(- text=TestWorkingMemoryData.SAMPLE_TEXT, concept_string=SpecialDomain.NATIVE + "." + NativeConceptCode.TEXT, name="sample_text"- )-- @pytest.fixture- def single_image_memory(self) -> WorkingMemory:- """Create WorkingMemory with single image content."""- return WorkingMemoryFactory.make_from_image(- image_url=TestWorkingMemoryData.SAMPLE_IMAGE_URL,- concept_string="gantt.GanttImage",- name="gantt_chart_image",- )-- @pytest.fixture- def single_pdf_memory(self) -> WorkingMemory:- """Create WorkingMemory with single PDF content."""- return WorkingMemoryFactory.make_from_pdf(- pdf_url=TestWorkingMemoryData.SAMPLE_PDF_URL, concept_string=NativeConceptCode.PDF, name="pdf_document"- )-- @pytest.fixture- def multiple_stuff_memory(self) -> WorkingMemory:- """Create WorkingMemory with multiple stuff items."""- text_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- name="question",- content=TextContent(text="What are the aerodynamic features?"),- )-- document_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- name="document",- content=TextContent(text=TestWorkingMemoryData.SAMPLE_TEXT),- )-- image_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),- name="diagram",- content=ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL),- )-- return WorkingMemoryFactory.make_from_multiple_stuffs(stuff_list=[text_stuff, document_stuff, image_stuff], main_name="document")-- @pytest.fixture- def memory_with_aliases(self) -> WorkingMemory:- """Create WorkingMemory with aliases."""- text_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- name="primary_text",- content=TextContent(text="Primary content"),- )-- secondary_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- name="secondary_text",- content=TextContent(text="Secondary content"),- )-- memory = WorkingMemoryFactory.make_from_multiple_stuffs(stuff_list=[text_stuff, secondary_stuff], main_name="primary_text")- memory.set_alias(alias="main_text", target="primary_text")- memory.set_alias(alias="backup_text", target="secondary_text")-- return memory-- @pytest.fixture- def complex_list_memory(self) -> WorkingMemory:- """Create WorkingMemory with complex list content containing mixed types."""- complex_content: ListContent[TextContent | ImageContent | NumberContent] = ListContent(- items=[- TextContent(text="The quick brown fox jumps over the lazy dog"),- ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL),- NumberContent(number=42.5),- ],- )-- complex_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make(- concept_code="List", domain=SpecialDomain.NATIVE, description="Lorem Ipsum", structure_class_name="ListContent"- ),- name="mixed_list",- content=complex_content,- )-- return WorkingMemoryFactory.make_from_single_stuff(stuff=complex_stuff)-- @pytest.fixture- def text_and_images_memory(self) -> WorkingMemory:- """Create WorkingMemory with text and images content."""- text_and_images_content = TextAndImagesContent(- text=TextContent(text="Project overview with diagrams"),- images=[ImageContent(url=TestWorkingMemoryData.SAMPLE_IMAGE_URL), ImageContent(url="assets/diagrams/architecture.png")],- )-- stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT_AND_IMAGES),- name="project_overview",- content=text_and_images_content,- )-- return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)-- @pytest.fixture- def html_content_memory(self) -> WorkingMemory:- """Create WorkingMemory with HTML content."""- html_content = HtmlContent(- inner_html="<h1>Test Report</h1><p>This is a <strong>test</strong> report.</p><ul><li>Item 1</li><li>Item 2</li></ul>",- css_class="report-content",- )-- stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make(- concept_code="Html", domain=SpecialDomain.NATIVE, description="Lorem Ipsum", structure_class_name="HtmlContent"- ),- name="test_report",- content=html_content,- )-- return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)-- @pytest.fixture- def number_content_memory(self) -> WorkingMemory:- """Create WorkingMemory with number content."""- number_content = NumberContent(number=3.14159)-- stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.NUMBER),- name="pi_value",- content=number_content,- )-- return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)-- def test_working_memory_basic_functionality(self, single_text_memory: WorkingMemory):- """Test basic WorkingMemory functionality."""- # Should have one entry for the text content- assert len(single_text_memory.root) == 1- assert "sample_text" in single_text_memory.root-- # Check stuff retrieval- stuff = single_text_memory.get_stuff("sample_text")- assert stuff.concept.code == NativeConceptCode.TEXT- assert isinstance(stuff.content, TextContent)- assert stuff.content.text == TestWorkingMemoryData.SAMPLE_TEXT-- def test_working_memory_aliases(self, memory_with_aliases: WorkingMemory):- # Should have two root items and two aliases other then the main stuff- assert MAIN_STUFF_NAME in memory_with_aliases.aliases- # Remove it from the aliases- aliases = memory_with_aliases.aliases.copy()- del aliases[MAIN_STUFF_NAME]- assert len(memory_with_aliases.root) == 2- assert len(aliases) == 2-- # Check aliases work- primary_stuff = memory_with_aliases.get_stuff("primary_text")- alias_stuff = memory_with_aliases.get_stuff("main_text")- assert primary_stuff == alias_stuff-- def test_working_memory_empty(self):- """Test empty WorkingMemory."""- empty_memory = WorkingMemoryFactory.make_empty()- assert len(empty_memory.root) == 0- assert len(empty_memory.aliases) == 0diff --git a/tests/unit/pipelex/core/memory/test_working_memory_basic.py b/tests/unit/pipelex/core/memory/test_working_memory_basic.pynew file mode 100644index 000000000..de907a377--- /dev/null+++ b/tests/unit/pipelex/core/memory/test_working_memory_basic.py@@ -0,0 +1,42 @@+from pipelex.core.concepts.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.text_content import TextContent+from tests.unit.pipelex.core.memory.conftest import TestWorkingMemoryData+++class TestWorkingMemoryBasic:+ """Unit tests for WorkingMemory basic functionality."""++ def test_working_memory_basic_functionality(self, single_text_memory: WorkingMemory):+ """Test basic WorkingMemory functionality."""+ # Should have one entry for the text content+ assert len(single_text_memory.root) == 1+ assert "sample_text" in single_text_memory.root++ # Check stuff retrieval+ stuff = single_text_memory.get_stuff("sample_text")+ assert stuff.concept.code == NativeConceptCode.TEXT+ assert isinstance(stuff.content, TextContent)+ assert stuff.content.text == TestWorkingMemoryData.SAMPLE_TEXT++ def test_working_memory_aliases(self, memory_with_aliases: WorkingMemory):+ """Test WorkingMemory with aliases."""+ # Should have two root items and two aliases other then the main stuff+ assert MAIN_STUFF_NAME in memory_with_aliases.aliases+ # Remove it from the aliases+ aliases = memory_with_aliases.aliases.copy()+ del aliases[MAIN_STUFF_NAME]+ assert len(memory_with_aliases.root) == 2+ assert len(aliases) == 2++ # Check aliases work+ primary_stuff = memory_with_aliases.get_stuff("primary_text")+ alias_stuff = memory_with_aliases.get_stuff("main_text")+ assert primary_stuff == alias_stuff++ def test_working_memory_empty(self):+ """Test empty WorkingMemory."""+ empty_memory = WorkingMemoryFactory.make_empty()+ assert len(empty_memory.root) == 0+ assert len(empty_memory.aliases) == 0diff --git a/tests/unit/pipelex/core/memory/test_working_memory_generate_context.py b/tests/unit/pipelex/core/memory/test_working_memory_generate_context.pynew file mode 100644index 000000000..4a895b892--- /dev/null+++ b/tests/unit/pipelex/core/memory/test_working_memory_generate_context.py@@ -0,0 +1,243 @@+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.html_content import HtmlContent+from pipelex.core.stuffs.image_content import ImageContent+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.stuff_artefact import StuffArtefact+from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent+from pipelex.core.stuffs.text_content import TextContent+from tests.unit.pipelex.core.memory.conftest import TestWorkingMemoryData+++class TestWorkingMemoryGenerateContext:+ """Unit tests for WorkingMemory.generate_context() method."""++ def test_generate_context_empty(self):+ """Test generate_context with empty WorkingMemory."""+ empty_memory = WorkingMemoryFactory.make_empty()+ context = empty_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 0++ def test_generate_context_single_text(self, single_text_memory: WorkingMemory):+ """Test generate_context with single text content."""+ context = single_text_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # sample_text + MAIN_STUFF_NAME alias+ assert "sample_text" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["sample_text"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], TextContent)++ # Verify actual content value+ assert artefact["content"].text == TestWorkingMemoryData.SAMPLE_TEXT++ # Verify MAIN_STUFF_NAME points to the same artefact+ assert context[MAIN_STUFF_NAME] is context["sample_text"]++ def test_generate_context_single_pdf(self, single_pdf_memory: WorkingMemory):+ """Test generate_context with single PDF content."""+ context = single_pdf_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # pdf_document + MAIN_STUFF_NAME alias+ assert "pdf_document" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["pdf_document"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact++ # Verify actual content value+ assert artefact["content"].url == TestWorkingMemoryData.SAMPLE_PDF_URL++ # Verify MAIN_STUFF_NAME points to the same artefact+ assert context[MAIN_STUFF_NAME] is context["pdf_document"]++ def test_generate_context_single_image(self, single_image_memory: WorkingMemory):+ """Test generate_context with single image content."""+ context = single_image_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # sample_image + MAIN_STUFF_NAME alias+ assert "sample_image" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["sample_image"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], ImageContent)++ # Verify actual content value+ assert artefact["content"].url == TestWorkingMemoryData.SAMPLE_IMAGE_URL++ # Verify MAIN_STUFF_NAME points to the same artefact+ assert context[MAIN_STUFF_NAME] is context["sample_image"]++ def test_generate_context_multiple_stuffs(self, multiple_stuff_memory: WorkingMemory):+ """Test generate_context with multiple stuff items."""+ context = multiple_stuff_memory.generate_context()++ assert isinstance(context, dict)+ # Should have 3 stuffs + MAIN_STUFF_NAME alias = 4 entries+ assert len(context) == 4+ assert "question" in context+ assert "document" in context+ assert "diagram" in context+ assert MAIN_STUFF_NAME in context++ # Verify each artefact is a StuffArtefact with content+ assert isinstance(context["question"], StuffArtefact)+ assert isinstance(context["document"], StuffArtefact)+ assert isinstance(context["diagram"], StuffArtefact)+ assert "content" in context["question"]+ assert "content" in context["document"]+ assert "content" in context["diagram"]++ # Verify actual content values+ assert isinstance(context["question"]["content"], TextContent)+ assert context["question"]["content"].text == "What are the aerodynamic features?"+ assert isinstance(context["document"]["content"], TextContent)+ assert context["document"]["content"].text == TestWorkingMemoryData.SAMPLE_TEXT+ assert isinstance(context["diagram"]["content"], ImageContent)+ assert context["diagram"]["content"].url == TestWorkingMemoryData.SAMPLE_IMAGE_URL++ # Verify MAIN_STUFF_NAME points to document (main_name="document" in fixture)+ assert context[MAIN_STUFF_NAME] is context["document"]++ def test_generate_context_with_aliases(self, memory_with_aliases: WorkingMemory):+ """Test generate_context with aliases."""+ context = memory_with_aliases.generate_context()++ assert isinstance(context, dict)+ # Should have 2 stuffs + 3 aliases (MAIN_STUFF_NAME, main_text, backup_text) = 5 entries+ assert len(context) == 5+ assert "primary_text" in context+ assert "secondary_text" in context+ assert "main_text" in context+ assert "backup_text" in context+ assert MAIN_STUFF_NAME in context++ # Verify actual content values+ assert isinstance(context["primary_text"]["content"], TextContent)+ assert context["primary_text"]["content"].text == "Primary content"+ assert isinstance(context["secondary_text"]["content"], TextContent)+ assert context["secondary_text"]["content"].text == "Secondary content"++ # Verify aliases point to the same artefact objects (using 'is')+ assert context["main_text"] is context["primary_text"]+ assert context["backup_text"] is context["secondary_text"]+ assert context[MAIN_STUFF_NAME] is context["primary_text"]++ def test_generate_context_complex_list(self, complex_list_memory: WorkingMemory):+ """Test generate_context with complex list content."""+ context = complex_list_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # mixed_list + MAIN_STUFF_NAME alias+ assert "mixed_list" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["mixed_list"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], ListContent)++ # Verify actual list content values+ list_content = artefact["content"]+ assert len(list_content.items) == 3+ assert isinstance(list_content.items[0], TextContent)+ assert list_content.items[0].text == "The quick brown fox jumps over the lazy dog"+ assert isinstance(list_content.items[1], ImageContent)+ assert list_content.items[1].url == TestWorkingMemoryData.SAMPLE_IMAGE_URL+ assert isinstance(list_content.items[2], NumberContent)+ assert list_content.items[2].number == 42.5++ def test_generate_context_text_and_images(self, text_and_images_memory: WorkingMemory):+ """Test generate_context with text and images content."""+ context = text_and_images_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # project_overview + MAIN_STUFF_NAME alias+ assert "project_overview" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["project_overview"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], TextAndImagesContent)++ # Verify actual content values+ content = artefact["content"]+ assert content.text is not None+ assert content.text.text == "Project overview with diagrams"+ assert content.images is not None+ assert len(content.images) == 2+ assert content.images[0].url == TestWorkingMemoryData.SAMPLE_IMAGE_URL+ assert content.images[1].url == "assets/diagrams/architecture.png"++ def test_generate_context_html_content(self, html_content_memory: WorkingMemory):+ """Test generate_context with HTML content."""+ context = html_content_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # test_report + MAIN_STUFF_NAME alias+ assert "test_report" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["test_report"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], HtmlContent)++ # Verify actual content values+ content = artefact["content"]+ assert content.inner_html == "<h1>Test Report</h1><p>This is a <strong>test</strong> report.</p><ul><li>Item 1</li><li>Item 2</li></ul>"+ assert content.css_class == "report-content"++ def test_generate_context_number_content(self, number_content_memory: WorkingMemory):+ """Test generate_context with number content."""+ context = number_content_memory.generate_context()++ assert isinstance(context, dict)+ assert len(context) == 2 # pi_value + MAIN_STUFF_NAME alias+ assert "pi_value" in context+ assert MAIN_STUFF_NAME in context++ # Verify artefact structure+ artefact = context["pi_value"]+ assert isinstance(artefact, StuffArtefact)+ assert "content" in artefact+ assert isinstance(artefact["content"], NumberContent)++ # Verify actual content value+ assert artefact["content"].number == 3.14159++ def test_generate_context_artefact_independence(self, multiple_stuff_memory: WorkingMemory):+ """Test that each stuff generates independent artefacts."""+ context = multiple_stuff_memory.generate_context()++ # Get artefacts for different stuffs+ question_artefact = context["question"]+ document_artefact = context["document"]+ diagram_artefact = context["diagram"]++ # Verify they are different objects+ assert question_artefact is not document_artefact+ assert question_artefact is not diagram_artefact+ assert document_artefact is not diagram_artefact++ # But aliases should point to the same object+ assert context[MAIN_STUFF_NAME] is document_artefactdiff --git a/tests/unit/pipelex/core/memory/test_working_memory_typed_access.py b/tests/unit/pipelex/core/memory/test_working_memory_typed_access.pynew file mode 100644index 000000000..4c1af32ef--- /dev/null+++ b/tests/unit/pipelex/core/memory/test_working_memory_typed_access.py@@ -0,0 +1,223 @@+"""Unit tests for WorkingMemory get_typed_object_or_attribute() method."""++from typing import cast++import pytest++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.domains.domain import SpecialDomain+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.image_content import ImageContent+from pipelex.core.stuffs.page_content import PageContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.exceptions import WorkingMemoryStuffAttributeNotFoundError, WorkingMemoryTypeError+from tests.unit.pipelex.core.memory.conftest import TestWorkingMemoryData+++def nested_content_memory() -> WorkingMemory:+ """Create WorkingMemory with nested content structure (PageContent)."""+ page_content = PageContent(+ text_and_images=TextAndImagesContent(+ text=TextContent(text="Sample page text"),+ images=[+ ImageContent(url="assets/image1.png"),+ ImageContent(url="assets/image2.png"),+ ],+ ),+ page_view=ImageContent(url="assets/page_view.png"),+ )++ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make(+ concept_code="Page",+ domain=SpecialDomain.NATIVE,+ description="A page with text and images",+ structure_class_name="PageContent",+ ),+ name="sample_page",+ content=page_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+++def nested_content_with_none_memory() -> WorkingMemory:+ """Create WorkingMemory with nested content where some fields are None."""+ page_content = PageContent(+ text_and_images=TextAndImagesContent(+ text=None, # Optional field set to None+ images=None, # Optional field set to None+ ),+ page_view=None, # Optional field set to None+ )++ stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make(+ concept_code="Page",+ domain=SpecialDomain.NATIVE,+ description="A page with minimal content",+ structure_class_name="PageContent",+ ),+ name="minimal_page",+ content=page_content,+ )++ return WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)+++class TestWorkingMemoryTypedAccess:+ """Unit tests for WorkingMemory.get_typed_object_or_attribute() method."""++ def test_simple_access_without_type(self, single_text_memory: WorkingMemory):+ """Test retrieving content by simple name without type validation."""+ result = single_text_memory.get_typed_object_or_attribute("sample_text")++ assert isinstance(result, TextContent)+ assert result.text == TestWorkingMemoryData.SAMPLE_TEXT++ def test_simple_access_with_matching_type(self, single_text_memory: WorkingMemory):+ """Test retrieving content with matching type validation."""+ result = single_text_memory.get_typed_object_or_attribute("sample_text", wanted_type=TextContent)++ assert isinstance(result, TextContent)+ assert result.text == TestWorkingMemoryData.SAMPLE_TEXT++ def test_simple_access_with_wrong_type(self, single_text_memory: WorkingMemory):+ """Test that type mismatch raises WorkingMemoryTypeError."""+ with pytest.raises(WorkingMemoryTypeError) as exc_info:+ single_text_memory.get_typed_object_or_attribute("sample_text", wanted_type=ImageContent)++ assert "sample_text" in str(exc_info.value)+ assert "TextContent" in str(exc_info.value)+ assert "ImageContent" in str(exc_info.value)++ def test_nested_attribute_access_one_level(self, nested_content_memory: WorkingMemory):+ """Test retrieving nested attribute (one level deep)."""+ result = nested_content_memory.get_typed_object_or_attribute("sample_page.text_and_images")++ assert isinstance(result, TextAndImagesContent)+ assert result.text is not None+ assert result.text.text == "Sample page text"++ def test_nested_attribute_access_two_levels(self, nested_content_memory: WorkingMemory):+ """Test retrieving deeply nested attribute (two levels deep)."""+ result = nested_content_memory.get_typed_object_or_attribute("sample_page.text_and_images.text")++ assert isinstance(result, TextContent)+ assert result.text == "Sample page text"++ def test_nested_attribute_access_list_field(self, nested_content_memory: WorkingMemory):+ """Test retrieving nested list field."""+ result = nested_content_memory.get_typed_object_or_attribute("sample_page.text_and_images.images")++ # Type narrowing and validation+ assert isinstance(result, list)+ # Cast to help type checker - we validate the types with assertions+ images = cast("list[ImageContent]", result)+ assert len(images) == 2+ assert all(isinstance(img, ImageContent) for img in images)+ assert images[0].url == "assets/image1.png"+ assert images[1].url == "assets/image2.png"++ def test_nested_attribute_optional_field(self, nested_content_memory: WorkingMemory):+ """Test retrieving optional nested field."""+ result = nested_content_memory.get_typed_object_or_attribute("sample_page.page_view")++ assert isinstance(result, ImageContent)+ assert result.url == "assets/page_view.png"++ def test_nested_attribute_with_type_validation(self, nested_content_memory: WorkingMemory):+ """Test retrieving nested attribute with type validation."""+ result = nested_content_memory.get_typed_object_or_attribute(+ "sample_page.text_and_images.text",+ wanted_type=TextContent,+ )++ assert isinstance(result, TextContent)+ assert result.text == "Sample page text"++ def test_nested_attribute_with_wrong_type(self, nested_content_memory: WorkingMemory):+ """Test that type mismatch on nested attribute raises WorkingMemoryTypeError."""+ with pytest.raises(WorkingMemoryTypeError) as exc_info:+ nested_content_memory.get_typed_object_or_attribute(+ "sample_page.text_and_images.text",+ wanted_type=ImageContent,+ )++ assert "sample_page.text_and_images.text" in str(exc_info.value)+ assert "TextContent" in str(exc_info.value)+ assert "ImageContent" in str(exc_info.value)++ def test_nested_attribute_not_found(self, nested_content_memory: WorkingMemory):+ """Test that accessing non-existent nested attribute raises WorkingMemoryStuffAttributeNotFoundError."""+ with pytest.raises(WorkingMemoryStuffAttributeNotFoundError) as exc_info:+ nested_content_memory.get_typed_object_or_attribute("sample_page.nonexistent_field")++ assert "sample_page.nonexistent_field" in str(exc_info.value)++ def test_nested_attribute_deep_not_found(self, nested_content_memory: WorkingMemory):+ """Test that accessing non-existent deeply nested attribute raises error."""+ with pytest.raises(WorkingMemoryStuffAttributeNotFoundError) as exc_info:+ nested_content_memory.get_typed_object_or_attribute("sample_page.text_and_images.nonexistent")++ assert "sample_page.text_and_images.nonexistent" in str(exc_info.value)++ def test_none_content_without_type(self, nested_content_with_none_memory: WorkingMemory):+ """Test retrieving None content without type validation."""+ result = nested_content_with_none_memory.get_typed_object_or_attribute("minimal_page.text_and_images.text")++ assert result is None++ def test_none_content_with_type_validation(self, nested_content_with_none_memory: WorkingMemory):+ """Test that None content is allowed even when wanted_type is specified."""+ # This should not raise an error because None is explicitly allowed for optional fields+ result = nested_content_with_none_memory.get_typed_object_or_attribute(+ "minimal_page.text_and_images.text",+ wanted_type=TextContent,+ )++ assert result is None++ def test_none_optional_field(self, nested_content_with_none_memory: WorkingMemory):+ """Test retrieving None from optional field."""+ result = nested_content_with_none_memory.get_typed_object_or_attribute("minimal_page.page_view")++ assert result is None++ def test_multiple_stuffs_access(self, multiple_stuff_memory: WorkingMemory):+ """Test accessing different stuffs in memory with multiple items."""+ question_result = multiple_stuff_memory.get_typed_object_or_attribute("question")+ document_result = multiple_stuff_memory.get_typed_object_or_attribute("document")+ diagram_result = multiple_stuff_memory.get_typed_object_or_attribute("diagram")++ assert isinstance(question_result, TextContent)+ assert question_result.text == "What are the aerodynamic features?"++ assert isinstance(document_result, TextContent)+ assert document_result.text == TestWorkingMemoryData.SAMPLE_TEXT++ assert isinstance(diagram_result, ImageContent)+ assert diagram_result.url == TestWorkingMemoryData.SAMPLE_IMAGE_URL++ def test_alias_access(self, memory_with_aliases: WorkingMemory):+ """Test accessing content through alias."""+ result = memory_with_aliases.get_typed_object_or_attribute("main_text")++ assert isinstance(result, TextContent)+ assert result.text == "Primary content"++ def test_html_content_nested_access(self, html_content_memory: WorkingMemory):+ """Test accessing nested attributes of HtmlContent."""+ inner_html = html_content_memory.get_typed_object_or_attribute("test_report.inner_html")+ css_class = html_content_memory.get_typed_object_or_attribute("test_report.css_class")++ assert isinstance(inner_html, str)+ assert "<h1>Test Report</h1>" in inner_html++ assert isinstance(css_class, str)+ assert css_class == "report-content"diff --git a/tests/unit/pipelex/core/pipes/__init__.py b/tests/unit/pipelex/core/pipes/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/pipes/pipe_operator/__init__.py b/tests/unit/pipelex/core/pipes/pipe_operator/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/__init__.py b/tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/stuffs/__init__.py b/tests/unit/pipelex/core/stuffs/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/core/test_data/__init__.py b/tests/unit/pipelex/core/test_data/__init__.pyindex 700c94538..663094fea 100644--- a/tests/unit/pipelex/core/test_data/__init__.py+++ b/tests/unit/pipelex/core/test_data/__init__.py@@ -1,22 +1,21 @@from typing import ClassVarfrom pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint--from .complex.multi_feature import COMPLEX_TEST_CASES-from .concepts.refining_concepts import REFINING_CONCEPT_TEST_CASES-from .concepts.simple_concepts import SIMPLE_CONCEPT_TEST_CASES-from .concepts.structured_concepts import STRUCTURED_CONCEPT_TEST_CASES-from .domain.simple_domains import DOMAIN_TEST_CASES-from .errors.invalid_plx import ERROR_TEST_CASES-from .pipes.controllers.batch.pipe_batch import PIPE_BATCH_TEST_CASES-from .pipes.controllers.condition.pipe_condition import PIPE_CONDITION_TEST_CASES-from .pipes.controllers.parallel.pipe_parallel import PIPE_PARALLEL_TEST_CASES-from .pipes.controllers.sequence.pipe_sequence import PIPE_SEQUENCE_TEST_CASES-from .pipes.operators.compose.pipe_compose import PIPE_COMPOSE_TEST_CASES-from .pipes.operators.extract.pipe_extract import PIPE_EXTRACT_TEST_CASES-from .pipes.operators.func.pipe_func import PIPE_FUNC_TEST_CASES-from .pipes.operators.img_gen.pipe_img_gen import PIPE_IMG_GEN_TEST_CASES-from .pipes.operators.llm.pipe_llm import PIPE_LLM_TEST_CASES+from tests.unit.pipelex.core.test_data.complex.multi_feature import COMPLEX_TEST_CASES+from tests.unit.pipelex.core.test_data.concepts.refining_concepts import REFINING_CONCEPT_TEST_CASES+from tests.unit.pipelex.core.test_data.concepts.simple_concepts import SIMPLE_CONCEPT_TEST_CASES+from tests.unit.pipelex.core.test_data.concepts.structured_concepts import STRUCTURED_CONCEPT_TEST_CASES+from tests.unit.pipelex.core.test_data.domain.simple_domains import DOMAIN_TEST_CASES+from tests.unit.pipelex.core.test_data.errors.invalid_plx import ERROR_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.controllers.batch.pipe_batch import PIPE_BATCH_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.controllers.condition.pipe_condition import PIPE_CONDITION_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.controllers.parallel.pipe_parallel import PIPE_PARALLEL_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.controllers.sequence.pipe_sequence import PIPE_SEQUENCE_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.operators.compose.pipe_compose import PIPE_COMPOSE_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.operators.extract.pipe_extract import PIPE_EXTRACT_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.operators.func.pipe_func import PIPE_FUNC_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.operators.img_gen.pipe_img_gen import PIPE_IMG_GEN_TEST_CASES+from tests.unit.pipelex.core.test_data.pipes.operators.llm.pipe_llm import PIPE_LLM_TEST_CASESclass InterpreterTestCases:diff --git a/tests/unit/pipelex/core/test_data/complex/__init__.py b/tests/unit/pipelex/core/test_data/complex/__init__.pyindex bbba9d0b6..e69de29bb 100644--- a/tests/unit/pipelex/core/test_data/complex/__init__.py+++ b/tests/unit/pipelex/core/test_data/complex/__init__.py@@ -1 +0,0 @@-"""Complex test scenarios for PipelexInterpreter tests."""diff --git a/tests/unit/pipelex/core/test_data/errors/invalid_plx.py b/tests/unit/pipelex/core/test_data/errors/invalid_plx.pyindex fa64ecd34..5d8ae047c 100644--- a/tests/unit/pipelex/core/test_data/errors/invalid_plx.py+++ b/tests/unit/pipelex/core/test_data/errors/invalid_plx.py@@ -188,7 +188,7 @@ description = "Domain with invalid array syntax"[concept]TestConcept = ["Unclosed array"""",- ValidationError,+ PLXDecodeError,)INVALID_ARRAY_SYNTAX2 = ("invalid_array_syntax",diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py b/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.pyindex aa7bc6dc6..e7df0c42d 100644--- a/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py+++ b/tests/unit/pipelex/core/test_data/pipes/operators/compose/pipe_compose.py@@ -40,7 +40,7 @@ description = "Process a Jinja2 template"output = "Text"[pipe.compose_output.template]-source = "Hello {{ name }}!"+template = "Hello {{ name }}!"category = "markdown"""",PipelexBundleBlueprint(@@ -52,7 +52,7 @@ category = "markdown"description="Process a Jinja2 template",output=NativeConceptCode.TEXT,template=TemplateBlueprint(- source="Hello {{ name }}!",+ template="Hello {{ name }}!",category=TemplateCategory.MARKDOWN,),),diff --git a/tests/unit/pipelex/kit/test_index_loader.py b/tests/unit/pipelex/kit/test_index_loader.pynew file mode 100644index 000000000..3f71fa6c5--- /dev/null+++ b/tests/unit/pipelex/kit/test_index_loader.py@@ -0,0 +1,39 @@+from pipelex.kit.index_loader import load_index+++class TestKitIndexLoader:+ """Test kit index loading and validation."""++ def test_load_index_succeeds(self):+ """Test that index.toml loads successfully."""+ idx = load_index()+ assert idx is not None+ assert idx.agent_rules is not None+ assert idx.meta is not None++ def test_index_has_required_structure(self):+ """Test that loaded index has expected structure."""+ idx = load_index()+ assert hasattr(idx.agent_rules, "sets")+ assert hasattr(idx.agent_rules, "default_set")+ assert hasattr(idx.agent_rules, "demote")+ assert hasattr(idx.agent_rules, "cursor")+ assert hasattr(idx.agent_rules, "targets")++ def test_index_sets_contain_expected_files(self):+ """Test that agent_rules sets reference expected markdown files."""+ idx = load_index()+ assert "pipelex_language" in idx.agent_rules.sets+ assert "all" in idx.agent_rules.sets+ assert len(idx.agent_rules.sets["all"]) > 0++ # Verify files in sets end with .md+ for file_list in idx.agent_rules.sets.values():+ for file_name in file_list:+ assert file_name.endswith(".md"), f"Expected .md file, got {file_name}"++ def test_index_has_valid_default_set(self):+ """Test that default_set points to an existing set."""+ idx = load_index()+ default_set = idx.agent_rules.default_set+ assert default_set in idx.agent_rules.sets, f"Default set '{default_set}' not found in sets"diff --git a/tests/unit/pipelex/kit/test_markers.py b/tests/unit/pipelex/kit/test_markers.pynew file mode 100644index 000000000..8c4fcb7fd--- /dev/null+++ b/tests/unit/pipelex/kit/test_markers.py@@ -0,0 +1,37 @@+from pipelex.kit.markers import find_span, replace_span, wrap+++class TestMarkers:+ """Test marker utilities for content insertion and replacement."""++ def test_find_span_with_markers(self):+ """Test finding span when markers exist."""+ text = "prefix\n<!-- BEGIN -->\ncontent\n<!-- END -->\nsuffix"+ span = find_span(text, "<!-- BEGIN -->", "<!-- END -->")+ assert span is not None+ assert text[span[0] : span[1]] == "<!-- BEGIN -->\ncontent\n<!-- END -->"++ def test_find_span_no_markers(self):+ """Test finding span when markers don't exist."""+ text = "no markers here"+ span = find_span(text, "<!-- BEGIN -->", "<!-- END -->")+ assert span is None++ def test_find_span_incomplete_markers(self):+ """Test finding span with only begin marker."""+ text = "prefix\n<!-- BEGIN -->\ncontent without end"+ span = find_span(text, "<!-- BEGIN -->", "<!-- END -->")+ assert span is None++ def test_wrap_content(self):+ """Test wrapping content with markers."""+ content = "line1\nline2"+ wrapped = wrap("<!-- BEGIN -->", "<!-- END -->", content)+ assert wrapped == "<!-- BEGIN -->\nline1\nline2\n<!-- END -->"++ def test_replace_span(self):+ """Test replacing content within a span."""+ text = "prefix\nOLD_CONTENT\nsuffix"+ span = (7, 18) # Position of OLD_CONTENT+ new_text = replace_span(text, span, "NEW_CONTENT")+ assert new_text == "prefix\nNEW_CONTENT\nsuffix"diff --git a/tests/unit/pipelex/kit/test_paths.py b/tests/unit/pipelex/kit/test_paths.pynew file mode 100644index 000000000..4713cc79b--- /dev/null+++ b/tests/unit/pipelex/kit/test_paths.py@@ -0,0 +1,23 @@+from pipelex.kit.paths import get_agents_dir, get_configs_dir, get_kit_root+++class TestKitPaths:+ """Test kit path utilities."""++ def test_get_kit_root(self):+ """Test that kit root path is valid."""+ kit_root = get_kit_root()+ assert kit_root is not None+ assert (kit_root / "index.toml").is_file()++ def test_get_agents_dir(self):+ """Test that agents directory path is valid."""+ agents_dir = get_agents_dir()+ assert agents_dir is not None+ assert agents_dir.is_dir()++ def test_get_configs_dir(self):+ """Test that configs directory path is valid."""+ configs_dir = get_configs_dir()+ assert configs_dir is not None+ assert configs_dir.is_dir()diff --git a/tests/unit/pipelex/language/__init__.py b/tests/unit/pipelex/language/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/language/test_plx_factory.py b/tests/unit/pipelex/language/test_plx_factory.pyindex 559ffa3b4..3c79312ac 100644--- a/tests/unit/pipelex/language/test_plx_factory.py+++ b/tests/unit/pipelex/language/test_plx_factory.py@@ -5,7 +5,7 @@ import tomlkitfrom pytest_mock import MockerFixturefrom pipelex.language.plx_config import PlxConfig, PlxConfigForConcepts, PlxConfigForPipes, PlxConfigInlineTables, PlxConfigStrings-from pipelex.language.plx_factory import PlxFactory+from pipelex.language.plx_factory import PIPE_CATEGORY_FIELD_KEY, PlxFactoryclass TestPlxFactoryUnit:@@ -206,13 +206,13 @@ class TestPlxFactoryUnit:"""Test that category field is skipped."""_mock_config = mocker.patch.object(PlxFactory, "_plx_config", return_value=mock_plx_config)- mapping = {"field1": "value1", "category": "should_be_skipped", "field2": "value2"}+ mapping = {"field1": "value1", PIPE_CATEGORY_FIELD_KEY: "should_be_skipped", "field2": "value2"}result = PlxFactory.convert_mapping_to_table(mapping)assert isinstance(result, tomlkit.items.Table) # pyright: ignore[reportAttributeAccessIssue, reportUnknownMemberType]assert "field1" in resultassert "field2" in result- assert "category" not in result+ assert PIPE_CATEGORY_FIELD_KEY not in resultdef test_add_spaces_to_inline_tables_simple(self):"""Test adding spaces to simple inline tables."""diff --git a/tests/unit/pipelex/libraries/pipelines/builder/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/concept/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/concept/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_batch/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_condition/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_parallel/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controller/pipe_sequence/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_compose/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_extract/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_func/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_img_gen/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/__init__.py b/tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_operator/pipe_llm/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/pipe_controllers/batch/data.py b/tests/unit/pipelex/pipe_controllers/batch/data.pynew file mode 100644index 000000000..f5ec2791e--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/batch/data.py@@ -0,0 +1,60 @@+from typing import ClassVar++from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint+++class PipeBatchInputTestCases:+ """Test cases for PipeBatch input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_SIMPLE_BATCH: ClassVar[tuple[str, PipeBatchBlueprint]] = (+ "valid_simple_batch",+ PipeBatchBlueprint(+ description="Test case: valid_simple_batch",+ inputs={"items": "native.Text"},+ output="native.Text",+ branch_pipe_code="process_item",+ input_list_name="items",+ input_item_name="item",+ ),+ )++ VALID_WITHOUT_EXPLICIT_NAMES: ClassVar[tuple[str, PipeBatchBlueprint]] = (+ "valid_without_explicit_names",+ PipeBatchBlueprint(+ description="Test case: valid_without_explicit_names",+ inputs={"data_list": "native.Text"},+ output="native.Text",+ branch_pipe_code="process_data",+ ),+ )++ VALID_WITH_INPUT_LIST_NAME_ONLY: ClassVar[tuple[str, PipeBatchBlueprint]] = (+ "valid_with_input_list_name_only",+ PipeBatchBlueprint(+ description="Test case: valid_with_input_list_name_only",+ inputs={"records": "native.Text"},+ output="native.Text",+ branch_pipe_code="process_record",+ input_list_name="records",+ ),+ )++ VALID_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeBatchBlueprint]] = (+ "valid_multiple_inputs",+ PipeBatchBlueprint(+ description="Test case: valid_multiple_inputs",+ inputs={"items": "native.Text", "config": "native.Text"},+ output="native.Text",+ branch_pipe_code="process_with_config",+ input_list_name="items",+ input_item_name="item",+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeBatchBlueprint]]] = [+ VALID_SIMPLE_BATCH,+ VALID_WITHOUT_EXPLICIT_NAMES,+ VALID_WITH_INPUT_LIST_NAME_ONLY,+ VALID_MULTIPLE_INPUTS,+ ]diff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.pynew file mode 100644index 000000000..2eef16fa6--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_input.py@@ -0,0 +1,30 @@+import pytest++from pipelex import log+from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint+from pipelex.pipe_controllers.batch.pipe_batch_factory import PipeBatchFactory+from tests.unit.pipelex.pipe_controllers.batch.data import PipeBatchInputTestCases+++class TestPipeBatchValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeBatchInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeBatchBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_batch = PipeBatchFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_batch is not None+ assert pipe_batch.code == f"test_pipe_{test_id}"diff --git a/tests/unit/pipelex/pipe_controllers/condition/data.py b/tests/unit/pipelex/pipe_controllers/condition/data.pynew file mode 100644index 000000000..11ff05efa--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/condition/data.py@@ -0,0 +1,144 @@+from typing import Any, ClassVar++from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome+++class PipeConditionInputTestCases:+ """Test cases for PipeCondition input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_WITH_EXPRESSION: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_with_expression",+ PipeConditionBlueprint(+ description="Test case: valid_with_expression",+ inputs={"status": "native.Text"},+ output="native.Text",+ expression="status",+ outcomes={"approved": "approve_pipe", "rejected": "reject_pipe"},+ default_outcome="fallback_pipe",+ ),+ )++ VALID_WITH_EXPRESSION_TEMPLATE: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_with_expression_template",+ PipeConditionBlueprint(+ description="Test case: valid_with_expression_template",+ inputs={"category": "native.Text"},+ output="native.Text",+ expression_template="{{ category }}",+ outcomes={"small": "process_small", "large": "process_large"},+ default_outcome="process_default",+ ),+ )++ VALID_WITH_MULTIPLE_OUTCOMES: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_with_multiple_outcomes",+ PipeConditionBlueprint(+ description="Test case: valid_with_multiple_outcomes",+ inputs={"priority": "native.Text"},+ output="native.Text",+ expression="priority",+ outcomes={+ "high": "urgent_handler",+ "medium": "normal_handler",+ "low": "delayed_handler",+ },+ default_outcome="default_handler",+ ),+ )++ VALID_WITH_SPECIAL_OUTCOME: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_with_special_outcome",+ PipeConditionBlueprint(+ description="Test case: valid_with_special_outcome",+ inputs={"should_process": "native.Text"},+ output="native.Text",+ expression="should_process",+ outcomes={"yes": "process_pipe"},+ default_outcome=SpecialOutcome.CONTINUE,+ ),+ )++ VALID_WITH_ALIAS: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_with_alias",+ PipeConditionBlueprint(+ description="Test case: valid_with_alias",+ inputs={"type": "native.Text"},+ output="native.Text",+ expression="type",+ outcomes={"A": "handle_a", "B": "handle_b"},+ default_outcome="handle_default",+ add_alias_from_expression_to="selected_type",+ ),+ )++ VALID_SINGLE_OUTCOME: ClassVar[tuple[str, PipeConditionBlueprint]] = (+ "valid_single_outcome",+ PipeConditionBlueprint(+ description="Test case: valid_single_outcome",+ inputs={"flag": "native.Text"},+ output="native.Text",+ expression="flag",+ outcomes={"true": "process_pipe"},+ default_outcome=SpecialOutcome.CONTINUE,+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeConditionBlueprint]]] = [+ VALID_WITH_EXPRESSION,+ VALID_WITH_EXPRESSION_TEMPLATE,+ VALID_WITH_MULTIPLE_OUTCOMES,+ VALID_WITH_SPECIAL_OUTCOME,+ VALID_WITH_ALIAS,+ VALID_SINGLE_OUTCOME,+ ]++ # Error test cases: (test_id, blueprint_dict, expected_error_message_fragment)+ # Using dicts instead of blueprints to avoid validation errors during import+ ERROR_BOTH_EXPRESSION_AND_TEMPLATE: ClassVar[tuple[str, dict[str, Any], str]] = (+ "both_expression_and_template",+ {+ "description": "Test case: both_expression_and_template",+ "inputs": {"data": "native.Text"},+ "output": "native.Text",+ "expression": "data",+ "expression_template": "{{ data }}",+ "outcomes": {"A": "pipe_a"},+ "default_outcome": "pipe_default",+ },+ "exactly one of expression_template or expression",+ )++ ERROR_NEITHER_EXPRESSION_NOR_TEMPLATE: ClassVar[tuple[str, dict[str, Any], str]] = (+ "neither_expression_nor_template",+ {+ "description": "Test case: neither_expression_nor_template",+ "inputs": {"data": "native.Text"},+ "output": "native.Text",+ "expression": None,+ "expression_template": None,+ "outcomes": {"A": "pipe_a"},+ "default_outcome": "pipe_default",+ },+ "exactly one of expression_template or expression",+ )++ ERROR_EMPTY_OUTCOMES: ClassVar[tuple[str, dict[str, Any], str]] = (+ "empty_outcomes",+ {+ "description": "Test case: empty_outcomes",+ "inputs": {"data": "native.Text"},+ "output": "native.Text",+ "expression": "data",+ "outcomes": {},+ "default_outcome": "pipe_default",+ },+ "must have at least one mapping in outcomes",+ )++ ERROR_CASES: ClassVar[list[tuple[str, dict[str, Any], str]]] = [+ ERROR_BOTH_EXPRESSION_AND_TEMPLATE,+ ERROR_NEITHER_EXPRESSION_NOR_TEMPLATE,+ ERROR_EMPTY_OUTCOMES,+ ]diff --git a/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.pynew file mode 100644index 000000000..fb810bf2e--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/condition/test_pipe_condition_input.py@@ -0,0 +1,59 @@+from typing import Any++import pytest++from pipelex import log+from pipelex.exceptions import PipeDefinitionError, StaticValidationError+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint+from pipelex.pipe_controllers.condition.pipe_condition_factory import PipeConditionFactory+from tests.unit.pipelex.pipe_controllers.condition.data import PipeConditionInputTestCases+++class TestPipeConditionValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeConditionInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeConditionBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_condition = PipeConditionFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_condition is not None+ assert pipe_condition.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint_dict", "expected_error_message_fragment"),+ PipeConditionInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint_dict: dict[str, Any],+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012+ # Construct blueprint from dict at test time to trigger validation+ blueprint = PipeConditionBlueprint.model_validate(blueprint_dict)+ PipeConditionFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/pipe_controllers/parallel/data.py b/tests/unit/pipelex/pipe_controllers/parallel/data.pynew file mode 100644index 000000000..bdbfda0f9--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/parallel/data.py@@ -0,0 +1,111 @@+from typing import Any, ClassVar++from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+++class PipeParallelInputTestCases:+ """Test cases for PipeParallel input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_WITH_ADD_EACH_OUTPUT: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_with_add_each_output",+ PipeParallelBlueprint(+ description="Test case: valid_with_add_each_output",+ inputs={"data": "native.Text"},+ output="native.Text",+ parallels=[+ SubPipeBlueprint(pipe="process_a", result="result_a"),+ SubPipeBlueprint(pipe="process_b", result="result_b"),+ ],+ add_each_output=True,+ ),+ )++ VALID_WITH_COMBINED_OUTPUT: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_with_combined_output",+ PipeParallelBlueprint(+ description="Test case: valid_with_combined_output",+ inputs={"data": "native.Text"},+ output="native.Text",+ parallels=[+ SubPipeBlueprint(pipe="analyze_1", result="analysis_1"),+ SubPipeBlueprint(pipe="analyze_2", result="analysis_2"),+ ],+ combined_output="native.Text",+ ),+ )++ VALID_WITH_BOTH_OUTPUT_OPTIONS: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_with_both_output_options",+ PipeParallelBlueprint(+ description="Test case: valid_with_both_output_options",+ inputs={"data": "native.Text"},+ output="native.Text",+ parallels=[+ SubPipeBlueprint(pipe="compute_x", result="x"),+ SubPipeBlueprint(pipe="compute_y", result="y"),+ ],+ add_each_output=True,+ combined_output="native.Text",+ ),+ )++ VALID_THREE_PARALLELS: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_three_parallels",+ PipeParallelBlueprint(+ description="Test case: valid_three_parallels",+ inputs={"input_data": "native.Text"},+ output="native.Text",+ parallels=[+ SubPipeBlueprint(pipe="branch_1", result="result_1"),+ SubPipeBlueprint(pipe="branch_2", result="result_2"),+ SubPipeBlueprint(pipe="branch_3", result="result_3"),+ ],+ add_each_output=True,+ ),+ )++ VALID_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_multiple_inputs",+ PipeParallelBlueprint(+ description="Test case: valid_multiple_inputs",+ inputs={"text_data": "native.Text", "image_data": "native.Image"},+ output="native.Text",+ parallels=[+ SubPipeBlueprint(pipe="process_text", result="text_result"),+ SubPipeBlueprint(pipe="process_image", result="image_result"),+ ],+ combined_output="native.Text",+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeParallelBlueprint]]] = [+ VALID_WITH_ADD_EACH_OUTPUT,+ VALID_WITH_COMBINED_OUTPUT,+ VALID_WITH_BOTH_OUTPUT_OPTIONS,+ VALID_THREE_PARALLELS,+ VALID_MULTIPLE_INPUTS,+ ]++ # Error test cases: (test_id, blueprint_dict, expected_error_message_fragment)+ # Using dicts instead of blueprints to avoid validation errors during import+ ERROR_NO_OUTPUT_OPTIONS: ClassVar[tuple[str, dict[str, Any], str]] = (+ "no_output_options",+ {+ "description": "Test case: no_output_options",+ "inputs": {"data": "native.Text"},+ "output": "native.Text",+ "parallels": [+ {"pipe": "process_a", "result": "result_a"},+ {"pipe": "process_b", "result": "result_b"},+ ],+ "add_each_output": False,+ "combined_output": None,+ },+ "requires either add_each_output to be True or combined_output to be set",+ )++ ERROR_CASES: ClassVar[list[tuple[str, dict[str, Any], str]]] = [+ ERROR_NO_OUTPUT_OPTIONS,+ ]diff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.pynew file mode 100644index 000000000..08497daf5--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_input.py@@ -0,0 +1,59 @@+from typing import Any++import pytest++from pipelex import log+from pipelex.exceptions import PipeDefinitionError, StaticValidationError+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.parallel.pipe_parallel_factory import PipeParallelFactory+from tests.unit.pipelex.pipe_controllers.parallel.data import PipeParallelInputTestCases+++class TestPipeParallelValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeParallelInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeParallelBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_parallel = PipeParallelFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_parallel is not None+ assert pipe_parallel.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint_dict", "expected_error_message_fragment"),+ PipeParallelInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint_dict: dict[str, Any],+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012+ # Construct blueprint from dict at test time to trigger validation+ blueprint = PipeParallelBlueprint.model_validate(blueprint_dict)+ PipeParallelFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/pipe_controllers/sequence/data.py b/tests/unit/pipelex/pipe_controllers/sequence/data.pynew file mode 100644index 000000000..53d853d9c--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/sequence/data.py@@ -0,0 +1,105 @@+from typing import Any, ClassVar++from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+++class PipeSequenceInputTestCases:+ """Test cases for PipeSequence input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_SIMPLE_SEQUENCE: ClassVar[tuple[str, PipeSequenceBlueprint]] = (+ "valid_simple_sequence",+ PipeSequenceBlueprint(+ description="Test case: valid_simple_sequence",+ inputs={"text": "native.Text"},+ output="native.Text",+ steps=[+ SubPipeBlueprint(pipe="step_1", result="result_1"),+ SubPipeBlueprint(pipe="step_2", result="result_2"),+ ],+ ),+ )++ VALID_THREE_STEPS: ClassVar[tuple[str, PipeSequenceBlueprint]] = (+ "valid_three_steps",+ PipeSequenceBlueprint(+ description="Test case: valid_three_steps",+ inputs={"input_data": "native.Text"},+ output="native.Text",+ steps=[+ SubPipeBlueprint(pipe="process_step_1", result="processed_1"),+ SubPipeBlueprint(pipe="process_step_2", result="processed_2"),+ SubPipeBlueprint(pipe="process_step_3", result="final_output"),+ ],+ ),+ )++ VALID_SINGLE_STEP: ClassVar[tuple[str, PipeSequenceBlueprint]] = (+ "valid_single_step",+ PipeSequenceBlueprint(+ description="Test case: valid_single_step",+ inputs={"data": "native.Text"},+ output="native.Text",+ steps=[+ SubPipeBlueprint(pipe="single_process", result="output"),+ ],+ ),+ )++ VALID_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeSequenceBlueprint]] = (+ "valid_multiple_inputs",+ PipeSequenceBlueprint(+ description="Test case: valid_multiple_inputs",+ inputs={"text_input": "native.Text", "image_input": "native.Image"},+ output="native.Text",+ steps=[+ SubPipeBlueprint(pipe="analyze_text", result="text_analysis"),+ SubPipeBlueprint(pipe="analyze_image", result="image_analysis"),+ SubPipeBlueprint(pipe="combine_results", result="final_result"),+ ],+ ),+ )++ VALID_WITH_BATCH: ClassVar[tuple[str, PipeSequenceBlueprint]] = (+ "valid_with_batch",+ PipeSequenceBlueprint(+ description="Test case: valid_with_batch",+ inputs={"items": "native.Text"},+ output="native.Text",+ steps=[+ SubPipeBlueprint(+ pipe="process_item",+ batch_over="items",+ batch_as="item",+ result="processed_items",+ ),+ SubPipeBlueprint(pipe="summarize_results", result="summary"),+ ],+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeSequenceBlueprint]]] = [+ VALID_SIMPLE_SEQUENCE,+ VALID_THREE_STEPS,+ VALID_SINGLE_STEP,+ VALID_MULTIPLE_INPUTS,+ VALID_WITH_BATCH,+ ]++ # Error test cases: (test_id, blueprint_dict, expected_error_message_fragment)+ # Using dicts instead of blueprints to avoid validation errors during import+ ERROR_EMPTY_STEPS: ClassVar[tuple[str, dict[str, Any], str]] = (+ "empty_steps",+ {+ "description": "Test case: empty_steps",+ "inputs": {"text": "native.Text"},+ "output": "native.Text",+ "steps": [],+ },+ "must have at least 1 step",+ )++ ERROR_CASES: ClassVar[list[tuple[str, dict[str, Any], str]]] = [+ ERROR_EMPTY_STEPS,+ ]diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.pynew file mode 100644index 000000000..9043917ae--- /dev/null+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_pipe_sequence_input.py@@ -0,0 +1,59 @@+from typing import Any++import pytest++from pipelex import log+from pipelex.exceptions import PipeDefinitionError, StaticValidationError+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint+from pipelex.pipe_controllers.sequence.pipe_sequence_factory import PipeSequenceFactory+from tests.unit.pipelex.pipe_controllers.sequence.data import PipeSequenceInputTestCases+++class TestPipeSequenceValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeSequenceInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeSequenceBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_sequence = PipeSequenceFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_sequence is not None+ assert pipe_sequence.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint_dict", "expected_error_message_fragment"),+ PipeSequenceInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint_dict: dict[str, Any],+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises((StaticValidationError, ValueError, PipeDefinitionError)) as exc_info: # noqa: PT012+ # Construct blueprint from dict at test time to trigger validation+ blueprint = PipeSequenceBlueprint.model_validate(blueprint_dict)+ PipeSequenceFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/data.py b/tests/unit/pipelex/pipe_operators/pipe_compose/data.pynew file mode 100644index 000000000..b0066b52b--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/data.py@@ -0,0 +1,140 @@+from typing import ClassVar++from pipelex.cogt.templating.template_blueprint import TemplateBlueprint+from pipelex.cogt.templating.template_category import TemplateCategory+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint+from pipelex.tools.jinja2.jinja2_errors import Jinja2TemplateSyntaxError+++class PipeComposeInputTestCases:+ """Test cases for PipeCompose input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_SIMPLE_TEMPLATE: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_simple_template",+ PipeComposeBlueprint(+ description="Test case: valid_simple_template",+ inputs={"name": "native.Text"},+ output="native.Text",+ template="Hello {{ name }}!",+ ),+ )++ VALID_NO_INPUTS: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_no_inputs",+ PipeComposeBlueprint(+ description="Test case: valid_no_inputs",+ inputs={},+ output="native.Text",+ template="Hello World!",+ ),+ )++ VALID_TWO_INPUTS: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_two_inputs",+ PipeComposeBlueprint(+ description="Test case: valid_two_inputs",+ inputs={"first_name": "native.Text", "last_name": "native.Text"},+ output="native.Text",+ template="Hello {{ first_name }} {{ last_name }}!",+ ),+ )++ VALID_WITH_TEMPLATE_BLUEPRINT: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_with_template_blueprint",+ PipeComposeBlueprint(+ description="Test case: valid_with_template_blueprint",+ inputs={"content": "native.Text"},+ output="native.Text",+ template=TemplateBlueprint(+ template="# Title\n\n{{ content }}",+ category=TemplateCategory.MARKDOWN,+ ),+ ),+ )++ VALID_WITH_JINJA2_CONTROL: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_with_jinja2_control",+ PipeComposeBlueprint(+ description="Test case: valid_with_jinja2_control",+ inputs={"items": "native.Text"},+ output="native.Text",+ template="{% for item in items %}{{ item }}{% endfor %}",+ ),+ )++ VALID_WITH_HTML_TEMPLATE: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_with_html_template",+ PipeComposeBlueprint(+ description="Test case: valid_with_html_template",+ inputs={"title": "native.Text", "body": "native.Text"},+ output="native.Text",+ template=TemplateBlueprint(+ template="<h1>{{ title }}</h1><p>{{ body }}</p>",+ category=TemplateCategory.HTML,+ ),+ ),+ )++ VALID_COMPLEX_JINJA2: ClassVar[tuple[str, PipeComposeBlueprint]] = (+ "valid_complex_jinja2",+ PipeComposeBlueprint(+ description="Test case: valid_complex_jinja2",+ inputs={"user": "native.Text", "items": "native.Text"},+ output="native.Text",+ template="Hello {{ user }}!\n{% if items %}Items: {{ items }}{% endif %}",+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeComposeBlueprint]]] = [+ VALID_SIMPLE_TEMPLATE,+ VALID_NO_INPUTS,+ VALID_TWO_INPUTS,+ VALID_WITH_TEMPLATE_BLUEPRINT,+ VALID_WITH_JINJA2_CONTROL,+ VALID_WITH_HTML_TEMPLATE,+ VALID_COMPLEX_JINJA2,+ ]++ # Error test cases: (test_id, blueprint, expected_error_type, expected_error_message_fragment)+ ERROR_MISSING_INPUT_IN_TEMPLATE: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (+ "missing_input_in_template",+ PipeComposeBlueprint(+ description="Test case: missing_input_in_template",+ inputs={},+ output="native.Text",+ template="Hello {{ name }}!",+ ),+ Exception, # Will be PipeDefinitionError+ "Required variable 'name' is not in the inputs",+ )++ ERROR_TWO_INPUTS_THREE_VARIABLES: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (+ "two_inputs_three_variables",+ PipeComposeBlueprint(+ description="Test case: two_inputs_three_variables",+ inputs={"first_name": "native.Text", "last_name": "native.Text"},+ output="native.Text",+ template="Hello {{ first_name }} {{ last_name }} from {{ city }}!",+ ),+ Exception, # Will be PipeDefinitionError+ "Required variable 'city' is not in the inputs",+ )++ ERROR_INVALID_JINJA2_SYNTAX: ClassVar[tuple[str, PipeComposeBlueprint, type[Exception], str]] = (+ "invalid_jinja2_syntax",+ PipeComposeBlueprint(+ description="Test case: invalid_jinja2_syntax",+ inputs={"name": "native.Text"},+ output="native.Text",+ template="Hello {{ name }!",+ ),+ Jinja2TemplateSyntaxError,+ "Could not parse Jinja2 template",+ )++ ERROR_CASES: ClassVar[list[tuple[str, PipeComposeBlueprint, type[Exception], str]]] = [+ ERROR_MISSING_INPUT_IN_TEMPLATE,+ ERROR_TWO_INPUTS_THREE_VARIABLES,+ ERROR_INVALID_JINJA2_SYNTAX,+ ]diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.pynew file mode 100644index 000000000..dbf8aa90c--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/test_pipe_compose_input.py@@ -0,0 +1,55 @@+import pytest++from pipelex import log+from pipelex.pipe_operators.compose.pipe_compose_blueprint import PipeComposeBlueprint+from pipelex.pipe_operators.compose.pipe_compose_factory import PipeComposeFactory+from tests.unit.pipelex.pipe_operators.pipe_compose.data import PipeComposeInputTestCases+++class TestPipeComposeValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeComposeInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeComposeBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_compose = PipeComposeFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_compose is not None+ assert pipe_compose.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint", "expected_error_type", "expected_error_message_fragment"),+ PipeComposeInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint: PipeComposeBlueprint,+ expected_error_type: type[Exception],+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises(expected_error_type) as exc_info:+ PipeComposeFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/data.py b/tests/unit/pipelex/pipe_operators/pipe_extract/data.pynew file mode 100644index 000000000..230517642--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/data.py@@ -0,0 +1,103 @@+from typing import ClassVar++from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint+++class PipeExtractInputTestCases:+ """Test cases for PipeExtract input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_IMAGE_INPUT: ClassVar[tuple[str, PipeExtractBlueprint]] = (+ "valid_image_input",+ PipeExtractBlueprint(+ description="Test case: valid_image_input",+ inputs={"document_image": "native.Image"},+ output="native.Page",+ ),+ )++ VALID_PDF_INPUT: ClassVar[tuple[str, PipeExtractBlueprint]] = (+ "valid_pdf_input",+ PipeExtractBlueprint(+ description="Test case: valid_pdf_input",+ inputs={"document": "native.PDF"},+ output="native.Page",+ ),+ )++ VALID_IMAGE_WITH_PAGE_IMAGES: ClassVar[tuple[str, PipeExtractBlueprint]] = (+ "valid_image_with_page_images",+ PipeExtractBlueprint(+ description="Test case: valid_image_with_page_images",+ inputs={"invoice_image": "native.Image"},+ output="native.Page",+ page_images=True,+ ),+ )++ VALID_PDF_WITH_PAGE_VIEWS: ClassVar[tuple[str, PipeExtractBlueprint]] = (+ "valid_pdf_with_page_views",+ PipeExtractBlueprint(+ description="Test case: valid_pdf_with_page_views",+ inputs={"contract": "native.PDF"},+ output="native.Page",+ page_views=True,+ page_views_dpi=150,+ ),+ )++ VALID_IMAGE_WITH_CAPTIONS: ClassVar[tuple[str, PipeExtractBlueprint]] = (+ "valid_image_with_captions",+ PipeExtractBlueprint(+ description="Test case: valid_image_with_captions",+ inputs={"report_image": "native.Image"},+ output="native.Page",+ page_images=True,+ page_image_captions=True,+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeExtractBlueprint]]] = [+ VALID_IMAGE_INPUT,+ VALID_PDF_INPUT,+ VALID_IMAGE_WITH_PAGE_IMAGES,+ VALID_PDF_WITH_PAGE_VIEWS,+ VALID_IMAGE_WITH_CAPTIONS,+ ]++ # Error test cases: (test_id, blueprint, expected_error_message_fragment)+ ERROR_NO_INPUT: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (+ "no_input",+ PipeExtractBlueprint(+ description="Test case: no_input",+ inputs={},+ output="native.Page",+ ),+ "missing_input_variable",+ )++ ERROR_TOO_MANY_INPUTS: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (+ "too_many_inputs",+ PipeExtractBlueprint(+ description="Test case: too_many_inputs",+ inputs={"image1": "native.Image", "image2": "native.Image"},+ output="native.Page",+ ),+ "too_many_candidate_inputs",+ )++ ERROR_WRONG_INPUT_TYPE: ClassVar[tuple[str, PipeExtractBlueprint, str]] = (+ "wrong_input_type",+ PipeExtractBlueprint(+ description="Test case: wrong_input_type",+ inputs={"text_doc": "native.Text"},+ output="native.Page",+ ),+ "inadequate_input_concept",+ )++ ERROR_CASES: ClassVar[list[tuple[str, PipeExtractBlueprint, str]]] = [+ ERROR_NO_INPUT,+ ERROR_TOO_MANY_INPUTS,+ ERROR_WRONG_INPUT_TYPE,+ ]diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.pynew file mode 100644index 000000000..359460d9b--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_input.py@@ -0,0 +1,55 @@+import pytest++from pipelex import log+from pipelex.exceptions import StaticValidationError+from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint+from pipelex.pipe_operators.extract.pipe_extract_factory import PipeExtractFactory+from tests.unit.pipelex.pipe_operators.pipe_extract.data import PipeExtractInputTestCases+++class TestPipeExtractValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeExtractInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeExtractBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ # Validation happens automatically during instantiation via model_validator+ pipe_extract = PipeExtractFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_extract is not None+ assert pipe_extract.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint", "expected_error_message_fragment"),+ PipeExtractInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint: PipeExtractBlueprint,+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises(StaticValidationError) as exc_info:+ PipeExtractFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/data.py b/tests/unit/pipelex/pipe_operators/pipe_func/data.pynew file mode 100644index 000000000..6572cff4b--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_func/data.py@@ -0,0 +1,71 @@+from typing import ClassVar++from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint+++class PipeFuncInputTestCases:+ """Test cases for PipeFunc input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_NO_INPUTS: ClassVar[tuple[str, PipeFuncBlueprint]] = (+ "valid_no_inputs",+ PipeFuncBlueprint(+ description="Test case: valid_no_inputs",+ inputs={},+ output="native.Text",+ function_name="my_function",+ ),+ )++ VALID_SINGLE_INPUT: ClassVar[tuple[str, PipeFuncBlueprint]] = (+ "valid_single_input",+ PipeFuncBlueprint(+ description="Test case: valid_single_input",+ inputs={"input_data": "native.Text"},+ output="native.Text",+ function_name="process_text",+ ),+ )++ VALID_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeFuncBlueprint]] = (+ "valid_multiple_inputs",+ PipeFuncBlueprint(+ description="Test case: valid_multiple_inputs",+ inputs={"text_input": "native.Text", "number_input": "native.Number"},+ output="native.Text",+ function_name="combine_data",+ ),+ )++ VALID_IMAGE_INPUT: ClassVar[tuple[str, PipeFuncBlueprint]] = (+ "valid_image_input",+ PipeFuncBlueprint(+ description="Test case: valid_image_input",+ inputs={"image": "native.Image"},+ output="native.Text",+ function_name="process_image",+ ),+ )++ VALID_MIXED_INPUTS: ClassVar[tuple[str, PipeFuncBlueprint]] = (+ "valid_mixed_inputs",+ PipeFuncBlueprint(+ description="Test case: valid_mixed_inputs",+ inputs={"text": "native.Text", "image": "native.Image", "number": "native.Number"},+ output="native.Text",+ function_name="process_all",+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeFuncBlueprint]]] = [+ VALID_NO_INPUTS,+ VALID_SINGLE_INPUT,+ VALID_MULTIPLE_INPUTS,+ VALID_IMAGE_INPUT,+ VALID_MIXED_INPUTS,+ ]++ # Note: PipeFunc has minimal validation since it's very flexible+ # The main validation is that function_name is required (enforced by Pydantic)+ # We don't have error cases for inputs since any input configuration is valid+ ERROR_CASES: ClassVar[list[tuple[str, PipeFuncBlueprint, str]]] = []diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.pynew file mode 100644index 000000000..cffd07d99--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_pipe_func_input.py@@ -0,0 +1,30 @@+import pytest++from pipelex import log+from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint+from pipelex.pipe_operators.func.pipe_func_factory import PipeFuncFactory+from tests.unit.pipelex.pipe_operators.pipe_func.data import PipeFuncInputTestCases+++class TestPipeFuncValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeFuncInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeFuncBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ pipe_func = PipeFuncFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_func is not None+ assert pipe_func.code == f"test_pipe_{test_id}"+ assert pipe_func.function_name == blueprint.function_namediff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.pynew file mode 100644index 000000000..5ab79f8a5--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/data.py@@ -0,0 +1,159 @@+from typing import ClassVar++from pipelex.cogt.img_gen.img_gen_job_components import AspectRatio, Background, OutputFormat+from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint+++class PipeImgGenInputTestCases:+ """Test cases for PipeImgGen input validation."""++ # Valid test cases: (test_id, blueprint)+ VALID_TEXT_INPUT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_text_input",+ PipeImgGenBlueprint(+ description="Test case: valid_text_input",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ ),+ )++ VALID_WITH_INLINE_PROMPT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_inline_prompt",+ PipeImgGenBlueprint(+ description="Test case: valid_with_inline_prompt",+ inputs={},+ output="native.Image",+ img_gen_prompt="A beautiful sunset over the ocean",+ ),+ )++ VALID_WITH_ASPECT_RATIO: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_aspect_ratio",+ PipeImgGenBlueprint(+ description="Test case: valid_with_aspect_ratio",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ aspect_ratio=AspectRatio.LANDSCAPE_16_9,+ ),+ )++ VALID_WITH_NB_OUTPUT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_nb_output",+ PipeImgGenBlueprint(+ description="Test case: valid_with_nb_output",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ nb_output=3,+ ),+ )++ VALID_WITH_SEED: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_seed",+ PipeImgGenBlueprint(+ description="Test case: valid_with_seed",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ seed=42,+ ),+ )++ VALID_WITH_SEED_AUTO: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_seed_auto",+ PipeImgGenBlueprint(+ description="Test case: valid_with_seed_auto",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ seed="auto",+ ),+ )++ VALID_WITH_BACKGROUND: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_background",+ PipeImgGenBlueprint(+ description="Test case: valid_with_background",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ background=Background.TRANSPARENT,+ ),+ )++ VALID_WITH_OUTPUT_FORMAT: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_output_format",+ PipeImgGenBlueprint(+ description="Test case: valid_with_output_format",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ output_format=OutputFormat.PNG,+ ),+ )++ VALID_WITH_IS_RAW: ClassVar[tuple[str, PipeImgGenBlueprint]] = (+ "valid_with_is_raw",+ PipeImgGenBlueprint(+ description="Test case: valid_with_is_raw",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ is_raw=True,+ ),+ )++ VALID_CASES: ClassVar[list[tuple[str, PipeImgGenBlueprint]]] = [+ VALID_TEXT_INPUT,+ VALID_WITH_INLINE_PROMPT,+ VALID_WITH_ASPECT_RATIO,+ VALID_WITH_NB_OUTPUT,+ VALID_WITH_SEED,+ VALID_WITH_SEED_AUTO,+ VALID_WITH_BACKGROUND,+ VALID_WITH_OUTPUT_FORMAT,+ VALID_WITH_IS_RAW,+ ]++ # Error test cases: (test_id, blueprint, expected_error_message_fragment)+ ERROR_NO_INPUT_NO_PROMPT: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (+ "no_input_no_prompt",+ PipeImgGenBlueprint(+ description="Test case: no_input_no_prompt",+ inputs={},+ output="native.Image",+ ),+ "missing_input_variable",+ )++ ERROR_MULTIPLE_INPUTS: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (+ "multiple_inputs",+ PipeImgGenBlueprint(+ description="Test case: multiple_inputs",+ inputs={"prompt1": "native.Text", "prompt2": "native.Text"},+ output="native.Image",+ ),+ "too_many_candidate_inputs",+ )++ ERROR_WRONG_INPUT_TYPE: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (+ "wrong_input_type",+ PipeImgGenBlueprint(+ description="Test case: wrong_input_type",+ inputs={"image": "native.Image"},+ output="native.Image",+ ),+ "inadequate_input_concept",+ )++ ERROR_BOTH_PROMPT_AND_INPUT: ClassVar[tuple[str, PipeImgGenBlueprint, str]] = (+ "both_prompt_and_input",+ PipeImgGenBlueprint(+ description="Test case: both_prompt_and_input",+ inputs={"prompt": "native.Text"},+ output="native.Image",+ img_gen_prompt="A beautiful sunset",+ ),+ "There must be no inputs if img_gen_prompt is provided",+ )++ ERROR_CASES: ClassVar[list[tuple[str, PipeImgGenBlueprint, str]]] = [+ ERROR_NO_INPUT_NO_PROMPT,+ ERROR_MULTIPLE_INPUTS,+ ERROR_WRONG_INPUT_TYPE,+ ERROR_BOTH_PROMPT_AND_INPUT,+ ]diff --git a/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.pynew file mode 100644index 000000000..ca07c75c5--- /dev/null+++ b/tests/unit/pipelex/pipe_operators/pipe_img_gen/test_pipe_img_gen_input.py@@ -0,0 +1,54 @@+import pytest++from pipelex import log+from pipelex.exceptions import PipeDefinitionError, StaticValidationError+from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint+from pipelex.pipe_operators.img_gen.pipe_img_gen_factory import PipeImgGenFactory+from tests.unit.pipelex.pipe_operators.pipe_img_gen.data import PipeImgGenInputTestCases+++class TestPipeImgGenValidateInputs:+ @pytest.mark.parametrize(+ ("test_id", "blueprint"),+ PipeImgGenInputTestCases.VALID_CASES,+ )+ def test_validate_inputs_valid_cases(+ self,+ test_id: str,+ blueprint: PipeImgGenBlueprint,+ ):+ log.debug(f"Testing valid case: {test_id}")++ pipe_img_gen = PipeImgGenFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ # Assert that the pipe was created successfully+ assert pipe_img_gen is not None+ assert pipe_img_gen.code == f"test_pipe_{test_id}"++ @pytest.mark.parametrize(+ ("test_id", "blueprint", "expected_error_message_fragment"),+ PipeImgGenInputTestCases.ERROR_CASES,+ )+ def test_validate_inputs_error_cases(+ self,+ test_id: str,+ blueprint: PipeImgGenBlueprint,+ expected_error_message_fragment: str,+ ):+ log.debug(f"Testing error case: {test_id}")++ with pytest.raises((StaticValidationError, PipeDefinitionError)) as exc_info:+ PipeImgGenFactory.make_from_blueprint(+ domain="test_domain",+ pipe_code=f"test_pipe_{test_id}",+ blueprint=blueprint,+ )++ error_str = str(exc_info.value)+ assert expected_error_message_fragment in error_str, (+ f"Expected fragment '{expected_error_message_fragment}' not found in error message: {error_str}"+ )diff --git a/tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/data.py b/tests/unit/pipelex/pipe_operators/pipe_llm/data.pysimilarity index 100%rename from tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/data.pyrename to tests/unit/pipelex/pipe_operators/pipe_llm/data.pydiff --git a/tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/test_pipe_llm_input.py b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.pysimilarity index 94%rename from tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/test_pipe_llm_input.pyrename to tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.pyindex 48de06393..ecfa7ebe7 100644--- a/tests/unit/pipelex/core/pipes/pipe_operator/pipe_llm/test_pipe_llm_input.py+++ b/tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_input.py@@ -4,7 +4,7 @@ from pipelex import logfrom pipelex.exceptions import StaticValidationErrorfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintfrom pipelex.pipe_operators.llm.pipe_llm_factory import PipeLLMFactory-from tests.unit.pipelex.core.pipes.pipe_operator.pipe_llm.data import PipeLLMInputTestCases+from tests.unit.pipelex.pipe_operators.pipe_llm.data import PipeLLMInputTestCasesclass TestPipeLLMValidateInputs:diff --git a/tests/unit/pipelex/core/pipes/data.py b/tests/unit/pipelex/pipe_run/data.pysimilarity index 100%rename from tests/unit/pipelex/core/pipes/data.pyrename to tests/unit/pipelex/pipe_run/data.pydiff --git a/tests/unit/pipelex/core/pipes/test_pipe_run_params.py b/tests/unit/pipelex/pipe_run/test_pipe_run_params.pysimilarity index 99%rename from tests/unit/pipelex/core/pipes/test_pipe_run_params.pyrename to tests/unit/pipelex/pipe_run/test_pipe_run_params.pyindex 43e14a3ec..a927f61cf 100644--- a/tests/unit/pipelex/core/pipes/test_pipe_run_params.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_run_params.py@@ -6,7 +6,7 @@ from pipelex.pipe_run.pipe_run_params import (make_output_multiplicity,output_multiplicity_to_apply,)-from tests.unit.pipelex.core.pipes.data import MAKE_OUTPUT_MULTIPLICITY_TEST_CASES, OUTPUT_MULTIPLICITY_TO_APPLY_TEST_CASES+from tests.unit.pipelex.pipe_run.data import MAKE_OUTPUT_MULTIPLICITY_TEST_CASES, OUTPUT_MULTIPLICITY_TO_APPLY_TEST_CASESclass TestMakeOutputMultiplicity:diff --git a/tests/unit/pipelex/tools/__init__.py b/tests/unit/pipelex/tools/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/config/__init__.py b/tests/unit/pipelex/tools/config/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/config/test_config_manager.py b/tests/unit/pipelex/tools/config/test_config_manager.pyindex 78c8c274e..1a13f1eb5 100644--- a/tests/unit/pipelex/tools/config/test_config_manager.py+++ b/tests/unit/pipelex/tools/config/test_config_manager.py@@ -1,7 +1,7 @@from pipelex import pretty_printfrom pipelex.config import PipelexConfigfrom pipelex.hub import get_pipelex_hub-from pipelex.tools.config.manager import config_manager+from pipelex.system.configuration.config_loader import config_managerclass TestConfigManager:@@ -12,4 +12,4 @@ class TestConfigManager:def test_load_pipelex_template_config(self):hub = get_pipelex_hub()- hub.setup_config(config_cls=PipelexConfig, specific_config_path="pipelex/config_template/pipelex.toml")+ hub.setup_config(config_cls=PipelexConfig, specific_config_path="pipelex/kit/configs/pipelex.toml")diff --git a/tests/unit/pipelex/tools/misc/__init__.py b/tests/unit/pipelex/tools/misc/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/misc/test_base_64_utils.py b/tests/unit/pipelex/tools/misc/test_base_64_utils.pyindex b63b9092e..779b3b5d0 100644--- a/tests/unit/pipelex/tools/misc/test_base_64_utils.py+++ b/tests/unit/pipelex/tools/misc/test_base_64_utils.py@@ -8,7 +8,7 @@ from pipelex.tools.misc.base_64_utils import (encode_to_base64_async,load_binary_as_base64,load_binary_as_base64_async,- save_base64_to_binary_file,+ save_base_64_str_to_binary_file,)from tests.cases import FileHelperTestCases@@ -53,7 +53,7 @@ class TestBase64Utils:b64 = base64.b64encode(data).decode()out_file = tmp_path / "out.bin"- save_base64_to_binary_file(b64=b64, file_path=str(out_file))+ save_base_64_str_to_binary_file(base_64_str=b64, file_path=str(out_file))with open(out_file, "rb") as f:assert f.read() == data@@ -64,7 +64,7 @@ class TestBase64Utils:data_url = f"data:application/octet-stream;base64,{b64}"out_file = tmp_path / "out.bin"- save_base64_to_binary_file(b64=data_url, file_path=str(out_file))+ save_base_64_str_to_binary_file(base_64_str=data_url, file_path=str(out_file))with open(out_file, "rb") as f:assert f.read() == datadiff --git a/tests/unit/pipelex/tools/misc/test_dict_utils.py b/tests/unit/pipelex/tools/misc/test_dict_utils.pyindex 4cc21158c..fe34164ed 100644--- a/tests/unit/pipelex/tools/misc/test_dict_utils.py+++ b/tests/unit/pipelex/tools/misc/test_dict_utils.py@@ -1,7 +1,10 @@from typing import Any+import pytest+from pipelex.core.concepts.concept_native import NativeConceptCode-from pipelex.tools.misc.dict_utils import apply_to_strings_in_list, apply_to_strings_recursive, insert_before+from pipelex.system.exceptions import NestedKeyConflictError+from pipelex.tools.misc.dict_utils import apply_to_strings_in_list, apply_to_strings_recursive, insert_before, substitute_nested_in_contextclass TestDictUtils:@@ -243,3 +246,106 @@ class TestDictUtils:assert result[2][0] == "list in list list"assert result[2][1][0]["very_deep"] == "very deep value very_deep"assert result[2][1][1] == "deep list item deep_list"++ def test_substitute_nested_in_context_basic(self) -> None:+ """Test basic nested substitution with dotted keys."""+ context: dict[str, Any] = {}+ extra_params = {"foo.bar": "hello"}++ result = substitute_nested_in_context(context, extra_params)++ assert result is context # Should mutate original+ assert context["foo"]["bar"] == "hello"++ def test_substitute_nested_in_context_deep_nesting(self) -> None:+ """Test deep nesting with multiple levels."""+ context: dict[str, Any] = {}+ extra_params = {"a.b.c.d": "deep_value"}++ substitute_nested_in_context(context, extra_params)++ assert context["a"]["b"]["c"]["d"] == "deep_value"++ def test_substitute_nested_in_context_multiple_keys(self) -> None:+ """Test multiple nested keys, some sharing prefixes."""+ context: dict[str, Any] = {}+ extra_params = {+ "foo.bar.nested": "value3",+ "foo.baz": "value2",+ "other.key": "value4",+ }++ substitute_nested_in_context(context, extra_params)++ assert context["foo"]["bar"]["nested"] == "value3"+ assert context["foo"]["baz"] == "value2"+ assert context["other"]["key"] == "value4"++ def test_substitute_nested_in_context_mixed_keys(self) -> None:+ """Test mixed keys with and without dots."""+ context: dict[str, Any] = {}+ extra_params = {+ "simple": "simple_value",+ "nested.key": "nested_value",+ "another": 42,+ }++ substitute_nested_in_context(context, extra_params)++ assert context["simple"] == "simple_value"+ assert context["nested"]["key"] == "nested_value"+ assert context["another"] == 42++ def test_substitute_nested_in_context_existing_nested(self) -> None:+ """Test extending existing nested structures."""+ context: dict[str, Any] = {"foo": {"existing": "old_value"}}+ extra_params = {"foo.bar": "new_value"}++ substitute_nested_in_context(context, extra_params)++ assert context["foo"]["existing"] == "old_value"+ assert context["foo"]["bar"] == "new_value"++ def test_substitute_nested_in_context_conflict(self) -> None:+ """Test error when trying to nest under a non-dict-like value."""+ context: dict[str, Any] = {"foo": "string_value"}+ extra_params = {"foo.bar": "new_value"}++ with pytest.raises(NestedKeyConflictError) as exc_info:+ substitute_nested_in_context(context, extra_params)++ assert "foo.bar" in exc_info.value.message+ assert "foo" in exc_info.value.message+ assert "dict-like" in exc_info.value.message++ def test_substitute_nested_in_context_none_extra_params(self) -> None:+ """Test function works when extra_params is None."""+ context: dict[str, Any] = {"existing": "value"}++ result = substitute_nested_in_context(context, None)++ assert result is context+ assert context == {"existing": "value"}++ def test_substitute_nested_in_context_empty_extra_params(self) -> None:+ """Test function works with empty extra_params."""+ context: dict[str, Any] = {"existing": "value"}+ extra_params: dict[str, Any] = {}++ result = substitute_nested_in_context(context, extra_params)++ assert result is context+ assert context == {"existing": "value"}++ def test_substitute_nested_in_context_mutation(self) -> None:+ """Test that the original context dict is mutated."""+ context: dict[str, Any] = {"existing": "value"}+ original_id = id(context)+ extra_params = {"foo.bar": "hello"}++ result = substitute_nested_in_context(context, extra_params)++ assert id(result) == original_id+ assert result is context+ assert context["foo"]["bar"] == "hello"+ assert context["existing"] == "value"diff --git a/tests/unit/pipelex/tools/secrets/__init__.py b/tests/unit/pipelex/tools/secrets/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/templating/__init__.py b/tests/unit/pipelex/tools/templating/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/test_class_registry_utils.py b/tests/unit/pipelex/tools/test_class_registry_utils.pyindex 58a01828f..846639f60 100644--- a/tests/unit/pipelex/tools/test_class_registry_utils.py+++ b/tests/unit/pipelex/tools/test_class_registry_utils.py@@ -3,7 +3,7 @@ from pathlib import Pathfrom pydantic import BaseModelfrom pytest_mock import MockerFixture-from pipelex.tools.class_registry_utils import ClassRegistryUtils+from pipelex.system.registries.class_registry_utils import ClassRegistryUtilsclass TestClassRegistryUtilsUnit:@@ -17,18 +17,18 @@ class TestClassRegistryUtilsUnit:# Mock the functions at the location where they're imported in ClassRegistryUtilsmock_import = mocker.patch(- "pipelex.tools.class_registry_utils.import_module_from_file",+ "pipelex.system.registries.class_registry_utils.import_module_from_file",return_value=mock_module,)mock_find = mocker.patch(- "pipelex.tools.class_registry_utils.find_classes_in_module",+ "pipelex.system.registries.class_registry_utils.find_classes_in_module",return_value=[str, int],)# Mock the global class registrymock_registry = mocker.MagicMock()mocker.patch(- "pipelex.tools.class_registry_utils.KajsonManager.get_class_registry",+ "pipelex.system.registries.class_registry_utils.KajsonManager.get_class_registry",return_value=mock_registry,)diff --git a/tests/unit/pipelex/tools/test_environment.py b/tests/unit/pipelex/tools/test_environment.pyindex e0f181f43..ea03ddb89 100644--- a/tests/unit/pipelex/tools/test_environment.py+++ b/tests/unit/pipelex/tools/test_environment.py@@ -3,7 +3,7 @@ import osimport pytestfrom pytest_mock import MockerFixture-from pipelex.tools.environment import (+from pipelex.system.environment import (EnvVarNotFoundError,all_env_vars_are_set,any_env_var_is_placeholder,diff --git a/tests/unit/pipelex/tools/test_func_registry.py b/tests/unit/pipelex/tools/test_func_registry.pyindex 1adabb6c2..aa6c41c4c 100644--- a/tests/unit/pipelex/tools/test_func_registry.py+++ b/tests/unit/pipelex/tools/test_func_registry.py@@ -7,7 +7,7 @@ from pytest import LogCaptureFixturefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.text_content import TextContent-from pipelex.tools.func_registry import FuncRegistry, FuncRegistryError+from pipelex.system.registries.func_registry import FuncRegistry, FuncRegistryErrordef sample_function():diff --git a/tests/unit/pipelex/tools/test_func_registry_utils.py b/tests/unit/pipelex/tools/test_func_registry_utils.pyindex 8b4880def..fa6a22721 100644--- a/tests/unit/pipelex/tools/test_func_registry_utils.py+++ b/tests/unit/pipelex/tools/test_func_registry_utils.py@@ -8,8 +8,8 @@ from pydantic import Fieldfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.structured_content import StructuredContentfrom pipelex.core.stuffs.text_content import TextContent-from pipelex.tools.func_registry import func_registry-from pipelex.tools.func_registry_utils import FuncRegistryUtils+from pipelex.system.registries.func_registry import func_registry+from pipelex.system.registries.func_registry_utils import FuncRegistryUtilsclass FilePath(StructuredContent):@@ -23,6 +23,9 @@ class CodebaseFileContent(StructuredContent):class TestCases:VALID_ASYNC_FUNCTION = """+from pipelex.system.registries.func_registry import pipe_func++@pipe_func()async def read_file_content(working_memory: WorkingMemory) -> ListContent[CodebaseFileContent]:'''Read the content of related codebase files.'''@@ -43,7 +46,10 @@ async def read_file_content(working_memory: WorkingMemory) -> ListContent[Codeba"""VALID_SYNC_FUNCTION = """+from pipelex.system.registries.func_registry import pipe_func+# Sync function - should be accepted+@pipe_func()def sync_function(working_memory: WorkingMemory) -> StructuredContent:'''This should be registered - sync functions are now eligible.'''pass@@ -172,7 +178,9 @@ class CodebaseFileContent(StructuredContent):root_file.write_text("""from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.system.registries.func_registry import pipe_func+@pipe_func()async def root_function(working_memory: WorkingMemory) -> TextContent:return TextContent(text="root")""")@@ -182,7 +190,9 @@ async def root_function(working_memory: WorkingMemory) -> TextContent:nested_file.write_text("""from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.stuffs.text_content import TextContent+from pipelex.system.registries.func_registry import pipe_func+@pipe_func()async def nested_function(working_memory: WorkingMemory) -> TextContent:return TextContent(text="nested")""")diff --git a/tests/unit/pipelex/tools/templating/test_jinja2.py b/tests/unit/pipelex/tools/test_jinja2.pysimilarity index 100%rename from tests/unit/pipelex/tools/templating/test_jinja2.pyrename to tests/unit/pipelex/tools/test_jinja2.pydiff --git a/tests/unit/pipelex/tools/test_testing.py b/tests/unit/pipelex/tools/test_testing.pyindex b3a2e62c8..139b0a430 100644--- a/tests/unit/pipelex/tools/test_testing.py+++ b/tests/unit/pipelex/tools/test_testing.py@@ -1,5 +1,5 @@-from pipelex.tools.environment import is_env_var_set-from pipelex.tools.runtime_manager import RunMode, runtime_manager+from pipelex.system.environment import is_env_var_set+from pipelex.system.runtime import RunMode, runtime_managerdef test_testing():diff --git a/tests/unit/pipelex/tools/typing/__init__.py b/tests/unit/pipelex/tools/typing/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/tests/unit/pipelex/tools/typing/test_convert_file_path_to_module_path.py b/tests/unit/pipelex/tools/typing/test_convert_file_path_to_module_path.pynew file mode 100644index 000000000..262306b07--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_convert_file_path_to_module_path.py@@ -0,0 +1,195 @@+import os+from pathlib import Path++from pipelex.tools.typing.module_inspector import (+ convert_file_path_to_module_path,+)+++class TestConvertFilePathToModulePath:+ def test_simple_relative_path(self):+ """Test conversion of a simple relative path."""+ result = convert_file_path_to_module_path("test_module.py")+ # Should convert to absolute path and replace separators with underscores+ assert result.endswith("test_module")+ assert ".py" not in result+ assert result.replace("_", "").replace(os.sep, "").isalnum()++ def test_absolute_path(self, tmp_path: Path):+ """Test conversion of an absolute path."""+ file_path = str(tmp_path / "test_file.py")+ result = convert_file_path_to_module_path(file_path)+ # Should contain the temp path components with separators replaced+ assert "test_file" in result+ assert ".py" not in result+ assert "/" not in result+ assert "\\" not in result++ def test_nested_path(self, tmp_path: Path):+ """Test conversion of a nested path."""+ file_path = str(tmp_path / "subdir" / "nested" / "module.py")+ result = convert_file_path_to_module_path(file_path)+ assert "module" in result+ assert "subdir" in result+ assert "nested" in result+ assert ".py" not in result+ assert "/" not in result+ assert "\\" not in result++ def test_removes_py_extension(self):+ """Test that .py extension is removed."""+ result = convert_file_path_to_module_path("my_module.py")+ assert not result.endswith(".py")+ assert not result.endswith("_py")++ def test_replaces_special_characters(self):+ """Test that special characters are replaced with underscores."""+ result = convert_file_path_to_module_path("my-special.module.py")+ # Hyphens and dots should be replaced with underscores+ assert "-" not in result+ # The .py extension is removed, but other dots become underscores+ assert result.count(".") == 0 or all(c.isalnum() or c == "_" for c in result)++ def test_replaces_spaces(self):+ """Test that spaces are replaced with underscores."""+ result = convert_file_path_to_module_path("my module.py")+ assert " " not in result+ assert "my" in result+ assert "module" in result++ def test_path_with_multiple_special_chars(self):+ """Test path with various special characters."""+ result = convert_file_path_to_module_path("my-file@name#test.py")+ # All special characters should be replaced with underscores+ assert "@" not in result+ assert "#" not in result+ assert "-" not in result+ assert "." not in result+ # Result should be valid Python identifier (alphanumeric + underscores)+ assert all(c.isalnum() or c == "_" for c in result)++ def test_number_at_start_adds_underscore(self):+ """Test that a path starting with a number gets underscore prefix."""+ result = convert_file_path_to_module_path("123_module.py")+ # Module names cannot start with numbers in Python+ assert result[0] == "_" or not result[0].isdigit()+ # If it was prefixed, check that the original number is still there+ if result[0] == "_":+ assert "123" in result++ def test_consistent_results_for_same_path(self):+ """Test that the same path produces the same result."""+ path = "test/module.py"+ result1 = convert_file_path_to_module_path(path)+ result2 = convert_file_path_to_module_path(path)+ assert result1 == result2++ def test_different_paths_produce_different_results(self, tmp_path: Path):+ """Test that different absolute paths produce different results."""+ path1 = str(tmp_path / "module1.py")+ path2 = str(tmp_path / "module2.py")+ result1 = convert_file_path_to_module_path(path1)+ result2 = convert_file_path_to_module_path(path2)+ assert result1 != result2++ def test_uses_absolute_path(self, tmp_path: Path):+ """Test that the function converts to absolute path."""+ # Create a temporary directory and file+ test_dir = tmp_path / "testdir"+ test_dir.mkdir()+ original_cwd = os.getcwd()+ try:+ # Change to test directory+ os.chdir(test_dir)+ # Use relative path+ result = convert_file_path_to_module_path("relative_module.py")+ # Result should contain the absolute path components+ assert str(test_dir).replace(os.sep, "_") in result or "testdir" in result+ finally:+ # Restore original directory+ os.chdir(original_cwd)++ def test_path_with_dots_in_directory(self):+ """Test path with dots in directory names."""+ result = convert_file_path_to_module_path("dir.name/module.py")+ # Dots should be replaced with underscores+ assert "." not in result+ assert all(c.isalnum() or c == "_" for c in result)++ def test_unicode_characters(self):+ """Test path with Unicode characters."""+ result = convert_file_path_to_module_path("modülé.py")+ # Unicode alphanumeric should be preserved+ assert ".py" not in result+ # Result should only contain valid identifier characters+ assert all(c.isalnum() or c == "_" for c in result)++ def test_minimal_path(self):+ """Test that even minimal paths like '.py' produce valid identifiers."""+ # Even a path like ".py" will be converted to absolute path first,+ # so it will include the current directory components+ result = convert_file_path_to_module_path(".py")+ # Should still be a valid identifier+ assert len(result) > 0+ assert result[0].isalpha() or result[0] == "_"+ assert all(c.isalnum() or c == "_" for c in result)++ def test_result_is_valid_python_identifier(self):+ """Test that the result is always a valid Python identifier."""+ test_paths = [+ "simple.py",+ "path/to/module.py",+ "my-module.py",+ "my_module_123.py",+ "123module.py",+ "special@chars#here.py",+ ]+ for path in test_paths:+ result = convert_file_path_to_module_path(path)+ # Check it's a valid identifier: starts with letter or underscore,+ # followed by letters, digits, or underscores+ assert result[0].isalpha() or result[0] == "_"+ assert all(c.isalnum() or c == "_" for c in result)++ def test_long_path(self, tmp_path: Path):+ """Test conversion of a very long path."""+ # Create a deeply nested path+ long_path = tmp_path+ for i in range(10):+ long_path = long_path / f"dir{i}"+ file_path = str(long_path / "module.py")+ result = convert_file_path_to_module_path(file_path)+ # Should contain elements from all directory levels+ assert "dir0" in result+ assert "dir9" in result+ assert "module" in result+ assert ".py" not in result++ def test_path_with_multiple_dots(self):+ """Test path with multiple dots in filename."""+ result = convert_file_path_to_module_path("my.module.test.py")+ # All dots except in .py extension should become underscores+ assert ".py" not in result+ assert "my" in result+ assert "module" in result+ assert "test" in result++ def test_windows_style_path(self):+ """Test Windows-style path with backslashes."""+ # Use a path that looks like Windows path+ result = convert_file_path_to_module_path("C:\\Users\\test\\module.py")+ # Backslashes should be replaced+ assert "\\" not in result+ assert ":" not in result # Colon from C: should be replaced+ assert all(c.isalnum() or c == "_" for c in result)++ def test_preserves_uniqueness(self, tmp_path: Path):+ """Test that different paths with similar names produce unique results."""+ path1 = str(tmp_path / "dir1" / "module.py")+ path2 = str(tmp_path / "dir2" / "module.py")+ result1 = convert_file_path_to_module_path(path1)+ result2 = convert_file_path_to_module_path(path2)+ # Even though they have the same filename, full paths should differ+ assert result1 != result2+ assert "dir1" in result1+ assert "dir2" in result2diff --git a/tests/unit/pipelex/tools/typing/test_find_class_names_in_file.py b/tests/unit/pipelex/tools/typing/test_find_class_names_in_file.pynew file mode 100644index 000000000..d2addd5aa--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_find_class_names_in_file.py@@ -0,0 +1,102 @@+from pathlib import Path++import pytest++from pipelex.tools.typing.module_inspector import (+ ModuleFileError,+ find_class_names_in_file,+)+++class TestFindClassNamesInFile:+ def test_find_all_class_names(self, tmp_path: Path):+ """Test finding all class names without filtering."""+ test_file_path = tmp_path / "test_classes.py"+ test_file_path.write_text("""+class ClassA:+ pass++class ClassB:+ pass++def some_function():+ pass+""")+ class_names = find_class_names_in_file(str(test_file_path))+ assert len(class_names) == 2+ assert "ClassA" in class_names+ assert "ClassB" in class_names++ def test_find_class_names_with_base_class_filter(self, tmp_path: Path):+ """Test finding classes that inherit from specific base classes."""+ test_file_path = tmp_path / "test_inheritance.py"+ test_file_path.write_text("""+class BaseContent:+ pass++class StructuredContent:+ pass++class MyContent(StructuredContent):+ pass++class OtherContent(BaseContent):+ pass++class UnrelatedClass:+ pass+""")+ class_names = find_class_names_in_file(+ str(test_file_path),+ base_class_names=["StructuredContent"],+ )+ assert len(class_names) == 1+ assert "MyContent" in class_names+ assert "OtherContent" not in class_names+ assert "UnrelatedClass" not in class_names++ def test_find_class_names_with_qualified_base_class(self, tmp_path: Path):+ """Test finding classes with qualified base class names."""+ test_file_path = tmp_path / "test_qualified.py"+ test_file_path.write_text("""+from pipelex.core.stuffs.structured_content import StructuredContent++class MyContent(StructuredContent):+ pass++class UnrelatedClass:+ pass+""")+ class_names = find_class_names_in_file(+ str(test_file_path),+ base_class_names=["StructuredContent"],+ )+ assert len(class_names) == 1+ assert "MyContent" in class_names++ def test_find_class_names_empty_file(self, tmp_path: Path):+ """Test with file containing no classes."""+ test_file_path = tmp_path / "test_empty.py"+ test_file_path.write_text("""+def some_function():+ pass++variable = 42+""")+ class_names = find_class_names_in_file(str(test_file_path))+ assert len(class_names) == 0++ def test_find_class_names_non_python_file_raises_error(self, tmp_path: Path):+ """Test that non-Python file raises error."""+ test_file_path = tmp_path / "test.txt"+ test_file_path.write_text("Not Python")+ with pytest.raises(ModuleFileError) as excinfo:+ find_class_names_in_file(str(test_file_path))+ assert "is not a Python file" in str(excinfo.value)++ def test_find_class_names_nonexistent_file_raises_error(self, tmp_path: Path):+ """Test that nonexistent file raises error."""+ nonexistent_file_path = tmp_path / "nonexistent.py"+ with pytest.raises(ModuleFileError) as excinfo:+ find_class_names_in_file(str(nonexistent_file_path))+ assert "does not exist" in str(excinfo.value)diff --git a/tests/unit/pipelex/tools/typing/test_module_inspector.py b/tests/unit/pipelex/tools/typing/test_find_classes_in_module.pysimilarity index 64%rename from tests/unit/pipelex/tools/typing/test_module_inspector.pyrename to tests/unit/pipelex/tools/typing/test_find_classes_in_module.pyindex ab35c63ad..ba9d9548d 100644--- a/tests/unit/pipelex/tools/typing/test_module_inspector.py+++ b/tests/unit/pipelex/tools/typing/test_find_classes_in_module.py@@ -1,104 +1,6 @@-import sysimport types-from pathlib import Path--import pytest--from pipelex.tools.typing.module_inspector import ModuleFileError, find_classes_in_module, import_module_from_file---class TestModuleFileError:- def test_exception_inheritance(self):- """Test that ModuleFileError inherits from Exception."""- error = ModuleFileError("test message")- assert isinstance(error, Exception)- assert str(error) == "test message"---class TestImportModuleFromFile:- @pytest.fixture(autouse=True)- def cleanup_sys_modules(self):- """Clean up sys.modules entries after each test."""- yield- # Clean up sys.modules entries for test modules- modules_to_remove = [name for name in sys.modules if "test_module_" in name or name == "test_module"]- for module_name in modules_to_remove:- del sys.modules[module_name]-- def test_import_valid_python_file(self, tmp_path: Path):- """Test importing a valid Python file."""- test_file_path = tmp_path / "test_module_valid.py"- test_file_path.write_text("""-def test_function():- return "test_value"--class TestClass:- pass-""")- module = import_module_from_file(str(test_file_path))- assert hasattr(module, "test_function")- assert hasattr(module, "TestClass")- assert module.test_function() == "test_value"-- def test_import_non_python_file_raises_error(self, tmp_path: Path):- """Test that importing a non-Python file raises ModuleFileError."""- test_file_path = tmp_path / "test_file.txt"- test_file_path.write_text("This is not Python code")- with pytest.raises(ModuleFileError) as excinfo:- import_module_from_file(str(test_file_path))- assert "is not a Python file" in str(excinfo.value)-- def test_import_nonexistent_file_raises_error(self, tmp_path: Path):- """Test that importing a nonexistent file raises FileNotFoundError."""- nonexistent_file_path = tmp_path / "nonexistent.py"- with pytest.raises(FileNotFoundError):- import_module_from_file(str(nonexistent_file_path))-- def test_import_file_with_syntax_error_raises_error(self, tmp_path: Path):- """Test that importing a file with syntax errors raises SyntaxError."""- test_file_path = tmp_path / "syntax_error.py"- test_file_path.write_text("""-def test_function(- return "missing closing parenthesis"-""")- with pytest.raises(SyntaxError):- import_module_from_file(str(test_file_path))-- def test_import_file_with_import_error_raises_error(self, tmp_path: Path):- """Test that importing a file with import errors raises ImportError."""- test_file_path = tmp_path / "import_error.py"- test_file_path.write_text("""-import nonexistent_module-""")- with pytest.raises(ImportError):- import_module_from_file(str(test_file_path))-- def test_module_added_to_sys_modules(self, tmp_path: Path):- """Test that imported module is added to sys.modules."""- test_file_path = tmp_path / "test_module_sys.py"- test_file_path.write_text("""-def test_function():- return "test_value"-""")- module = import_module_from_file(str(test_file_path))- # The module name is derived from the full file path- module_name = module.__name__- assert module_name.endswith("test_module_sys")- assert module_name in sys.modules- assert sys.modules[module_name] is module-- def test_module_name_with_path_separators(self, tmp_path: Path):- """Test that module name is correctly formatted with path separators (should use file basename)."""- nested_dir = tmp_path / "nested" / "subdir"- nested_dir.mkdir(parents=True)- test_file_path = nested_dir / "test_module_nested.py"- test_file_path.write_text("""-def test_function():- return "test_value"-""")- module = import_module_from_file(str(test_file_path))- assert hasattr(module, "test_function")- assert module.test_function() == "test_value"++from pipelex.tools.typing.module_inspector import find_classes_in_moduleclass TestFindClassesInModule:diff --git a/tests/unit/pipelex/tools/typing/test_import_module_from_file.py b/tests/unit/pipelex/tools/typing/test_import_module_from_file.pynew file mode 100644index 000000000..f73ad7b32--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_import_module_from_file.py@@ -0,0 +1,95 @@+import sys+from pathlib import Path++import pytest++from pipelex.tools.typing.module_inspector import (+ ModuleFileError,+ import_module_from_file,+)+++class TestImportModuleFromFile:+ @pytest.fixture(autouse=True)+ def cleanup_sys_modules(self):+ """Clean up sys.modules entries after each test."""+ yield+ # Clean up sys.modules entries for test modules+ modules_to_remove = [name for name in sys.modules if "test_module_" in name or name == "test_module"]+ for module_name in modules_to_remove:+ del sys.modules[module_name]++ def test_import_valid_python_file(self, tmp_path: Path):+ """Test importing a valid Python file."""+ test_file_path = tmp_path / "test_module_valid.py"+ test_file_path.write_text("""+def test_function():+ return "test_value"++class TestClass:+ pass+""")+ module = import_module_from_file(str(test_file_path))+ assert hasattr(module, "test_function")+ assert hasattr(module, "TestClass")+ assert module.test_function() == "test_value"++ def test_import_non_python_file_raises_error(self, tmp_path: Path):+ """Test that importing a non-Python file raises ModuleFileError."""+ test_file_path = tmp_path / "test_file.txt"+ test_file_path.write_text("This is not Python code")+ with pytest.raises(ModuleFileError) as excinfo:+ import_module_from_file(str(test_file_path))+ assert "is not a Python file" in str(excinfo.value)++ def test_import_nonexistent_file_raises_error(self, tmp_path: Path):+ """Test that importing a nonexistent file raises FileNotFoundError."""+ nonexistent_file_path = tmp_path / "nonexistent.py"+ with pytest.raises(FileNotFoundError):+ import_module_from_file(str(nonexistent_file_path))++ def test_import_file_with_syntax_error_raises_error(self, tmp_path: Path):+ """Test that importing a file with syntax errors raises SyntaxError."""+ test_file_path = tmp_path / "syntax_error.py"+ test_file_path.write_text("""+def test_function(+ return "missing closing parenthesis"+""")+ with pytest.raises(SyntaxError):+ import_module_from_file(str(test_file_path))++ def test_import_file_with_import_error_raises_error(self, tmp_path: Path):+ """Test that importing a file with import errors raises ImportError."""+ test_file_path = tmp_path / "import_error.py"+ test_file_path.write_text("""+import nonexistent_module+""")+ with pytest.raises(ImportError):+ import_module_from_file(str(test_file_path))++ def test_module_added_to_sys_modules(self, tmp_path: Path):+ """Test that imported module is added to sys.modules."""+ test_file_path = tmp_path / "test_module_sys.py"+ test_file_path.write_text("""+def test_function():+ return "test_value"+""")+ module = import_module_from_file(str(test_file_path))+ # The module name is derived from the full file path+ module_name = module.__name__+ assert module_name.endswith("test_module_sys")+ assert module_name in sys.modules+ assert sys.modules[module_name] is module++ def test_module_name_with_path_separators(self, tmp_path: Path):+ """Test that module name is correctly formatted with path separators (should use file basename)."""+ nested_dir = tmp_path / "nested" / "subdir"+ nested_dir.mkdir(parents=True)+ test_file_path = nested_dir / "test_module_nested.py"+ test_file_path.write_text("""+def test_function():+ return "test_value"+""")+ module = import_module_from_file(str(test_file_path))+ assert hasattr(module, "test_function")+ assert module.test_function() == "test_value"diff --git a/tests/unit/pipelex/tools/typing/test_import_module_from_file_if_has_classes.py b/tests/unit/pipelex/tools/typing/test_import_module_from_file_if_has_classes.pynew file mode 100644index 000000000..ec2b13e05--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_import_module_from_file_if_has_classes.py@@ -0,0 +1,80 @@+import sys+from pathlib import Path++import pytest++from pipelex.tools.typing.module_inspector import import_module_from_file_if_has_classes+++class TestImportModuleFromFileIfHasClasses:+ @pytest.fixture(autouse=True)+ def cleanup_sys_modules(self):+ """Clean up sys.modules entries after each test."""+ yield+ # Clean up sys.modules entries for test modules+ modules_to_remove = [name for name in sys.modules if "test_module_" in name or name == "test_module"]+ for module_name in modules_to_remove:+ del sys.modules[module_name]++ def test_import_file_with_matching_classes(self, tmp_path: Path):+ """Test that file with matching classes is imported."""+ test_file_path = tmp_path / "test_module_with_class.py"+ test_file_path.write_text("""+class StructuredContent:+ pass++class MyContent(StructuredContent):+ value = "imported"+""")+ module = import_module_from_file_if_has_classes(+ str(test_file_path),+ base_class_names=["StructuredContent"],+ )+ assert module is not None+ assert hasattr(module, "MyContent")+ assert module.MyContent.value == "imported"++ def test_skip_file_without_matching_classes(self, tmp_path: Path):+ """Test that file without matching classes is not imported."""+ test_file_path = tmp_path / "test_module_no_match.py"+ # Add code that would execute and cause side effects+ test_file_path.write_text("""+print("This should not execute!")++class UnrelatedClass:+ pass++def some_function():+ pass+""")+ module = import_module_from_file_if_has_classes(+ str(test_file_path),+ base_class_names=["StructuredContent"],+ )+ assert module is None+ # Verify the module was NOT loaded into sys.modules+ assert not any("test_module_no_match" in name for name in sys.modules)++ def test_import_all_files_with_classes_when_no_filter(self, tmp_path: Path):+ """Test that any file with classes is imported when no filter is provided."""+ test_file_path = tmp_path / "test_module_any_class.py"+ test_file_path.write_text("""+class AnyClass:+ value = "any_class"+""")+ module = import_module_from_file_if_has_classes(str(test_file_path))+ assert module is not None+ assert hasattr(module, "AnyClass")+ assert module.AnyClass.value == "any_class"++ def test_skip_file_with_no_classes_when_no_filter(self, tmp_path: Path):+ """Test that file with no classes is skipped even without filter."""+ test_file_path = tmp_path / "test_module_no_classes.py"+ test_file_path.write_text("""+def some_function():+ pass++variable = 42+""")+ module = import_module_from_file_if_has_classes(str(test_file_path))+ assert module is Nonediff --git a/tests/unit/pipelex/tools/typing/test_module_file_error.py b/tests/unit/pipelex/tools/typing/test_module_file_error.pynew file mode 100644index 000000000..03d72d01f--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_module_file_error.py@@ -0,0 +1,9 @@+from pipelex.tools.typing.module_inspector import ModuleFileError+++class TestModuleFileError:+ def test_exception_inheritance(self):+ """Test that ModuleFileError inherits from Exception."""+ error = ModuleFileError("test message")+ assert isinstance(error, Exception)+ assert str(error) == "test message"diff --git a/uv.lock b/uv.lockindex 6663e9aee..47ac2ecd8 100644--- a/uv.lock+++ b/uv.lock@@ -47,11 +47,11 @@ boto3 = [[[package]]name = "aiofiles"-version = "24.1.0"+version = "25.1.0"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/0b/03/a88171e277e8caa88a4c77808c20ebb04ba74cc4681bf1e9416c862de237/aiofiles-24.1.0.tar.gz", hash = "sha256:22a075c9e5a3810f0c2e48f3008c94d68c65d763b9b03857924c99e57355166c", size = 30247, upload-time = "2024-06-24T11:02:03.584Z" }+sdist = { url = "https://files.pythonhosted.org/packages/41/c3/534eac40372d8ee36ef40df62ec129bee4fdb5ad9706e58a29be53b2c970/aiofiles-25.1.0.tar.gz", hash = "sha256:a8d728f0a29de45dc521f18f07297428d56992a742f0cd2701ba86e44d23d5b2", size = 46354, upload-time = "2025-10-09T20:51:04.358Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/a5/45/30bb92d442636f570cb5651bc661f52b610e2eec3f891a5dc3a4c3667db0/aiofiles-24.1.0-py3-none-any.whl", hash = "sha256:b4ec55f4195e3eb5d7abd1bf7e061763e864dd4954231fb8539a0ef8bb8260e5", size = 15896, upload-time = "2024-06-24T11:02:01.529Z" },+ { url = "https://files.pythonhosted.org/packages/bc/8a/340a1555ae33d7354dbca4faa54948d76d89a27ceef032c8c3bc661d003e/aiofiles-25.1.0-py3-none-any.whl", hash = "sha256:abe311e527c862958650f9438e859c1fa7568a141b22abcd015e120e86a85695", size = 14668, upload-time = "2025-10-09T20:51:03.174Z" },][[package]]@@ -250,14 +250,14 @@ wheels = [[[package]]name = "astroid"-version = "3.3.11"+version = "4.0.1"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "typing-extensions", marker = "python_full_version < '3.11'" },]-sdist = { url = "https://files.pythonhosted.org/packages/18/74/dfb75f9ccd592bbedb175d4a32fc643cf569d7c218508bfbd6ea7ef9c091/astroid-3.3.11.tar.gz", hash = "sha256:1e5a5011af2920c7c67a53f65d536d65bfa7116feeaf2354d8b94f29573bb0ce", size = 400439, upload-time = "2025-07-13T18:04:23.177Z" }+sdist = { url = "https://files.pythonhosted.org/packages/a7/d1/6eee8726a863f28ff50d26c5eacb1a590f96ccbb273ce0a8c047ffb10f5a/astroid-4.0.1.tar.gz", hash = "sha256:0d778ec0def05b935e198412e62f9bcca8b3b5c39fdbe50b0ba074005e477aab", size = 405414, upload-time = "2025-10-11T15:15:42.6Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/af/0f/3b8fdc946b4d9cc8cc1e8af42c4e409468c84441b933d037e101b3d72d86/astroid-3.3.11-py3-none-any.whl", hash = "sha256:54c760ae8322ece1abd213057c4b5bba7c49818853fc901ef09719a60dbf9dec", size = 275612, upload-time = "2025-07-13T18:04:21.07Z" },+ { url = "https://files.pythonhosted.org/packages/47/f4/034361a9cbd9284ef40c8ad107955ede4efae29cbc17a059f63f6569c06a/astroid-4.0.1-py3-none-any.whl", hash = "sha256:37ab2f107d14dc173412327febf6c78d39590fdafcb44868f03b6c03452e3db0", size = 276268, upload-time = "2025-10-11T15:15:40.585Z" },][[package]]@@ -335,16 +335,16 @@ wheels = [[[package]]name = "boto3-stubs"-version = "1.40.47"+version = "1.40.51"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "botocore-stubs" },{ name = "types-s3transfer" },{ name = "typing-extensions", marker = "python_full_version < '3.12'" },]-sdist = { url = "https://files.pythonhosted.org/packages/c8/57/94842f8f5cdf79d1df73423e4e8eac63c42d7a997bc3e59d2d737aa91922/boto3_stubs-1.40.47.tar.gz", hash = "sha256:643ee24bf8b7651130e99e152380566bd34ab35d102f594f3336ba61cc9e7aec", size = 100830, upload-time = "2025-10-07T19:42:28.533Z" }+sdist = { url = "https://files.pythonhosted.org/packages/82/4d/b07f9ee0fe432fa8ec6dc368ee7a0409e2b6d9df2c5a2a88265c9b6fd878/boto3_stubs-1.40.51.tar.gz", hash = "sha256:0281e820813a310954e15fb7c1d470c24c34c1cccc7b1ddad977fa293a1080a9", size = 100890, upload-time = "2025-10-13T19:25:36.126Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/c4/1e/95ae2cdde2a9d409070533e6acf84a5dfffbe1823781f626618747a49504/boto3_stubs-1.40.47-py3-none-any.whl", hash = "sha256:77b0caa066ff407eaf7a8fced06b6a9d929f970c0b166e15dc09af17e683b3b7", size = 69688, upload-time = "2025-10-07T19:42:23.099Z" },+ { url = "https://files.pythonhosted.org/packages/d3/2e/4476431f11fc3bf7a7e0f4f5c275f17607aa127da7c0d8685a4dc6bf6291/boto3_stubs-1.40.51-py3-none-any.whl", hash = "sha256:896d0ffaa298ce1749eea1a54946320a0f4e07c6912f8e1f8c0744a708ee25a4", size = 69709, upload-time = "2025-10-13T19:25:23.116Z" },][[package]]@@ -363,23 +363,23 @@ wheels = [[[package]]name = "botocore-stubs"-version = "1.40.33"+version = "1.40.50"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "types-awscrt" },]-sdist = { url = "https://files.pythonhosted.org/packages/ae/94/16f8e1f41feaa38f1350aa5a4c60c5724b6c8524ca0e6c28523bf5070e74/botocore_stubs-1.40.33.tar.gz", hash = "sha256:89c51ae0b28d9d79fde8c497cf908ddf872ce027d2737d4d4ba473fde9cdaa82", size = 42742, upload-time = "2025-09-17T20:25:56.388Z" }+sdist = { url = "https://files.pythonhosted.org/packages/20/4b/86ad2d24ea36eed159c8e1f85a2645bfeedae34ccb8c77ea8c99abbd66d1/botocore_stubs-1.40.50.tar.gz", hash = "sha256:d772b2d3aea6b4e464963fe45b2d504eee7bc3842f047cebbae5492b3993e0fd", size = 42250, upload-time = "2025-10-11T23:08:59.925Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/af/7b/6d8fe12a955b16094460e89ea7c4e063f131f4b3bd461b96bcd625d0c79e/botocore_stubs-1.40.33-py3-none-any.whl", hash = "sha256:ad21fee32cbdc7ad4730f29baf88424c7086bf88a745f8e43660ca3e9a7e5f89", size = 66843, upload-time = "2025-09-17T20:25:54.052Z" },+ { url = "https://files.pythonhosted.org/packages/b9/c1/4a736155b2d5dd7fdd09af8fba9ed59693c565d6e2bc1b5adc769da36cb5/botocore_stubs-1.40.50-py3-none-any.whl", hash = "sha256:7cb8d636e061e600929cd03339c3bbc162c21435b4bfeb6413cf7b0b612e7de0", size = 66541, upload-time = "2025-10-11T23:08:57.678Z" },][[package]]name = "cachetools"-version = "6.2.0"+version = "6.2.1"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/9d/61/e4fad8155db4a04bfb4734c7c8ff0882f078f24294d42798b3568eb63bff/cachetools-6.2.0.tar.gz", hash = "sha256:38b328c0889450f05f5e120f56ab68c8abaf424e1275522b138ffc93253f7e32", size = 30988, upload-time = "2025-08-25T18:57:30.924Z" }+sdist = { url = "https://files.pythonhosted.org/packages/cc/7e/b975b5814bd36faf009faebe22c1072a1fa1168db34d285ef0ba071ad78c/cachetools-6.2.1.tar.gz", hash = "sha256:3f391e4bd8f8bf0931169baf7456cc822705f4e2a31f840d218f445b9a854201", size = 31325, upload-time = "2025-10-12T14:55:30.139Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/6c/56/3124f61d37a7a4e7cc96afc5492c78ba0cb551151e530b54669ddd1436ef/cachetools-6.2.0-py3-none-any.whl", hash = "sha256:1c76a8960c0041fcc21097e357f882197c79da0dbff766e7317890a65d7d8ba6", size = 11276, upload-time = "2025-08-25T18:57:29.684Z" },+ { url = "https://files.pythonhosted.org/packages/96/c5/1e741d26306c42e2bf6ab740b2202872727e0f606033c9dd713f8b93f5a8/cachetools-6.2.1-py3-none-any.whl", hash = "sha256:09868944b6dde876dfd44e1d47e18484541eaf12f26f29b7af91b26cc892d701", size = 11280, upload-time = "2025-10-12T14:55:28.382Z" },][[package]]@@ -1023,7 +1023,7 @@ wheels = [[[package]]name = "google-genai"-version = "1.41.0"+version = "1.43.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "anyio" },@@ -1035,9 +1035,9 @@ dependencies = [{ name = "typing-extensions" },{ name = "websockets" },]-sdist = { url = "https://files.pythonhosted.org/packages/72/8b/ee20bcf707769b3b0e1106c3b5c811507736af7e8a60f29a70af1750ba19/google_genai-1.41.0.tar.gz", hash = "sha256:134f861bb0ace4e34af0501ecb75ceee15f7662fd8120698cd185e8cb39f2800", size = 245812, upload-time = "2025-10-02T22:30:29.699Z" }+sdist = { url = "https://files.pythonhosted.org/packages/c1/75/992ca4462682949750709678b8efbc865222c9a16cf34504b69c5459606c/google_genai-1.43.0.tar.gz", hash = "sha256:84eb219d320759c5882bc2cdb4e2ac84544d00f5d12c7892c79fb03d71bfc9a4", size = 236132, upload-time = "2025-10-10T23:16:40.131Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/15/14/e5e8fbca8863fee718208566c4e927b8e9f45fd46ec5cf89e24759da545b/google_genai-1.41.0-py3-none-any.whl", hash = "sha256:111a3ee64c1a0927d3879faddb368234594432479a40c311e5fe4db338ca8778", size = 245931, upload-time = "2025-10-02T22:30:27.885Z" },+ { url = "https://files.pythonhosted.org/packages/61/85/e90dda488d5044e6e4cd1b49e7e7f0cc7f4a2a1c8004e88a5122d42ea024/google_genai-1.43.0-py3-none-any.whl", hash = "sha256:be1d4b1acab268125d536fd81b73c38694a70cb08266759089154718924434fd", size = 236733, upload-time = "2025-10-10T23:16:38.809Z" },][[package]]@@ -1079,20 +1079,20 @@ wheels = [[[package]]name = "httpx-sse"-version = "0.4.2"+version = "0.4.3"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/63/7a/280d644f906f077e4f4a6d327e9b6e5a936624395ad1bf6ee9165a9d9959/httpx_sse-0.4.2.tar.gz", hash = "sha256:5bb6a2771a51e6c7a5f5c645e40b8a5f57d8de708f46cb5f3868043c3c18124e", size = 16000, upload-time = "2025-10-07T08:10:05.219Z" }+sdist = { url = "https://files.pythonhosted.org/packages/0f/4c/751061ffa58615a32c31b2d82e8482be8dd4a89154f003147acee90f2be9/httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d", size = 15943, upload-time = "2025-10-10T21:48:22.271Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/4f/e5/ec31165492ecc52426370b9005e0637d6da02f9579283298affcb1ab614d/httpx_sse-0.4.2-py3-none-any.whl", hash = "sha256:a9fa4afacb293fa50ef9bacb6cae8287ba5fd1f4b1c2d10a35bb981c41da31ab", size = 9018, upload-time = "2025-10-07T08:10:04.257Z" },+ { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" },][[package]]name = "idna"-version = "3.10"+version = "3.11"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/f1/70/7703c29685631f5a7590aa73f1f1d3fa9a380e654b86af429e0934a32f7d/idna-3.10.tar.gz", hash = "sha256:12f65c9b470abda6dc35cf8e63cc574b1c52b11df2c86030af0ac09b01b13ea9", size = 190490, upload-time = "2024-09-15T18:07:39.745Z" }+sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/76/c6/c88e154df9c4e1a2a66ccf0005a88dfb2650c1dffb6f5ce603dfbd452ce3/idna-3.10-py3-none-any.whl", hash = "sha256:946d195a0d259cbba61165e88e65941f16e9b36ea6ddb97f00452bae8b1287d3", size = 70442, upload-time = "2024-09-15T18:07:37.964Z" },+ { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" },][[package]]@@ -1135,11 +1135,11 @@ google-genai = [[[package]]name = "isort"-version = "6.1.0"+version = "7.0.0"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/1e/82/fa43935523efdfcce6abbae9da7f372b627b27142c3419fcf13bf5b0c397/isort-6.1.0.tar.gz", hash = "sha256:9b8f96a14cfee0677e78e941ff62f03769a06d412aabb9e2a90487b3b7e8d481", size = 824325, upload-time = "2025-10-01T16:26:45.027Z" }+sdist = { url = "https://files.pythonhosted.org/packages/63/53/4f3c058e3bace40282876f9b553343376ee687f3c35a525dc79dbd450f88/isort-7.0.0.tar.gz", hash = "sha256:5513527951aadb3ac4292a41a16cbc50dd1642432f5e8c20057d414bdafb4187", size = 805049, upload-time = "2025-10-11T13:30:59.107Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/7f/cc/9b681a170efab4868a032631dea1e8446d8ec718a7f657b94d49d1a12643/isort-6.1.0-py3-none-any.whl", hash = "sha256:58d8927ecce74e5087aef019f778d4081a3b6c98f15a80ba35782ca8a2097784", size = 94329, upload-time = "2025-10-01T16:26:43.291Z" },+ { url = "https://files.pythonhosted.org/packages/7f/ed/e3705d6d02b4f7aea715a353c8ce193efd0b5db13e204df895d38734c244/isort-7.0.0-py3-none-any.whl", hash = "sha256:1bcabac8bc3c36c7fb7b98a76c8abb18e0f841a3ba81decac7691008592499c1", size = 94672, upload-time = "2025-10-11T13:30:57.665Z" },][[package]]@@ -2164,7 +2164,7 @@ dependencies = [{ name = "pyyaml" },{ name = "rich" },{ name = "shortuuid" },- { name = "toml" },+ { name = "tomli" },{ name = "tomlkit" },{ name = "typer" },{ name = "typing-extensions" },@@ -2195,13 +2195,10 @@ dev = [{ name = "ruff" },{ name = "types-aioboto3", extra = ["bedrock", "bedrock-runtime"] },{ name = "types-aiofiles" },- { name = "types-beautifulsoup4" },{ name = "types-markdown" },{ name = "types-networkx" },{ name = "types-openpyxl" },{ name = "types-pyyaml" },- { name = "types-requests" },- { name = "types-toml" },]docs = [{ name = "mkdocs" },@@ -2227,7 +2224,7 @@ mistralai = [requires-dist = [{ name = "aioboto3", marker = "extra == 'bedrock'", specifier = ">=13.4.0" },{ name = "aiofiles", specifier = ">=23.2.1" },- { name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.49.0" },+ { name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.60.0" },{ name = "backports-strenum", marker = "python_full_version < '3.11'", specifier = ">=1.3.0" },{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },{ name = "boto3-stubs", marker = "extra == 'dev'", specifier = ">=1.35.24" },@@ -2271,18 +2268,15 @@ requires-dist = [{ name = "rich", specifier = ">=13.8.1" },{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6.8" },{ name = "shortuuid", specifier = ">=1.0.13" },- { name = "toml", specifier = ">=0.10.2" },+ { name = "tomli", specifier = ">=2.3.0" },{ name = "tomlkit", specifier = ">=0.13.2" },{ name = "typer", specifier = ">=0.16.0" },{ name = "types-aioboto3", extras = ["bedrock", "bedrock-runtime"], marker = "extra == 'dev'", specifier = ">=13.4.0" },{ name = "types-aiofiles", marker = "extra == 'dev'", specifier = ">=24.1.0.20240626" },- { name = "types-beautifulsoup4", marker = "extra == 'dev'", specifier = ">=4.12.0.20240907" },{ name = "types-markdown", marker = "extra == 'dev'", specifier = ">=3.6.0.20240316" },{ name = "types-networkx", marker = "extra == 'dev'", specifier = ">=3.3.0.20241020" },{ name = "types-openpyxl", marker = "extra == 'dev'", specifier = ">=3.1.5.20250306" },{ name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6.0.12.20250326" },- { name = "types-requests", marker = "extra == 'dev'", specifier = ">=2.32.0.2024091" },- { name = "types-toml", marker = "extra == 'dev'", specifier = ">=0.10.8.20240310" },{ name = "typing-extensions", specifier = ">=4.13.2" },{ name = "yattag", specifier = ">=1.15.2" },]@@ -2290,11 +2284,11 @@ provides-extras = ["anthropic", "bedrock", "fal", "google", "google-genai", "mis[[package]]name = "platformdirs"-version = "4.4.0"+version = "4.5.0"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/23/e8/21db9c9987b0e728855bd57bff6984f67952bea55d6f75e055c46b5383e8/platformdirs-4.4.0.tar.gz", hash = "sha256:ca753cf4d81dc309bc67b0ea38fd15dc97bc30ce419a7f58d13eb3bf14c4febf", size = 21634, upload-time = "2025-08-26T14:32:04.268Z" }+sdist = { url = "https://files.pythonhosted.org/packages/61/33/9611380c2bdb1225fdef633e2a9610622310fed35ab11dac9620972ee088/platformdirs-4.5.0.tar.gz", hash = "sha256:70ddccdd7c99fc5942e9fc25636a8b34d04c24b335100223152c2803e4063312", size = 21632, upload-time = "2025-10-08T17:44:48.791Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/40/4b/2028861e724d3bd36227adfa20d3fd24c3fc6d52032f4a93c133be5d17ce/platformdirs-4.4.0-py3-none-any.whl", hash = "sha256:abd01743f24e5287cd7a5db3752faf1a2d65353f38ec26d98e25a6db65958c85", size = 18654, upload-time = "2025-08-26T14:32:02.735Z" },+ { url = "https://files.pythonhosted.org/packages/73/cb/ac7874b3e5d58441674fb70742e6c374b28b0c7cb988d37d991cde47166c/platformdirs-4.5.0-py3-none-any.whl", hash = "sha256:e578a81bb873cbb89a41fcc904c7ef523cc18284b7e3b3ccf06aca1403b7ebd3", size = 18651, upload-time = "2025-10-08T17:44:47.223Z" },][[package]]@@ -2321,116 +2315,116 @@ wheels = [[[package]]name = "propcache"-version = "0.4.0"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/ea/c8/d70cd26d845c6d85479d8f5a11a0fd7151e9bc4794cc5e6eb5a790f12df8/propcache-0.4.0.tar.gz", hash = "sha256:c1ad731253eb738f9cadd9fa1844e019576c70bca6a534252e97cf33a57da529", size = 45187, upload-time = "2025-10-04T21:57:39.546Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/d1/7b/4bd85fea3dc58b6f246abf0e6c9e44adca26f6817e6c136780315d723b82/propcache-0.4.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:779aaae64089e2f4992e993faea801925395d26bb5de4a47df7ef7f942c14f80", size = 79437, upload-time = "2025-10-04T21:54:49.766Z" },- { url = "https://files.pythonhosted.org/packages/e0/91/379ecc1ab37fe33648c7cb2d2252f58969adac1edcd6ec74682d7fb2d920/propcache-0.4.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:566552ed9b003030745e5bc7b402b83cf3cecae1bade95262d78543741786db5", size = 45369, upload-time = "2025-10-04T21:54:51.688Z" },- { url = "https://files.pythonhosted.org/packages/de/8e/2e002e59e359bbc6ababbb7da168226f93e0533429ea1e93989a7eedcb2a/propcache-0.4.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:944de70384c62d16d4a00c686b422aa75efbc67c4addaebefbb56475d1c16034", size = 47191, upload-time = "2025-10-04T21:54:52.915Z" },- { url = "https://files.pythonhosted.org/packages/b5/9a/f56eef9932dc3cbc63df4716f09fbaefec7a475608b643842784a01351b6/propcache-0.4.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e878553543ece1f8006d0ba4d096b40290580db173bfb18e16158045b9371335", size = 201000, upload-time = "2025-10-04T21:54:54.556Z" },- { url = "https://files.pythonhosted.org/packages/6e/84/e7ad1e09c13f0574dbad261441f6a7f1fb8cc1e2fcb23ec4d4b3e4c7dc67/propcache-0.4.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8659f995b19185179474b18de8755689e1f71e1334d05c14e1895caa4e409cf7", size = 209175, upload-time = "2025-10-04T21:54:55.996Z" },- { url = "https://files.pythonhosted.org/packages/4a/74/0b785ac0fbb44a5a7c267efc409b7a62d7a03b17c6442ecb52fd29152314/propcache-0.4.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7aa8cc5c94e682dce91cb4d12d7b81c01641f4ef5b3b3dc53325d43f0e3b9f2e", size = 214874, upload-time = "2025-10-04T21:54:57.831Z" },- { url = "https://files.pythonhosted.org/packages/b4/fd/e8d795def2b1d8dc1dc4731d36da1f6111d7c73212909e79462172d0434c/propcache-0.4.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da584d917a1a17f690fc726617fd2c3f3006ea959dae5bb07a5630f7b16f9f5f", size = 196686, upload-time = "2025-10-04T21:54:59.218Z" },- { url = "https://files.pythonhosted.org/packages/79/c2/dc992c712c3a1bfaa11d13ff177dbdf9b8b272e7bd443601e37f35728338/propcache-0.4.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:892a072e5b19c3f324a4f8543c9f7e8fc2b0aa08579e46f69bdf0cfc1b440454", size = 192000, upload-time = "2025-10-04T21:55:00.645Z" },- { url = "https://files.pythonhosted.org/packages/b3/de/bb108dbdfae594148b033ff283d9fa6e4b0906a99f2c03b98b526883149d/propcache-0.4.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:c20d796210720455086ef3f85adc413d1e41d374742f9b439354f122bbc3b528", size = 190310, upload-time = "2025-10-04T21:55:02.107Z" },- { url = "https://files.pythonhosted.org/packages/ba/7b/1bdb5d44ba4c87d270bcf11354950f8f7fbc9ace1fbe7745e683fcb57b5a/propcache-0.4.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:df7107a91126a495880576610ae989f19106e1900dd5218d08498391fa43b31d", size = 199646, upload-time = "2025-10-04T21:55:03.55Z" },- { url = "https://files.pythonhosted.org/packages/e5/04/44beda877f779f49f5b8c0ff4817a62b5f90a2dfac1ec5311df15a9dfceb/propcache-0.4.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:0b04ac2120c161416c866d0b6a4259e47e92231ff166b518cc0efb95777367c3", size = 200507, upload-time = "2025-10-04T21:55:04.914Z" },- { url = "https://files.pythonhosted.org/packages/d4/62/a13ad0a63e06f3695fcaeaeeeb62e2cc685181a1248b23a2bc877c8b7111/propcache-0.4.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:1e7fa29c71ffa8d6a37324258737d09475f84715a6e8c350f67f0bc8e5e44993", size = 192787, upload-time = "2025-10-04T21:55:06.385Z" },- { url = "https://files.pythonhosted.org/packages/9b/07/386246b3b4a6b11208bcbf57580210fb8c923ab26759389fe594e5615cd7/propcache-0.4.0-cp310-cp310-win32.whl", hash = "sha256:01c0ebc172ca28e9d62876832befbf7f36080eee6ed9c9e00243de2a8089ad57", size = 38004, upload-time = "2025-10-04T21:55:07.692Z" },- { url = "https://files.pythonhosted.org/packages/a4/f2/e1fcb9694f590bc443ae5044f982546bb01cbaa3cdf05286e9473a9874bf/propcache-0.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:84f847e64f4d1a232e50460eebc1196642ee9b4c983612f41cd2d44fd2fe7c71", size = 41516, upload-time = "2025-10-04T21:55:08.854Z" },- { url = "https://files.pythonhosted.org/packages/15/f4/d211744d41d72fbb89d3ee53963c1dc26892c49f53ae3c49fbc15cfb2548/propcache-0.4.0-cp310-cp310-win_arm64.whl", hash = "sha256:2166466a666a5bebc332cd209cad77d996fad925ca7e8a2a6310ba9e851ae641", size = 38122, upload-time = "2025-10-04T21:55:10.044Z" },- { url = "https://files.pythonhosted.org/packages/f9/c4/72b8d41bdbae8aea9c25b869d7cdc3ab5f281f979d8aea30f4646ad12743/propcache-0.4.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6a6a36b94c09711d6397d79006ca47901539fbc602c853d794c39abd6a326549", size = 80035, upload-time = "2025-10-04T21:55:11.266Z" },- { url = "https://files.pythonhosted.org/packages/e9/f8/f87115733e221408a363f3a9753419cf2d4be7a8a7ec9dc0788325cd23f1/propcache-0.4.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:da47070e1340a1639aca6b1c18fe1f1f3d8d64d3a1f9ddc67b94475f44cd40f3", size = 45622, upload-time = "2025-10-04T21:55:12.41Z" },- { url = "https://files.pythonhosted.org/packages/5d/cc/391f883248faa2efdf6886bdb12ac8edf20eac0863770d8d925450d8cc76/propcache-0.4.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:de536cf796abc5b58d11c0ad56580215d231d9554ea4bb6b8b1b3bed80aa3234", size = 47517, upload-time = "2025-10-04T21:55:13.819Z" },- { url = "https://files.pythonhosted.org/packages/3e/d2/5593b59999f42d1044c5ab5f238be1f9d537ab91b0c910727986d520a6e9/propcache-0.4.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f5c82af8e329c3cdc3e717dd3c7b2ff1a218b6de611f6ce76ee34967570a9de9", size = 214540, upload-time = "2025-10-04T21:55:15.206Z" },- { url = "https://files.pythonhosted.org/packages/bb/5d/028cdc0eaa1a66ee2ec339a08b5e6ec15e7e71dac86103bebe53ba10dc0f/propcache-0.4.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:abe04e7aa5ab2e4056fcf3255ebee2071e4a427681f76d4729519e292c46ecc1", size = 221603, upload-time = "2025-10-04T21:55:16.704Z" },- { url = "https://files.pythonhosted.org/packages/e8/f8/e30aee5f59ea21647faef9c82bd67fa510295c34908a7a38571def555881/propcache-0.4.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:075ca32384294434344760fdcb95f7833e1d7cf7c4e55f0e726358140179da35", size = 227749, upload-time = "2025-10-04T21:55:18.082Z" },- { url = "https://files.pythonhosted.org/packages/d7/85/0757dfc73931bea63b18d26b2c5e7bf13113ca60fe0e5f19905f104bcf6a/propcache-0.4.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:626ec13592928b677f48ff5861040b604b635e93d8e2162fb638397ea83d07e8", size = 209792, upload-time = "2025-10-04T21:55:19.475Z" },- { url = "https://files.pythonhosted.org/packages/d2/45/35a6a6241f46948c0ac2418d5bf50cfbcd9735739f42028a1c11e9066a72/propcache-0.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:02e071548b6a376e173b0102c3f55dc16e7d055b5307d487e844c320e38cacf2", size = 207979, upload-time = "2025-10-04T21:55:21.164Z" },- { url = "https://files.pythonhosted.org/packages/e3/d1/5930396e75c9ed477958eac1496e6fb08794d823e9b14a459f1c0e20f338/propcache-0.4.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:2af6de831a26f42a3f94592964becd8d7f238551786d7525807f02e53defbd13", size = 201923, upload-time = "2025-10-04T21:55:22.5Z" },- { url = "https://files.pythonhosted.org/packages/98/72/675455f22bcefeda16907461f9a9a4a93709ff2095e8cf799bdb6c78e030/propcache-0.4.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:bd6c6dba1a3b8949e08c4280071c86e38cb602f02e0ed6659234108c7a7cd710", size = 212117, upload-time = "2025-10-04T21:55:23.858Z" },- { url = "https://files.pythonhosted.org/packages/13/27/c533302ff80a49a848c3dbd01bb18f87b06826602b3b37043ff00d6b5005/propcache-0.4.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:783e91595cf9b66c2deda17f2e8748ae8591aa9f7c65dcab038872bfe83c5bb1", size = 216594, upload-time = "2025-10-04T21:55:25.169Z" },- { url = "https://files.pythonhosted.org/packages/63/91/8250fbb601fd16c427e5f469132f27e175c6692dbfa784ef1266dc652e55/propcache-0.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c3f4b125285d354a627eb37f3ea7c13b8842c7c0d47783581d0df0e272dbf5f0", size = 204863, upload-time = "2025-10-04T21:55:26.511Z" },- { url = "https://files.pythonhosted.org/packages/34/c4/fd945a9a25845aafb6094b9fa6a88286e4e1c55686e60172c60fe669e0d1/propcache-0.4.0-cp311-cp311-win32.whl", hash = "sha256:71c45f02ffbb8a21040ae816ceff7f6cd749ffac29fc0f9daa42dc1a9652d577", size = 37948, upload-time = "2025-10-04T21:55:27.719Z" },- { url = "https://files.pythonhosted.org/packages/42/02/f30e7304661ffe8d51ff4050e06765ac2df6d95cf23c999dfe5a0cd0eb4c/propcache-0.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:7d51f70f77950f8efafed4383865d3533eeee52d8a0dd1c35b65f24de41de4e0", size = 41511, upload-time = "2025-10-04T21:55:29.15Z" },- { url = "https://files.pythonhosted.org/packages/a5/f2/edd329d86085438a1ba32cf4cf45fc982d18343bed1f16b218b516c3340d/propcache-0.4.0-cp311-cp311-win_arm64.whl", hash = "sha256:858eaabd2191dd0da5272993ad08a748b5d3ae1aefabea8aee619b45c2af4a64", size = 37957, upload-time = "2025-10-04T21:55:30.31Z" },- { url = "https://files.pythonhosted.org/packages/b3/cf/3f88344261d69f8021256f20e82e820c5df3aba96e5ba9b5fdd3685d3a9f/propcache-0.4.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:381c84a445efb8c9168f1393a5a7c566de22edc42bfe207a142fff919b37f5d9", size = 79846, upload-time = "2025-10-04T21:55:31.447Z" },- { url = "https://files.pythonhosted.org/packages/be/fa/0286fc92764eead9dcfee639b67828daa32e61dd0f1618831547141eb28b/propcache-0.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5a531d29d7b873b12730972237c48b1a4e5980b98cf21b3f09fa4710abd3a8c3", size = 45850, upload-time = "2025-10-04T21:55:32.637Z" },- { url = "https://files.pythonhosted.org/packages/c7/83/57840656f972f8a67992eee40781e4066657776dcb889f49df0e8eecb112/propcache-0.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:cd6e22255ed73efeaaeb1765505a66a48a9ec9ebc919fce5ad490fe5e33b1555", size = 47171, upload-time = "2025-10-04T21:55:33.819Z" },- { url = "https://files.pythonhosted.org/packages/9f/8e/e0a0bd376c3440476b924eca517589ee535bb4520420d178268bf88558ba/propcache-0.4.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9a8d277dc218ddf04ec243a53ac309b1afcebe297c0526a8f82320139b56289", size = 225306, upload-time = "2025-10-04T21:55:35.312Z" },- { url = "https://files.pythonhosted.org/packages/84/fe/76884442da1bab6d4353ba1c43fdc4a770c3b3973f3ac7620a7205402fdd/propcache-0.4.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:399c73201d88c856a994916200d7cba41d7687096f8eb5139eb68f02785dc3f7", size = 230013, upload-time = "2025-10-04T21:55:37.005Z" },- { url = "https://files.pythonhosted.org/packages/f4/b7/322af273bd1136bb7e13628821fb855c9f61d64651c73fea71dded68dda5/propcache-0.4.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a1d5e474d43c238035b74ecf997f655afa67f979bae591ac838bb3fbe3076392", size = 238331, upload-time = "2025-10-04T21:55:38.713Z" },- { url = "https://files.pythonhosted.org/packages/84/5e/036d2b105927ae7f179346c9911d16c345f4dba5a19a063f23a8d28acfbd/propcache-0.4.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:22f589652ee38de96aa58dd219335604e09666092bc250c1d9c26a55bcef9932", size = 221461, upload-time = "2025-10-04T21:55:40.034Z" },- { url = "https://files.pythonhosted.org/packages/63/0d/babd038efb12a87a46ab070438c52daeac6bed0a930693a418feef8cb8a6/propcache-0.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e5227da556b2939da6125cda1d5eecf9e412e58bc97b41e2f192605c3ccbb7c2", size = 216707, upload-time = "2025-10-04T21:55:41.455Z" },- { url = "https://files.pythonhosted.org/packages/ab/68/dd075a037381581f16e7e504a6da9c1d7e415e945dd8ed67905d608f0687/propcache-0.4.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:92bc43a1ab852310721ce856f40a3a352254aa6f5e26f0fad870b31be45bba2e", size = 212591, upload-time = "2025-10-04T21:55:42.938Z" },- { url = "https://files.pythonhosted.org/packages/ff/43/22698f28fc8e04c32b109cb9cb81305a4873b77c907b17484566b6133aef/propcache-0.4.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:83ae2f5343f6f06f4c91ae530d95f56b415f768f9c401a5ee2a10459cf74370b", size = 220188, upload-time = "2025-10-04T21:55:44.53Z" },- { url = "https://files.pythonhosted.org/packages/96/7a/27886e4a4c69598a38fbeeed64f9b8ddfa6f08fe3452035845a1fe90336f/propcache-0.4.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:077a32977399dc05299b16e793210341a0b511eb0a86d1796873e83ce47334cc", size = 226736, upload-time = "2025-10-04T21:55:46.348Z" },- { url = "https://files.pythonhosted.org/packages/5b/c7/313c632b5888db3c9f4cb262420dcd5e57cf858d939d6ad9c3b1b90c12af/propcache-0.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:94a278c45e6463031b5a8278e40a07edf2bcc3b5379510e22b6c1a6e6498c194", size = 216363, upload-time = "2025-10-04T21:55:47.768Z" },- { url = "https://files.pythonhosted.org/packages/7a/5d/5aaf82bd1542aedb47d10483b84f49ee8f00d970a58e27534cd241e9c5ac/propcache-0.4.0-cp312-cp312-win32.whl", hash = "sha256:4c491462e1dc80f9deb93f428aad8d83bb286de212837f58eb48e75606e7726c", size = 37945, upload-time = "2025-10-04T21:55:49.104Z" },- { url = "https://files.pythonhosted.org/packages/4c/67/47ffff6eb176f383f56319f31c0e1bcf7500cb94ffb7582efc600c6b3c73/propcache-0.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:cdb0cecafb528ab15ed89cdfed183074d15912d046d3e304955513b50a34b907", size = 41530, upload-time = "2025-10-04T21:55:50.261Z" },- { url = "https://files.pythonhosted.org/packages/f3/7e/61b70306b9d7527286ce887a8ff28c304ab2514e5893eea36b5bdf7a21af/propcache-0.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:b2f29697d1110e8cdf7a39cc630498df0082d7898b79b731c1c863f77c6e8cfc", size = 37662, upload-time = "2025-10-04T21:55:51.35Z" },- { url = "https://files.pythonhosted.org/packages/cd/dd/f405b0fe84d29d356895bc048404d3321a2df849281cf3f932158c9346ac/propcache-0.4.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e2d01fd53e89cb3d71d20b8c225a8c70d84660f2d223afc7ed7851a4086afe6d", size = 77565, upload-time = "2025-10-04T21:55:52.907Z" },- { url = "https://files.pythonhosted.org/packages/c0/48/dfb2c45e1b0d92228c9c66fa929af7316c15cbe69a7e438786aaa60c1b3c/propcache-0.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7dfa60953169d2531dd8ae306e9c27c5d4e5efe7a2ba77049e8afdaece062937", size = 44602, upload-time = "2025-10-04T21:55:54.406Z" },- { url = "https://files.pythonhosted.org/packages/d0/d9/b15e88b4463df45a7793fb04e2b5497334f8fcc24e281c221150a0af9aff/propcache-0.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:227892597953611fce2601d49f1d1f39786a6aebc2f253c2de775407f725a3f6", size = 46168, upload-time = "2025-10-04T21:55:55.537Z" },- { url = "https://files.pythonhosted.org/packages/40/ac/983e69cce8800251aab85858069cf9359b22222a9cda47591e03e2f24eec/propcache-0.4.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e0a5bc019014531308fb67d86066d235daa7551baf2e00e1ea7b00531f6ea85", size = 207997, upload-time = "2025-10-04T21:55:57.022Z" },- { url = "https://files.pythonhosted.org/packages/ae/9c/5586a7a54e7e0b9a87fdd8ba935961f398c0e6eaecd57baaa8eca468a236/propcache-0.4.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6ebc6e2e65c31356310ddb6519420eaa6bb8c30fbd809d0919129c89dcd70f4c", size = 210948, upload-time = "2025-10-04T21:55:58.397Z" },- { url = "https://files.pythonhosted.org/packages/5f/ba/644e367f8a86461d45bd023ace521180938e76515040550af9b44085e99a/propcache-0.4.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1927b78dd75fc31a7fdc76cc7039e39f3170cb1d0d9a271e60f0566ecb25211a", size = 217988, upload-time = "2025-10-04T21:56:00.251Z" },- { url = "https://files.pythonhosted.org/packages/24/0e/1e21af74b4732d002b0452605bdf31d6bf990fd8b720cb44e27a97d80db5/propcache-0.4.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5b113feeda47f908562d9a6d0e05798ad2f83d4473c0777dafa2bc7756473218", size = 204442, upload-time = "2025-10-04T21:56:01.93Z" },- { url = "https://files.pythonhosted.org/packages/fd/30/ae2eec96995a8a760acb9a0b6c92b9815f1fc885c7d8481237ccb554eab0/propcache-0.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4596c12aa7e3bb2abf158ea8f79eb0fb4851606695d04ab846b2bb386f5690a1", size = 199371, upload-time = "2025-10-04T21:56:03.25Z" },- { url = "https://files.pythonhosted.org/packages/45/1d/a18fac8cb04f8379ccb79cf15aac31f4167a270d1cd1111f33c0d38ce4fb/propcache-0.4.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:6d1f67dad8cc36e8abc2207a77f3f952ac80be7404177830a7af4635a34cbc16", size = 196638, upload-time = "2025-10-04T21:56:04.619Z" },- { url = "https://files.pythonhosted.org/packages/48/45/3549a2b6f74dce6f21b2664d078bd26ceb876aae9c58f3c017cf590f0ee3/propcache-0.4.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:e6229ad15366cd8b6d6b4185c55dd48debf9ca546f91416ba2e5921ad6e210a6", size = 203651, upload-time = "2025-10-04T21:56:06.153Z" },- { url = "https://files.pythonhosted.org/packages/7d/f0/90ea14d518c919fc154332742a9302db3004af4f1d3df688676959733283/propcache-0.4.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:2a4bf309d057327f1f227a22ac6baf34a66f9af75e08c613e47c4d775b06d6c7", size = 205726, upload-time = "2025-10-04T21:56:07.955Z" },- { url = "https://files.pythonhosted.org/packages/f6/de/8efc1dbafeb42108e7af744822cdca944b990869e9da70e79efb21569d6b/propcache-0.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:c2e274f3d1cbb2ddcc7a55ce3739af0f8510edc68a7f37981b2258fa1eedc833", size = 199576, upload-time = "2025-10-04T21:56:09.43Z" },- { url = "https://files.pythonhosted.org/packages/d7/38/4d79fe3477b050398fb8d8f59301ed116d8c6ea3c4dbf09498c679103f90/propcache-0.4.0-cp313-cp313-win32.whl", hash = "sha256:f114a3e1f8034e2957d34043b7a317a8a05d97dfe8fddb36d9a2252c0117dbbc", size = 37474, upload-time = "2025-10-04T21:56:10.74Z" },- { url = "https://files.pythonhosted.org/packages/36/9b/a283daf665a1945cff1b03d1104e7c9ee92bb7b6bbcc6518b24fcdac8bd0/propcache-0.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:9ba68c57cde9c667f6b65b98bc342dfa7240b1272ffb2c24b32172ee61b6d281", size = 40685, upload-time = "2025-10-04T21:56:11.896Z" },- { url = "https://files.pythonhosted.org/packages/e9/f7/def8fc0b4d7a89f1628f337cb122bb9a946c5ed97760f2442b27b7fa5a69/propcache-0.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:eb77a85253174bf73e52c968b689d64be62d71e8ac33cabef4ca77b03fb4ef92", size = 37046, upload-time = "2025-10-04T21:56:13.021Z" },- { url = "https://files.pythonhosted.org/packages/ca/6b/f6e8b36b58d17dfb6c505b9ae1163fcf7a4cf98825032fdc77bba4ab5c4a/propcache-0.4.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c0e1c218fff95a66ad9f2f83ad41a67cf4d0a3f527efe820f57bde5fda616de4", size = 81274, upload-time = "2025-10-04T21:56:14.206Z" },- { url = "https://files.pythonhosted.org/packages/8e/c5/1fd0baa222b8faf53ba04dd4f34de33ea820b80e34f87c7960666bae5f4f/propcache-0.4.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:5710b1c01472542bb024366803812ca13e8774d21381bcfc1f7ae738eeb38acc", size = 46232, upload-time = "2025-10-04T21:56:15.337Z" },- { url = "https://files.pythonhosted.org/packages/cb/6b/7aa5324983cab7666ed58fc32c68a0430468a18e02e3f04e7a879c002414/propcache-0.4.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d7f008799682e8826ce98f25e8bc43532d2cd26c187a1462499fa8d123ae054f", size = 48239, upload-time = "2025-10-04T21:56:16.768Z" },- { url = "https://files.pythonhosted.org/packages/24/0f/58c192301c0436762ed5fed5a3edadb0ae399cb73528fb9c1b5cb8e53523/propcache-0.4.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0596d2ae99d74ca436553eb9ce11fe4163dc742fcf8724ebe07d7cb0db679bb1", size = 275804, upload-time = "2025-10-04T21:56:18.066Z" },- { url = "https://files.pythonhosted.org/packages/f7/b9/092ee32064ebfabedae4251952787e63e551075af1a1205e8061b3ed5838/propcache-0.4.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab9c1bd95ebd1689f0e24f2946c495808777e9e8df7bb3c1dfe3e9eb7f47fe0d", size = 273996, upload-time = "2025-10-04T21:56:19.801Z" },- { url = "https://files.pythonhosted.org/packages/43/82/becf618ed28e732f3bba3df172cd290a1afbd99f291074f747fd5bd031bb/propcache-0.4.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a8ef2ea819549ae2e8698d2ec229ae948d7272feea1cb2878289f767b6c585a4", size = 280266, upload-time = "2025-10-04T21:56:21.136Z" },- { url = "https://files.pythonhosted.org/packages/51/be/b370930249a9332a81b5c4c550dac614b7e11b6c160080777e903d57e197/propcache-0.4.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:71a400b2f0b079438cc24f9a27f02eff24d8ef78f2943f949abc518b844ade3d", size = 263186, upload-time = "2025-10-04T21:56:22.787Z" },- { url = "https://files.pythonhosted.org/packages/33/b6/546fd3e31770aed3aed1c01b120944c689edb510aeb7a25472edc472ce23/propcache-0.4.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4c2735d3305e6cecab6e53546909edf407ad3da5b9eeaf483f4cf80142bb21be", size = 260721, upload-time = "2025-10-04T21:56:24.22Z" },- { url = "https://files.pythonhosted.org/packages/80/70/3751930d16e5984490c73ca65b80777e4b26e7a0015f2d41f31d75959a71/propcache-0.4.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:72b51340047ac43b3cf388eebd362d052632260c9f73a50882edbb66e589fd44", size = 247516, upload-time = "2025-10-04T21:56:25.577Z" },- { url = "https://files.pythonhosted.org/packages/59/90/4bc96ce6476f67e2e6b72469f328c92b53259a0e4d1d5386d71a36e9258c/propcache-0.4.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:184c779363740d6664982ad05699f378f7694220e2041996f12b7c2a4acdcad0", size = 262675, upload-time = "2025-10-04T21:56:27.065Z" },- { url = "https://files.pythonhosted.org/packages/6f/d1/f16d096869c5f1c93d67fc37488c0c814add0560574f6877653a10239cde/propcache-0.4.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:a60634a9de41f363923c6adfb83105d39e49f7a3058511563ed3de6748661af6", size = 263379, upload-time = "2025-10-04T21:56:28.517Z" },- { url = "https://files.pythonhosted.org/packages/ab/2a/da5cd1bc1c6412939c457ea65bbe7e034045c395d98ff8ff880d06ec4553/propcache-0.4.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c9b8119244d122241a9c4566bce49bb20408a6827044155856735cf14189a7da", size = 257694, upload-time = "2025-10-04T21:56:30.051Z" },- { url = "https://files.pythonhosted.org/packages/a5/11/938e67c07189b662a6c72551d48285a02496de885408392447c25657dd47/propcache-0.4.0-cp313-cp313t-win32.whl", hash = "sha256:515b610a364c8cdd2b72c734cc97dece85c416892ea8d5c305624ac8734e81db", size = 41321, upload-time = "2025-10-04T21:56:31.406Z" },- { url = "https://files.pythonhosted.org/packages/f4/6e/72b11a4dcae68c728b15126cc5bc830bf275c84836da2633412b768d07e0/propcache-0.4.0-cp313-cp313t-win_amd64.whl", hash = "sha256:7ea86eb32e74f9902df57e8608e8ac66f1e1e1d24d1ed2ddeb849888413b924d", size = 44846, upload-time = "2025-10-04T21:56:32.5Z" },- { url = "https://files.pythonhosted.org/packages/94/09/0ef3c025e0621e703ef71b69e0085181a3124bcc1beef29e0ffef59ed7f4/propcache-0.4.0-cp313-cp313t-win_arm64.whl", hash = "sha256:c1443fa4bb306461a3a8a52b7de0932a2515b100ecb0ebc630cc3f87d451e0a9", size = 39689, upload-time = "2025-10-04T21:56:33.686Z" },- { url = "https://files.pythonhosted.org/packages/60/89/7699d8e9f8c222bbef1fae26afd72d448353f164a52125d5f87dd9fec2c7/propcache-0.4.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:de8e310d24b5a61de08812dd70d5234da1458d41b059038ee7895a9e4c8cae79", size = 77977, upload-time = "2025-10-04T21:56:34.836Z" },- { url = "https://files.pythonhosted.org/packages/77/c5/2758a498199ce46d6d500ba4391a8594df35400cc85738aa9f0c9b8366db/propcache-0.4.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:55a54de5266bc44aa274915cdf388584fa052db8748a869e5500ab5993bac3f4", size = 44715, upload-time = "2025-10-04T21:56:36.075Z" },- { url = "https://files.pythonhosted.org/packages/0d/da/5a44e10282a28c2dd576e5e1a2c7bb8145587070ddab7375fb643f7129d7/propcache-0.4.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:88d50d662c917ec2c9d3858920aa7b9d5bfb74ab9c51424b775ccbe683cb1b4e", size = 46463, upload-time = "2025-10-04T21:56:37.227Z" },- { url = "https://files.pythonhosted.org/packages/d5/5a/b2c314f655f46c10c204dc0d69e19fadfb1cc4d40ab33f403698a35c3281/propcache-0.4.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ae3adf88a66f5863cf79394bc359da523bb27a2ed6ba9898525a6a02b723bfc5", size = 206980, upload-time = "2025-10-04T21:56:38.828Z" },- { url = "https://files.pythonhosted.org/packages/7c/4e/f6643ec2cd5527b92c93488f9b67a170494736bb1c5460136399d709ce5a/propcache-0.4.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7f088e21d15b3abdb9047e4b7b7a0acd79bf166893ac2b34a72ab1062feb219e", size = 211385, upload-time = "2025-10-04T21:56:40.2Z" },- { url = "https://files.pythonhosted.org/packages/71/41/362766a346c3f8d3bbeb7899e1ff40f18844e0fe37e9f6f536553cf6b6be/propcache-0.4.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a4efbaf10793fd574c76a5732c75452f19d93df6e0f758c67dd60552ebd8614b", size = 215315, upload-time = "2025-10-04T21:56:41.574Z" },- { url = "https://files.pythonhosted.org/packages/ff/98/17385d51816d56fa6acc035d8625fbf833b6a795d7ef7fb37ea3f62db6c9/propcache-0.4.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:681a168d06284602d56e97f09978057aa88bcc4177352b875b3d781df4efd4cb", size = 201416, upload-time = "2025-10-04T21:56:42.947Z" },- { url = "https://files.pythonhosted.org/packages/7a/83/801178ca1c29e217564ee507ff2a49d3f24a4dd85c9b9d681fd1d62b15f2/propcache-0.4.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a7f06f077fc4ef37e8a37ca6bbb491b29e29db9fb28e29cf3896aad10dbd4137", size = 197726, upload-time = "2025-10-04T21:56:44.313Z" },- { url = "https://files.pythonhosted.org/packages/d2/38/c8743917bca92b7e5474366b6b04c7b3982deac32a0fe4b705f2e92c09bb/propcache-0.4.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:082a643479f49a6778dcd68a80262fc324b14fd8e9b1a5380331fe41adde1738", size = 192819, upload-time = "2025-10-04T21:56:45.702Z" },- { url = "https://files.pythonhosted.org/packages/0b/74/3de3ef483e8615aaaf62026fcdcb20cbfc4535ea14871b12f72d52c1d6dc/propcache-0.4.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:26692850120241a99bb4a4eec675cd7b4fdc431144f0d15ef69f7f8599f6165f", size = 202492, upload-time = "2025-10-04T21:56:47.388Z" },- { url = "https://files.pythonhosted.org/packages/46/86/a130dd85199d651a6986ba6bf1ce297b7bbcafc01c8e139e6ba2b8218a20/propcache-0.4.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:33ad7d37b9a386f97582f5d042cc7b8d4b3591bb384cf50866b749a17e4dba90", size = 204106, upload-time = "2025-10-04T21:56:49.139Z" },- { url = "https://files.pythonhosted.org/packages/b2/f7/44eab58659d71d21995146c94139e63882bac280065b3a9ed10376897bcc/propcache-0.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1e7fd82d4a5b7583588f103b0771e43948532f1292105f13ee6f3b300933c4ca", size = 198043, upload-time = "2025-10-04T21:56:50.561Z" },- { url = "https://files.pythonhosted.org/packages/96/14/df37be1bf1423d2dda201a4cdb1c5cb44048d34e31a97df227cc25b0a55c/propcache-0.4.0-cp314-cp314-win32.whl", hash = "sha256:213eb0d3bc695a70cffffe11a1c2e1c2698d89ffd8dba35a49bc44a035d45c93", size = 38036, upload-time = "2025-10-04T21:56:51.868Z" },- { url = "https://files.pythonhosted.org/packages/99/96/9cea65d6c50224737e80c57a3f3db4ca81bc7b1b52bc73346df8c50db400/propcache-0.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:087e2d3d7613e1b59b2ffca0daabd500c1a032d189c65625ee05ea114afcad0b", size = 41156, upload-time = "2025-10-04T21:56:53.242Z" },- { url = "https://files.pythonhosted.org/packages/52/4d/91523dcbe23cc127b097623a6ba177da51fca6b7c979082aa49745b527b7/propcache-0.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:94b0f7407d18001dbdcbb239512e753b1b36725a6e08a4983be1c948f5435f79", size = 37976, upload-time = "2025-10-04T21:56:54.351Z" },- { url = "https://files.pythonhosted.org/packages/ec/f7/7118a944cb6cdb548c9333cf311bda120f9793ecca54b2ca4a3f7e58723e/propcache-0.4.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:b730048ae8b875e2c0af1a09ca31b303fc7b5ed27652beec03fa22b29545aec9", size = 81270, upload-time = "2025-10-04T21:56:55.516Z" },- { url = "https://files.pythonhosted.org/packages/ab/f9/04a8bc9977ea201783f3ccb04106f44697f635f70439a208852d4d08554d/propcache-0.4.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:f495007ada16a4e16312b502636fafff42a9003adf1d4fb7541e0a0870bc056f", size = 46224, upload-time = "2025-10-04T21:56:56.695Z" },- { url = "https://files.pythonhosted.org/packages/0f/3d/808b074034156f130a0047304d811a5a5df3bb0976c9adfb9383718fd888/propcache-0.4.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:659a0ea6d9017558ed7af00fb4028186f64d0ba9adfc70a4d2c85fcd3d026321", size = 48246, upload-time = "2025-10-04T21:56:57.926Z" },- { url = "https://files.pythonhosted.org/packages/66/eb/e311f3a59ddc93078cb079b12699af9fd844142c4b4d382b386ee071d921/propcache-0.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d74aa60b1ec076d4d5dcde27c9a535fc0ebb12613f599681c438ca3daa68acac", size = 275562, upload-time = "2025-10-04T21:56:59.221Z" },- { url = "https://files.pythonhosted.org/packages/f4/05/a146094d6a00bb2f2036dd2a2f4c2b2733ff9574b59ce53bd8513edfca5d/propcache-0.4.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:34000e31795bdcda9826e0e70e783847a42e3dcd0d6416c5d3cb717905ebaec0", size = 273627, upload-time = "2025-10-04T21:57:00.582Z" },- { url = "https://files.pythonhosted.org/packages/91/95/a6d138f6e3d5f6c9b34dbd336b964a1293f2f1a79cafbe70ae3403d7cc46/propcache-0.4.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bcb5bfac5b9635e6fc520c8af6efc7a0a56f12a1fe9e9d3eb4328537e316dd6a", size = 279778, upload-time = "2025-10-04T21:57:01.944Z" },- { url = "https://files.pythonhosted.org/packages/ac/09/19594a20da0519bfa00deef8cf35dda6c9a5b51bba947f366e85ea59b3de/propcache-0.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0ea11fceb31fa95b0fa2007037f19e922e2caceb7dc6c6cac4cb56e2d291f1a2", size = 262833, upload-time = "2025-10-04T21:57:03.326Z" },- { url = "https://files.pythonhosted.org/packages/b5/92/60d2ddc7662f7b2720d3b628ad8ce888015f4ab5c335b7b1b50183194e68/propcache-0.4.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:cd8684f628fe285ea5c86f88e1c30716239dc9d6ac55e7851a4b7f555b628da3", size = 260456, upload-time = "2025-10-04T21:57:05.159Z" },- { url = "https://files.pythonhosted.org/packages/6f/e2/4c2e25c77cf43add2e05a86c4fcf51107edc4d92318e5c593bbdc2515d57/propcache-0.4.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:790286d3d542c0ef9f6d0280d1049378e5e776dcba780d169298f664c39394db", size = 247284, upload-time = "2025-10-04T21:57:06.566Z" },- { url = "https://files.pythonhosted.org/packages/dc/3e/c273ab8edc80683ec8b15b486e95c03096ef875d99e4b0ab0a36c1e42c94/propcache-0.4.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:009093c9b5dbae114a5958e6a649f8a5d94dd6866b0f82b60395eb92c58002d4", size = 262368, upload-time = "2025-10-04T21:57:08.231Z" },- { url = "https://files.pythonhosted.org/packages/ac/a9/3fa231f65a9f78614c5aafa9cee788d7f55c22187cc2f33e86c7c16d0262/propcache-0.4.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:728d98179e92d77096937fdfecd2c555a3d613abe56c9909165c24196a3b5012", size = 263010, upload-time = "2025-10-04T21:57:09.641Z" },- { url = "https://files.pythonhosted.org/packages/38/a0/f4f5d368e60c9dc04d3158eaf1ca0ad899b40ac3d29c015bf62735225a6f/propcache-0.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a9725d96a81e17e48a0fe82d0c3de2f5e623d7163fec70a6c7df90753edd1bec", size = 257298, upload-time = "2025-10-04T21:57:11.125Z" },- { url = "https://files.pythonhosted.org/packages/c7/30/f78d6758dc36a98f1cddc39b3185cefde616cc58248715b7c65495491cb1/propcache-0.4.0-cp314-cp314t-win32.whl", hash = "sha256:0964c55c95625193defeb4fd85f8f28a9a754ed012cab71127d10e3dc66b1373", size = 42484, upload-time = "2025-10-04T21:57:12.652Z" },- { url = "https://files.pythonhosted.org/packages/4e/ad/de0640e9b56d2caa796c4266d7d1e6cc4544cc327c25b7ced5c59893b625/propcache-0.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:24403152e41abf09488d3ae9c0c3bf7ff93e2fb12b435390718f21810353db28", size = 46229, upload-time = "2025-10-04T21:57:14.034Z" },- { url = "https://files.pythonhosted.org/packages/da/bf/5aed62dddbf2bbe62a3564677436261909c9dd63a0fa1fb6cf0629daa13c/propcache-0.4.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0363a696a9f24b37a04ed5e34c2e07ccbe92798c998d37729551120a1bb744c4", size = 40329, upload-time = "2025-10-04T21:57:15.198Z" },- { url = "https://files.pythonhosted.org/packages/c7/16/794c114f6041bbe2de23eb418ef58a0f45de27224d5540f5dbb266a73d72/propcache-0.4.0-py3-none-any.whl", hash = "sha256:015b2ca2f98ea9e08ac06eecc409d5d988f78c5fd5821b2ad42bc9afcd6b1557", size = 13183, upload-time = "2025-10-04T21:57:38.054Z" },+version = "0.4.1"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/9e/da/e9fc233cf63743258bff22b3dfa7ea5baef7b5bc324af47a0ad89b8ffc6f/propcache-0.4.1.tar.gz", hash = "sha256:f48107a8c637e80362555f37ecf49abe20370e557cc4ab374f04ec4423c97c3d", size = 46442, upload-time = "2025-10-08T19:49:02.291Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/3c/0e/934b541323035566a9af292dba85a195f7b78179114f2c6ebb24551118a9/propcache-0.4.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:7c2d1fa3201efaf55d730400d945b5b3ab6e672e100ba0f9a409d950ab25d7db", size = 79534, upload-time = "2025-10-08T19:46:02.083Z" },+ { url = "https://files.pythonhosted.org/packages/a1/6b/db0d03d96726d995dc7171286c6ba9d8d14251f37433890f88368951a44e/propcache-0.4.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1eb2994229cc8ce7fe9b3db88f5465f5fd8651672840b2e426b88cdb1a30aac8", size = 45526, upload-time = "2025-10-08T19:46:03.884Z" },+ { url = "https://files.pythonhosted.org/packages/e4/c3/82728404aea669e1600f304f2609cde9e665c18df5a11cdd57ed73c1dceb/propcache-0.4.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:66c1f011f45a3b33d7bcb22daed4b29c0c9e2224758b6be00686731e1b46f925", size = 47263, upload-time = "2025-10-08T19:46:05.405Z" },+ { url = "https://files.pythonhosted.org/packages/df/1b/39313ddad2bf9187a1432654c38249bab4562ef535ef07f5eb6eb04d0b1b/propcache-0.4.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9a52009f2adffe195d0b605c25ec929d26b36ef986ba85244891dee3b294df21", size = 201012, upload-time = "2025-10-08T19:46:07.165Z" },+ { url = "https://files.pythonhosted.org/packages/5b/01/f1d0b57d136f294a142acf97f4ed58c8e5b974c21e543000968357115011/propcache-0.4.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5d4e2366a9c7b837555cf02fb9be2e3167d333aff716332ef1b7c3a142ec40c5", size = 209491, upload-time = "2025-10-08T19:46:08.909Z" },+ { url = "https://files.pythonhosted.org/packages/a1/c8/038d909c61c5bb039070b3fb02ad5cccdb1dde0d714792e251cdb17c9c05/propcache-0.4.1-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9d2b6caef873b4f09e26ea7e33d65f42b944837563a47a94719cc3544319a0db", size = 215319, upload-time = "2025-10-08T19:46:10.7Z" },+ { url = "https://files.pythonhosted.org/packages/08/57/8c87e93142b2c1fa2408e45695205a7ba05fb5db458c0bf5c06ba0e09ea6/propcache-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2b16ec437a8c8a965ecf95739448dd938b5c7f56e67ea009f4300d8df05f32b7", size = 196856, upload-time = "2025-10-08T19:46:12.003Z" },+ { url = "https://files.pythonhosted.org/packages/42/df/5615fec76aa561987a534759b3686008a288e73107faa49a8ae5795a9f7a/propcache-0.4.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:296f4c8ed03ca7476813fe666c9ea97869a8d7aec972618671b33a38a5182ef4", size = 193241, upload-time = "2025-10-08T19:46:13.495Z" },+ { url = "https://files.pythonhosted.org/packages/d5/21/62949eb3a7a54afe8327011c90aca7e03547787a88fb8bd9726806482fea/propcache-0.4.1-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:1f0978529a418ebd1f49dad413a2b68af33f85d5c5ca5c6ca2a3bed375a7ac60", size = 190552, upload-time = "2025-10-08T19:46:14.938Z" },+ { url = "https://files.pythonhosted.org/packages/30/ee/ab4d727dd70806e5b4de96a798ae7ac6e4d42516f030ee60522474b6b332/propcache-0.4.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:fd138803047fb4c062b1c1dd95462f5209456bfab55c734458f15d11da288f8f", size = 200113, upload-time = "2025-10-08T19:46:16.695Z" },+ { url = "https://files.pythonhosted.org/packages/8a/0b/38b46208e6711b016aa8966a3ac793eee0d05c7159d8342aa27fc0bc365e/propcache-0.4.1-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:8c9b3cbe4584636d72ff556d9036e0c9317fa27b3ac1f0f558e7e84d1c9c5900", size = 200778, upload-time = "2025-10-08T19:46:18.023Z" },+ { url = "https://files.pythonhosted.org/packages/cf/81/5abec54355ed344476bee711e9f04815d4b00a311ab0535599204eecc257/propcache-0.4.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f93243fdc5657247533273ac4f86ae106cc6445a0efacb9a1bfe982fcfefd90c", size = 193047, upload-time = "2025-10-08T19:46:19.449Z" },+ { url = "https://files.pythonhosted.org/packages/ec/b6/1f237c04e32063cb034acd5f6ef34ef3a394f75502e72703545631ab1ef6/propcache-0.4.1-cp310-cp310-win32.whl", hash = "sha256:a0ee98db9c5f80785b266eb805016e36058ac72c51a064040f2bc43b61101cdb", size = 38093, upload-time = "2025-10-08T19:46:20.643Z" },+ { url = "https://files.pythonhosted.org/packages/a6/67/354aac4e0603a15f76439caf0427781bcd6797f370377f75a642133bc954/propcache-0.4.1-cp310-cp310-win_amd64.whl", hash = "sha256:1cdb7988c4e5ac7f6d175a28a9aa0c94cb6f2ebe52756a3c0cda98d2809a9e37", size = 41638, upload-time = "2025-10-08T19:46:21.935Z" },+ { url = "https://files.pythonhosted.org/packages/e0/e1/74e55b9fd1a4c209ff1a9a824bf6c8b3d1fc5a1ac3eabe23462637466785/propcache-0.4.1-cp310-cp310-win_arm64.whl", hash = "sha256:d82ad62b19645419fe79dd63b3f9253e15b30e955c0170e5cebc350c1844e581", size = 38229, upload-time = "2025-10-08T19:46:23.368Z" },+ { url = "https://files.pythonhosted.org/packages/8c/d4/4e2c9aaf7ac2242b9358f98dccd8f90f2605402f5afeff6c578682c2c491/propcache-0.4.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:60a8fda9644b7dfd5dece8c61d8a85e271cb958075bfc4e01083c148b61a7caf", size = 80208, upload-time = "2025-10-08T19:46:24.597Z" },+ { url = "https://files.pythonhosted.org/packages/c2/21/d7b68e911f9c8e18e4ae43bdbc1e1e9bbd971f8866eb81608947b6f585ff/propcache-0.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c30b53e7e6bda1d547cabb47c825f3843a0a1a42b0496087bb58d8fedf9f41b5", size = 45777, upload-time = "2025-10-08T19:46:25.733Z" },+ { url = "https://files.pythonhosted.org/packages/d3/1d/11605e99ac8ea9435651ee71ab4cb4bf03f0949586246476a25aadfec54a/propcache-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:6918ecbd897443087a3b7cd978d56546a812517dcaaca51b49526720571fa93e", size = 47647, upload-time = "2025-10-08T19:46:27.304Z" },+ { url = "https://files.pythonhosted.org/packages/58/1a/3c62c127a8466c9c843bccb503d40a273e5cc69838805f322e2826509e0d/propcache-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3d902a36df4e5989763425a8ab9e98cd8ad5c52c823b34ee7ef307fd50582566", size = 214929, upload-time = "2025-10-08T19:46:28.62Z" },+ { url = "https://files.pythonhosted.org/packages/56/b9/8fa98f850960b367c4b8fe0592e7fc341daa7a9462e925228f10a60cf74f/propcache-0.4.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a9695397f85973bb40427dedddf70d8dc4a44b22f1650dd4af9eedf443d45165", size = 221778, upload-time = "2025-10-08T19:46:30.358Z" },+ { url = "https://files.pythonhosted.org/packages/46/a6/0ab4f660eb59649d14b3d3d65c439421cf2f87fe5dd68591cbe3c1e78a89/propcache-0.4.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2bb07ffd7eaad486576430c89f9b215f9e4be68c4866a96e97db9e97fead85dc", size = 228144, upload-time = "2025-10-08T19:46:32.607Z" },+ { url = "https://files.pythonhosted.org/packages/52/6a/57f43e054fb3d3a56ac9fc532bc684fc6169a26c75c353e65425b3e56eef/propcache-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fd6f30fdcf9ae2a70abd34da54f18da086160e4d7d9251f81f3da0ff84fc5a48", size = 210030, upload-time = "2025-10-08T19:46:33.969Z" },+ { url = "https://files.pythonhosted.org/packages/40/e2/27e6feebb5f6b8408fa29f5efbb765cd54c153ac77314d27e457a3e993b7/propcache-0.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:fc38cba02d1acba4e2869eef1a57a43dfbd3d49a59bf90dda7444ec2be6a5570", size = 208252, upload-time = "2025-10-08T19:46:35.309Z" },+ { url = "https://files.pythonhosted.org/packages/9e/f8/91c27b22ccda1dbc7967f921c42825564fa5336a01ecd72eb78a9f4f53c2/propcache-0.4.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:67fad6162281e80e882fb3ec355398cf72864a54069d060321f6cd0ade95fe85", size = 202064, upload-time = "2025-10-08T19:46:36.993Z" },+ { url = "https://files.pythonhosted.org/packages/f2/26/7f00bd6bd1adba5aafe5f4a66390f243acab58eab24ff1a08bebb2ef9d40/propcache-0.4.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f10207adf04d08bec185bae14d9606a1444715bc99180f9331c9c02093e1959e", size = 212429, upload-time = "2025-10-08T19:46:38.398Z" },+ { url = "https://files.pythonhosted.org/packages/84/89/fd108ba7815c1117ddca79c228f3f8a15fc82a73bca8b142eb5de13b2785/propcache-0.4.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:e9b0d8d0845bbc4cfcdcbcdbf5086886bc8157aa963c31c777ceff7846c77757", size = 216727, upload-time = "2025-10-08T19:46:39.732Z" },+ { url = "https://files.pythonhosted.org/packages/79/37/3ec3f7e3173e73f1d600495d8b545b53802cbf35506e5732dd8578db3724/propcache-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:981333cb2f4c1896a12f4ab92a9cc8f09ea664e9b7dbdc4eff74627af3a11c0f", size = 205097, upload-time = "2025-10-08T19:46:41.025Z" },+ { url = "https://files.pythonhosted.org/packages/61/b0/b2631c19793f869d35f47d5a3a56fb19e9160d3c119f15ac7344fc3ccae7/propcache-0.4.1-cp311-cp311-win32.whl", hash = "sha256:f1d2f90aeec838a52f1c1a32fe9a619fefd5e411721a9117fbf82aea638fe8a1", size = 38084, upload-time = "2025-10-08T19:46:42.693Z" },+ { url = "https://files.pythonhosted.org/packages/f4/78/6cce448e2098e9f3bfc91bb877f06aa24b6ccace872e39c53b2f707c4648/propcache-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:364426a62660f3f699949ac8c621aad6977be7126c5807ce48c0aeb8e7333ea6", size = 41637, upload-time = "2025-10-08T19:46:43.778Z" },+ { url = "https://files.pythonhosted.org/packages/9c/e9/754f180cccd7f51a39913782c74717c581b9cc8177ad0e949f4d51812383/propcache-0.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:e53f3a38d3510c11953f3e6a33f205c6d1b001129f972805ca9b42fc308bc239", size = 38064, upload-time = "2025-10-08T19:46:44.872Z" },+ { url = "https://files.pythonhosted.org/packages/a2/0f/f17b1b2b221d5ca28b4b876e8bb046ac40466513960646bda8e1853cdfa2/propcache-0.4.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:e153e9cd40cc8945138822807139367f256f89c6810c2634a4f6902b52d3b4e2", size = 80061, upload-time = "2025-10-08T19:46:46.075Z" },+ { url = "https://files.pythonhosted.org/packages/76/47/8ccf75935f51448ba9a16a71b783eb7ef6b9ee60f5d14c7f8a8a79fbeed7/propcache-0.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:cd547953428f7abb73c5ad82cbb32109566204260d98e41e5dfdc682eb7f8403", size = 46037, upload-time = "2025-10-08T19:46:47.23Z" },+ { url = "https://files.pythonhosted.org/packages/0a/b6/5c9a0e42df4d00bfb4a3cbbe5cf9f54260300c88a0e9af1f47ca5ce17ac0/propcache-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f048da1b4f243fc44f205dfd320933a951b8d89e0afd4c7cacc762a8b9165207", size = 47324, upload-time = "2025-10-08T19:46:48.384Z" },+ { url = "https://files.pythonhosted.org/packages/9e/d3/6c7ee328b39a81ee877c962469f1e795f9db87f925251efeb0545e0020d0/propcache-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ec17c65562a827bba85e3872ead335f95405ea1674860d96483a02f5c698fa72", size = 225505, upload-time = "2025-10-08T19:46:50.055Z" },+ { url = "https://files.pythonhosted.org/packages/01/5d/1c53f4563490b1d06a684742cc6076ef944bc6457df6051b7d1a877c057b/propcache-0.4.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:405aac25c6394ef275dee4c709be43745d36674b223ba4eb7144bf4d691b7367", size = 230242, upload-time = "2025-10-08T19:46:51.815Z" },+ { url = "https://files.pythonhosted.org/packages/20/e1/ce4620633b0e2422207c3cb774a0ee61cac13abc6217763a7b9e2e3f4a12/propcache-0.4.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0013cb6f8dde4b2a2f66903b8ba740bdfe378c943c4377a200551ceb27f379e4", size = 238474, upload-time = "2025-10-08T19:46:53.208Z" },+ { url = "https://files.pythonhosted.org/packages/46/4b/3aae6835b8e5f44ea6a68348ad90f78134047b503765087be2f9912140ea/propcache-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15932ab57837c3368b024473a525e25d316d8353016e7cc0e5ba9eb343fbb1cf", size = 221575, upload-time = "2025-10-08T19:46:54.511Z" },+ { url = "https://files.pythonhosted.org/packages/6e/a5/8a5e8678bcc9d3a1a15b9a29165640d64762d424a16af543f00629c87338/propcache-0.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:031dce78b9dc099f4c29785d9cf5577a3faf9ebf74ecbd3c856a7b92768c3df3", size = 216736, upload-time = "2025-10-08T19:46:56.212Z" },+ { url = "https://files.pythonhosted.org/packages/f1/63/b7b215eddeac83ca1c6b934f89d09a625aa9ee4ba158338854c87210cc36/propcache-0.4.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:ab08df6c9a035bee56e31af99be621526bd237bea9f32def431c656b29e41778", size = 213019, upload-time = "2025-10-08T19:46:57.595Z" },+ { url = "https://files.pythonhosted.org/packages/57/74/f580099a58c8af587cac7ba19ee7cb418506342fbbe2d4a4401661cca886/propcache-0.4.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4d7af63f9f93fe593afbf104c21b3b15868efb2c21d07d8732c0c4287e66b6a6", size = 220376, upload-time = "2025-10-08T19:46:59.067Z" },+ { url = "https://files.pythonhosted.org/packages/c4/ee/542f1313aff7eaf19c2bb758c5d0560d2683dac001a1c96d0774af799843/propcache-0.4.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:cfc27c945f422e8b5071b6e93169679e4eb5bf73bbcbf1ba3ae3a83d2f78ebd9", size = 226988, upload-time = "2025-10-08T19:47:00.544Z" },+ { url = "https://files.pythonhosted.org/packages/8f/18/9c6b015dd9c6930f6ce2229e1f02fb35298b847f2087ea2b436a5bfa7287/propcache-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:35c3277624a080cc6ec6f847cbbbb5b49affa3598c4535a0a4682a697aaa5c75", size = 215615, upload-time = "2025-10-08T19:47:01.968Z" },+ { url = "https://files.pythonhosted.org/packages/80/9e/e7b85720b98c45a45e1fca6a177024934dc9bc5f4d5dd04207f216fc33ed/propcache-0.4.1-cp312-cp312-win32.whl", hash = "sha256:671538c2262dadb5ba6395e26c1731e1d52534bfe9ae56d0b5573ce539266aa8", size = 38066, upload-time = "2025-10-08T19:47:03.503Z" },+ { url = "https://files.pythonhosted.org/packages/54/09/d19cff2a5aaac632ec8fc03737b223597b1e347416934c1b3a7df079784c/propcache-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:cb2d222e72399fcf5890d1d5cc1060857b9b236adff2792ff48ca2dfd46c81db", size = 41655, upload-time = "2025-10-08T19:47:04.973Z" },+ { url = "https://files.pythonhosted.org/packages/68/ab/6b5c191bb5de08036a8c697b265d4ca76148efb10fa162f14af14fb5f076/propcache-0.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:204483131fb222bdaaeeea9f9e6c6ed0cac32731f75dfc1d4a567fc1926477c1", size = 37789, upload-time = "2025-10-08T19:47:06.077Z" },+ { url = "https://files.pythonhosted.org/packages/bf/df/6d9c1b6ac12b003837dde8a10231a7344512186e87b36e855bef32241942/propcache-0.4.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:43eedf29202c08550aac1d14e0ee619b0430aaef78f85864c1a892294fbc28cf", size = 77750, upload-time = "2025-10-08T19:47:07.648Z" },+ { url = "https://files.pythonhosted.org/packages/8b/e8/677a0025e8a2acf07d3418a2e7ba529c9c33caf09d3c1f25513023c1db56/propcache-0.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d62cdfcfd89ccb8de04e0eda998535c406bf5e060ffd56be6c586cbcc05b3311", size = 44780, upload-time = "2025-10-08T19:47:08.851Z" },+ { url = "https://files.pythonhosted.org/packages/89/a4/92380f7ca60f99ebae761936bc48a72a639e8a47b29050615eef757cb2a7/propcache-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cae65ad55793da34db5f54e4029b89d3b9b9490d8abe1b4c7ab5d4b8ec7ebf74", size = 46308, upload-time = "2025-10-08T19:47:09.982Z" },+ { url = "https://files.pythonhosted.org/packages/2d/48/c5ac64dee5262044348d1d78a5f85dd1a57464a60d30daee946699963eb3/propcache-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:333ddb9031d2704a301ee3e506dc46b1fe5f294ec198ed6435ad5b6a085facfe", size = 208182, upload-time = "2025-10-08T19:47:11.319Z" },+ { url = "https://files.pythonhosted.org/packages/c6/0c/cd762dd011a9287389a6a3eb43aa30207bde253610cca06824aeabfe9653/propcache-0.4.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:fd0858c20f078a32cf55f7e81473d96dcf3b93fd2ccdb3d40fdf54b8573df3af", size = 211215, upload-time = "2025-10-08T19:47:13.146Z" },+ { url = "https://files.pythonhosted.org/packages/30/3e/49861e90233ba36890ae0ca4c660e95df565b2cd15d4a68556ab5865974e/propcache-0.4.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:678ae89ebc632c5c204c794f8dab2837c5f159aeb59e6ed0539500400577298c", size = 218112, upload-time = "2025-10-08T19:47:14.913Z" },+ { url = "https://files.pythonhosted.org/packages/f1/8b/544bc867e24e1bd48f3118cecd3b05c694e160a168478fa28770f22fd094/propcache-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d472aeb4fbf9865e0c6d622d7f4d54a4e101a89715d8904282bb5f9a2f476c3f", size = 204442, upload-time = "2025-10-08T19:47:16.277Z" },+ { url = "https://files.pythonhosted.org/packages/50/a6/4282772fd016a76d3e5c0df58380a5ea64900afd836cec2c2f662d1b9bb3/propcache-0.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4d3df5fa7e36b3225954fba85589da77a0fe6a53e3976de39caf04a0db4c36f1", size = 199398, upload-time = "2025-10-08T19:47:17.962Z" },+ { url = "https://files.pythonhosted.org/packages/3e/ec/d8a7cd406ee1ddb705db2139f8a10a8a427100347bd698e7014351c7af09/propcache-0.4.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:ee17f18d2498f2673e432faaa71698032b0127ebf23ae5974eeaf806c279df24", size = 196920, upload-time = "2025-10-08T19:47:19.355Z" },+ { url = "https://files.pythonhosted.org/packages/f6/6c/f38ab64af3764f431e359f8baf9e0a21013e24329e8b85d2da32e8ed07ca/propcache-0.4.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:580e97762b950f993ae618e167e7be9256b8353c2dcd8b99ec100eb50f5286aa", size = 203748, upload-time = "2025-10-08T19:47:21.338Z" },+ { url = "https://files.pythonhosted.org/packages/d6/e3/fa846bd70f6534d647886621388f0a265254d30e3ce47e5c8e6e27dbf153/propcache-0.4.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:501d20b891688eb8e7aa903021f0b72d5a55db40ffaab27edefd1027caaafa61", size = 205877, upload-time = "2025-10-08T19:47:23.059Z" },+ { url = "https://files.pythonhosted.org/packages/e2/39/8163fc6f3133fea7b5f2827e8eba2029a0277ab2c5beee6c1db7b10fc23d/propcache-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9a0bd56e5b100aef69bd8562b74b46254e7c8812918d3baa700c8a8009b0af66", size = 199437, upload-time = "2025-10-08T19:47:24.445Z" },+ { url = "https://files.pythonhosted.org/packages/93/89/caa9089970ca49c7c01662bd0eeedfe85494e863e8043565aeb6472ce8fe/propcache-0.4.1-cp313-cp313-win32.whl", hash = "sha256:bcc9aaa5d80322bc2fb24bb7accb4a30f81e90ab8d6ba187aec0744bc302ad81", size = 37586, upload-time = "2025-10-08T19:47:25.736Z" },+ { url = "https://files.pythonhosted.org/packages/f5/ab/f76ec3c3627c883215b5c8080debb4394ef5a7a29be811f786415fc1e6fd/propcache-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:381914df18634f5494334d201e98245c0596067504b9372d8cf93f4bb23e025e", size = 40790, upload-time = "2025-10-08T19:47:26.847Z" },+ { url = "https://files.pythonhosted.org/packages/59/1b/e71ae98235f8e2ba5004d8cb19765a74877abf189bc53fc0c80d799e56c3/propcache-0.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:8873eb4460fd55333ea49b7d189749ecf6e55bf85080f11b1c4530ed3034cba1", size = 37158, upload-time = "2025-10-08T19:47:27.961Z" },+ { url = "https://files.pythonhosted.org/packages/83/ce/a31bbdfc24ee0dcbba458c8175ed26089cf109a55bbe7b7640ed2470cfe9/propcache-0.4.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:92d1935ee1f8d7442da9c0c4fa7ac20d07e94064184811b685f5c4fada64553b", size = 81451, upload-time = "2025-10-08T19:47:29.445Z" },+ { url = "https://files.pythonhosted.org/packages/25/9c/442a45a470a68456e710d96cacd3573ef26a1d0a60067e6a7d5e655621ed/propcache-0.4.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:473c61b39e1460d386479b9b2f337da492042447c9b685f28be4f74d3529e566", size = 46374, upload-time = "2025-10-08T19:47:30.579Z" },+ { url = "https://files.pythonhosted.org/packages/f4/bf/b1d5e21dbc3b2e889ea4327044fb16312a736d97640fb8b6aa3f9c7b3b65/propcache-0.4.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:c0ef0aaafc66fbd87842a3fe3902fd889825646bc21149eafe47be6072725835", size = 48396, upload-time = "2025-10-08T19:47:31.79Z" },+ { url = "https://files.pythonhosted.org/packages/f4/04/5b4c54a103d480e978d3c8a76073502b18db0c4bc17ab91b3cb5092ad949/propcache-0.4.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f95393b4d66bfae908c3ca8d169d5f79cd65636ae15b5e7a4f6e67af675adb0e", size = 275950, upload-time = "2025-10-08T19:47:33.481Z" },+ { url = "https://files.pythonhosted.org/packages/b4/c1/86f846827fb969c4b78b0af79bba1d1ea2156492e1b83dea8b8a6ae27395/propcache-0.4.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c07fda85708bc48578467e85099645167a955ba093be0a2dcba962195676e859", size = 273856, upload-time = "2025-10-08T19:47:34.906Z" },+ { url = "https://files.pythonhosted.org/packages/36/1d/fc272a63c8d3bbad6878c336c7a7dea15e8f2d23a544bda43205dfa83ada/propcache-0.4.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:af223b406d6d000830c6f65f1e6431783fc3f713ba3e6cc8c024d5ee96170a4b", size = 280420, upload-time = "2025-10-08T19:47:36.338Z" },+ { url = "https://files.pythonhosted.org/packages/07/0c/01f2219d39f7e53d52e5173bcb09c976609ba30209912a0680adfb8c593a/propcache-0.4.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a78372c932c90ee474559c5ddfffd718238e8673c340dc21fe45c5b8b54559a0", size = 263254, upload-time = "2025-10-08T19:47:37.692Z" },+ { url = "https://files.pythonhosted.org/packages/2d/18/cd28081658ce597898f0c4d174d4d0f3c5b6d4dc27ffafeef835c95eb359/propcache-0.4.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:564d9f0d4d9509e1a870c920a89b2fec951b44bf5ba7d537a9e7c1ccec2c18af", size = 261205, upload-time = "2025-10-08T19:47:39.659Z" },+ { url = "https://files.pythonhosted.org/packages/7a/71/1f9e22eb8b8316701c2a19fa1f388c8a3185082607da8e406a803c9b954e/propcache-0.4.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:17612831fda0138059cc5546f4d12a2aacfb9e47068c06af35c400ba58ba7393", size = 247873, upload-time = "2025-10-08T19:47:41.084Z" },+ { url = "https://files.pythonhosted.org/packages/4a/65/3d4b61f36af2b4eddba9def857959f1016a51066b4f1ce348e0cf7881f58/propcache-0.4.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:41a89040cb10bd345b3c1a873b2bf36413d48da1def52f268a055f7398514874", size = 262739, upload-time = "2025-10-08T19:47:42.51Z" },+ { url = "https://files.pythonhosted.org/packages/2a/42/26746ab087faa77c1c68079b228810436ccd9a5ce9ac85e2b7307195fd06/propcache-0.4.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:e35b88984e7fa64aacecea39236cee32dd9bd8c55f57ba8a75cf2399553f9bd7", size = 263514, upload-time = "2025-10-08T19:47:43.927Z" },+ { url = "https://files.pythonhosted.org/packages/94/13/630690fe201f5502d2403dd3cfd451ed8858fe3c738ee88d095ad2ff407b/propcache-0.4.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6f8b465489f927b0df505cbe26ffbeed4d6d8a2bbc61ce90eb074ff129ef0ab1", size = 257781, upload-time = "2025-10-08T19:47:45.448Z" },+ { url = "https://files.pythonhosted.org/packages/92/f7/1d4ec5841505f423469efbfc381d64b7b467438cd5a4bbcbb063f3b73d27/propcache-0.4.1-cp313-cp313t-win32.whl", hash = "sha256:2ad890caa1d928c7c2965b48f3a3815c853180831d0e5503d35cf00c472f4717", size = 41396, upload-time = "2025-10-08T19:47:47.202Z" },+ { url = "https://files.pythonhosted.org/packages/48/f0/615c30622316496d2cbbc29f5985f7777d3ada70f23370608c1d3e081c1f/propcache-0.4.1-cp313-cp313t-win_amd64.whl", hash = "sha256:f7ee0e597f495cf415bcbd3da3caa3bd7e816b74d0d52b8145954c5e6fd3ff37", size = 44897, upload-time = "2025-10-08T19:47:48.336Z" },+ { url = "https://files.pythonhosted.org/packages/fd/ca/6002e46eccbe0e33dcd4069ef32f7f1c9e243736e07adca37ae8c4830ec3/propcache-0.4.1-cp313-cp313t-win_arm64.whl", hash = "sha256:929d7cbe1f01bb7baffb33dc14eb5691c95831450a26354cd210a8155170c93a", size = 39789, upload-time = "2025-10-08T19:47:49.876Z" },+ { url = "https://files.pythonhosted.org/packages/8e/5c/bca52d654a896f831b8256683457ceddd490ec18d9ec50e97dfd8fc726a8/propcache-0.4.1-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:3f7124c9d820ba5548d431afb4632301acf965db49e666aa21c305cbe8c6de12", size = 78152, upload-time = "2025-10-08T19:47:51.051Z" },+ { url = "https://files.pythonhosted.org/packages/65/9b/03b04e7d82a5f54fb16113d839f5ea1ede58a61e90edf515f6577c66fa8f/propcache-0.4.1-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:c0d4b719b7da33599dfe3b22d3db1ef789210a0597bc650b7cee9c77c2be8c5c", size = 44869, upload-time = "2025-10-08T19:47:52.594Z" },+ { url = "https://files.pythonhosted.org/packages/b2/fa/89a8ef0468d5833a23fff277b143d0573897cf75bd56670a6d28126c7d68/propcache-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9f302f4783709a78240ebc311b793f123328716a60911d667e0c036bc5dcbded", size = 46596, upload-time = "2025-10-08T19:47:54.073Z" },+ { url = "https://files.pythonhosted.org/packages/86/bd/47816020d337f4a746edc42fe8d53669965138f39ee117414c7d7a340cfe/propcache-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c80ee5802e3fb9ea37938e7eecc307fb984837091d5fd262bb37238b1ae97641", size = 206981, upload-time = "2025-10-08T19:47:55.715Z" },+ { url = "https://files.pythonhosted.org/packages/df/f6/c5fa1357cc9748510ee55f37173eb31bfde6d94e98ccd9e6f033f2fc06e1/propcache-0.4.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ed5a841e8bb29a55fb8159ed526b26adc5bdd7e8bd7bf793ce647cb08656cdf4", size = 211490, upload-time = "2025-10-08T19:47:57.499Z" },+ { url = "https://files.pythonhosted.org/packages/80/1e/e5889652a7c4a3846683401a48f0f2e5083ce0ec1a8a5221d8058fbd1adf/propcache-0.4.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:55c72fd6ea2da4c318e74ffdf93c4fe4e926051133657459131a95c846d16d44", size = 215371, upload-time = "2025-10-08T19:47:59.317Z" },+ { url = "https://files.pythonhosted.org/packages/b2/f2/889ad4b2408f72fe1a4f6a19491177b30ea7bf1a0fd5f17050ca08cfc882/propcache-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8326e144341460402713f91df60ade3c999d601e7eb5ff8f6f7862d54de0610d", size = 201424, upload-time = "2025-10-08T19:48:00.67Z" },+ { url = "https://files.pythonhosted.org/packages/27/73/033d63069b57b0812c8bd19f311faebeceb6ba31b8f32b73432d12a0b826/propcache-0.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:060b16ae65bc098da7f6d25bf359f1f31f688384858204fe5d652979e0015e5b", size = 197566, upload-time = "2025-10-08T19:48:02.604Z" },+ { url = "https://files.pythonhosted.org/packages/dc/89/ce24f3dc182630b4e07aa6d15f0ff4b14ed4b9955fae95a0b54c58d66c05/propcache-0.4.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:89eb3fa9524f7bec9de6e83cf3faed9d79bffa560672c118a96a171a6f55831e", size = 193130, upload-time = "2025-10-08T19:48:04.499Z" },+ { url = "https://files.pythonhosted.org/packages/a9/24/ef0d5fd1a811fb5c609278d0209c9f10c35f20581fcc16f818da959fc5b4/propcache-0.4.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:dee69d7015dc235f526fe80a9c90d65eb0039103fe565776250881731f06349f", size = 202625, upload-time = "2025-10-08T19:48:06.213Z" },+ { url = "https://files.pythonhosted.org/packages/f5/02/98ec20ff5546f68d673df2f7a69e8c0d076b5abd05ca882dc7ee3a83653d/propcache-0.4.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:5558992a00dfd54ccbc64a32726a3357ec93825a418a401f5cc67df0ac5d9e49", size = 204209, upload-time = "2025-10-08T19:48:08.432Z" },+ { url = "https://files.pythonhosted.org/packages/a0/87/492694f76759b15f0467a2a93ab68d32859672b646aa8a04ce4864e7932d/propcache-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c9b822a577f560fbd9554812526831712c1436d2c046cedee4c3796d3543b144", size = 197797, upload-time = "2025-10-08T19:48:09.968Z" },+ { url = "https://files.pythonhosted.org/packages/ee/36/66367de3575db1d2d3f3d177432bd14ee577a39d3f5d1b3d5df8afe3b6e2/propcache-0.4.1-cp314-cp314-win32.whl", hash = "sha256:ab4c29b49d560fe48b696cdcb127dd36e0bc2472548f3bf56cc5cb3da2b2984f", size = 38140, upload-time = "2025-10-08T19:48:11.232Z" },+ { url = "https://files.pythonhosted.org/packages/0c/2a/a758b47de253636e1b8aef181c0b4f4f204bf0dd964914fb2af90a95b49b/propcache-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:5a103c3eb905fcea0ab98be99c3a9a5ab2de60228aa5aceedc614c0281cf6153", size = 41257, upload-time = "2025-10-08T19:48:12.707Z" },+ { url = "https://files.pythonhosted.org/packages/34/5e/63bd5896c3fec12edcbd6f12508d4890d23c265df28c74b175e1ef9f4f3b/propcache-0.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:74c1fb26515153e482e00177a1ad654721bf9207da8a494a0c05e797ad27b992", size = 38097, upload-time = "2025-10-08T19:48:13.923Z" },+ { url = "https://files.pythonhosted.org/packages/99/85/9ff785d787ccf9bbb3f3106f79884a130951436f58392000231b4c737c80/propcache-0.4.1-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:824e908bce90fb2743bd6b59db36eb4f45cd350a39637c9f73b1c1ea66f5b75f", size = 81455, upload-time = "2025-10-08T19:48:15.16Z" },+ { url = "https://files.pythonhosted.org/packages/90/85/2431c10c8e7ddb1445c1f7c4b54d886e8ad20e3c6307e7218f05922cad67/propcache-0.4.1-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c2b5e7db5328427c57c8e8831abda175421b709672f6cfc3d630c3b7e2146393", size = 46372, upload-time = "2025-10-08T19:48:16.424Z" },+ { url = "https://files.pythonhosted.org/packages/01/20/b0972d902472da9bcb683fa595099911f4d2e86e5683bcc45de60dd05dc3/propcache-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6f6ff873ed40292cd4969ef5310179afd5db59fdf055897e282485043fc80ad0", size = 48411, upload-time = "2025-10-08T19:48:17.577Z" },+ { url = "https://files.pythonhosted.org/packages/e2/e3/7dc89f4f21e8f99bad3d5ddb3a3389afcf9da4ac69e3deb2dcdc96e74169/propcache-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49a2dc67c154db2c1463013594c458881a069fcf98940e61a0569016a583020a", size = 275712, upload-time = "2025-10-08T19:48:18.901Z" },+ { url = "https://files.pythonhosted.org/packages/20/67/89800c8352489b21a8047c773067644e3897f02ecbbd610f4d46b7f08612/propcache-0.4.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:005f08e6a0529984491e37d8dbc3dd86f84bd78a8ceb5fa9a021f4c48d4984be", size = 273557, upload-time = "2025-10-08T19:48:20.762Z" },+ { url = "https://files.pythonhosted.org/packages/e2/a1/b52b055c766a54ce6d9c16d9aca0cad8059acd9637cdf8aa0222f4a026ef/propcache-0.4.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5c3310452e0d31390da9035c348633b43d7e7feb2e37be252be6da45abd1abcc", size = 280015, upload-time = "2025-10-08T19:48:22.592Z" },+ { url = "https://files.pythonhosted.org/packages/48/c8/33cee30bd890672c63743049f3c9e4be087e6780906bfc3ec58528be59c1/propcache-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4c3c70630930447f9ef1caac7728c8ad1c56bc5015338b20fed0d08ea2480b3a", size = 262880, upload-time = "2025-10-08T19:48:23.947Z" },+ { url = "https://files.pythonhosted.org/packages/0c/b1/8f08a143b204b418285c88b83d00edbd61afbc2c6415ffafc8905da7038b/propcache-0.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e57061305815dfc910a3634dcf584f08168a8836e6999983569f51a8544cd89", size = 260938, upload-time = "2025-10-08T19:48:25.656Z" },+ { url = "https://files.pythonhosted.org/packages/cf/12/96e4664c82ca2f31e1c8dff86afb867348979eb78d3cb8546a680287a1e9/propcache-0.4.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:521a463429ef54143092c11a77e04056dd00636f72e8c45b70aaa3140d639726", size = 247641, upload-time = "2025-10-08T19:48:27.207Z" },+ { url = "https://files.pythonhosted.org/packages/18/ed/e7a9cfca28133386ba52278136d42209d3125db08d0a6395f0cba0c0285c/propcache-0.4.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:120c964da3fdc75e3731aa392527136d4ad35868cc556fd09bb6d09172d9a367", size = 262510, upload-time = "2025-10-08T19:48:28.65Z" },+ { url = "https://files.pythonhosted.org/packages/f5/76/16d8bf65e8845dd62b4e2b57444ab81f07f40caa5652b8969b87ddcf2ef6/propcache-0.4.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:d8f353eb14ee3441ee844ade4277d560cdd68288838673273b978e3d6d2c8f36", size = 263161, upload-time = "2025-10-08T19:48:30.133Z" },+ { url = "https://files.pythonhosted.org/packages/e7/70/c99e9edb5d91d5ad8a49fa3c1e8285ba64f1476782fed10ab251ff413ba1/propcache-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ab2943be7c652f09638800905ee1bab2c544e537edb57d527997a24c13dc1455", size = 257393, upload-time = "2025-10-08T19:48:31.567Z" },+ { url = "https://files.pythonhosted.org/packages/08/02/87b25304249a35c0915d236575bc3574a323f60b47939a2262b77632a3ee/propcache-0.4.1-cp314-cp314t-win32.whl", hash = "sha256:05674a162469f31358c30bcaa8883cb7829fa3110bf9c0991fe27d7896c42d85", size = 42546, upload-time = "2025-10-08T19:48:32.872Z" },+ { url = "https://files.pythonhosted.org/packages/cb/ef/3c6ecf8b317aa982f309835e8f96987466123c6e596646d4e6a1dfcd080f/propcache-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:990f6b3e2a27d683cb7602ed6c86f15ee6b43b1194736f9baaeb93d0016633b1", size = 46259, upload-time = "2025-10-08T19:48:34.226Z" },+ { url = "https://files.pythonhosted.org/packages/c4/2d/346e946d4951f37eca1e4f55be0f0174c52cd70720f84029b02f296f4a38/propcache-0.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:ecef2343af4cc68e05131e45024ba34f6095821988a9d0a02aa7c73fcc448aa9", size = 40428, upload-time = "2025-10-08T19:48:35.441Z" },+ { url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" },][[package]]@@ -2594,7 +2588,7 @@ crypto = [[[package]]name = "pylint"-version = "3.3.9"+version = "4.0.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "astroid" },@@ -2606,9 +2600,9 @@ dependencies = [{ name = "tomli", marker = "python_full_version < '3.11'" },{ name = "tomlkit" },]-sdist = { url = "https://files.pythonhosted.org/packages/04/9d/81c84a312d1fa8133b0db0c76148542a98349298a01747ab122f9314b04e/pylint-3.3.9.tar.gz", hash = "sha256:d312737d7b25ccf6b01cc4ac629b5dcd14a0fcf3ec392735ac70f137a9d5f83a", size = 1525946, upload-time = "2025-10-05T18:41:43.786Z" }+sdist = { url = "https://files.pythonhosted.org/packages/b6/2f/e80cc4301c81c41a8836d726377daeebf5901a33c06ba8c2d5afb94f7612/pylint-4.0.0.tar.gz", hash = "sha256:62da212808c0681e49ffb125f0a994c685d912cf19ae373075649ebb5870ec28", size = 1567676, upload-time = "2025-10-12T15:21:15.165Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/1a/a7/69460c4a6af7575449e615144aa2205b89408dc2969b87bc3df2f262ad0b/pylint-3.3.9-py3-none-any.whl", hash = "sha256:01f9b0462c7730f94786c283f3e52a1fbdf0494bbe0971a78d7277ef46a751e7", size = 523465, upload-time = "2025-10-05T18:41:41.766Z" },+ { url = "https://files.pythonhosted.org/packages/42/af/068a0b92c49927ada0e177561244157dc9d122eeea5987e34c423172a296/pylint-4.0.0-py3-none-any.whl", hash = "sha256:196b92a85204bb0c0a416a6bb324f6185e59ff1d687ee1d614bf0abf34a348e8", size = 535836, upload-time = "2025-10-12T15:21:13.041Z" },][[package]]@@ -2914,15 +2908,15 @@ wheels = [[[package]]name = "rich"-version = "14.1.0"+version = "14.2.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "markdown-it-py" },{ name = "pygments" },]-sdist = { url = "https://files.pythonhosted.org/packages/fe/75/af448d8e52bf1d8fa6a9d089ca6c07ff4453d86c65c145d0a300bb073b9b/rich-14.1.0.tar.gz", hash = "sha256:e497a48b844b0320d45007cdebfeaeed8db2a4f4bcf49f15e455cfc4af11eaa8", size = 224441, upload-time = "2025-07-25T07:32:58.125Z" }+sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/e3/30/3c4d035596d3cf444529e0b2953ad0466f6049528a879d27534700580395/rich-14.1.0-py3-none-any.whl", hash = "sha256:536f5f1785986d6dbdea3c75205c473f970777b4a0d6c6dd1b696aa05a3fa04f", size = 243368, upload-time = "2025-07-25T07:32:56.73Z" },+ { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" },][[package]]@@ -3029,52 +3023,53 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/4f/bd/de8d508070629b6d84a30d01d57e4a65c69aa7f5abe7560b8fad3b50ea59/termcolor-3.1.0-py3-none-any.whl", hash = "sha256:591dd26b5c2ce03b9e43f391264626557873ce1d379019786f99b0c2bee140aa", size = 7684, upload-time = "2025-04-30T11:37:52.382Z" },]-[[package]]-name = "toml"-version = "0.10.2"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f", size = 22253, upload-time = "2020-11-01T01:40:22.204Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" },-]-[[package]]name = "tomli"-version = "2.2.1"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/18/87/302344fed471e44a87289cf4967697d07e532f2421fdaf868a303cbae4ff/tomli-2.2.1.tar.gz", hash = "sha256:cd45e1dc79c835ce60f7404ec8119f2eb06d38b1deba146f07ced3bbc44505ff", size = 17175, upload-time = "2024-11-27T22:38:36.873Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/43/ca/75707e6efa2b37c77dadb324ae7d9571cb424e61ea73fad7c56c2d14527f/tomli-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:678e4fa69e4575eb77d103de3df8a895e1591b48e740211bd1067378c69e8249", size = 131077, upload-time = "2024-11-27T22:37:54.956Z" },- { url = "https://files.pythonhosted.org/packages/c7/16/51ae563a8615d472fdbffc43a3f3d46588c264ac4f024f63f01283becfbb/tomli-2.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:023aa114dd824ade0100497eb2318602af309e5a55595f76b626d6d9f3b7b0a6", size = 123429, upload-time = "2024-11-27T22:37:56.698Z" },- { url = "https://files.pythonhosted.org/packages/f1/dd/4f6cd1e7b160041db83c694abc78e100473c15d54620083dbd5aae7b990e/tomli-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ece47d672db52ac607a3d9599a9d48dcb2f2f735c6c2d1f34130085bb12b112a", size = 226067, upload-time = "2024-11-27T22:37:57.63Z" },- { url = "https://files.pythonhosted.org/packages/a9/6b/c54ede5dc70d648cc6361eaf429304b02f2871a345bbdd51e993d6cdf550/tomli-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6972ca9c9cc9f0acaa56a8ca1ff51e7af152a9f87fb64623e31d5c83700080ee", size = 236030, upload-time = "2024-11-27T22:37:59.344Z" },- { url = "https://files.pythonhosted.org/packages/1f/47/999514fa49cfaf7a92c805a86c3c43f4215621855d151b61c602abb38091/tomli-2.2.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c954d2250168d28797dd4e3ac5cf812a406cd5a92674ee4c8f123c889786aa8e", size = 240898, upload-time = "2024-11-27T22:38:00.429Z" },- { url = "https://files.pythonhosted.org/packages/73/41/0a01279a7ae09ee1573b423318e7934674ce06eb33f50936655071d81a24/tomli-2.2.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8dd28b3e155b80f4d54beb40a441d366adcfe740969820caf156c019fb5c7ec4", size = 229894, upload-time = "2024-11-27T22:38:02.094Z" },- { url = "https://files.pythonhosted.org/packages/55/18/5d8bc5b0a0362311ce4d18830a5d28943667599a60d20118074ea1b01bb7/tomli-2.2.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:e59e304978767a54663af13c07b3d1af22ddee3bb2fb0618ca1593e4f593a106", size = 245319, upload-time = "2024-11-27T22:38:03.206Z" },- { url = "https://files.pythonhosted.org/packages/92/a3/7ade0576d17f3cdf5ff44d61390d4b3febb8a9fc2b480c75c47ea048c646/tomli-2.2.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:33580bccab0338d00994d7f16f4c4ec25b776af3ffaac1ed74e0b3fc95e885a8", size = 238273, upload-time = "2024-11-27T22:38:04.217Z" },- { url = "https://files.pythonhosted.org/packages/72/6f/fa64ef058ac1446a1e51110c375339b3ec6be245af9d14c87c4a6412dd32/tomli-2.2.1-cp311-cp311-win32.whl", hash = "sha256:465af0e0875402f1d226519c9904f37254b3045fc5084697cefb9bdde1ff99ff", size = 98310, upload-time = "2024-11-27T22:38:05.908Z" },- { url = "https://files.pythonhosted.org/packages/6a/1c/4a2dcde4a51b81be3530565e92eda625d94dafb46dbeb15069df4caffc34/tomli-2.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:2d0f2fdd22b02c6d81637a3c95f8cd77f995846af7414c5c4b8d0545afa1bc4b", size = 108309, upload-time = "2024-11-27T22:38:06.812Z" },- { url = "https://files.pythonhosted.org/packages/52/e1/f8af4c2fcde17500422858155aeb0d7e93477a0d59a98e56cbfe75070fd0/tomli-2.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:4a8f6e44de52d5e6c657c9fe83b562f5f4256d8ebbfe4ff922c495620a7f6cea", size = 132762, upload-time = "2024-11-27T22:38:07.731Z" },- { url = "https://files.pythonhosted.org/packages/03/b8/152c68bb84fc00396b83e7bbddd5ec0bd3dd409db4195e2a9b3e398ad2e3/tomli-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8d57ca8095a641b8237d5b079147646153d22552f1c637fd3ba7f4b0b29167a8", size = 123453, upload-time = "2024-11-27T22:38:09.384Z" },- { url = "https://files.pythonhosted.org/packages/c8/d6/fc9267af9166f79ac528ff7e8c55c8181ded34eb4b0e93daa767b8841573/tomli-2.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e340144ad7ae1533cb897d406382b4b6fede8890a03738ff1683af800d54192", size = 233486, upload-time = "2024-11-27T22:38:10.329Z" },- { url = "https://files.pythonhosted.org/packages/5c/51/51c3f2884d7bab89af25f678447ea7d297b53b5a3b5730a7cb2ef6069f07/tomli-2.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:db2b95f9de79181805df90bedc5a5ab4c165e6ec3fe99f970d0e302f384ad222", size = 242349, upload-time = "2024-11-27T22:38:11.443Z" },- { url = "https://files.pythonhosted.org/packages/ab/df/bfa89627d13a5cc22402e441e8a931ef2108403db390ff3345c05253935e/tomli-2.2.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:40741994320b232529c802f8bc86da4e1aa9f413db394617b9a256ae0f9a7f77", size = 252159, upload-time = "2024-11-27T22:38:13.099Z" },- { url = "https://files.pythonhosted.org/packages/9e/6e/fa2b916dced65763a5168c6ccb91066f7639bdc88b48adda990db10c8c0b/tomli-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:400e720fe168c0f8521520190686ef8ef033fb19fc493da09779e592861b78c6", size = 237243, upload-time = "2024-11-27T22:38:14.766Z" },- { url = "https://files.pythonhosted.org/packages/b4/04/885d3b1f650e1153cbb93a6a9782c58a972b94ea4483ae4ac5cedd5e4a09/tomli-2.2.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:02abe224de6ae62c19f090f68da4e27b10af2b93213d36cf44e6e1c5abd19fdd", size = 259645, upload-time = "2024-11-27T22:38:15.843Z" },- { url = "https://files.pythonhosted.org/packages/9c/de/6b432d66e986e501586da298e28ebeefd3edc2c780f3ad73d22566034239/tomli-2.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b82ebccc8c8a36f2094e969560a1b836758481f3dc360ce9a3277c65f374285e", size = 244584, upload-time = "2024-11-27T22:38:17.645Z" },- { url = "https://files.pythonhosted.org/packages/1c/9a/47c0449b98e6e7d1be6cbac02f93dd79003234ddc4aaab6ba07a9a7482e2/tomli-2.2.1-cp312-cp312-win32.whl", hash = "sha256:889f80ef92701b9dbb224e49ec87c645ce5df3fa2cc548664eb8a25e03127a98", size = 98875, upload-time = "2024-11-27T22:38:19.159Z" },- { url = "https://files.pythonhosted.org/packages/ef/60/9b9638f081c6f1261e2688bd487625cd1e660d0a85bd469e91d8db969734/tomli-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:7fc04e92e1d624a4a63c76474610238576942d6b8950a2d7f908a340494e67e4", size = 109418, upload-time = "2024-11-27T22:38:20.064Z" },- { url = "https://files.pythonhosted.org/packages/04/90/2ee5f2e0362cb8a0b6499dc44f4d7d48f8fff06d28ba46e6f1eaa61a1388/tomli-2.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f4039b9cbc3048b2416cc57ab3bda989a6fcf9b36cf8937f01a6e731b64f80d7", size = 132708, upload-time = "2024-11-27T22:38:21.659Z" },- { url = "https://files.pythonhosted.org/packages/c0/ec/46b4108816de6b385141f082ba99e315501ccd0a2ea23db4a100dd3990ea/tomli-2.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:286f0ca2ffeeb5b9bd4fcc8d6c330534323ec51b2f52da063b11c502da16f30c", size = 123582, upload-time = "2024-11-27T22:38:22.693Z" },- { url = "https://files.pythonhosted.org/packages/a0/bd/b470466d0137b37b68d24556c38a0cc819e8febe392d5b199dcd7f578365/tomli-2.2.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a92ef1a44547e894e2a17d24e7557a5e85a9e1d0048b0b5e7541f76c5032cb13", size = 232543, upload-time = "2024-11-27T22:38:24.367Z" },- { url = "https://files.pythonhosted.org/packages/d9/e5/82e80ff3b751373f7cead2815bcbe2d51c895b3c990686741a8e56ec42ab/tomli-2.2.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9316dc65bed1684c9a98ee68759ceaed29d229e985297003e494aa825ebb0281", size = 241691, upload-time = "2024-11-27T22:38:26.081Z" },- { url = "https://files.pythonhosted.org/packages/05/7e/2a110bc2713557d6a1bfb06af23dd01e7dde52b6ee7dadc589868f9abfac/tomli-2.2.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e85e99945e688e32d5a35c1ff38ed0b3f41f43fad8df0bdf79f72b2ba7bc5272", size = 251170, upload-time = "2024-11-27T22:38:27.921Z" },- { url = "https://files.pythonhosted.org/packages/64/7b/22d713946efe00e0adbcdfd6d1aa119ae03fd0b60ebed51ebb3fa9f5a2e5/tomli-2.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ac065718db92ca818f8d6141b5f66369833d4a80a9d74435a268c52bdfa73140", size = 236530, upload-time = "2024-11-27T22:38:29.591Z" },- { url = "https://files.pythonhosted.org/packages/38/31/3a76f67da4b0cf37b742ca76beaf819dca0ebef26d78fc794a576e08accf/tomli-2.2.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:d920f33822747519673ee656a4b6ac33e382eca9d331c87770faa3eef562aeb2", size = 258666, upload-time = "2024-11-27T22:38:30.639Z" },- { url = "https://files.pythonhosted.org/packages/07/10/5af1293da642aded87e8a988753945d0cf7e00a9452d3911dd3bb354c9e2/tomli-2.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a198f10c4d1b1375d7687bc25294306e551bf1abfa4eace6650070a5c1ae2744", size = 243954, upload-time = "2024-11-27T22:38:31.702Z" },- { url = "https://files.pythonhosted.org/packages/5b/b9/1ed31d167be802da0fc95020d04cd27b7d7065cc6fbefdd2f9186f60d7bd/tomli-2.2.1-cp313-cp313-win32.whl", hash = "sha256:d3f5614314d758649ab2ab3a62d4f2004c825922f9e370b29416484086b264ec", size = 98724, upload-time = "2024-11-27T22:38:32.837Z" },- { url = "https://files.pythonhosted.org/packages/c7/32/b0963458706accd9afcfeb867c0f9175a741bf7b19cd424230714d722198/tomli-2.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:a38aa0308e754b0e3c67e344754dff64999ff9b513e691d0e786265c93583c69", size = 109383, upload-time = "2024-11-27T22:38:34.455Z" },- { url = "https://files.pythonhosted.org/packages/6e/c2/61d3e0f47e2b74ef40a68b9e6ad5984f6241a942f7cd3bbfbdbd03861ea9/tomli-2.2.1-py3-none-any.whl", hash = "sha256:cb55c73c5f4408779d0cf3eef9f762b9c9f147a77de7b258bef0a5628adc85cc", size = 14257, upload-time = "2024-11-27T22:38:35.385Z" },+version = "2.3.0"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/52/ed/3f73f72945444548f33eba9a87fc7a6e969915e7b1acc8260b30e1f76a2f/tomli-2.3.0.tar.gz", hash = "sha256:64be704a875d2a59753d80ee8a533c3fe183e3f06807ff7dc2232938ccb01549", size = 17392, upload-time = "2025-10-08T22:01:47.119Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/b3/2e/299f62b401438d5fe1624119c723f5d877acc86a4c2492da405626665f12/tomli-2.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:88bd15eb972f3664f5ed4b57c1634a97153b4bac4479dcb6a495f41921eb7f45", size = 153236, upload-time = "2025-10-08T22:01:00.137Z" },+ { url = "https://files.pythonhosted.org/packages/86/7f/d8fffe6a7aefdb61bced88fcb5e280cfd71e08939da5894161bd71bea022/tomli-2.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:883b1c0d6398a6a9d29b508c331fa56adbcdff647f6ace4dfca0f50e90dfd0ba", size = 148084, upload-time = "2025-10-08T22:01:01.63Z" },+ { url = "https://files.pythonhosted.org/packages/47/5c/24935fb6a2ee63e86d80e4d3b58b222dafaf438c416752c8b58537c8b89a/tomli-2.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d1381caf13ab9f300e30dd8feadb3de072aeb86f1d34a8569453ff32a7dea4bf", size = 234832, upload-time = "2025-10-08T22:01:02.543Z" },+ { url = "https://files.pythonhosted.org/packages/89/da/75dfd804fc11e6612846758a23f13271b76d577e299592b4371a4ca4cd09/tomli-2.3.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a0e285d2649b78c0d9027570d4da3425bdb49830a6156121360b3f8511ea3441", size = 242052, upload-time = "2025-10-08T22:01:03.836Z" },+ { url = "https://files.pythonhosted.org/packages/70/8c/f48ac899f7b3ca7eb13af73bacbc93aec37f9c954df3c08ad96991c8c373/tomli-2.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0a154a9ae14bfcf5d8917a59b51ffd5a3ac1fd149b71b47a3a104ca4edcfa845", size = 239555, upload-time = "2025-10-08T22:01:04.834Z" },+ { url = "https://files.pythonhosted.org/packages/ba/28/72f8afd73f1d0e7829bfc093f4cb98ce0a40ffc0cc997009ee1ed94ba705/tomli-2.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:74bf8464ff93e413514fefd2be591c3b0b23231a77f901db1eb30d6f712fc42c", size = 245128, upload-time = "2025-10-08T22:01:05.84Z" },+ { url = "https://files.pythonhosted.org/packages/b6/eb/a7679c8ac85208706d27436e8d421dfa39d4c914dcf5fa8083a9305f58d9/tomli-2.3.0-cp311-cp311-win32.whl", hash = "sha256:00b5f5d95bbfc7d12f91ad8c593a1659b6387b43f054104cda404be6bda62456", size = 96445, upload-time = "2025-10-08T22:01:06.896Z" },+ { url = "https://files.pythonhosted.org/packages/0a/fe/3d3420c4cb1ad9cb462fb52967080575f15898da97e21cb6f1361d505383/tomli-2.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:4dc4ce8483a5d429ab602f111a93a6ab1ed425eae3122032db7e9acf449451be", size = 107165, upload-time = "2025-10-08T22:01:08.107Z" },+ { url = "https://files.pythonhosted.org/packages/ff/b7/40f36368fcabc518bb11c8f06379a0fd631985046c038aca08c6d6a43c6e/tomli-2.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d7d86942e56ded512a594786a5ba0a5e521d02529b3826e7761a05138341a2ac", size = 154891, upload-time = "2025-10-08T22:01:09.082Z" },+ { url = "https://files.pythonhosted.org/packages/f9/3f/d9dd692199e3b3aab2e4e4dd948abd0f790d9ded8cd10cbaae276a898434/tomli-2.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:73ee0b47d4dad1c5e996e3cd33b8a76a50167ae5f96a2607cbe8cc773506ab22", size = 148796, upload-time = "2025-10-08T22:01:10.266Z" },+ { url = "https://files.pythonhosted.org/packages/60/83/59bff4996c2cf9f9387a0f5a3394629c7efa5ef16142076a23a90f1955fa/tomli-2.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:792262b94d5d0a466afb5bc63c7daa9d75520110971ee269152083270998316f", size = 242121, upload-time = "2025-10-08T22:01:11.332Z" },+ { url = "https://files.pythonhosted.org/packages/45/e5/7c5119ff39de8693d6baab6c0b6dcb556d192c165596e9fc231ea1052041/tomli-2.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4f195fe57ecceac95a66a75ac24d9d5fbc98ef0962e09b2eddec5d39375aae52", size = 250070, upload-time = "2025-10-08T22:01:12.498Z" },+ { url = "https://files.pythonhosted.org/packages/45/12/ad5126d3a278f27e6701abde51d342aa78d06e27ce2bb596a01f7709a5a2/tomli-2.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e31d432427dcbf4d86958c184b9bfd1e96b5b71f8eb17e6d02531f434fd335b8", size = 245859, upload-time = "2025-10-08T22:01:13.551Z" },+ { url = "https://files.pythonhosted.org/packages/fb/a1/4d6865da6a71c603cfe6ad0e6556c73c76548557a8d658f9e3b142df245f/tomli-2.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7b0882799624980785240ab732537fcfc372601015c00f7fc367c55308c186f6", size = 250296, upload-time = "2025-10-08T22:01:14.614Z" },+ { url = "https://files.pythonhosted.org/packages/a0/b7/a7a7042715d55c9ba6e8b196d65d2cb662578b4d8cd17d882d45322b0d78/tomli-2.3.0-cp312-cp312-win32.whl", hash = "sha256:ff72b71b5d10d22ecb084d345fc26f42b5143c5533db5e2eaba7d2d335358876", size = 97124, upload-time = "2025-10-08T22:01:15.629Z" },+ { url = "https://files.pythonhosted.org/packages/06/1e/f22f100db15a68b520664eb3328fb0ae4e90530887928558112c8d1f4515/tomli-2.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:1cb4ed918939151a03f33d4242ccd0aa5f11b3547d0cf30f7c74a408a5b99878", size = 107698, upload-time = "2025-10-08T22:01:16.51Z" },+ { url = "https://files.pythonhosted.org/packages/89/48/06ee6eabe4fdd9ecd48bf488f4ac783844fd777f547b8d1b61c11939974e/tomli-2.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5192f562738228945d7b13d4930baffda67b69425a7f0da96d360b0a3888136b", size = 154819, upload-time = "2025-10-08T22:01:17.964Z" },+ { url = "https://files.pythonhosted.org/packages/f1/01/88793757d54d8937015c75dcdfb673c65471945f6be98e6a0410fba167ed/tomli-2.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:be71c93a63d738597996be9528f4abe628d1adf5e6eb11607bc8fe1a510b5dae", size = 148766, upload-time = "2025-10-08T22:01:18.959Z" },+ { url = "https://files.pythonhosted.org/packages/42/17/5e2c956f0144b812e7e107f94f1cc54af734eb17b5191c0bbfb72de5e93e/tomli-2.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4665508bcbac83a31ff8ab08f424b665200c0e1e645d2bd9ab3d3e557b6185b", size = 240771, upload-time = "2025-10-08T22:01:20.106Z" },+ { url = "https://files.pythonhosted.org/packages/d5/f4/0fbd014909748706c01d16824eadb0307115f9562a15cbb012cd9b3512c5/tomli-2.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4021923f97266babc6ccab9f5068642a0095faa0a51a246a6a02fccbb3514eaf", size = 248586, upload-time = "2025-10-08T22:01:21.164Z" },+ { url = "https://files.pythonhosted.org/packages/30/77/fed85e114bde5e81ecf9bc5da0cc69f2914b38f4708c80ae67d0c10180c5/tomli-2.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4ea38c40145a357d513bffad0ed869f13c1773716cf71ccaa83b0fa0cc4e42f", size = 244792, upload-time = "2025-10-08T22:01:22.417Z" },+ { url = "https://files.pythonhosted.org/packages/55/92/afed3d497f7c186dc71e6ee6d4fcb0acfa5f7d0a1a2878f8beae379ae0cc/tomli-2.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ad805ea85eda330dbad64c7ea7a4556259665bdf9d2672f5dccc740eb9d3ca05", size = 248909, upload-time = "2025-10-08T22:01:23.859Z" },+ { url = "https://files.pythonhosted.org/packages/f8/84/ef50c51b5a9472e7265ce1ffc7f24cd4023d289e109f669bdb1553f6a7c2/tomli-2.3.0-cp313-cp313-win32.whl", hash = "sha256:97d5eec30149fd3294270e889b4234023f2c69747e555a27bd708828353ab606", size = 96946, upload-time = "2025-10-08T22:01:24.893Z" },+ { url = "https://files.pythonhosted.org/packages/b2/b7/718cd1da0884f281f95ccfa3a6cc572d30053cba64603f79d431d3c9b61b/tomli-2.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:0c95ca56fbe89e065c6ead5b593ee64b84a26fca063b5d71a1122bf26e533999", size = 107705, upload-time = "2025-10-08T22:01:26.153Z" },+ { url = "https://files.pythonhosted.org/packages/19/94/aeafa14a52e16163008060506fcb6aa1949d13548d13752171a755c65611/tomli-2.3.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:cebc6fe843e0733ee827a282aca4999b596241195f43b4cc371d64fc6639da9e", size = 154244, upload-time = "2025-10-08T22:01:27.06Z" },+ { url = "https://files.pythonhosted.org/packages/db/e4/1e58409aa78eefa47ccd19779fc6f36787edbe7d4cd330eeeedb33a4515b/tomli-2.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4c2ef0244c75aba9355561272009d934953817c49f47d768070c3c94355c2aa3", size = 148637, upload-time = "2025-10-08T22:01:28.059Z" },+ { url = "https://files.pythonhosted.org/packages/26/b6/d1eccb62f665e44359226811064596dd6a366ea1f985839c566cd61525ae/tomli-2.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c22a8bf253bacc0cf11f35ad9808b6cb75ada2631c2d97c971122583b129afbc", size = 241925, upload-time = "2025-10-08T22:01:29.066Z" },+ { url = "https://files.pythonhosted.org/packages/70/91/7cdab9a03e6d3d2bb11beae108da5bdc1c34bdeb06e21163482544ddcc90/tomli-2.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0eea8cc5c5e9f89c9b90c4896a8deefc74f518db5927d0e0e8d4a80953d774d0", size = 249045, upload-time = "2025-10-08T22:01:31.98Z" },+ { url = "https://files.pythonhosted.org/packages/15/1b/8c26874ed1f6e4f1fcfeb868db8a794cbe9f227299402db58cfcc858766c/tomli-2.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b74a0e59ec5d15127acdabd75ea17726ac4c5178ae51b85bfe39c4f8a278e879", size = 245835, upload-time = "2025-10-08T22:01:32.989Z" },+ { url = "https://files.pythonhosted.org/packages/fd/42/8e3c6a9a4b1a1360c1a2a39f0b972cef2cc9ebd56025168c4137192a9321/tomli-2.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b5870b50c9db823c595983571d1296a6ff3e1b88f734a4c8f6fc6188397de005", size = 253109, upload-time = "2025-10-08T22:01:34.052Z" },+ { url = "https://files.pythonhosted.org/packages/22/0c/b4da635000a71b5f80130937eeac12e686eefb376b8dee113b4a582bba42/tomli-2.3.0-cp314-cp314-win32.whl", hash = "sha256:feb0dacc61170ed7ab602d3d972a58f14ee3ee60494292d384649a3dc38ef463", size = 97930, upload-time = "2025-10-08T22:01:35.082Z" },+ { url = "https://files.pythonhosted.org/packages/b9/74/cb1abc870a418ae99cd5c9547d6bce30701a954e0e721821df483ef7223c/tomli-2.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:b273fcbd7fc64dc3600c098e39136522650c49bca95df2d11cf3b626422392c8", size = 107964, upload-time = "2025-10-08T22:01:36.057Z" },+ { url = "https://files.pythonhosted.org/packages/54/78/5c46fff6432a712af9f792944f4fcd7067d8823157949f4e40c56b8b3c83/tomli-2.3.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:940d56ee0410fa17ee1f12b817b37a4d4e4dc4d27340863cc67236c74f582e77", size = 163065, upload-time = "2025-10-08T22:01:37.27Z" },+ { url = "https://files.pythonhosted.org/packages/39/67/f85d9bd23182f45eca8939cd2bc7050e1f90c41f4a2ecbbd5963a1d1c486/tomli-2.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f85209946d1fe94416debbb88d00eb92ce9cd5266775424ff81bc959e001acaf", size = 159088, upload-time = "2025-10-08T22:01:38.235Z" },+ { url = "https://files.pythonhosted.org/packages/26/5a/4b546a0405b9cc0659b399f12b6adb750757baf04250b148d3c5059fc4eb/tomli-2.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a56212bdcce682e56b0aaf79e869ba5d15a6163f88d5451cbde388d48b13f530", size = 268193, upload-time = "2025-10-08T22:01:39.712Z" },+ { url = "https://files.pythonhosted.org/packages/42/4f/2c12a72ae22cf7b59a7fe75b3465b7aba40ea9145d026ba41cb382075b0e/tomli-2.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5f3ffd1e098dfc032d4d3af5c0ac64f6d286d98bc148698356847b80fa4de1b", size = 275488, upload-time = "2025-10-08T22:01:40.773Z" },+ { url = "https://files.pythonhosted.org/packages/92/04/a038d65dbe160c3aa5a624e93ad98111090f6804027d474ba9c37c8ae186/tomli-2.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5e01decd096b1530d97d5d85cb4dff4af2d8347bd35686654a004f8dea20fc67", size = 272669, upload-time = "2025-10-08T22:01:41.824Z" },+ { url = "https://files.pythonhosted.org/packages/be/2f/8b7c60a9d1612a7cbc39ffcca4f21a73bf368a80fc25bccf8253e2563267/tomli-2.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8a35dd0e643bb2610f156cca8db95d213a90015c11fee76c946aa62b7ae7e02f", size = 279709, upload-time = "2025-10-08T22:01:43.177Z" },+ { url = "https://files.pythonhosted.org/packages/7e/46/cc36c679f09f27ded940281c38607716c86cf8ba4a518d524e349c8b4874/tomli-2.3.0-cp314-cp314t-win32.whl", hash = "sha256:a1f7f282fe248311650081faafa5f4732bdbfef5d45fe3f2e702fbc6f2d496e0", size = 107563, upload-time = "2025-10-08T22:01:44.233Z" },+ { url = "https://files.pythonhosted.org/packages/84/ff/426ca8683cf7b753614480484f6437f568fd2fda2edbdf57a2d3d8b27a0b/tomli-2.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:70a251f8d4ba2d9ac2542eecf008b3c8a9fc5c3f9f02c56a9d7952612be2fdba", size = 119756, upload-time = "2025-10-08T22:01:45.234Z" },+ { url = "https://files.pythonhosted.org/packages/77/b8/0135fadc89e73be292b473cb820b4f5a08197779206b33191e801feeae40/tomli-2.3.0-py3-none-any.whl", hash = "sha256:e95b1af3c5b07d9e643909b5abbec77cd9f1217e6d0bca72b0234736b9fb1f1b", size = 14408, upload-time = "2025-10-08T22:01:46.04Z" },][[package]]@@ -3138,15 +3133,15 @@ bedrock-runtime = [[[package]]name = "types-aiobotocore"-version = "2.24.3"+version = "2.25.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "botocore-stubs" },{ name = "typing-extensions", marker = "python_full_version < '3.12'" },]-sdist = { url = "https://files.pythonhosted.org/packages/d4/ec/bdd9468061247fe48b5158ece8c39ed6ff21336a55bc49529e3980063d76/types_aiobotocore-2.24.3.tar.gz", hash = "sha256:b2b43b2e3102c2c4d4c3bca4a7ec97f89856cdde000355512e019e0dbdcd4a36", size = 86847, upload-time = "2025-10-08T01:43:02.483Z" }+sdist = { url = "https://files.pythonhosted.org/packages/51/0e/ce075f02b7b99cf06607c06d85f731cf70d22a628a4c05b14f8fdbd7a852/types_aiobotocore-2.25.0.tar.gz", hash = "sha256:7e5e96568935d5255095b5f8aaedc0c1b265770a260a2ab6ed7d4c7ea7fe8228", size = 86814, upload-time = "2025-10-11T01:37:34.424Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/23/ab/95b02273a34ea6a359d6488daff3061fa8915fb021d18ab0d61620dcfc09/types_aiobotocore-2.24.3-py3-none-any.whl", hash = "sha256:313250d1dfc8392248a0083eb158f39616fe7e6593ceca7d670ec6da5c02e9a8", size = 54296, upload-time = "2025-10-08T01:42:58.977Z" },+ { url = "https://files.pythonhosted.org/packages/4e/fb/1b78fc895712e48d9a3b94aeee9aed31eee5682a447844f8873f036186fb/types_aiobotocore-2.25.0-py3-none-any.whl", hash = "sha256:7a9efa7e8240774546b95ae0db3b6f7cbf9f05c89db317612ec1d678649a88ff", size = 54291, upload-time = "2025-10-11T01:37:29.108Z" },][[package]]@@ -3175,41 +3170,20 @@ wheels = [[[package]]name = "types-aiofiles"-version = "24.1.0.20250822"+version = "25.1.0.20251011"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/19/48/c64471adac9206cc844afb33ed311ac5a65d2f59df3d861e0f2d0cad7414/types_aiofiles-24.1.0.20250822.tar.gz", hash = "sha256:9ab90d8e0c307fe97a7cf09338301e3f01a163e39f3b529ace82466355c84a7b", size = 14484, upload-time = "2025-08-22T03:02:23.039Z" }+sdist = { url = "https://files.pythonhosted.org/packages/84/6c/6d23908a8217e36704aa9c79d99a620f2fdd388b66a4b7f72fbc6b6ff6c6/types_aiofiles-25.1.0.20251011.tar.gz", hash = "sha256:1c2b8ab260cb3cd40c15f9d10efdc05a6e1e6b02899304d80dfa0410e028d3ff", size = 14535, upload-time = "2025-10-11T02:44:51.237Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/bc/8e/5e6d2215e1d8f7c2a94c6e9d0059ae8109ce0f5681956d11bb0a228cef04/types_aiofiles-24.1.0.20250822-py3-none-any.whl", hash = "sha256:0ec8f8909e1a85a5a79aed0573af7901f53120dd2a29771dd0b3ef48e12328b0", size = 14322, upload-time = "2025-08-22T03:02:21.918Z" },+ { url = "https://files.pythonhosted.org/packages/71/0f/76917bab27e270bb6c32addd5968d69e558e5b6f7fb4ac4cbfa282996a96/types_aiofiles-25.1.0.20251011-py3-none-any.whl", hash = "sha256:8ff8de7f9d42739d8f0dadcceeb781ce27cd8d8c4152d4a7c52f6b20edb8149c", size = 14338, upload-time = "2025-10-11T02:44:50.054Z" },][[package]]name = "types-awscrt"-version = "0.27.6"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/56/ce/5d84526a39f44c420ce61b16654193f8437d74b54f21597ea2ac65d89954/types_awscrt-0.27.6.tar.gz", hash = "sha256:9d3f1865a93b8b2c32f137514ac88cb048b5bc438739945ba19d972698995bfb", size = 16937, upload-time = "2025-08-13T01:54:54.659Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/ac/af/e3d20e3e81d235b3964846adf46a334645a8a9b25a0d3d472743eb079552/types_awscrt-0.27.6-py3-none-any.whl", hash = "sha256:18aced46da00a57f02eb97637a32e5894dc5aa3dc6a905ba3e5ed85b9f3c526b", size = 39626, upload-time = "2025-08-13T01:54:53.454Z" },-]--[[package]]-name = "types-beautifulsoup4"-version = "4.12.0.20250516"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "types-html5lib" },-]-sdist = { url = "https://files.pythonhosted.org/packages/6d/d1/32b410f6d65eda94d3dfb0b3d0ca151f12cb1dc4cef731dcf7cbfd8716ff/types_beautifulsoup4-4.12.0.20250516.tar.gz", hash = "sha256:aa19dd73b33b70d6296adf92da8ab8a0c945c507e6fb7d5db553415cc77b417e", size = 16628, upload-time = "2025-05-16T03:09:09.93Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/7c/79/d84de200a80085b32f12c5820d4fd0addcbe7ba6dce8c1c9d8605e833c8e/types_beautifulsoup4-4.12.0.20250516-py3-none-any.whl", hash = "sha256:5923399d4a1ba9cc8f0096fe334cc732e130269541d66261bb42ab039c0376ee", size = 16879, upload-time = "2025-05-16T03:09:09.051Z" },-]--[[package]]-name = "types-html5lib"-version = "1.1.11.20250917"+version = "0.28.1"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/68/4b/a970718e8bd9324ee8fb8eaf02ff069f6d03c20d4523bb4232892ecc3d06/types_html5lib-1.1.11.20250917.tar.gz", hash = "sha256:7b52743377f33f9b4fd7385afbd2d457b8864ee51f90ff2a795ad9e8c053373a", size = 16868, upload-time = "2025-09-17T02:47:41.18Z" }+sdist = { url = "https://files.pythonhosted.org/packages/60/19/a3a6377c9e2e389c1421c033a1830c29cac08f2e1e05a082ea84eb22c75f/types_awscrt-0.28.1.tar.gz", hash = "sha256:66d77ec283e1dc907526a44511a12624118723a396c36d3f3dd9855cb614ce14", size = 17410, upload-time = "2025-10-11T21:55:07.443Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/78/8a/da91a9c64dcb5e69beb567519857411996d8ecae9f6f128bcef8260e7a8d/types_html5lib-1.1.11.20250917-py3-none-any.whl", hash = "sha256:b294fd06d60da205daeb2f615485ca4d475088d2eff1009cf427f4a80fcd5346", size = 22908, upload-time = "2025-09-17T02:47:40.39Z" },+ { url = "https://files.pythonhosted.org/packages/ea/c7/0266b797d19b82aebe0e177efe35de7aabdc192bc1605ce3309331f0a505/types_awscrt-0.28.1-py3-none-any.whl", hash = "sha256:d88f43ef779f90b841ba99badb72fe153077225a4e426ae79e943184827b4443", size = 41851, upload-time = "2025-10-11T21:55:06.235Z" },][[package]]@@ -3261,34 +3235,13 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/bd/e0/1eed384f02555dde685fff1a1ac805c1c7dcb6dd019c916fe659b1c1f9ec/types_pyyaml-6.0.12.20250915-py3-none-any.whl", hash = "sha256:e7d4d9e064e89a3b3cae120b4990cd370874d2bf12fa5f46c97018dd5d3c9ab6", size = 20338, upload-time = "2025-09-15T03:00:59.218Z" },]-[[package]]-name = "types-requests"-version = "2.32.4.20250913"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "urllib3" },-]-sdist = { url = "https://files.pythonhosted.org/packages/36/27/489922f4505975b11de2b5ad07b4fe1dca0bca9be81a703f26c5f3acfce5/types_requests-2.32.4.20250913.tar.gz", hash = "sha256:abd6d4f9ce3a9383f269775a9835a4c24e5cd6b9f647d64f88aa4613c33def5d", size = 23113, upload-time = "2025-09-13T02:40:02.309Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/2a/20/9a227ea57c1285986c4cf78400d0a91615d25b24e257fd9e2969606bdfae/types_requests-2.32.4.20250913-py3-none-any.whl", hash = "sha256:78c9c1fffebbe0fa487a418e0fa5252017e9c60d1a2da394077f1780f655d7e1", size = 20658, upload-time = "2025-09-13T02:40:01.115Z" },-]-[[package]]name = "types-s3transfer"-version = "0.13.1"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/a5/c5/23946fac96c9dd5815ec97afd1c8ad6d22efa76c04a79a4823f2f67692a5/types_s3transfer-0.13.1.tar.gz", hash = "sha256:ce488d79fdd7d3b9d39071939121eca814ec65de3aa36bdce1f9189c0a61cc80", size = 14181, upload-time = "2025-08-31T16:57:06.93Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/8e/dc/b3f9b5c93eed6ffe768f4972661250584d5e4f248b548029026964373bcd/types_s3transfer-0.13.1-py3-none-any.whl", hash = "sha256:4ff730e464a3fd3785b5541f0f555c1bd02ad408cf82b6b7a95429f6b0d26b4a", size = 19617, upload-time = "2025-08-31T16:57:05.73Z" },-]--[[package]]-name = "types-toml"-version = "0.10.8.20240310"+version = "0.14.0"source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/86/47/3e4c75042792bff8e90d7991aa5c51812cc668828cc6cce711e97f63a607/types-toml-0.10.8.20240310.tar.gz", hash = "sha256:3d41501302972436a6b8b239c850b26689657e25281b48ff0ec06345b8830331", size = 4392, upload-time = "2024-03-10T02:18:37.518Z" }+sdist = { url = "https://files.pythonhosted.org/packages/8e/9b/8913198b7fc700acc1dcb84827137bb2922052e43dde0f4fb0ed2dc6f118/types_s3transfer-0.14.0.tar.gz", hash = "sha256:17f800a87c7eafab0434e9d87452c809c290ae906c2024c24261c564479e9c95", size = 14218, upload-time = "2025-10-11T21:11:27.892Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/da/a2/d32ab58c0b216912638b140ab2170ee4b8644067c293b170e19fba340ccc/types_toml-0.10.8.20240310-py3-none-any.whl", hash = "sha256:627b47775d25fa29977d9c70dc0cbab3f314f32c8d8d0c012f2ef5de7aaec05d", size = 4777, upload-time = "2024-03-10T02:18:36.568Z" },+ { url = "https://files.pythonhosted.org/packages/92/c3/4dfb2e87c15ca582b7d956dfb7e549de1d005c758eb9a305e934e1b83fda/types_s3transfer-0.14.0-py3-none-any.whl", hash = "sha256:108134854069a38b048e9b710b9b35904d22a9d0f37e4e1889c2e6b58e5b3253", size = 19697, upload-time = "2025-10-11T21:11:26.749Z" },][[package]]