Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/doc (#253)
Thomas Hebrard472fedf3ea8f
diff --git a/.cursor/rules/base_models.mdc b/.cursor/rules/base_models.mdcdeleted file mode 100644index 777b15464..000000000--- a/.cursor/rules/base_models.mdc+++ /dev/null@@ -1,14 +0,0 @@-----description:-globs:-alwaysApply: true-----Rule to create BaseModels:--- Respect Pydantic v2 standards-- Keep models focused and single-purpose-- Use descriptive field names-- Use type hints for all fields-- Document complex validations-- Use Optional[] for nullable fields-- Use Field(default_factory=...) for mutable defaultsdiff --git a/.cursor/rules/best_practices.mdc b/.cursor/rules/best_practices.mdcdeleted file mode 100644index caea92693..000000000--- a/.cursor/rules/best_practices.mdc+++ /dev/null@@ -1,77 +0,0 @@-----description:-globs:-alwaysApply: false-----# Best Practices Guide--This document outlines the core best practices and patterns used in our codebase.--## Type Hints--1. **Always Use Type Hints**- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious--2. **StrEnum**- If you want to use StrEnum, import it from `pipelex.types`- ```python- from pipelex.types import StrEnum- ```--## Factory Pattern--1. **Use Factory Pattern for Object Creation**- - Create factories when dealing with multiple implementations--## Documentation--1. **Docstring Format**- - Quick description of the function/class- - List args and their types- - Document return values- - Example:- ```python- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:- """Process and resize an image.-- Args:- image_path: Path to the source image- size: Tuple of (width, height) for resizing-- Returns:- Processed image as bytes- """- pass- ```--2. **Class Documentation**- - Document class purpose and behavior- - Include examples if complex- ```python- class ImageProcessor:- """Handles image processing operations.-- Provides methods for resizing, converting, and optimizing images.- """- ```--## Custom Exceptions--1. **Graceful Error Handling**- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to custom ones- ```python- try:- from fal_client import AsyncClient as FalAsyncClient- except ImportError as exc:- raise MissingDependencyError(- "fal-client", "fal",- "The fal-client SDK is required to use FAL models."- ) from exc- ```--## Pipelines--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/coding_standards.mdc b/.cursor/rules/coding_standards.mdcnew file mode 100644index 000000000..c94a3de96--- /dev/null+++ b/.cursor/rules/coding_standards.mdc@@ -0,0 +1,124 @@+---+description:+globs:+alwaysApply: true+---+# Coding Standards & Best Practices++This document outlines the core coding standards, best practices, and quality control procedures for the codebase.++## Type Hints++1. **Always Use Type Hints**+ - Every function parameter must be typed+ - Every function return must be typed+ - Use type hints for all variables where type is not obvious+ - Use types with Uppercase first letter (Dict[], List[], etc.)++2. **StrEnum**+ - Import from `pipelex.types`:+ ```python+ from pipelex.types import StrEnum+ ```++## BaseModel Standards++- Respect Pydantic v2 standards+- Keep models focused and single-purpose+- Use descriptive field names+- Use type hints for all fields+- Document complex validations+- Use Optional[] for nullable fields+- Use Field(default_factory=...) for mutable defaults++## Factory Pattern++- Use Factory Pattern for object creation when dealing with multiple implementations++## Documentation++1. **Docstring Format**+ ```python+ def process_image(image_path: str, size: Tuple[int, int]) -> bytes:+ """Process and resize an image.++ Args:+ image_path: Path to the source image+ size: Tuple of (width, height) for resizing++ Returns:+ Processed image as bytes+ """+ pass+ ```++2. **Class Documentation**+ ```python+ class ImageProcessor:+ """Handles image processing operations.++ Provides methods for resizing, converting, and optimizing images.+ """+ ```++## Error Handling++1. **Graceful Error Handling**+ - Use try/except blocks with specific exceptions+ - Convert third-party exceptions to custom ones+ ```python+ try:+ from fal_client import AsyncClient as FalAsyncClient+ except ImportError as exc:+ raise MissingDependencyError(+ "fal-client", "fal",+ "The fal-client SDK is required to use FAL models."+ ) from exc+ ```++## Code Quality Checks++### Linting and Type Checking++Before finalizing a task, run:+```bash+make fix-unused-imports+make check+```++This runs multiple code quality tools:+- Pyright: Static type checking+- Ruff: Fast Python linter+- Mypy: Static type checker++Always fix any issues reported by these tools before proceeding.++### Running Tests++1. **Quick Test Run** (no LLM/image generation):+ ```bash+ make tp+ ```+ Runs tests with markers: `(dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)`++2. **Specific Tests**:+ ```bash+ make tp TEST=TestClassName+ # or+ make tp TEST=test_function_name+ ```+ Note: Matches names starting with the provided string.++**Important**: Never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-imgg` - these use costly inference.++## Pipelines++- All pipeline definitions go in `pipelex/libraries/pipelines/`+- Always validate pipelines after creation/edit with `make validate`.+ Iterate if there are errors.++## Project Structure++- **Pipelines**: `pipelex/libraries/pipelines/`+- **Tests**: `tests/` directory+- **Documentation**: `docs/` directory\ No newline at end of filediff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdcindex f8a11cc97..e814eb827 100644--- a/.cursor/rules/llms.mdc+++ b/.cursor/rules/llms.mdc@@ -1,6 +1,5 @@----description:-globs:+globs: *.plx,*.tomlalwaysApply: false---# Rules to choose LLM models used in PipeLLMs.@@ -39,7 +38,7 @@ Here is an example of using an llm_handle to specify which LLM to use in a PipeL```plx[pipe.hello_world]type = "PipeLLM"-description = "Write text about Hello World."+definition = "Write text about Hello World."output = "Text"llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }prompt_template = """@@ -64,7 +63,7 @@ The interest is that these presets can be used to set the LLM choice in a PipeLL```plx[pipe.extract_invoice]type = "PipeLLM"-description = "Extract invoice information from an invoice text transcript"+definition = "Extract invoice information from an invoice text transcript"inputs = { invoice_text = "InvoiceText" }output = "Invoice"llm = "llm_to_extract_invoice"@@ -81,4 +80,4 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.-You can override the predefined llm presets in [overrides.toml](mdc:pipelex/libraries/llm_deck/overrides.toml).+You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).diff --git a/.cursor/rules/pipe-batch.mdc b/.cursor/rules/pipe-batch.mdcdeleted file mode 100644index 729c31c0f..000000000--- a/.cursor/rules/pipe-batch.mdc+++ /dev/null@@ -1,32 +0,0 @@-----description:-globs:-alwaysApply: false-----# PipeBatch Controller--The PipeBatch controller allows you to apply a pipe operation to each element in a list of inputs in parallele. It is created via a PipeSequence.--## Usage in PLX Configuration--```plx-[pipe.sequence_with_batch]-type = "Sequence"-description = "A Sequence of pipes"-inputs = { input_data = "ConceptName" }-output = "OutputConceptName"-steps = [- { pipe = "pipe_to_apply", batch_over = "input_list", batch_as = "current_item", result = "batch_results" }-]-```--## Key Parameters--- `pipe`: The pipe operation to apply to each element in the batch-- `batch_over`: The name of the list in the context to iterate over-- `batch_as`: The name to use for the current element in the pipe's context-- `result`: Where to store the results of the batch operation--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/pipe-condition.mdc b/.cursor/rules/pipe-condition.mdcdeleted file mode 100644index bfae8ac98..000000000--- a/.cursor/rules/pipe-condition.mdc+++ /dev/null@@ -1,52 +0,0 @@-----description:-globs:-alwaysApply: false-----# PipeCondition Controller--The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.--## Usage in PLX Configuration--### Basic Usage with Direct Expression--```plx-[pipe.conditional_operation]-type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."-inputs = { input_data = "CategoryInput" }-output = "native.Text"-expression = "input_data.category"--[pipe.conditional_operation.pipe_map]-small = "process_small"-medium = "process_medium"-large = "process_large"-```-or-```plx-[pipe.conditional_operation]-type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."-inputs = { input_data = "CategoryInput" }-output = "native.Text"-expression_template = "{{ input_data.category }}" # Jinja2 code--[pipe.conditional_operation.pipe_map]-small = "process_small"-medium = "process_medium"-large = "process_large"-```--## Key Parameters--- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)-- `pipe_map`: Dictionary mapping expression results to pipe codes :-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/pipe-func.mdc b/.cursor/rules/pipe-func.mdcdeleted file mode 100644index e69de29bb..000000000diff --git a/.cursor/rules/pipe-imgg.mdc b/.cursor/rules/pipe-imgg.mdcdeleted file mode 100644index b93c988bf..000000000--- a/.cursor/rules/pipe-imgg.mdc+++ /dev/null@@ -1,5 +0,0 @@-----description:-globs:-alwaysApply: false----diff --git a/.cursor/rules/pipe-llm.mdc b/.cursor/rules/pipe-llm.mdcdeleted file mode 100644index cb6bf7cad..000000000--- a/.cursor/rules/pipe-llm.mdc+++ /dev/null@@ -1,114 +0,0 @@-----description:-globs:-alwaysApply: false-----# PipeLLM Guide--## Purpose--PipeLLM is used to:-1. Generate text or objects with LLMs-2. Process images with Vision LLMs--## Basic Usage--### Simple Text Generation-```plx-[pipe.write_story]-type = "PipeLLM"-description = "Write a short story"-output = "Text"-prompt_template = """-Write a short story about a programmer.-"""-```--### Structured Data Extraction-```plx-[pipe.extract_info]-type = "PipeLLM"-description = "Extract information"-inputs = { text = "Text" }-output = "PersonInfo"-prompt_template = """-Extract person information from this text:-@text-"""-```--### Where to Put Structured Objects-Place your Pydantic models in `pipelex_libraries/pipelines/your_models.py`:--```python-from pipelex.core.stuffs.stuff_content import StructuredContent--class PersonInfo(StructuredContent): # The output models always have to be subclass of StructuredContent- name: str- age: int- email: str-```--## Advanced Features--### LLM Settings--You can specify LLM settings in two ways:--1. **Direct in the pipe**:-```plx-[pipe.analyze]-type = "PipeLLM"-description = "Analyze text"-output = "Analysis"-llm = { llm_handle = "gpt-4", temperature = 0.7 }-prompt_template = "Analyze this text"-```--2. **Using predefined settings** from `pipelex_libraries/llm_deck/base_llm_deck.toml`:-```plx-[pipe.analyze]-type = "PipeLLM"-description = "Analyze text"-output = "Analysis"-llm = "llm_for_analysis" # References a preset from llm_deck-prompt_template = "Analyze this text"-```--### System Prompts-Add system-level instructions:-```plx-[pipe.expert_analysis]-type = "PipeLLM"-description = "Expert analysis"-output = "Analysis"-system_prompt = "You are a data analysis expert"-prompt_template = "Analyze this data"-```--### Multiple Outputs-Generate multiple results:-```plx-[pipe.generate_ideas]-type = "PipeLLM"-description = "Generate ideas"-output = "Idea"-nb_output = 3 # Generate exactly 3 ideas-# OR-multiple_output = true # Let the LLM decide how many to generate-```--### Vision Tasks-Process images with VLMs:-```plx-[pipe.analyze_image]-type = "PipeLLM"-description = "Analyze image"-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in a stuff, you can add something like `{ "page.image": "Image" }-output = "ImageAnalysis"-prompt_template = "Describe what you see in this image"-```--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/pipe-ocr.mdc b/.cursor/rules/pipe-ocr.mdcdeleted file mode 100644index 1e21281aa..000000000--- a/.cursor/rules/pipe-ocr.mdc+++ /dev/null@@ -1,40 +0,0 @@-----description:-globs:-alwaysApply: false-----# PipeOCR Guide--## Purpose--Extract text and images from an image or a PDF--## Basic Usage--### Simple Text Generation-```plx-[pipe.extract_info]-type = "PipeOcr"-description = "extract the information"-inputs = { ocr_input = "PDF" } # or { ocr_input = "Image" } if its an image. This is the only input-output = "Page"-```--The output concept `Page` is a native concept, with the structure `PageContent`:-It corresponds to 1 page. Therefore, the PipeOcr is outputing a `ListContent` of `Page`--```python-class TextAndImagesContent(StuffContent):- text: Optional[TextContent]- images: Optional[List[ImageContent]]--class PageContent(StructuredContent):- text_and_images: TextAndImagesContent- page_view: Optional[ImageContent] = None-```-- `text_and_images` are the text, and the related images found in the input image or PDF.-- `page_view` is the screenshot of the whole pdf page/image.--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/pipe-parallel.mdc b/.cursor/rules/pipe-parallel.mdcdeleted file mode 100644index b93c988bf..000000000--- a/.cursor/rules/pipe-parallel.mdc+++ /dev/null@@ -1,5 +0,0 @@-----description:-globs:-alwaysApply: false----diff --git a/.cursor/rules/pipe-sequence.mdc b/.cursor/rules/pipe-sequence.mdcdeleted file mode 100644index d1ab1786d..000000000--- a/.cursor/rules/pipe-sequence.mdc+++ /dev/null@@ -1,60 +0,0 @@-----description:-globs:-alwaysApply: false-----# PipeSequence Guide--## Purpose-PipeSequence executes multiple pipes in a defined order, where each step can use results from previous steps.--## Basic Structure-```plx-[pipe.your_sequence_name]-type = "PipeSequence"-description = "Description of what this sequence does"-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps-output = "OutputType"-steps = [- { pipe = "first_pipe", result = "first_result" },- { pipe = "second_pipe", result = "second_result" },- { pipe = "final_pipe", result = "final_result" }-]-```--## Key Components--1. **Steps Array**: List of pipes to execute in sequence- - `pipe`: Name of the pipe to execute- - `result`: Name to assign to the pipe's output that will be in the working memory--2. **Working Memory**: Each step can access:- - Original sequence inputs- - Results from previous steps- - Use the result names in subsequent steps- - The output concept of the last step should be the output of the sequence--## Using PipeBatch in Steps--You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:--```plx-steps = [- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"- }-]-```--1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.- - Must be a `ListContent` type containing the items to process- - Can reference inputs or results from previous steps--2. **batch_as**: Defines the name that will be used to reference the current item being processed- - This name can be used in the pipe's input mappings- - Makes each item from the batch available as a single element--The result of a batched step will be a `ListContent` containing the outputs from processing each item.--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/pipelex.mdc b/.cursor/rules/pipelex.mdcindex 3f578db37..4c0f29238 100644--- a/.cursor/rules/pipelex.mdc+++ b/.cursor/rules/pipelex.mdc@@ -4,7 +4,7 @@ alwaysApply: true# Pipeline Guide- Always first write your "plan" in natural langage, then transcribe it in pipelex.-- You should ALWAYS RUN the terminal command `pipelex validate all -c pipelex/libraries` when you are writing a `.toml` file. It will ensure the pipe is runnable. If not, iterate.+- You should ALWAYS RUN the terminal command `make validate` when you are writing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)# Pipeline Structure Guide@@ -46,7 +46,7 @@ yes```plx[pipe.your_pipe_name]type = "PipeLLM"-description = "A description of what your pipe does"+definition = "A description of what your pipe does"inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }output = "ConceptName"```@@ -62,7 +62,7 @@ But it should be:```plx[pipe.your_pipe_name]type = "PipeSequence"-description = "....."+definition = "....."```The pipes will all have at least this base structure.@@ -155,7 +155,7 @@ PipeSequence executes multiple pipes in a defined order, where each step can use```plx[pipe.your_sequence_name]type = "PipeSequence"-description = "Description of what this sequence does"+definition = "Description of what this sequence does"inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate stepsoutput = "OutputType"steps = [@@ -203,7 +203,7 @@ The PipeCondition controller allows you to implement conditional logic in your p```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+definition = "A conditonal pipe to decide wheter..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"@@ -217,7 +217,7 @@ or```plx[pipe.conditional_operation]type = "PipeCondition"-description = "A conditonal pipe to decide wheter..."+definition = "A conditonal pipe to decide wheter..."inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}" # Jinja2 code@@ -245,7 +245,7 @@ The PipeBatch controller allows you to apply a pipe operation to each element in```plx[pipe.sequence_with_batch]type = "PipeSequence"-description = "A Sequence of pipes"+definition = "A Sequence of pipes"inputs = { input_data = "ConceptName" }output = "OutputConceptName"steps = [@@ -274,7 +274,7 @@ PipeLLM is used to:```plx[pipe.write_story]type = "PipeLLM"-description = "Write a short story"+definition = "Write a short story"output = "Text"prompt_template = """Write a short story about a programmer.@@ -285,7 +285,7 @@ Write a short story about a programmer.```plx[pipe.extract_info]type = "PipeLLM"-description = "Extract information"+definition = "Extract information"inputs = { text = "Text" }output = "PersonInfo"prompt_template = """@@ -299,7 +299,7 @@ Add system-level instructions:```plx[pipe.expert_analysis]type = "PipeLLM"-description = "Expert analysis"+definition = "Expert analysis"output = "Analysis"system_prompt = "You are a data analysis expert"prompt_template = "Analyze this data"@@ -310,7 +310,7 @@ Generate multiple results:```plx[pipe.generate_ideas]type = "PipeLLM"-description = "Generate ideas"+definition = "Generate ideas"output = "Idea"nb_output = 3 # Generate exactly 3 ideas# OR@@ -322,7 +322,7 @@ Process images with VLMs:```plx[pipe.analyze_image]type = "PipeLLM"-description = "Analyze image"+definition = "Analyze image"inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in a stuff, you can add something like `{ "page.image": "Image" }output = "ImageAnalysis"prompt_template = "Describe what you see in this image"@@ -340,7 +340,7 @@ Extract text and images from an image or a PDF```plx[pipe.extract_info]type = "PipeOcr"-description = "extract the information"+definition = "extract the information"inputs = { ocr_input = "PDF" } # or { ocr_input = "Image" } if its an image. This is the only inputoutput = "Page"```@@ -515,6 +515,6 @@ So here are a few concrete examples of calls to execute_pipeline with various wa)```-ALWAYS RUN `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.Then, create an example file to run the pipeline in the `examples` folder.But don't write documentation unless asked explicitly to.diff --git a/.cursor/rules/pipes.mdc b/.cursor/rules/pipes.mdcdeleted file mode 100644index e7b81982f..000000000--- a/.cursor/rules/pipes.mdc+++ /dev/null@@ -1,81 +0,0 @@-----description: Build a Pipelex pipeline-globs:-alwaysApply: false-----# Pipeline Guide--- Always first write your "plan" in natural langage, then transcribe it in pipelex.-- Always run `pipelex validate all -c pipelex/libraries` before finishing a task. It will ensure the pipe is runnable--# Pipeline Structure Guide--## Pipeline File Naming-- Files must be `.plx` for pipelines-- Files must be `.py` for structures-- Use descriptive names in `snake_case`--## Pipeline File Structure-A pipeline file has three main sections:-1. Domain statement-2. Concept definitions-3. Pipe definitions--### Domain Statement-```plx-domain = "domain_name"-definition = "Description of the domain" # Optional-```-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.--### Concept Definitions-```plx-[concept]-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output-```--Important Rules:-- Use PascalCase for concept names-- Never use plurals (no "Stories", use "Story")-- Avoid adjectives (no "LargeText", use "Text")-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)--### Pipe Definitions--## Pipe Base Structure--```plx-[pipe.your_pipe_name]-type = "PipeLLM"-description = "A description of what your pipe does"-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }-output = "ConceptName"-```--The pipes will all have at least this base structure.-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output. Look on how to do that here: [structures.mdc](structures.mdc)--## Pipe Controllers and Pipe Operator--Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:--1. **Controllers** - For flow control:- - `PipeSequence` - See [pipe-sequence.mdc](pipe-sequence.mdc): For creating a sequence of multiple steps- - `PipeCondition` - See [pipe-condition.mdc](pipe-condition.mdc): If the next pipe depends of the expression of a stuff in the working memory- - `PipeParallel` - See [pipe-parallel.mdc](pipe-parallel.mdc): For parallelizing pipes- - `PipeBatch` - See [pipe-batch.mdc](pipe-batch.mdc): For running pipes in Batch over a ListContent--2. **Operators** - For specific tasks:- - `PipeLLM` - See [pipe-llm.mdc](pipe-llm.mdc): Generate Text and Objects (include Vision LLM)- - `PipeOcr` - See [pipe-ocr.mdc](pipe-ocr.mdc): OCR Pipe- - `PipeImgGen` - See [pipe-imgg.mdc](pipe-imgg.mdc): Generate Images- - `PipeFunc` - See [pipe-func.mdc](pipe-func.mdc): For running classic python scripts--When building a pipeline:-1. Always define controllers first, then operators-2. Each pipe must have a clear, single responsibility--# Important tip--Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.\ No newline at end of filediff --git a/.cursor/rules/prompt-templates.mdc b/.cursor/rules/prompt-templates.mdcdeleted file mode 100644index a3ab438e4..000000000--- a/.cursor/rules/prompt-templates.mdc+++ /dev/null@@ -1,75 +0,0 @@-----description:-globs: pipelex/libraries/pipelines/**/*.plx-alwaysApply: false-----This rule explains how to write prompt templates in PipeLLM definitions.--## Insert stuff inside a tagged block--If the inserted text is supposedly long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.--Example template:-```plx-prompt_template = """-Match the expense with its corresponding invoice:--@expense--@invoices-"""-```-In this example, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.--**DO NOT write things like "Here is the expense: @expense".**-**DO write simply "@expense" alone in an isolated line.**--## Insert stuff inside a tagged block only if it exists (optional)--If you want to conditionally insert a block only when the variable exists and is not empty, use the "@?" prefix. This is useful for optional fields that may or may not be present in your data.--Example template:-```plx-prompt_template = """-User Profile:--@name--@?bio--@?contact_info--Please summarize this profile.-"""-```-In this example, the bio and contact_info will only be inserted if they exist. If they don't exist or are empty, those sections will be completely omitted from the rendered template.--**DO use "@?optional_field" when the field might not always be present.**-**DO NOT use "@optional_field" for fields that might be missing - this would cause an error.**--## Insert stuff inline--If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.--Example template:-```plx-prompt_template = """-Your goal is to summarize everything related to $topic in the provided text:--@text--Please provide only the summary, with no additional text or explanations.-Your summary should not be longer than 2 sentences.-"""-```--Here, $topic will be inserted inline, whereas @text will be a a delimited block.-Be sure to make the proper choice of prefix for each insertion.--**DO NOT write "$topic" alone in an isolated line.**-**DO write things like "Write an essay about $topic" included in an actual sentence.**--## Style rule--If you need to add more complexe jinja2 template, we support the classic double curly brackets `{{ }}`-diff --git a/.cursor/rules/standards.mdc b/.cursor/rules/standards.mdcdeleted file mode 100644index 19972fed3..000000000--- a/.cursor/rules/standards.mdc+++ /dev/null@@ -1,61 +0,0 @@-----description:-globs:-alwaysApply: true-----# Coding Standards--This document outlines the coding standards and quality control procedures that must be followed when contributing to this project.--## Style--Always use type hints. Use the types with Uppercase first letter for types like Dict[], List[] etc.--## Code Quality Checks--### Linting and Type Checking--Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:--```bash-make fix-unused-imports-make check-```--This command runs multiple code quality tools:-- Pyright: Static type checking-- Ruff: Fast Python linter-- Mypy: Static type checker--Always fix any issues reported by these tools before proceeding.--### Running Tests--We have several make commands for running tests:--1. `make tp`: Runs all tests with these markers:- ```- (dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)- ```- Use this for quick test runs that don't require LLM or image generation.--2. To run specific tests:- ```bash- make tp TEST=TestClassName- # or- make tp TEST=test_function_name- ```- It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.--Note: never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-imgg`: these all use inference which is costly.--## Important Project Directories--### Pipelines Directory-- All pipeline definitions go in `pipelex/libraries/pipelines/`--### Tests Directory-- All tests are located in the `tests/` directory--### Documentation Directory-- All documentation is located in the `docs/` directorydiff --git a/.cursor/rules/structures.mdc b/.cursor/rules/structures.mdcdeleted file mode 100644index 8de441eeb..000000000--- a/.cursor/rules/structures.mdc+++ /dev/null@@ -1,62 +0,0 @@-----description:-globs: pipelex/libraries/pipelines/**/*.py-alwaysApply: false-----Rules to write structure classes for concepts used in pipes with structured generations.-In particular, these structures are generated by PipeLLM using structured generation.--# Structured Models Rules--## Model Location and Registration--- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`-- Models must inherit from `StructuredContent` or appropriate content type--## Model Structure--Concepts and their structure classes are meant to indicate an idea.-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.--```python-from datetime import datetime-from typing import List, Optional-from pydantic import BaseModel, Field--from pipelex.core.stuffs.stuff_content import StructuredContent--class YourModel(StructuredContent): # Always be a subclass of StructuredContent- # Required fields- field1: str- field2: int-- # Optional fields with defaults- field3: Optional[str] = Field(None, "Description of field3")- field4: List[str] = Field(default_factory=list)-- # Date fields should remove timezone- date_field: Optional[datetime] = None-- @field_validator("date_field")- @classmethod- def remove_tzinfo(cls, v: Optional[datetime]) -> Optional[datetime]:- if v is not None:- return v.replace(tzinfo=None)- return v-```-## Usage--Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.--Structure classes defined within `pipelex/libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.---## Best Practices for structures--- Respect Pydantic v2 standards-- Use type hints for all fields-- Use `Field` declaration and write the description-- Use Pydantic validators for data cleaning/validation-- Remove timezone info from datetime fields-- Respect rules of @base_models.-diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdcindex 8c15b54b2..575544a6f 100644--- a/.cursor/rules/tdd.mdc+++ b/.cursor/rules/tdd.mdc@@ -18,10 +18,10 @@ This document outlines our test-driven development (TDD) process and the tools a- Keep it simple - don't write more than needed3. **Run Linting and Type Checking**-[standards.mdc](standards.mdc)+[coding_standards.mdc](coding_standards.mdc)4. **Refactor if needed**-If the code needs refactoring, with the best practices [best_practices.mdc](best_practices.mdc)+If the code needs refactoring, with the best practices [coding_standards.mdc](coding_standards.mdc)5. **Validate tests**diff --git a/AGENTS.md b/AGENTS.mdindex 3077b4310..9725b0307 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -30,7 +30,7 @@ This python >=3.10 code is in the `pipelex` directory.## Testing- Always test with `make codex-tests` -> it runs pytest on our `tests/` directory using proper settings-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood)+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood) in the python environment.## PR Instructionsdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 11c3885b6..653a09b7f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,10 @@# Changelog+## [v0.9.1] - 2025-09-05++### Fixed+- Fixed all inconsistencies in the documentation.+## [v0.9.0] - 2025-09-02### Refactodiff --git a/CLAUDE.md b/CLAUDE.mdindex 9da6c7969..f2d28e1e2 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -82,7 +82,7 @@ class TestFooBar:- Always test with `make t` -> it runs pytest using proper settings- If some pytest tests fail, run pytest on the failed ones with the required verbosity to diagnose the issue-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood)+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood) in the python environment.## PR Instructionsdiff --git a/README.md b/README.mdindex 14803d455..287554677 100644--- a/README.md+++ b/README.md@@ -73,7 +73,7 @@ PurchaseDocumentText = "Transcript of a receipt, invoice, or order confirmation"[pipe.extract_buyer]type = "PipeLLM"-description = "Extract buyer from purchase document"+definition = "Extract buyer from purchase document"inputs = { purchase_document_text = "PurchaseDocumentText" }output = "Buyer"llm = "llm_to_extract_info"@@ -174,7 +174,7 @@ WritingStyle = "A style of writing"[pipe][pipe.analyze_tweet]type = "PipeLLM"-description = "Analyze the draft tweet and identify areas for improvement"+definition = "Analyze the draft tweet and identify areas for improvement"inputs = { draft_tweet = "DraftTweet" }output = "TweetAnalysis"llm = "llm_for_writing_analysis"@@ -203,7 +203,7 @@ without providing an actual rewrite[pipe.optimize_tweet]type = "PipeLLM"-description = "Optimize the tweet based on the analysis"+definition = "Optimize the tweet based on the analysis"inputs = { draft_tweet = "DraftTweet", tweet_analysis = "TweetAnalysis", writing_style = "WritingStyle" }output = "OptimizedTweet"llm = "llm_for_social_post_writing"@@ -238,7 +238,7 @@ Keep the core meaning of the original tweet.[pipe.optimize_tweet_sequence]type = "PipeSequence"-description = "Analyze and optimize a tech tweet in sequence"+definition = "Analyze and optimize a tech tweet in sequence"inputs = { draft_tweet = "DraftTweet", writing_style = "WritingStyle" }output = "OptimizedTweet"steps = [diff --git a/docs/pages/advanced-customization/index.md b/docs/pages/advanced-customization/index.mdindex 33fb3e03c..f68371a1d 100644--- a/docs/pages/advanced-customization/index.md+++ b/docs/pages/advanced-customization/index.md@@ -14,6 +14,7 @@ There are two main ways to inject custom implementations:from pipelex import Pipelexpipelex = Pipelex(+ config_dir_path="./pipelex_libraries",template_provider=MyTemplateProvider(),llm_model_provider=MyLLMProvider(),inference_manager=MyInferenceManager(),@@ -76,13 +77,13 @@ Pipelex supports injection of the following components:1. **Template Provider** (`TemplateLibrary`)- - Protocol: `TemplateLibraryProtocol`+ - Protocol: `TemplateProviderAbstract`- Default: `TemplateLibrary`- [Details](template-provider-injection.md)2. **LLM Model Provider** (`LLMModelLibrary`)- - Protocol: `LLMModelLibraryProtocol`+ - Protocol: `LLMModelProviderAbstract`- Default: `LLMModelLibrary`- [Details](llm-model-provider-injection.md)@@ -112,7 +113,7 @@ Pipelex supports injection of the following components:7. **Secrets Provider** (`EnvSecretsProvider`)- - Protocol: `SecretsProviderProtocol`+ - Protocol: `SecretsProviderAbstract`- Default: `EnvSecretsProvider`- [Details](secrets-provider-injection.md)diff --git a/docs/pages/advanced-customization/storage-provider-ingestion.md b/docs/pages/advanced-customization/storage-provider-injection.mdsimilarity index 100%rename from docs/pages/advanced-customization/storage-provider-ingestion.mdrename to docs/pages/advanced-customization/storage-provider-injection.mddiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.mdindex 0ccc8f508..37321a0ac 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md@@ -65,7 +65,7 @@ Here's how to use these configurations in your pipelines:```plx[pipe.generate_response]type = "PipeLLM"-description = "Generate a creative response"+definition = "Generate a creative response"inputs = { question = "Question" }output = "Response"llm = {@@ -81,7 +81,7 @@ Generate a creative response to this question:[pipe.extract_weather_data]type = "PipeLLM"-description = "Extract structured weather data from text"+definition = "Extract structured weather data from text"inputs = { text = "Text" }output = "WeatherData"llm = "llm_to_extract" # Using a presetdiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.mdindex 212b84431..8b8b0234b 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md@@ -55,7 +55,7 @@ Group concepts that naturally belong together in the same domain. A domain acts```plx# pipelex_libraries/pipelines/finance.plxdomain = "finance"-description = "Financial document processing"+definition = "Financial document processing"[concept]Invoice = "A commercial document issued by a seller to a buyer"@@ -221,7 +221,7 @@ For example, a pipe that processes `Document` can also process `Contract` or `Em```plx[pipe.extract_key_points]type = "PipeLLM"-description = "Extract main points from any document"+definition = "Extract main points from any document"inputs = { doc = "Document" } # Can accept Document, Contract, or EmploymentContractoutput = "KeyPoints"```@@ -308,21 +308,21 @@ Native concepts can be used directly in your pipeline definitions without any ad```plx[pipe.analyze_document]type = "PipeLLM"-description = "Analyze a PDF document"+definition = "Analyze a PDF document"inputs = { document = "PDF" }output = "Text"prompt_template = "Analyze this document and provide a summary"[pipe.process_image]type = "PipeLLM"-description = "Describe an image"+definition = "Describe an image"inputs = { photo = "Image" }output = "Text"prompt_template = "Describe what you see in this image"[pipe.extract_from_page]type = "PipeLLM"-description = "Extract information from a document page"+definition = "Extract information from a document page"inputs = { page_content = "Page" }output = "ExtractedInfo"prompt_template = "Extract key information from this page content"diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.mdindex 1d2523434..6e5217159 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md@@ -37,7 +37,7 @@ Tagline = "A catchy marketing tagline"# 2. Define the pipe that does the work[pipe.generate_tagline]type = "PipeLLM"-description = "Generate a catchy tagline for a product"+definition = "Generate a catchy tagline for a product"inputs = { description = "ProductDescription" }output = "Tagline"prompt_template = """@@ -72,7 +72,7 @@ Tagline = "A catchy marketing tagline"# 2. Define operator pipes[pipe.extract_keywords]type = "PipeLLM"-description = "Extract keywords from a product description"+definition = "Extract keywords from a product description"inputs = { description = "ProductDescription" }output = "Keyword"multiple_output = true@@ -86,7 +86,7 @@ Focus on features, benefits, and unique selling points.[pipe.generate_tagline_from_keywords]type = "PipeLLM"-description = "Generate a tagline from keywords"+definition = "Generate a tagline from keywords"inputs = { keywords = "Keyword" }output = "Tagline"prompt_template = """@@ -100,7 +100,7 @@ The tagline should be memorable, concise (under 10 words), and highlight the mai# 3. This controller pipe defines the two-step pipeline[pipe.description_to_tagline]type = "PipeSequence"-description = "From product description to tagline"+definition = "From product description to tagline"inputs = { description = "ProductDescription" }output = "Tagline"steps = [diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.mdindex cd405ee91..55c4cfc63 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md@@ -18,7 +18,7 @@ Answer = "A response to a question"[pipe][pipe.answer_question]type = "PipeLLM"-description = "Answer a question"+definition = "Answer a question"inputs = { question = "tutorial.Question" }output = "tutorial.Answer"prompt_template = """@@ -64,7 +64,7 @@ A domain consists of:[pipe][pipe.summarize_invoice]type = "PipeLLM"- description = "Summarize an invoice to extract key information"+ definition = "Summarize an invoice to extract key information"inputs = { invoice = "finance.Invoice" }output = "finance.InvoiceSummary"```@@ -115,8 +115,8 @@ When using a domain in your code, you refer to concepts with `domain.ConceptNamefrom pipelex.core.stuffs.stuff_factory import StuffFactory# The concept_code combines domain and concept names-invoice_stuff = StuffFactory.make_stuff(- concept_str="finance.Invoice", # domain.ConceptName+invoice_stuff = StuffFactory.make_from_concept_string(+ concept_string="finance.Invoice", # domain.ConceptNamename="invoice_123",content=invoice_data # dictionary or Invoice object)diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.mdindex 6b0220916..432ce9ab5 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md@@ -17,7 +17,7 @@ class StructureConfig(ConfigModel):```toml[pipelex.structure_config]-is_default_text_then_structure = true+is_default_text_then_structure = false # Set to true for better results: generates text before structuring```## Processing Flow@@ -36,7 +36,7 @@ The `is_default_text_then_structure` flag determines how LLM pipes generate stru```Your job is to extract and structure information from a text.Here is the text:- {preliminary_text}+ {{ preliminary_text|tag("text") }}Now generate the JSON in the required format.Do not create information that is not in the text.diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.mdindex 7b471ecbc..486f6ec48 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md@@ -23,7 +23,7 @@ This is the ideal controller for processing collections of documents, images, or| Parameter | Type | Description | Required || ------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------- || `type` | string | The type of the pipe: `PipeBatch` | Yes |-| `description` | string | A description of the batch operation. | Yes |+| `definition` | string | A description of the batch operation. | Yes || `inputs` | dictionary | The input concept(s) for the batch operation, as a dictionary mapping input names to concept codes. | Yes || `output` | string | The output concept produced by the batch operation. | Yes || `branch_pipe_code` | string | The name of the single pipe to execute for each item in the input list. | Yes |@@ -44,7 +44,7 @@ Suppose you have a list of articles and you want to generate a summary for each# The pipe that knows how to summarize one article[pipe.summarize_one_article]type = "PipeLLM"-description = "Summarize a single article"+definition = "Summarize a single article"inputs = { article = "ArticleText" }output = "ArticleSummary"prompt_template = "Please provide a one-sentence summary of the following article:\n\n@article_text"@@ -52,19 +52,18 @@ prompt_template = "Please provide a one-sentence summary of the following articl# The PipeBatch definition[pipe.summarize_all_articles]type = "PipeBatch"-description = "Summarize a batch of articles in parallel"+definition = "Summarize a batch of articles in parallel"inputs = { articles = "ArticleList" } # This is the list to iterate overoutput = "SummaryList" # This will be the list of summariesbranch_pipe_code = "summarize_one_article"-[pipe.summarize_all_articles.batch_params]-input_item_stuff_name = "ArticleText" # Name of an item within the branch+input_item_name = "ArticleText" # Name of an item within the branch```How this works:1. The `summarize_all_articles` pipe receives an `ArticleList`. Let's say it contains 10 articles.2. `PipeBatch` creates 10 parallel branches.-3. In branch #1, it takes the first article from `ArticleList`, puts it into the branch's isolated working memory, and gives it the name `ArticleText` (as specified by `input_item_stuff_name`).+3. In branch #1, it takes the first article from `ArticleList`, puts it into the branch's isolated working memory, and gives it the name `ArticleText` (as specified by `input_item_name`).4. The `summarize_one_article` pipe is then executed in branch #1. It looks for an input named `ArticleText`, finds the injected article, and produces a summary.5. Steps 3 and 4 happen simultaneously for all 10 articles in their respective branches.6. Once all `summarize_one_article` pipes are done, `PipeBatch` collects the 10 `ArticleSummary` outputs and bundles them into a single `SummaryList`. This list is the final result.\ No newline at end of filediff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.mdindex 0edc60df9..741729e5d 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md@@ -20,7 +20,7 @@ The `PipeCondition` controller adds branching logic to your pipelines. It evalua| Parameter | Type | Description | Required || ------------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ || `type` | string | The type of the pipe: `PipeCondition` | Yes |-| `description` | string | A description of the condition operation. | Yes |+| `definition` | string | A description of the condition operation. | Yes || `inputs` | dictionary | The input concept(s) for the condition, as a dictionary mapping input names to concept codes. | Yes || `output` | string | The output concept produced by the selected pipe. | Yes || `expression` | string | A simple Jinja2 expression. `{{ ... }}` are automatically added. Good for simple variable access like `"my_var.category"`. | Yes (or `expression_template`) |@@ -46,7 +46,7 @@ CategoryInput = "Input with a category field"# Define the PipeCondition first[pipe.route_by_category]type = "PipeCondition"-description = "Route based on category field"+definition = "Route based on category field"inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"@@ -59,7 +59,7 @@ large = "process_large"# Define the pipes that PipeCondition can route to[pipe.process_small]type = "PipeLLM"-description = "Handle small category"+definition = "Handle small category"output = "native.Text"prompt_template = """Output this only: "small"@@ -67,7 +67,7 @@ Output this only: "small"[pipe.process_medium]type = "PipeLLM"-description = "Handle medium category"+definition = "Handle medium category"output = "native.Text"prompt_template = """Output this only: "medium"@@ -75,7 +75,7 @@ Output this only: "medium"[pipe.process_large]type = "PipeLLM"-description = "Handle large category"+definition = "Handle large category"output = "native.Text"prompt_template = """Output this only: "large"@@ -94,7 +94,7 @@ How this works:```plx[pipe.route_with_fallback]type = "PipeCondition"-description = "Route with default handling"+definition = "Route with default handling"inputs = { classification = "DocumentType" }output = "ProcessedDocument"expression = "classification.type"@@ -106,7 +106,7 @@ receipt = "process_receipt"[pipe.process_invoice]type = "PipeLLM"-description = "Process invoice documents"+definition = "Process invoice documents"inputs = { classification = "DocumentType" }output = "ProcessedDocument"prompt_template = """@@ -115,7 +115,7 @@ Process this invoice document...[pipe.process_receipt]type = "PipeLLM"-description = "Process receipt documents"+definition = "Process receipt documents"inputs = { classification = "DocumentType" }output = "ProcessedDocument"prompt_template = """@@ -124,7 +124,7 @@ Process this receipt document...[pipe.process_unknown]type = "PipeLLM"-description = "Handle unknown document types"+definition = "Handle unknown document types"inputs = { classification = "DocumentType" }output = "ProcessedDocument"prompt_template = """diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.mdindex c0108dcd2..cd7b8fd7a 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md@@ -23,11 +23,11 @@ You must use `add_each_output`, `combined_output`, or both.| Parameter | Type | Description | Required || ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- || `type` | string | The type of the pipe: `PipeParallel` | Yes |-| `description` | string | A description of the parallel operation. | Yes |+| `definition` | string | A description of the parallel operation. | Yes || `inputs` | dictionary | The input concept(s) for the parallel operation, as a dictionary mapping input names to concept codes. | Yes || `output` | string | The output concept produced by the parallel operation. | Yes || `parallels` | array of tables| An array defining the pipes to run in parallel. Each table is a sub-pipe definition. | Yes |-| `add_each_output` | boolean | If `true`, adds the output of each parallel pipe to the working memory individually. Defaults to `false`. | No |+| `add_each_output` | boolean | If `true`, adds the output of each parallel pipe to the working memory individually. Defaults to `true`. | No || `combined_output` | string | The name of a concept to use for a single, combined output object. The structure of this concept must have fields that match the `result` names from the `parallels` array. | No |### Parallel Step Configuration@@ -49,23 +49,23 @@ structure = "ProductAnalysis" # A Pydantic model with 'features' and 'sentiment'[pipe.extract_features]type = "PipeLLM"-description = "Extract features from text"+definition = "Extract features from text"inputs = { description = "ProductDescription" }output = "ProductFeatures"[pipe.analyze_sentiment]type = "PipeLLM"-description = "Analyze sentiment of text"+definition = "Analyze sentiment of text"inputs = { description = "ProductDescription" }output = "ProductSentiment"# The PipeParallel definition[pipe.analyze_product_in_parallel]type = "PipeParallel"-description = "Extract features and sentiment at the same time"+definition = "Extract features and sentiment at the same time"inputs = { description = "ProductDescription" }output = "ProductAnalysis" # This name is for the combined output-add_each_output = false+add_each_output = truecombined_output = "ProductAnalysis"parallels = [{ pipe = "extract_features", result = "features" },diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.mdindex 062b53964..288647577 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md@@ -19,7 +19,7 @@ A `PipeSequence` defines a list of `steps`. Each step calls another pipe and giv| Parameter | Type | Description | Required || ---------- | --------------- | -------------------------------------------------------------------------------------------------------------- | -------- || `type` | string | The type of the pipe: `PipeSequence` | Yes |-| `description` | string | A description of the sequence operation. | Yes |+| `definition` | string | A description of the sequence operation. | Yes || `inputs` | dictionary | The input concept(s) for the *first* pipe in the sequence, as a dictionary mapping input names to concept codes. | No || `output` | string | The output concept produced by the *last* pipe in the sequence. | Yes || `steps` | array of tables | An ordered list of the pipes to execute. Each table in the array defines a single step. | Yes |@@ -42,25 +42,25 @@ Let's imagine a pipeline that first extracts text from an image, then summarizes```plx[pipe.extract_text_from_image]type = "PipeOcr"-description = "Extract text from an image"+definition = "Extract text from an image"output = "Text"[pipe.summarize_text]type = "PipeLLM"-description = "Summarize text"+definition = "Summarize text"inputs = { text = "Text" }output = "Text"[pipe.translate_to_french]type = "PipeLLM"-description = "Translate text to French"+definition = "Translate text to French"inputs = { text = "Text" }output = "Text"[pipe.image_to_french_summary]type = "PipeSequence"-description = "Extract, summarize, and translate text from an image"+definition = "Extract, summarize, and translate text from an image"inputs = { image = "source.Image" }output = "target.FrenchText"steps = [diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.mdindex 6281a9a0d..aadb71b8e 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md@@ -27,8 +27,8 @@ Here is an example of a function and its registration:```python# in a file like my_custom_functions.py-from pipelex.core.working_memory import WorkingMemory-from pipelex.core.stuff_content import TextContent+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.stuff_content import TextContentfrom pipelex.tools.func_registry import func_registrydef concatenate_texts(working_memory: WorkingMemory) -> TextContent:@@ -60,7 +60,7 @@ Once the function is registered, you can use it in your `.plx` file.| Parameter | Type | Description | Required || --------------- | ------ | --------------------------------------------------------------------------- | -------- || `type` | string | The type of the pipe: `PipeFunc` | Yes |-| `description` | string | A description of the function operation. | Yes |+| `definition` | string | A description of the function operation. | Yes || `function_name` | string | The unique name used to register the Python function (e.g., "combine_two_texts"). | Yes || `output` | string | The concept to associate with the function's return value. | Yes |@@ -71,7 +71,7 @@ This PLX snippet shows how to use the `combine_two_texts` function defined above```plx[pipe.combine_them]type = "PipeFunc"-description = "Combine two text inputs using a custom Python function"+definition = "Combine two text inputs using a custom Python function"function_name = "combine_two_texts"output = "ConcatenatedText"```diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.mdindex be5ee2f2e..a6ad8bb3c 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md@@ -21,11 +21,11 @@ The pipe can be configured to generate a single image or a list of images.| Parameter | Type | Description | Required || ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------- || `type` | string | The type of the pipe: `PipeImgGen` | Yes |-| `description` | string | A description of the image generation operation. | Yes |+| `definition` | string | A description of the image generation operation. | Yes || `inputs` | dictionary | The input concept(s) for the image generation operation, as a dictionary mapping input names to concept codes. | Yes || `output` | string | The output concept produced by the image generation operation. | Yes |-| `imgg_prompt` | string | A static text prompt for image generation. Use this *or* `input`. | No |-| `output_multiplicity` | integer | The number of images to generate. If omitted, a single image is generated. | No |+| `imgg_prompt` | string | A static text prompt for image generation. Use this *or* `input`. | No |+| `nb_output` | integer | The number of images to generate. If omitted, a single image is generated. | No || `imgg_handle` | string | The handle for the image generation model to use (e.g., `"dall-e-3"`). Defaults to the model specified in the global config. | No || `aspect_ratio` | string | The desired aspect ratio of the image (e.g., `"16:9"`, `"1:1"`). | No || `quality` | string | The quality of the generated image (e.g., `"standard"`, `"hd"`). | No |@@ -41,9 +41,9 @@ This pipe generates one image of a futuristic car without requiring any input.```plx[pipe.generate_car_image]type = "PipeImgGen"-description = "Generate a futuristic car image"+definition = "Generate a futuristic car image"output = "Image"-imgg_prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."+img_gen_prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."imgg_handle = "dall-e-3"aspect_ratio = "16:9"quality = "hd"@@ -59,7 +59,7 @@ ImagePrompt = "A text prompt for generating an image"[pipe.generate_logo_variations]type = "PipeImgGen"-description = "Generate three logo variations from a prompt"+definition = "Generate three logo variations from a prompt"inputs = { prompt = "images.ImgGenPrompt" }output = "Image"nb_output = 3diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.mdindex 16ac09305..56e16d569 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md@@ -23,7 +23,7 @@ The Jinja2 template has access to all the "stuffs" currently in the working memo| Parameter | Type | Description | Required || --------------- | --------------- | --------------------------------------------------------------------------------------------------------- | --------------------------- || `type` | string | The type of the pipe: `PipeJinja2` | Yes |-| `description` | string | A description of the Jinja2 operation. | Yes |+| `definition` | string | A description of the Jinja2 operation. | Yes || `output` | string | The concept for the output. Defaults to `native.Text`. | No || `jinja2_name` | string | The name of a pre-loaded template file. | Yes (or `jinja2`) || `jinja2` | string | An inline Jinja2 template string. | Yes (or `jinja2_name`) |@@ -54,7 +54,7 @@ Report generated on: {{ report_date }}```plx[pipe.generate_weekly_report]type = "PipeJinja2"-description = "Generate a formatted weekly report for a user"+definition = "Generate a formatted weekly report for a user"output = "WeeklyReportText"jinja2_name = "weekly_report.md"extra_context = { report_date = "2023-10-27" }diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.mdindex 4b50efedb..f74e06469 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md@@ -24,7 +24,7 @@ Images must be declared in the `inputs` section of your pipe definition. The ima```plx[pipe.describe_image]type = "PipeLLM"-description = "Describe an image"+definition = "Describe an image"inputs = { image = "Image" }output = "VisualDescription"prompt_template = """@@ -41,7 +41,7 @@ You can use any concept that refines `Image` as an input, and choose descriptive```plx[pipe.analyze_wedding]type = "PipeLLM"-description = "Analyze wedding photo"+definition = "Analyze wedding photo"inputs = { wedding_photo = "images.Photo" }output = "PhotoAnalysis"prompt_template = """@@ -56,7 +56,7 @@ When working with structured content that contains image fields (like `PageConte```plx[pipe.analyze_page_view]type = "PipeLLM"-description = "Analyze the visual layout of a page"+definition = "Analyze the visual layout of a page"inputs = { "page_content.page_view" = "Image" }output = "LayoutAnalysis"prompt_template = """@@ -77,7 +77,7 @@ You can include multiple images in a single prompt by listing them in the inputs```plx[pipe.compare_images]type = "PipeLLM"-description = "Compare two images"+definition = "Compare two images"inputs = {first_image = "Image",second_image = "Image"@@ -95,7 +95,7 @@ You can mix any stuff and image inputs in the same pipe:```plx[pipe.analyze_document_with_context]type = "PipeLLM"-description = "Analyze a document page with additional context"+definition = "Analyze a document page with additional context"inputs = {context = "Text",document.page_view = "Image"@@ -117,8 +117,8 @@ Analyze the document page shown in the image and explain how it relates to the p| Parameter | Type | Description | Required || --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- || `type` | string | The type of the pipe: `PipeLLM` | Yes |-| `description` | string | A description of the LLM operation. | Yes |-| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image"`). | Yes |+| `definition` | string | A description of the LLM operation. | Yes |+| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image_argurment"`)| `output` | string | The output concept produced by the LLM operation. | Yes || `llm` | string or table | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`). | No || `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`). | No |@@ -149,7 +149,7 @@ This pipe takes no input and writes a poem.```plx[pipe.write_poem]type = "PipeLLM"-description = "Write a short poem"+definition = "Write a short poem"output = "Text"llm = "llm_for_creative_writing"prompt = """@@ -164,7 +164,7 @@ This pipe summarizes an input text, using a `prompt_template` to inject the inpu```plx[pipe.summarize_text]type = "PipeLLM"-description = "Summarize a text"+definition = "Summarize a text"inputs = { text = "TextToSummarize" }output = "TextSummary"prompt_template = """@@ -183,7 +183,7 @@ This pipe takes an image of a table and uses a VLM to extract the content as an```plx[pipe.extract_table_from_image]type = "PipeLLM"-description = "Extract table data from an image"+definition = "Extract table data from an image"inputs = { image = "TableScreenshot" }output = "TableData"prompt_template = """@@ -201,7 +201,7 @@ structure = "Expense" # Assumes a Pydantic model 'Expense' is defined[pipe.process_expense_report]type = "PipeLLM"-description = "Process an expense report"+definition = "Process an expense report"inputs = { report = "ExpenseReport" }output = "ProcessedExpenseReport"prompt_template = """diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.mdindex 643037dbc..29d3eda81 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md@@ -26,13 +26,13 @@ The `PageContent` object has the following structure:| Parameter | Type | Description | Required || --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------- || `type` | string | The type of the pipe: `PipeOcr` | Yes |-| `description` | string | A description of the OCR operation. | Yes |+| `definition` | string | A description of the OCR operation. | Yes || `inputs` | Fixed | The input for the PipeOcr is the key `ocr_input` and the value is either of concept `Image` or `Pdf`. | Yes || `output` | string | The output concept produced by the OCR operation. | Yes |-| `should_include_images` | boolean | If `true`, any images found within the document pages will be extracted and included in the output. Defaults to `false`. | No |-| `should_include_page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No |+| `page_images` | boolean | If `true`, any images found within the document pages will be extracted and included in the output. Defaults to `false`. | No |+| `page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No || `page_views_dpi` | integer | The resolution (in Dots Per Inch) for the generated page views when processing a PDF. Defaults to `150`. | No |-| `should_caption_images` | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.* | No |+| `page_image_captions` | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.* | No |### Example: Processing a PDF@@ -48,13 +48,13 @@ refines = "Page"[pipe.extract_text_from_document]type = "PipeOcr"-description = "Extract text from a scanned document"+definition = "Extract text from a scanned document"inputs = { ocr_input = "ScannedDocument" }-output = "ExtractedText"-should_include_page_views = true+output = "Page"+page_views = truepage_views_dpi = 200```-The output of the PipeOcr has to be a concept compatible with the native `Page` concept. A concept is compatible with the `Page` concept if it refines the `Page` concept.+The output of the PipeOcr must be exactly the native `Page` concept.To use this pipe, you would first need to load a PDF into the `ScannedDocument` concept. After the pipe runs, the `ExtractedPages` concept will contain a list of `PageContent` objects, where each object has the extracted text and a 200 DPI image of the corresponding page.diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.mdindex 5dca68640..7a0099869 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md@@ -128,7 +128,7 @@ OppositePhotoPrompt = "Prompt for generating opposite version of photo"[pipe.gen_photopposite]type = "PipeSequence"-description = "Generate opposite version of photo's main feature"+definition = "Generate opposite version of photo's main feature"output = "Image"steps = [{ pipe = "analyze_photo", result = "analysis" },@@ -138,7 +138,7 @@ steps = [[pipe.analyze_photo]type = "PipeLLM"-description = "Analyze photo content and identify main feature"+definition = "Analyze photo content and identify main feature"inputs = { photo = "Image" }output = "PhotoAnalysis"prompt_template = """@@ -151,7 +151,7 @@ Focus on the dominant element, color, mood, or characteristic that defines this[pipe.create_opposite_concept]type = "PipeLLM"-description = "Create concept for opposite version"+definition = "Create concept for opposite version"inputs = { analysis = "PhotoAnalysis" }output = "OppositePhotoPrompt"prompt_template = """@@ -164,7 +164,7 @@ Describe what the opposite would look like, focusing on reversing the main featu[pipe.render_opposite]type = "PipeImgGen"-description = "Generate the opposite photo"+definition = "Generate the opposite photo"inputs = { opposite_concept = "OppositePhotoPrompt" }output = "Image"```@@ -271,13 +271,13 @@ Generated blueprints automatically integrate with:```bash# Validate your entire setup-pipelex validate all -c pipelex/libraries+pipelex validate all -c path/to/your/pipelex/config/folder# Test a specific pipe-pipelex validate pipe your_pipe_name -c pipelex/libraries+pipelex validate pipe your_pipe_name -c path/to/your/pipelex/config/folder# Check blueprint structure-pipelex validate blueprint your_blueprint.plx -c pipelex/libraries --no-fix+pipelex validate blueprint your_blueprint.plx -c path/to/your/pipelex/config/folder --no-fix```## Next Stepsdiff --git a/docs/pages/configuration/config-advanced/feature-config.md b/docs/pages/configuration/config-advanced/feature-config.mdindex 1997cab6a..574e1d46e 100644--- a/docs/pages/configuration/config-advanced/feature-config.md+++ b/docs/pages/configuration/config-advanced/feature-config.md@@ -32,24 +32,24 @@ The feature flags directly affect which implementation is used for certain compo### Pipeline Tracking```toml-is_pipeline_tracking_enabled = false+is_pipeline_tracking_enabled = true```- Controls whether pipeline execution tracking is enabled- When enabled, tracks the flow and execution of pipelines using by default mermaid chart:- View and edit charts at [Mermaid Live Editor](https://mermaid.live)- Useful for debugging and monitoring pipeline behavior-- Default: `false`+- Default: `true`### Activity Tracking```toml-is_activity_tracking_enabled = false+is_activity_tracking_enabled = true```- Controls whether activity tracking is enabled- When enabled, tracks detailed information about system activities-- Default: `false`+- Default: `true`### Reporting@@ -68,8 +68,8 @@ is_reporting_enabled = true# Enable pipeline tracking for debuggingis_pipeline_tracking_enabled = true-# Disable activity tracking for performance-is_activity_tracking_enabled = false+# Enable activity tracking+is_activity_tracking_enabled = true# Enable reporting for cost monitoringis_reporting_enabled = truediff --git a/docs/pages/configuration/config-pipeline-validation/dry-run-config.md b/docs/pages/configuration/config-pipeline-validation/dry-run-config.mdindex 6cb4998d8..435245640 100644--- a/docs/pages/configuration/config-pipeline-validation/dry-run-config.md+++ b/docs/pages/configuration/config-pipeline-validation/dry-run-config.md@@ -8,12 +8,20 @@ The `DryRunConfig` class controls how Pipelex behaves during dry runs.class DryRunConfig(ConfigModel):apply_to_jinja2_rendering: booltext_gen_truncate_length: int+ nb_list_items: int+ nb_ocr_pages: int+ image_urls: List[str]+ allowed_to_fail_pipes: List[str] = Field(default_factory=list)```### Fields- `apply_to_jinja2_rendering`: When true, simulates Jinja2 template rendering during dry runs- `text_gen_truncate_length`: Maximum length of generated text during dry runs+- `nb_list_items`: Number of items to generate for list content during dry runs+- `nb_ocr_pages`: Number of pages to simulate for OCR operations during dry runs+- `image_urls`: List of image URLs to use for dry run testing (must be non-empty)+- `allowed_to_fail_pipes`: List of pipe names that are allowed to fail during dry runs (optional)## Example Configuration@@ -21,6 +29,10 @@ class DryRunConfig(ConfigModel):[pipelex.dry_run_config]apply_to_jinja2_rendering = truetext_gen_truncate_length = 100+nb_list_items = 3+nb_ocr_pages = 2+image_urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]+allowed_to_fail_pipes = ["optional_pipe", "experimental_pipe"]```## Dry Run Behaviordiff --git a/docs/pages/configuration/config-pipeline-validation/static-validation-config.md b/docs/pages/configuration/config-pipeline-validation/static-validation-config.mdindex 371ca1c72..a5023edb4 100644--- a/docs/pages/configuration/config-pipeline-validation/static-validation-config.md+++ b/docs/pages/configuration/config-pipeline-validation/static-validation-config.md@@ -35,9 +35,11 @@ Each validation error type can be configured to have one of three reactions:default_reaction = "raise"[pipelex.static_validation_config.reactions]-missing_input_variable = "log"-extraneous_input_variable = "log"-inadequate_input_concept = "log"+# Enable one of these to tolerate some static validation errors, like you would for pyright or some linters+# missing_input_variable = "log"+# extraneous_input_variable = "log"+# inadequate_input_concept = "log"+# too_many_candidate_inputs = "log"```## Validation Processdiff --git a/docs/pages/configuration/config-practical/logging-config.md b/docs/pages/configuration/config-practical/logging-config.mdindex c8b23553d..a7636fe5f 100644--- a/docs/pages/configuration/config-practical/logging-config.md+++ b/docs/pages/configuration/config-practical/logging-config.md@@ -143,6 +143,7 @@ keywords_to_hilight = []```toml[pipelex.log_config]default_log_level = "INFO"+log_mode = "rich"is_console_logging_enabled = truejson_logs_indent = 4presentation_line_width = 120diff --git a/docs/pages/configuration/config-practical/pipe-run-config.md b/docs/pages/configuration/config-practical/pipe-run-config.mdindex 13acc214f..1cc9a7226 100644--- a/docs/pages/configuration/config-practical/pipe-run-config.md+++ b/docs/pages/configuration/config-practical/pipe-run-config.md@@ -17,7 +17,7 @@ class PipeRunConfig(ConfigModel):```toml[pipelex.pipe_run_config]-pipe_stack_limit = 100+pipe_stack_limit = 20```## Stack Limitdiff --git a/docs/pages/configuration/config-practical/tracker-config.md b/docs/pages/configuration/config-practical/tracker-config.mdindex 902eebf02..ecd9e0039 100644--- a/docs/pages/configuration/config-practical/tracker-config.md+++ b/docs/pages/configuration/config-practical/tracker-config.md@@ -73,18 +73,18 @@ The pipeline tracker visualizes:```toml[tracker]is_debug_mode = false-is_include_text_preview = true-is_include_interactivity = true-theme = "auto"-layout = "auto"+is_include_text_preview = false+is_include_interactivity = false+theme = "base"+layout = "dagre"wrapping_width = "auto"nb_items_limit = "unlimited"sub_graph_colors = ["#1f77b4", "#ff7f0e", "#2ca02c"]-pipe_edge_style = "solid"-branch_edge_style = "dashed"-aggregate_edge_style = "dotted"-condition_edge_style = "dashdot"-choice_edge_style = "solid"+pipe_edge_style = "---->"+branch_edge_style = "-...-"+aggregate_edge_style = "-...-"+condition_edge_style = "-----"+choice_edge_style = "-----"```## Property Accessorsdiff --git a/docs/pages/configuration/config-technical/aws-config.md b/docs/pages/configuration/config-technical/aws-config.mdindex 4f9826247..e845b81ee 100644--- a/docs/pages/configuration/config-technical/aws-config.md+++ b/docs/pages/configuration/config-technical/aws-config.md@@ -35,7 +35,7 @@ When using `api_key_method = "secret_provider"`, Pipelex will:- `AWS_REGION`⚠️ **Important**: To use the secret provider method, you must:-1. Configure a secret provider in your project using the `SecretsProviderProtocol`: See more in the [Secrets](../../advanced-customization/secrets-provider-injection.md) documentation.+1. Configure a secret provider in your project using the `SecretsProviderAbstract`: See more in the [Secrets](../../advanced-customization/secrets-provider-injection.md) documentation.2. Store your AWS credentials in your secret provider3. Ensure your secret provider is properly authenticateddiff --git a/docs/pages/configuration/config-technical/cogt-config.md b/docs/pages/configuration/config-technical/cogt-config.mdindex d8e6e3d89..4c1cac795 100644--- a/docs/pages/configuration/config-technical/cogt-config.md+++ b/docs/pages/configuration/config-technical/cogt-config.md@@ -5,12 +5,12 @@ The Cogt configuration manages all cognitive tools in Pipelex, including LLM (La## Overview```toml-[pipelex.cogt]+[cogt]# Main Cogt configuration sections-[pipelex.cogt.inference_manager_config]-[pipelex.cogt.llm_config]-[pipelex.cogt.imgg_config]-[pipelex.cogt.ocr_config]+[cogt.inference_manager_config]+[cogt.llm_config]+[cogt.imgg_config]+[cogt.ocr_config]```## Inference Manager Configuration@@ -18,7 +18,7 @@ The Cogt configuration manages all cognitive tools in Pipelex, including LLM (LaControls automatic setup of various cognitive tools:```toml-[pipelex.cogt.inference_manager_config]+[cogt.inference_manager_config]is_auto_setup_preset_llm = trueis_auto_setup_preset_imgg = trueis_auto_setup_preset_ocr = true@@ -29,22 +29,22 @@ is_auto_setup_preset_ocr = trueConfiguration for all Language Model interactions:```toml-[pipelex.cogt.llm_config]-default_max_images = 4 # Maximum number of images in prompts+[cogt.llm_config]+default_max_images = 100 # Maximum number of images in prompts# Platform preferences for different LLMs-[pipelex.cogt.llm_config.preferred_platforms]+[cogt.llm_config.preferred_platforms]gpt-4 = "openai"claude-3-opus = "anthropic"# Job configuration-[pipelex.cogt.llm_config.llm_job_config]-is_streaming_enabled = true+[cogt.llm_config.llm_job_config]+is_streaming_enabled = falsemax_retries = 3 # Between 1 and 10# Instructor settings-[pipelex.cogt.llm_config.instructor_config]-is_openai_structured_output_enabled = true+[cogt.llm_config.instructor_config]+is_openai_structured_output_enabled = false```### LLM Job Parameters@@ -60,25 +60,25 @@ When configuring LLM jobs, you can set:Configuration for image generation capabilities:```toml-[pipelex.cogt.imgg_config]-default_imgg_handle = "stable_diffusion"-imgg_handles = ["stable_diffusion", "dall_e"]+[cogt.imgg_config]+default_imgg_handle = "fal-ai/flux-pro"+imgg_handles = ["fal-ai/flux-pro", "fal-ai/fast-lightning-sdxl"]-[pipelex.cogt.imgg_config.imgg_job_config]-is_sync_mode = true+[cogt.imgg_config.imgg_job_config]+is_sync_mode = false# Default parameters for image generation-[pipelex.cogt.imgg_config.imgg_param_defaults]+[cogt.imgg_config.imgg_param_defaults]aspect_ratio = "square" # Options: square, landscape_4_3, landscape_3_2, landscape_16_9, landscape_21_9,- # portrait_4_3, portrait_2_3, portrait_9_16, portrait_9_21+ # portrait_3_4, portrait_2_3, portrait_9_16, portrait_9_21background = "auto" # Options: transparent, opaque, auto-quality = "high" # Options: low, medium, high-nb_steps = 50 # Number of diffusion steps-guidance_scale = 7.5 # Controls adherence to prompt+quality = "low" # Options: low, medium, high+# nb_steps = 1 # Number of diffusion steps (28 is good for Flux, [1,2,4,8] for SDXL Lightning)+guidance_scale = 3.5 # Controls adherence to promptis_moderated = true # Enable content moderation-safety_tolerance = 3 # Safety level (1-6)+safety_tolerance = 5 # Safety level (1-6)is_raw = false # Raw output mode-output_format = "png" # Options: png, jpg, webp+output_format = "jpg" # Options: png, jpg, webpseed = "auto" # "auto" or specific integer```@@ -113,9 +113,10 @@ Image generation jobs support these parameters:Configuration for Optical Character Recognition:```toml-[pipelex.cogt.ocr_config]-ocr_handles = ["tesseract", "azure_ocr"]-page_output_text_file_name = "page_text.txt"+[cogt.ocr_config]+ocr_handles = ["mistral/mistral-ocr-latest"]+page_output_text_file_name = "page_text.md"+default_page_views_dpi = 72```## Validation Rules@@ -155,43 +156,43 @@ page_output_text_file_name = "page_text.txt"## Example Complete Configuration```toml-[pipelex.cogt]-[pipelex.cogt.inference_manager_config]+[cogt]+[cogt.inference_manager_config]is_auto_setup_preset_llm = trueis_auto_setup_preset_imgg = trueis_auto_setup_preset_ocr = true-[pipelex.cogt.llm_config]-default_max_images = 4+[cogt.llm_config]+default_max_images = 100preferred_platforms = { "gpt-4" = "openai", "claude-3-opus" = "anthropic" }-[pipelex.cogt.llm_config.llm_job_config]-is_streaming_enabled = true+[cogt.llm_config.llm_job_config]+is_streaming_enabled = falsemax_retries = 3-[pipelex.cogt.llm_config.instructor_config]-is_openai_structured_output_enabled = true+[cogt.llm_config.instructor_config]+is_openai_structured_output_enabled = false-[pipelex.cogt.imgg_config]-default_imgg_handle = "stable_diffusion"-imgg_handles = ["stable_diffusion", "dall_e"]+[cogt.imgg_config]+default_imgg_handle = "fal-ai/flux-pro/v1.1-ultra"+imgg_handles = ["fal-ai/flux-pro", "fal-ai/fast-lightning-sdxl"]-[pipelex.cogt.imgg_config.imgg_job_config]-is_sync_mode = true+[cogt.imgg_config.imgg_job_config]+is_sync_mode = false-[pipelex.cogt.imgg_config.imgg_param_defaults]+[cogt.imgg_config.imgg_param_defaults]aspect_ratio = "square"background = "auto"-quality = "high"-nb_steps = 50-guidance_scale = 7.5+quality = "low"+# nb_steps = 1+guidance_scale = 3.5is_moderated = true-safety_tolerance = 3+safety_tolerance = 5is_raw = false-output_format = "png"+output_format = "jpg"seed = "auto"-[pipelex.cogt.ocr_config]-ocr_handles = ["tesseract", "azure_ocr"]-page_output_text_file_name = "page_text.txt"+[cogt.ocr_config]+ocr_handles = ["mistral/mistral-ocr-latest"]+page_output_text_file_name = "page_text.md"```diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.mdindex f1e7615a9..7b7bd9a2c 100644--- a/docs/pages/configuration/config-technical/library-config.md+++ b/docs/pages/configuration/config-technical/library-config.md@@ -41,22 +41,6 @@ pipelex_libraries/ # Exported library root- Validates pipe configurations- Links pipes with their respective domains-## Configuration Options--### Path Configuration--All paths are configurable through class variables in `LibraryConfig`:--```python-package_name = "pipelex"-internal_library_root = "libraries"-exported_library_root = "pipelex_libraries"-exported_pipelines_path = "pipelex_libraries/pipelines"-exported_llm_integrations_path = "pipelex_libraries/llm_integrations"-exported_llm_deck_path = "pipelex_libraries/llm_deck"-exported_templates_path = "pipelex_libraries/templates"-```-### Library InitializationUse the CLI command to initialize libraries:@@ -122,7 +106,7 @@ The library system includes specific error types:2. **Validation**:- - Run `pipelex validate all -c pipelex/libraries` after making changes+ - Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes- Check for domain consistency- Verify concept relationshipsdiff --git a/docs/pages/configuration/index.md b/docs/pages/configuration/index.mdindex 73d353d06..2c0cf18e2 100644--- a/docs/pages/configuration/index.md+++ b/docs/pages/configuration/index.md@@ -51,7 +51,7 @@ Each subsequent configuration file in this sequence can override settings from t- Base config: `pipelex.toml`- Local overrides: `pipelex_local.toml`- Environment overrides: `pipelex_dev.toml`, `pipelex_staging.toml`, `pipelex_prod.toml`, etc.-- Run mode overrides: `pipelex_normal.toml`, `pipelex_unit_test.toml`, etc.+- Run mode overrides: `pipelex_normal.toml`, `tests/pipelex_unit_test.toml`, etc.- Super user overrides: `pipelex_super.toml`NB: The run_mode unit_test is used for testing purposes.@@ -69,5 +69,5 @@ NB: The run_mode unit_test is used for testing purposes.1. **Version Control**: Include your base `pipelex.toml` in version control2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings3. **Documentation**: Comment any custom settings for team reference-4. **Validation**: Run `pipelex validate all -c pipelex/libraries` after making configuration changes+4. **Validation**: Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making configuration changes5. **Gitignore**: Add local and sensitive override files to `.gitignore`diff --git a/docs/pages/cookbook-examples/extract-dpe.md b/docs/pages/cookbook-examples/extract-dpe.mdindex 685ac210b..5e11cf882 100644--- a/docs/pages/cookbook-examples/extract-dpe.md+++ b/docs/pages/cookbook-examples/extract-dpe.md@@ -14,7 +14,7 @@ The pipeline `power_extractor_dpe` is designed to recognize and extract the keyasync def extract_dpe(pdf_url: str) -> Dpe:working_memory = WorkingMemoryFactory.make_from_pdf(pdf_url=pdf_url,- concept_str="PDF",+ concept_string="PDF",name="pdf",)pipe_output = await execute_pipeline(@@ -61,7 +61,7 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informa```plx[pipe.write_markdown_from_page_content_dpe]type = "PipeLLM"-description = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"+definition = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"inputs = { page_content = "Page" }output = "Dpe" # The output is structured as a Dpe objectllm = "llm_for_img_to_text"diff --git a/docs/pages/cookbook-examples/extract-gantt.md b/docs/pages/cookbook-examples/extract-gantt.mdindex 2bbe8ac83..450ca3b63 100644--- a/docs/pages/cookbook-examples/extract-gantt.md+++ b/docs/pages/cookbook-examples/extract-gantt.md@@ -15,7 +15,7 @@ async def extract_gantt(image_url: str) -> GanttChart:# Create Working Memoryworking_memory = WorkingMemoryFactory.make_from_image(image_url=image_url,- concept_str="gantt.GanttImage",+ concept_string="gantt.GanttImage",name="gantt_chart_image",)@@ -60,7 +60,7 @@ The `extract_gantt_by_steps` pipeline is a sequence of smaller, focused pipes. T```plx[pipe.extract_gantt_by_steps]type = "PipeSequence"-description = "Extract all details from a gantt chart"+definition = "Extract all details from a gantt chart"inputs = { gantt_chart_image = "GanttChartImage" }output = "GanttChart"steps = [@@ -77,7 +77,7 @@ steps = [# This is the pipe that extracts the details for a single task[pipe.extract_details_of_task]type = "PipeLLM"-description = "Extract the precise dates of the task, start_date and end_date"+definition = "Extract the precise dates of the task, start_date and end_date"inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails objectstructuring_method = "preliminary_text"diff --git a/docs/pages/cookbook-examples/extract-generic.md b/docs/pages/cookbook-examples/extract-generic.mdindex cf2f5a642..a08e5e586 100644--- a/docs/pages/cookbook-examples/extract-generic.md+++ b/docs/pages/cookbook-examples/extract-generic.md@@ -14,7 +14,7 @@ The `power_extractor` pipeline is at the heart of this example. After its executasync def extract_generic(pdf_url: str) -> TextAndImagesContent:working_memory = WorkingMemoryFactory.make_from_pdf(pdf_url=pdf_url,- concept_str="PDF",+ concept_string="PDF",name="pdf",)pipe_output = await execute_pipeline(@@ -31,9 +31,9 @@ The `merge_markdown_and_images` function is a great example of how you can add y```pythondef merge_markdown_and_images(working_memory: WorkingMemory) -> TextAndImagesContent:# Pages extracted from the PDF by PipeOCR- page_contents_list = working_memory.get_stuff_as_list(item_type=PageContent, name="page_contents")+ page_contents_list = working_memory.get_stuff_as_list(name="page_contents", item_type=PageContent)# Markdown text extracted from the Pages by PipeLLM- page_markdown_list = working_memory.get_stuff_as_list(item_type=TextContent, name="markdowns")+ page_markdown_list = working_memory.get_stuff_as_list(name="markdowns", item_type=TextContent)# ... (check for length equality)diff --git a/docs/pages/cookbook-examples/extract-proof-of-purchase.md b/docs/pages/cookbook-examples/extract-proof-of-purchase.mdindex 792a4d27c..d952d908d 100644--- a/docs/pages/cookbook-examples/extract-proof-of-purchase.md+++ b/docs/pages/cookbook-examples/extract-proof-of-purchase.md@@ -14,7 +14,7 @@ The pipeline `power_extractor_proof_of_purchase` is specifically designed to hanasync def extract_proof_of_purchase(pdf_url: str) -> ProofOfPurchase:working_memory = WorkingMemoryFactory.make_from_pdf(pdf_url=pdf_url,- concept_str="PDF",+ concept_string="PDF",name="pdf",)pipe_output = await execute_pipeline(@@ -57,7 +57,7 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d```plx[pipe.write_markdown_from_page_content_proof_of_purchase]type = "PipeLLM"-description = "Write markdown from page content"+definition = "Write markdown from page content"inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the pageoutput = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase objectllm = "llm_for_img_to_text"diff --git a/docs/pages/cookbook-examples/extract-table.md b/docs/pages/cookbook-examples/extract-table.mdindex 54b58a591..b212e0d70 100644--- a/docs/pages/cookbook-examples/extract-table.md+++ b/docs/pages/cookbook-examples/extract-table.md@@ -14,7 +14,7 @@ The pipeline `extract_html_table_and_review` takes an image of a table, processeasync def extract_table(table_screenshot: str) -> HtmlTable:working_memory = WorkingMemoryFactory.make_from_image(image_url=table_screenshot,- concept_str="tables.TableScreenshot",+ concept_string="tables.TableScreenshot",name="table_screenshot",)pipe_output = await execute_pipeline(@@ -62,7 +62,7 @@ The pipeline uses a two-step "extract and review" pattern. The first pipe does t```plx[pipe.extract_html_table_and_review]type = "PipeSequence"-description = "Get an HTML table and review it"+definition = "Get an HTML table and review it"inputs = { table_screenshot = "TableScreenshot" }output = "HtmlTable"steps = [@@ -74,7 +74,7 @@ steps = [[pipe.review_html_table]type = "PipeLLM"-description = "Review an HTML table"+definition = "Review an HTML table"inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }output = "HtmlTable"prompt_template = """diff --git a/docs/pages/cookbook-examples/hello-world.md b/docs/pages/cookbook-examples/hello-world.mdindex 2882a2289..98602b9a3 100644--- a/docs/pages/cookbook-examples/hello-world.md+++ b/docs/pages/cookbook-examples/hello-world.md@@ -17,7 +17,7 @@ The `hello_world` function demonstrates the simplest possible Pipelex pipeline.```pythonimport asyncio-from pipelex import pretty_print+from pipelex.tools.misc.pretty import pretty_printfrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipeline@@ -52,7 +52,7 @@ definition = "Discovering Pipelex"[pipe][pipe.hello_world]type = "PipeLLM"-description = "Write text about Hello World."+definition = "Write text about Hello World."output = "Text"llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }prompt = """diff --git a/docs/pages/cookbook-examples/invoice-extractor.md b/docs/pages/cookbook-examples/invoice-extractor.mdindex 5824c5362..78086b372 100644--- a/docs/pages/cookbook-examples/invoice-extractor.md+++ b/docs/pages/cookbook-examples/invoice-extractor.md@@ -65,7 +65,7 @@ The entire workflow is defined in a PLX file. This declarative approach makes th# The main pipeline, a sequence of steps[pipe.process_invoice]type = "PipeSequence"-description = "Process relevant information from an invoice"+definition = "Process relevant information from an invoice"inputs = { invoice_pdf = "PDF" }output = "Invoice"steps = [@@ -78,7 +78,7 @@ steps = [# A sub-pipeline that uses an LLM to extract the data[pipe.extract_invoice_data]type = "PipeLLM"-description = "Extract invoice information from an invoice text transcript"+definition = "Extract invoice information from an invoice text transcript"inputs = { "invoice_page.page_view" = "Page", invoice_details = "InvoiceDetails" }output = "Invoice"# The output is constrained to the "Invoice" modeldiff --git a/docs/pages/cookbook-examples/simple-ocr.md b/docs/pages/cookbook-examples/simple-ocr.mdindex aa6350944..c2844f100 100644--- a/docs/pages/cookbook-examples/simple-ocr.md+++ b/docs/pages/cookbook-examples/simple-ocr.md@@ -16,7 +16,7 @@ The core of this example is a simple function that creates a "working memory" frasync def simple_ocr(pdf_url: str):working_memory = WorkingMemoryFactory.make_from_pdf(pdf_url=pdf_url,- concept_str="PDF",+ concept_string="PDF",name="pdf",)pipe_output = await execute_pipeline(diff --git a/docs/pages/cookbook-examples/write-screenplay.md b/docs/pages/cookbook-examples/write-screenplay.mdindex 6cc8db9b0..6b324ca1f 100644--- a/docs/pages/cookbook-examples/write-screenplay.md+++ b/docs/pages/cookbook-examples/write-screenplay.md@@ -15,9 +15,9 @@ async def generate_screenplay(pitch: str):"""Generate a screenplay from a pitch using the pipeline."""# Create Stuff object for the pitch- pitch_stuff = StuffFactory.make_from_str(- str_value=pitch,- concept_str="screenplay.Pitch",+ pitch_stuff = StuffFactory.make_from_concept_string(+ concept_string="screenplay.Pitch",+ content=TextContent(text=pitch),name="pitch",)@@ -82,7 +82,7 @@ The `generate_screenplay` pipeline is a master `PipeSequence` that orchestrates```plx[pipe.generate_screenplay]type = "PipeSequence"-description = "Generate a complete screenplay from a pitch"+definition = "Generate a complete screenplay from a pitch"inputs = { pitch = "Pitch" }output = "FormattedScreenplay"steps = [@@ -101,7 +101,7 @@ steps = [# This is the nested sequence that creates all the scenes for a single chapter.[pipe.create_scenes_sequence]type = "PipeSequence"-description = "Create all scenes for a chapter sequentially"+definition = "Create all scenes for a chapter sequentially"inputs = { chapter = "Chapter", characters = "CharacterList", detailed_pitch = "DetailedPitch" }output = "Chapter"steps = [diff --git a/docs/pages/cookbook-examples/write-tweet.md b/docs/pages/cookbook-examples/write-tweet.mdindex 224d84f0f..81c22cd1c 100644--- a/docs/pages/cookbook-examples/write-tweet.md+++ b/docs/pages/cookbook-examples/write-tweet.md@@ -13,13 +13,13 @@ The `optimize_tweet` function is the core of this example. It takes two strings,```pythonasync def optimize_tweet(draft_tweet_str: str, writing_style_str: str) -> OptimizedTweet:# Create the draft tweet stuff- draft_tweet = StuffFactory.make_stuff(- concept_str="tech_tweet.DraftTweet",+ draft_tweet = StuffFactory.make_from_concept_string(+ concept_string="tech_tweet.DraftTweet",content=TextContent(text=draft_tweet_str),name="draft_tweet",)- writing_style = StuffFactory.make_stuff(- concept_str="tech_tweet.WritingStyle",+ writing_style = StuffFactory.make_from_concept_string(+ concept_string="tech_tweet.WritingStyle",content=TextContent(text=writing_style_str),name="writing_style",)@@ -62,7 +62,7 @@ This pipeline uses a two-step "analyze and optimize" sequence. The first pipe an```plx[pipe.optimize_tweet_sequence]type = "PipeSequence"-description = "Analyze and optimize a tech tweet in sequence"+definition = "Analyze and optimize a tech tweet in sequence"inputs = { draft_tweet = "DraftTweet", writing_style = "WritingStyle" }output = "OptimizedTweet"steps = [@@ -75,7 +75,7 @@ steps = [# This is the pipe that analyzes the draft tweet.[pipe.analyze_tweet]type = "PipeLLM"-description = "Analyze the draft tweet and identify areas for improvement"+definition = "Analyze the draft tweet and identify areas for improvement"inputs = { draft_tweet = "DraftTweet" }output = "TweetAnalysis"system_prompt = """diff --git a/docs/pages/migration/index.md b/docs/pages/migration/index.mdindex 92c6b2a81..252ea9112 100644--- a/docs/pages/migration/index.md+++ b/docs/pages/migration/index.md@@ -40,15 +40,15 @@ Use the built-in migration command to automatically update your PLX files:```bash# Preview changes without applying them-pipelex migrate --dry-run+pipelex migrate run --dry-run# Apply the migration to all PLX files-pipelex migrate+pipelex migrate run```The migration command will:-- Find all `.plx` files in your configured pipelines directory+- Find all `.toml` files in your configured pipelines directory- Replace `Concept =` with `definition =` in concept definitions- Preserve all other formatting and comments- Create backups of modified files (with `.backup` extension)@@ -59,7 +59,7 @@ If you prefer to migrate manually or need to handle special cases:1. **Locate your PLX files**: Find all pipeline library files (typically in your configured pipelines directory)2. **Update concept definitions**: Change `Concept =` to `definition =` in all `[concept.ConceptName]` sections-3. **Validate syntax**: Run `pipelex validate all -c pipelex/libraries` to ensure your files are correctly formatted+3. **Validate syntax**: Run `pipelex validate all -c path/to/your/pipelex/config/folder -c ./pipelex_libraries` to ensure your files are correctly formatted### Examples@@ -103,10 +103,10 @@ After migration, validate your pipeline files:```bash# Validate all pipeline files-pipelex validate all+pipelex validate all -c path/to/your/pipelex/config/folder# Run a specific pipeline to test-pipelex run your-pipeline-name+# Pipeline execution is done through Python code, not CLI (coming soon)```### Troubleshooting@@ -169,15 +169,15 @@ Use the built-in migration command to automatically update your PLX files:```bash# Preview changes without applying them-pipelex migrate --dry-run+pipelex migrate run --dry-run# Apply the migration to all PLX files-pipelex migrate+pipelex migrate run```The migration command will:-- Find all `.plx` files in your configured pipelines directory+- Find all `.toml` files in your configured pipelines directory- Convert `PipeClassName = "description"` to `type = "PipeClassName"` and `definition = "description"`- Preserve all other formatting and comments- Create backups of modified files (with `.backup` extension)@@ -192,7 +192,7 @@ If you prefer to migrate manually:- Replace `PipeClassName = "description"` with:- `type = "PipeClassName"`- `definition = "description"`-3. **Validate syntax**: Run `pipelex validate all -c pipelex/libraries` to ensure your files are correctly formatted+3. **Validate syntax**: Run `pipelex validate all -c path/to/your/pipelex/config/folder -c ./pipelex_libraries` to ensure your files are correctly formatted### Examples@@ -254,10 +254,10 @@ After migration, validate your pipeline files:```bash# Validate all pipeline files-pipelex validate all+pipelex validate all -c path/to/your/pipelex/config/folder# Run a specific pipeline to test-pipelex run your-pipeline-name+# Pipeline execution is done through Python code, not CLI (coming soon)```---diff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.mdindex 1f5ef9420..8cb6c5632 100644--- a/docs/pages/quick-start/index.md+++ b/docs/pages/quick-start/index.md@@ -23,7 +23,7 @@ domain = "characters"[pipe][pipe.create_character]type = "PipeLLM"-description = "Creates a character."+definition = "Creates a character."output = "Text"prompt_template = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""@@ -69,7 +69,7 @@ python character.py```plx[pipe.create_character]type = "PipeLLM"-description = "Create a character."+definition = "Create a character."output = "Text"llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }prompt_template = """You are a book writer. Your task is to create a character.@@ -81,7 +81,7 @@ Think of it and then output the character description."""```plx[pipe.create_character]type = "PipeLLM"-description = "Create a character."+definition = "Create a character."output = "Text"llm = "llm_for_creative_writing"prompt_template = """You are a book writer. Your task is to create a character.@@ -134,7 +134,7 @@ Character = "A character is a fiction story" # <- Define here your output concep[pipe][pipe.create_character]type = "PipeLLM"-description = "Create a character. Get a structured result."+definition = "Create a character. Get a structured result."output = "Character" # <- This is the output concept for your pipeprompt_template = """You are a book writer. Your task is to create a character.Think of it and then output the character description."""@@ -191,7 +191,7 @@ CharacterMetadata = "Metadata regarding a character."[pipe][pipe.extract_character_1]type = "PipeLLM"-description = "Get character information from a description."+definition = "Get character information from a description."inputs = { character = "Character" } # <- These are the inputs of your pipe, usable in the prompt_templateoutput = "CharacterMetadata"prompt_template = """@@ -235,8 +235,8 @@ async def process_existing_character():He moves with quiet purpose and speaks with a calm, thoughtful cadence that suggests he's always listening for more than just what's said.""",)# Wrap it into a stuff object- character_stuff = StuffFactory.make_stuff(- concept_code="character.Character", # <- `character` is the domain, `Character` is the concept name+ character_stuff = StuffFactory.make_from_concept_string(+ concept_string="character.Character", # <- `character` is the domain, `Character` is the concept namename="character",content=character,)@@ -245,7 +245,7 @@ async def process_existing_character():stuff=character_stuff,)# Run the pipe identified by its pipe_code (it's the name of the pipe)- pipe_output, pipeline_run_id = await execute_pipeline(+ pipe_output = await execute_pipeline(pipe_code="extract_character_1",working_memory=working_memory,)diff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.mdindex 9ad9e9bf3..37b27f9f7 100644--- a/docs/pages/tools/cli.md+++ b/docs/pages/tools/cli.md@@ -37,14 +37,14 @@ pipelex show pipe PIPE_CODE [-c/--config-folder-path PATH]Migrate TOML files to the new syntax.```bash-pipelex migrate run [-c/--config-folder-path PATH] [--dry-run] [--backups/--no-backups]+pipelex migrate run [-p/--path PATH] [--dry-run] [--backups/--no-backups]```## Usage Tips-1. Always run `pipelex validate all -c pipelex/libraries` after making changes to your configuration or pipelines+1. Always run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes to your configuration or pipelines2. Use `pipelex show config` to debug configuration issues3. When initializing a new project:- Start with `pipelex init config`- Then run `pipelex init libraries`- - Finally, validate your setup with `pipelex validate all -c pipelex/libraries`+ - Finally, validate your setup with `pipelex validate all -c path/to/your/pipelex/config/folder`diff --git a/mkdocs.yml b/mkdocs.ymlindex c56c4d6c3..0bb589d3a 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -154,6 +154,7 @@ nav:- Reporting Delegate: pages/advanced-customization/reporting-delegate-injection.md- Pipeline Tracker: pages/advanced-customization/pipeline-tracker-injection.md- Activity Manager: pages/advanced-customization/activity-manager-injection.md+ - Storage Provider: pages/advanced-customization/storage-provider-injection.md- Template Provider: pages/advanced-customization/template-provider-injection.md- Content Generator: pages/advanced-customization/content-generator-injection.md- LLM Model Provider: pages/advanced-customization/llm-model-provider-injection.mddiff --git a/pipelex/cogt/imgg/imgg_job_components.py b/pipelex/cogt/imgg/imgg_job_components.pyindex 3eb32e539..1987672f3 100644--- a/pipelex/cogt/imgg/imgg_job_components.py+++ b/pipelex/cogt/imgg/imgg_job_components.py@@ -12,7 +12,7 @@ class AspectRatio(StrEnum):LANDSCAPE_3_2 = "landscape_3_2"LANDSCAPE_16_9 = "landscape_16_9"LANDSCAPE_21_9 = "landscape_21_9"- PORTRAIT_3_4 = "portrait_4_3"+ PORTRAIT_3_4 = "portrait_3_4"PORTRAIT_2_3 = "portrait_2_3"PORTRAIT_9_16 = "portrait_9_16"PORTRAIT_9_21 = "portrait_9_21"diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex b9ca86d27..5d686394c 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -163,7 +163,7 @@ structure_from_preliminary_text_user = "structure_from_preliminary_text_user"default_reaction = "raise"[pipelex.static_validation_config.reactions]-# enable one of these to tolerate some static validation errors, like you would for pyright or some linters+# default_reaction is on raise# missing_input_variable = "log"# extraneous_input_variable = "log"# inadequate_input_concept = "log"diff --git a/pipelex/tools/config/manager.py b/pipelex/tools/config/manager.pyindex 76e8ff580..454ff0bc9 100644--- a/pipelex/tools/config/manager.py+++ b/pipelex/tools/config/manager.py@@ -104,7 +104,6 @@ class ConfigManager:- pipelex_local.toml (before env)- pipelex_{environment}.toml- pipelex_{run_mode}.toml- - pipelex_user.toml (after env)Returns:Dict[str, Any]: The merged configuration dictionarydiff --git a/pyproject.toml b/pyproject.tomlindex 61abb009b..bd920bbdf 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "pipelex"-version = "0.9.0"+version = "0.9.1"description = "Pipelex is an open-source dev tool based on a simple declarative language that lets you define replicable, structured, composable LLM pipelines."authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]diff --git a/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.pyindex 561a64696..086ae324c 100644--- a/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py+++ b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py@@ -131,7 +131,7 @@ refines = "Text""""Test that migration doesn't affect non-concept Concept = lines."""# These should NOT be changedtest_content = """# This is a comment about Concept = something-description = "This pipe works with Concept = something"+definition = "This pipe works with Concept = something"prompt_template = "Define the Concept = whatever""""diff --git a/tests/unit/pipelex/tools/misc/test_toml_utils.py b/tests/unit/pipelex/tools/misc/test_toml_utils.pyindex d9ac548eb..8124dbbdb 100644--- a/tests/unit/pipelex/tools/misc/test_toml_utils.py+++ b/tests/unit/pipelex/tools/misc/test_toml_utils.py@@ -60,7 +60,7 @@ definition = "Test"[pipe.test_pipe]type = "PipeLLM"-description = "Test"+definition = "Test"prompt_template = """Output this only: "test""""diff --git a/uv.lock b/uv.lockindex 1ac2febb0..c2f48854d 100644--- a/uv.lock+++ b/uv.lock@@ -1865,7 +1865,7 @@ wheels = [[[package]]name = "pipelex"-version = "0.9.0"+version = "0.9.1"source = { editable = "." }dependencies = [{ name = "aiofiles" },