Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Dev (#254)

Thomas Hebrard1d0e2d8b9e7e
diff --git a/.cursor/rules/base_models.mdc b/.cursor/rules/base_models.mdc
deleted file mode 100644
index 777b15464..000000000
--- a/.cursor/rules/base_models.mdc
+++ /dev/null
@@ -1,14 +0,0 @@
----
-description:
-globs:
-alwaysApply: true
----
-Rule to create BaseModels:
-
-- Respect Pydantic v2 standards
-- Keep models focused and single-purpose
-- Use descriptive field names
-- Use type hints for all fields
-- Document complex validations
-- Use Optional[] for nullable fields
-- Use Field(default_factory=...) for mutable defaults
diff --git a/.cursor/rules/best_practices.mdc b/.cursor/rules/best_practices.mdc
deleted file mode 100644
index caea92693..000000000
--- a/.cursor/rules/best_practices.mdc
+++ /dev/null
@@ -1,77 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# Best Practices Guide
-
-This document outlines the core best practices and patterns used in our codebase.
-
-## Type Hints
-
-1. **Always Use Type Hints**
- - Every function parameter must be typed
- - Every function return must be typed
- - Use type hints for all variables where type is not obvious
-
-2. **StrEnum**
- If you want to use StrEnum, import it from `pipelex.types`
- ```python
- from pipelex.types import StrEnum
- ```
-
-## Factory Pattern
-
-1. **Use Factory Pattern for Object Creation**
- - Create factories when dealing with multiple implementations
-
-## Documentation
-
-1. **Docstring Format**
- - Quick description of the function/class
- - List args and their types
- - Document return values
- - Example:
- ```python
- def process_image(image_path: str, size: Tuple[int, int]) -> bytes:
- """Process and resize an image.
-
- Args:
- image_path: Path to the source image
- size: Tuple of (width, height) for resizing
-
- Returns:
- Processed image as bytes
- """
- pass
- ```
-
-2. **Class Documentation**
- - Document class purpose and behavior
- - Include examples if complex
- ```python
- class ImageProcessor:
- """Handles image processing operations.
-
- Provides methods for resizing, converting, and optimizing images.
- """
- ```
-
-## Custom Exceptions
-
-1. **Graceful Error Handling**
- - Use try/except blocks with specific exceptions
- - Convert third-party exceptions to custom ones
- ```python
- try:
- from fal_client import AsyncClient as FalAsyncClient
- except ImportError as exc:
- raise MissingDependencyError(
- "fal-client", "fal",
- "The fal-client SDK is required to use FAL models."
- ) from exc
- ```
-
-## Pipelines
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/coding_standards.mdc b/.cursor/rules/coding_standards.mdc
new file mode 100644
index 000000000..c94a3de96
--- /dev/null
+++ b/.cursor/rules/coding_standards.mdc
@@ -0,0 +1,124 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+# Coding Standards & Best Practices
+
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
+
+## Type Hints
+
+1. **Always Use Type Hints**
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use types with Uppercase first letter (Dict[], List[], etc.)
+
+2. **StrEnum**
+ - Import from `pipelex.types`:
+ ```python
+ from pipelex.types import StrEnum
+ ```
+
+## BaseModel Standards
+
+- Respect Pydantic v2 standards
+- Keep models focused and single-purpose
+- Use descriptive field names
+- Use type hints for all fields
+- Document complex validations
+- Use Optional[] for nullable fields
+- Use Field(default_factory=...) for mutable defaults
+
+## Factory Pattern
+
+- Use Factory Pattern for object creation when dealing with multiple implementations
+
+## Documentation
+
+1. **Docstring Format**
+ ```python
+ def process_image(image_path: str, size: Tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
+ pass
+ ```
+
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
+
+## Error Handling
+
+1. **Graceful Error Handling**
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to custom ones
+ ```python
+ try:
+ from fal_client import AsyncClient as FalAsyncClient
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "fal-client", "fal",
+ "The fal-client SDK is required to use FAL models."
+ ) from exc
+ ```
+
+## Code Quality Checks
+
+### Linting and Type Checking
+
+Before finalizing a task, run:
+```bash
+make fix-unused-imports
+make check
+```
+
+This runs multiple code quality tools:
+- Pyright: Static type checking
+- Ruff: Fast Python linter
+- Mypy: Static type checker
+
+Always fix any issues reported by these tools before proceeding.
+
+### Running Tests
+
+1. **Quick Test Run** (no LLM/image generation):
+ ```bash
+ make tp
+ ```
+ Runs tests with markers: `(dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)`
+
+2. **Specific Tests**:
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ Note: Matches names starting with the provided string.
+
+**Important**: Never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-imgg` - these use costly inference.
+
+## Pipelines
+
+- All pipeline definitions go in `pipelex/libraries/pipelines/`
+- Always validate pipelines after creation/edit with `make validate`.
+ Iterate if there are errors.
+
+## Project Structure
+
+- **Pipelines**: `pipelex/libraries/pipelines/`
+- **Tests**: `tests/` directory
+- **Documentation**: `docs/` directory
\ No newline at end of file
diff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdc
index f8a11cc97..e814eb827 100644
--- a/.cursor/rules/llms.mdc
+++ b/.cursor/rules/llms.mdc
@@ -1,6 +1,5 @@
---
-description:
-globs:
+globs: *.plx,*.toml
alwaysApply: false
---
# Rules to choose LLM models used in PipeLLMs.
@@ -39,7 +38,7 @@ Here is an example of using an llm_handle to specify which LLM to use in a PipeL
```plx
[pipe.hello_world]
type = "PipeLLM"
-description = "Write text about Hello World."
+definition = "Write text about Hello World."
output = "Text"
llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }
prompt_template = """
@@ -64,7 +63,7 @@ The interest is that these presets can be used to set the LLM choice in a PipeLL
```plx
[pipe.extract_invoice]
type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
+definition = "Extract invoice information from an invoice text transcript"
inputs = { invoice_text = "InvoiceText" }
output = "Invoice"
llm = "llm_to_extract_invoice"
@@ -81,4 +80,4 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_
You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-You can override the predefined llm presets in [overrides.toml](mdc:pipelex/libraries/llm_deck/overrides.toml).
+You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).
diff --git a/.cursor/rules/pipe-batch.mdc b/.cursor/rules/pipe-batch.mdc
deleted file mode 100644
index 729c31c0f..000000000
--- a/.cursor/rules/pipe-batch.mdc
+++ /dev/null
@@ -1,32 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# PipeBatch Controller
-
-The PipeBatch controller allows you to apply a pipe operation to each element in a list of inputs in parallele. It is created via a PipeSequence.
-
-## Usage in PLX Configuration
-
-```plx
-[pipe.sequence_with_batch]
-type = "Sequence"
-description = "A Sequence of pipes"
-inputs = { input_data = "ConceptName" }
-output = "OutputConceptName"
-steps = [
- { pipe = "pipe_to_apply", batch_over = "input_list", batch_as = "current_item", result = "batch_results" }
-]
-```
-
-## Key Parameters
-
-- `pipe`: The pipe operation to apply to each element in the batch
-- `batch_over`: The name of the list in the context to iterate over
-- `batch_as`: The name to use for the current element in the pipe's context
-- `result`: Where to store the results of the batch operation
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/pipe-condition.mdc b/.cursor/rules/pipe-condition.mdc
deleted file mode 100644
index bfae8ac98..000000000
--- a/.cursor/rules/pipe-condition.mdc
+++ /dev/null
@@ -1,52 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# PipeCondition Controller
-
-The PipeCondition controller allows you to implement conditional logic in your pipeline, choosing which pipe to execute based on an evaluated expression. It supports both direct expressions and expression templates.
-
-## Usage in PLX Configuration
-
-### Basic Usage with Direct Expression
-
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditonal pipe to decide wheter..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression = "input_data.category"
-
-[pipe.conditional_operation.pipe_map]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-or
-```plx
-[pipe.conditional_operation]
-type = "PipeCondition"
-description = "A conditonal pipe to decide wheter..."
-inputs = { input_data = "CategoryInput" }
-output = "native.Text"
-expression_template = "{{ input_data.category }}" # Jinja2 code
-
-[pipe.conditional_operation.pipe_map]
-small = "process_small"
-medium = "process_medium"
-large = "process_large"
-```
-
-## Key Parameters
-
-- `expression`: Direct boolean or string expression (mutually exclusive with expression_template)
-- `expression_template`: Jinja2 template for more complex conditional logic (mutually exclusive with expression)
-- `pipe_map`: Dictionary mapping expression results to pipe codes :
-1 - The key on the left (`small`, `medium`) is the result of `expression` or `expression_template`.
-2 - The value on the right (`process_small`, `process_medium`, ..) is the name of the pipce to trigger
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/pipe-func.mdc b/.cursor/rules/pipe-func.mdc
deleted file mode 100644
index e69de29bb..000000000
diff --git a/.cursor/rules/pipe-imgg.mdc b/.cursor/rules/pipe-imgg.mdc
deleted file mode 100644
index b93c988bf..000000000
--- a/.cursor/rules/pipe-imgg.mdc
+++ /dev/null
@@ -1,5 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
diff --git a/.cursor/rules/pipe-llm.mdc b/.cursor/rules/pipe-llm.mdc
deleted file mode 100644
index cb6bf7cad..000000000
--- a/.cursor/rules/pipe-llm.mdc
+++ /dev/null
@@ -1,114 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# PipeLLM Guide
-
-## Purpose
-
-PipeLLM is used to:
-1. Generate text or objects with LLMs
-2. Process images with Vision LLMs
-
-## Basic Usage
-
-### Simple Text Generation
-```plx
-[pipe.write_story]
-type = "PipeLLM"
-description = "Write a short story"
-output = "Text"
-prompt_template = """
-Write a short story about a programmer.
-"""
-```
-
-### Structured Data Extraction
-```plx
-[pipe.extract_info]
-type = "PipeLLM"
-description = "Extract information"
-inputs = { text = "Text" }
-output = "PersonInfo"
-prompt_template = """
-Extract person information from this text:
-@text
-"""
-```
-
-### Where to Put Structured Objects
-Place your Pydantic models in `pipelex_libraries/pipelines/your_models.py`:
-
-```python
-from pipelex.core.stuffs.stuff_content import StructuredContent
-
-class PersonInfo(StructuredContent): # The output models always have to be subclass of StructuredContent
- name: str
- age: int
- email: str
-```
-
-## Advanced Features
-
-### LLM Settings
-
-You can specify LLM settings in two ways:
-
-1. **Direct in the pipe**:
-```plx
-[pipe.analyze]
-type = "PipeLLM"
-description = "Analyze text"
-output = "Analysis"
-llm = { llm_handle = "gpt-4", temperature = 0.7 }
-prompt_template = "Analyze this text"
-```
-
-2. **Using predefined settings** from `pipelex_libraries/llm_deck/base_llm_deck.toml`:
-```plx
-[pipe.analyze]
-type = "PipeLLM"
-description = "Analyze text"
-output = "Analysis"
-llm = "llm_for_analysis" # References a preset from llm_deck
-prompt_template = "Analyze this text"
-```
-
-### System Prompts
-Add system-level instructions:
-```plx
-[pipe.expert_analysis]
-type = "PipeLLM"
-description = "Expert analysis"
-output = "Analysis"
-system_prompt = "You are a data analysis expert"
-prompt_template = "Analyze this data"
-```
-
-### Multiple Outputs
-Generate multiple results:
-```plx
-[pipe.generate_ideas]
-type = "PipeLLM"
-description = "Generate ideas"
-output = "Idea"
-nb_output = 3 # Generate exactly 3 ideas
-# OR
-multiple_output = true # Let the LLM decide how many to generate
-```
-
-### Vision Tasks
-Process images with VLMs:
-```plx
-[pipe.analyze_image]
-type = "PipeLLM"
-description = "Analyze image"
-inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in a stuff, you can add something like `{ "page.image": "Image" }
-output = "ImageAnalysis"
-prompt_template = "Describe what you see in this image"
-```
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/pipe-ocr.mdc b/.cursor/rules/pipe-ocr.mdc
deleted file mode 100644
index 1e21281aa..000000000
--- a/.cursor/rules/pipe-ocr.mdc
+++ /dev/null
@@ -1,40 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# PipeOCR Guide
-
-## Purpose
-
-Extract text and images from an image or a PDF
-
-## Basic Usage
-
-### Simple Text Generation
-```plx
-[pipe.extract_info]
-type = "PipeOcr"
-description = "extract the information"
-inputs = { ocr_input = "PDF" } # or { ocr_input = "Image" } if its an image. This is the only input
-output = "Page"
-```
-
-The output concept `Page` is a native concept, with the structure `PageContent`:
-It corresponds to 1 page. Therefore, the PipeOcr is outputing a `ListContent` of `Page`
-
-```python
-class TextAndImagesContent(StuffContent):
- text: Optional[TextContent]
- images: Optional[List[ImageContent]]
-
-class PageContent(StructuredContent):
- text_and_images: TextAndImagesContent
- page_view: Optional[ImageContent] = None
-```
-- `text_and_images` are the text, and the related images found in the input image or PDF.
-- `page_view` is the screenshot of the whole pdf page/image.
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/pipe-parallel.mdc b/.cursor/rules/pipe-parallel.mdc
deleted file mode 100644
index b93c988bf..000000000
--- a/.cursor/rules/pipe-parallel.mdc
+++ /dev/null
@@ -1,5 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
diff --git a/.cursor/rules/pipe-sequence.mdc b/.cursor/rules/pipe-sequence.mdc
deleted file mode 100644
index d1ab1786d..000000000
--- a/.cursor/rules/pipe-sequence.mdc
+++ /dev/null
@@ -1,60 +0,0 @@
----
-description:
-globs:
-alwaysApply: false
----
-# PipeSequence Guide
-
-## Purpose
-PipeSequence executes multiple pipes in a defined order, where each step can use results from previous steps.
-
-## Basic Structure
-```plx
-[pipe.your_sequence_name]
-type = "PipeSequence"
-description = "Description of what this sequence does"
-inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
-output = "OutputType"
-steps = [
- { pipe = "first_pipe", result = "first_result" },
- { pipe = "second_pipe", result = "second_result" },
- { pipe = "final_pipe", result = "final_result" }
-]
-```
-
-## Key Components
-
-1. **Steps Array**: List of pipes to execute in sequence
- - `pipe`: Name of the pipe to execute
- - `result`: Name to assign to the pipe's output that will be in the working memory
-
-2. **Working Memory**: Each step can access:
- - Original sequence inputs
- - Results from previous steps
- - Use the result names in subsequent steps
- - The output concept of the last step should be the output of the sequence
-
-## Using PipeBatch in Steps
-
-You can use PipeBatch functionality within steps using `batch_over` and `batch_as`:
-
-```plx
-steps = [
- { pipe = "process_items", batch_over = "input_list", batch_as = "current_item", result = "processed_items"
- }
-]
-```
-
-1. **batch_over**: Specifies a `ListContent` field to iterate over. Each item in the list will be processed individually and IN PARALLEL by the pipe.
- - Must be a `ListContent` type containing the items to process
- - Can reference inputs or results from previous steps
-
-2. **batch_as**: Defines the name that will be used to reference the current item being processed
- - This name can be used in the pipe's input mappings
- - Makes each item from the batch available as a single element
-
-The result of a batched step will be a `ListContent` containing the outputs from processing each item.
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/pipelex.mdc b/.cursor/rules/pipelex.mdc
index 3f578db37..4c0f29238 100644
--- a/.cursor/rules/pipelex.mdc
+++ b/.cursor/rules/pipelex.mdc
@@ -4,7 +4,7 @@ alwaysApply: true
# Pipeline Guide
- Always first write your "plan" in natural langage, then transcribe it in pipelex.
-- You should ALWAYS RUN the terminal command `pipelex validate all -c pipelex/libraries` when you are writing a `.toml` file. It will ensure the pipe is runnable. If not, iterate.
+- You should ALWAYS RUN the terminal command `make validate` when you are writing a `.plx` file. It will ensure the pipe is runnable. If not, iterate.
- Please use POSIX standard for files. (enmpty lines, no trailing whitespaces, etc.)
# Pipeline Structure Guide
@@ -46,7 +46,7 @@ yes
```plx
[pipe.your_pipe_name]
type = "PipeLLM"
-description = "A description of what your pipe does"
+definition = "A description of what your pipe does"
inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
output = "ConceptName"
```
@@ -62,7 +62,7 @@ But it should be:
```plx
[pipe.your_pipe_name]
type = "PipeSequence"
-description = "....."
+definition = "....."
```
The pipes will all have at least this base structure.
@@ -155,7 +155,7 @@ PipeSequence executes multiple pipes in a defined order, where each step can use
```plx
[pipe.your_sequence_name]
type = "PipeSequence"
-description = "Description of what this sequence does"
+definition = "Description of what this sequence does"
inputs = { input_name = "InputType" } # All the inputs of the sub pipes, except the ones generated by intermediate steps
output = "OutputType"
steps = [
@@ -203,7 +203,7 @@ The PipeCondition controller allows you to implement conditional logic in your p
```plx
[pipe.conditional_operation]
type = "PipeCondition"
-description = "A conditonal pipe to decide wheter..."
+definition = "A conditonal pipe to decide wheter..."
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression = "input_data.category"
@@ -217,7 +217,7 @@ or
```plx
[pipe.conditional_operation]
type = "PipeCondition"
-description = "A conditonal pipe to decide wheter..."
+definition = "A conditonal pipe to decide wheter..."
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression_template = "{{ input_data.category }}" # Jinja2 code
@@ -245,7 +245,7 @@ The PipeBatch controller allows you to apply a pipe operation to each element in
```plx
[pipe.sequence_with_batch]
type = "PipeSequence"
-description = "A Sequence of pipes"
+definition = "A Sequence of pipes"
inputs = { input_data = "ConceptName" }
output = "OutputConceptName"
steps = [
@@ -274,7 +274,7 @@ PipeLLM is used to:
```plx
[pipe.write_story]
type = "PipeLLM"
-description = "Write a short story"
+definition = "Write a short story"
output = "Text"
prompt_template = """
Write a short story about a programmer.
@@ -285,7 +285,7 @@ Write a short story about a programmer.
```plx
[pipe.extract_info]
type = "PipeLLM"
-description = "Extract information"
+definition = "Extract information"
inputs = { text = "Text" }
output = "PersonInfo"
prompt_template = """
@@ -299,7 +299,7 @@ Add system-level instructions:
```plx
[pipe.expert_analysis]
type = "PipeLLM"
-description = "Expert analysis"
+definition = "Expert analysis"
output = "Analysis"
system_prompt = "You are a data analysis expert"
prompt_template = "Analyze this data"
@@ -310,7 +310,7 @@ Generate multiple results:
```plx
[pipe.generate_ideas]
type = "PipeLLM"
-description = "Generate ideas"
+definition = "Generate ideas"
output = "Idea"
nb_output = 3 # Generate exactly 3 ideas
# OR
@@ -322,7 +322,7 @@ Process images with VLMs:
```plx
[pipe.analyze_image]
type = "PipeLLM"
-description = "Analyze image"
+definition = "Analyze image"
inputs = { image = "Image" } # `image` is the name of the stuff that contains the Image. If its in a stuff, you can add something like `{ "page.image": "Image" }
output = "ImageAnalysis"
prompt_template = "Describe what you see in this image"
@@ -340,7 +340,7 @@ Extract text and images from an image or a PDF
```plx
[pipe.extract_info]
type = "PipeOcr"
-description = "extract the information"
+definition = "extract the information"
inputs = { ocr_input = "PDF" } # or { ocr_input = "Image" } if its an image. This is the only input
output = "Page"
```
@@ -515,6 +515,6 @@ So here are a few concrete examples of calls to execute_pipeline with various wa
)
```
-ALWAYS RUN `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
+ALWAYS RUN `make validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
Then, create an example file to run the pipeline in the `examples` folder.
But don't write documentation unless asked explicitly to.
diff --git a/.cursor/rules/pipes.mdc b/.cursor/rules/pipes.mdc
deleted file mode 100644
index e7b81982f..000000000
--- a/.cursor/rules/pipes.mdc
+++ /dev/null
@@ -1,81 +0,0 @@
----
-description: Build a Pipelex pipeline
-globs:
-alwaysApply: false
----
-# Pipeline Guide
-
-- Always first write your "plan" in natural langage, then transcribe it in pipelex.
-- Always run `pipelex validate all -c pipelex/libraries` before finishing a task. It will ensure the pipe is runnable
-
-# Pipeline Structure Guide
-
-## Pipeline File Naming
-- Files must be `.plx` for pipelines
-- Files must be `.py` for structures
-- Use descriptive names in `snake_case`
-
-## Pipeline File Structure
-A pipeline file has three main sections:
-1. Domain statement
-2. Concept definitions
-3. Pipe definitions
-
-### Domain Statement
-```plx
-domain = "domain_name"
-definition = "Description of the domain" # Optional
-```
-Note: The domain name usually matches the plx filename for single-file domains. For multi-file domains, use the subdirectory name.
-
-### Concept Definitions
-```plx
-[concept]
-ConceptName = "Description of the concept" # Should be the same name as the Structure ClassName you want to output
-```
-
-Important Rules:
-- Use PascalCase for concept names
-- Never use plurals (no "Stories", use "Story")
-- Avoid adjectives (no "LargeText", use "Text")
-- Don't redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page)
-
-### Pipe Definitions
-
-## Pipe Base Structure
-
-```plx
-[pipe.your_pipe_name]
-type = "PipeLLM"
-description = "A description of what your pipe does"
-inputs = { input_1 = "ConceptName1", input_2 = "ConceptName2" }
-output = "ConceptName"
-```
-
-The pipes will all have at least this base structure.
-- `inputs`: Dictionnary of key behing the variable used in the prompts, and the value behing the ConceptName
-- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output. Look on how to do that here: [structures.mdc](structures.mdc)
-
-## Pipe Controllers and Pipe Operator
-
-Look at the Pipes we have in order to adapt it. Pipes are organized in two categories:
-
-1. **Controllers** - For flow control:
- - `PipeSequence` - See [pipe-sequence.mdc](pipe-sequence.mdc): For creating a sequence of multiple steps
- - `PipeCondition` - See [pipe-condition.mdc](pipe-condition.mdc): If the next pipe depends of the expression of a stuff in the working memory
- - `PipeParallel` - See [pipe-parallel.mdc](pipe-parallel.mdc): For parallelizing pipes
- - `PipeBatch` - See [pipe-batch.mdc](pipe-batch.mdc): For running pipes in Batch over a ListContent
-
-2. **Operators** - For specific tasks:
- - `PipeLLM` - See [pipe-llm.mdc](pipe-llm.mdc): Generate Text and Objects (include Vision LLM)
- - `PipeOcr` - See [pipe-ocr.mdc](pipe-ocr.mdc): OCR Pipe
- - `PipeImgGen` - See [pipe-imgg.mdc](pipe-imgg.mdc): Generate Images
- - `PipeFunc` - See [pipe-func.mdc](pipe-func.mdc): For running classic python scripts
-
-When building a pipeline:
-1. Always define controllers first, then operators
-2. Each pipe must have a clear, single responsibility
-
-# Important tip
-
-Always run the cli `pipelex validate all -c pipelex/libraries` when you are finished writing pipelines: This checks for errors. If there are errors, iterate.
\ No newline at end of file
diff --git a/.cursor/rules/prompt-templates.mdc b/.cursor/rules/prompt-templates.mdc
deleted file mode 100644
index a3ab438e4..000000000
--- a/.cursor/rules/prompt-templates.mdc
+++ /dev/null
@@ -1,75 +0,0 @@
----
-description:
-globs: pipelex/libraries/pipelines/**/*.plx
-alwaysApply: false
----
-This rule explains how to write prompt templates in PipeLLM definitions.
-
-## Insert stuff inside a tagged block
-
-If the inserted text is supposedly long text, made of several lines or paragraphs, you want it inserted inside a block, possibly a block tagged and delimlited with proper syntax as one would do in a markdown documentation. To include stuff as a block, use the "@" prefix.
-
-Example template:
-```plx
-prompt_template = """
-Match the expense with its corresponding invoice:
-
-@expense
-
-@invoices
-"""
-```
-In this example, the expense data and the invoices data are obviously made of several lines each, that's why it makes sense to use the "@" prefix in order to have them delimited inside a block. Note that our preprocessor will automatically include the block's title, so it doens't need to be explictly written in the prompt template.
-
-**DO NOT write things like "Here is the expense: @expense".**
-**DO write simply "@expense" alone in an isolated line.**
-
-## Insert stuff inside a tagged block only if it exists (optional)
-
-If you want to conditionally insert a block only when the variable exists and is not empty, use the "@?" prefix. This is useful for optional fields that may or may not be present in your data.
-
-Example template:
-```plx
-prompt_template = """
-User Profile:
-
-@name
-
-@?bio
-
-@?contact_info
-
-Please summarize this profile.
-"""
-```
-In this example, the bio and contact_info will only be inserted if they exist. If they don't exist or are empty, those sections will be completely omitted from the rendered template.
-
-**DO use "@?optional_field" when the field might not always be present.**
-**DO NOT use "@optional_field" for fields that might be missing - this would cause an error.**
-
-## Insert stuff inline
-
-If the inserted text is short text and it makes sense to have it inserted directly into a sentence, you want it inserted inline. To insert stuff inline, use the "$" prefix. This will insert the stuff without delimiters and the content will be rendered as plain text.
-
-Example template:
-```plx
-prompt_template = """
-Your goal is to summarize everything related to $topic in the provided text:
-
-@text
-
-Please provide only the summary, with no additional text or explanations.
-Your summary should not be longer than 2 sentences.
-"""
-```
-
-Here, $topic will be inserted inline, whereas @text will be a a delimited block.
-Be sure to make the proper choice of prefix for each insertion.
-
-**DO NOT write "$topic" alone in an isolated line.**
-**DO write things like "Write an essay about $topic" included in an actual sentence.**
-
-## Style rule
-
-If you need to add more complexe jinja2 template, we support the classic double curly brackets `{{ }}`
-
diff --git a/.cursor/rules/standards.mdc b/.cursor/rules/standards.mdc
deleted file mode 100644
index 19972fed3..000000000
--- a/.cursor/rules/standards.mdc
+++ /dev/null
@@ -1,61 +0,0 @@
----
-description:
-globs:
-alwaysApply: true
----
-# Coding Standards
-
-This document outlines the coding standards and quality control procedures that must be followed when contributing to this project.
-
-## Style
-
-Always use type hints. Use the types with Uppercase first letter for types like Dict[], List[] etc.
-
-## Code Quality Checks
-
-### Linting and Type Checking
-
-Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:
-
-```bash
-make fix-unused-imports
-make check
-```
-
-This command runs multiple code quality tools:
-- Pyright: Static type checking
-- Ruff: Fast Python linter
-- Mypy: Static type checker
-
-Always fix any issues reported by these tools before proceeding.
-
-### Running Tests
-
-We have several make commands for running tests:
-
-1. `make tp`: Runs all tests with these markers:
- ```
- (dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)
- ```
- Use this for quick test runs that don't require LLM or image generation.
-
-2. To run specific tests:
- ```bash
- make tp TEST=TestClassName
- # or
- make tp TEST=test_function_name
- ```
- It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.
-
-Note: never run `make ti`, `make test-inference`, `make to`, `make test-ocr`, `make tg`, or `make test-imgg`: these all use inference which is costly.
-
-## Important Project Directories
-
-### Pipelines Directory
-- All pipeline definitions go in `pipelex/libraries/pipelines/`
-
-### Tests Directory
-- All tests are located in the `tests/` directory
-
-### Documentation Directory
-- All documentation is located in the `docs/` directory
diff --git a/.cursor/rules/structures.mdc b/.cursor/rules/structures.mdc
deleted file mode 100644
index 8de441eeb..000000000
--- a/.cursor/rules/structures.mdc
+++ /dev/null
@@ -1,62 +0,0 @@
----
-description:
-globs: pipelex/libraries/pipelines/**/*.py
-alwaysApply: false
----
-Rules to write structure classes for concepts used in pipes with structured generations.
-In particular, these structures are generated by PipeLLM using structured generation.
-
-# Structured Models Rules
-
-## Model Location and Registration
-
-- Create models for structured generations related to "some_domain" in `pipelex_libraries/pipelines/<some_domain>.py`
-- Models must inherit from `StructuredContent` or appropriate content type
-
-## Model Structure
-
-Concepts and their structure classes are meant to indicate an idea.
-A Concept MUST NEVER be a plural noun and you should never create a SomeConceptList: lists and arrays are implicitly handled by Pipelex according to the context. Just define SomeConcept.
-
-```python
-from datetime import datetime
-from typing import List, Optional
-from pydantic import BaseModel, Field
-
-from pipelex.core.stuffs.stuff_content import StructuredContent
-
-class YourModel(StructuredContent): # Always be a subclass of StructuredContent
- # Required fields
- field1: str
- field2: int
-
- # Optional fields with defaults
- field3: Optional[str] = Field(None, "Description of field3")
- field4: List[str] = Field(default_factory=list)
-
- # Date fields should remove timezone
- date_field: Optional[datetime] = None
-
- @field_validator("date_field")
- @classmethod
- def remove_tzinfo(cls, v: Optional[datetime]) -> Optional[datetime]:
- if v is not None:
- return v.replace(tzinfo=None)
- return v
-```
-## Usage
-
-Structures are meant to indicate what class to use for a particular Concept. In general they use the same name as the concept.
-
-Structure classes defined within `pipelex/libraries/pipelines/` are automatically loaded into the class_registry when setting up Pipelex, no need to do it manually.
-
-
-## Best Practices for structures
-
-- Respect Pydantic v2 standards
-- Use type hints for all fields
-- Use `Field` declaration and write the description
-- Use Pydantic validators for data cleaning/validation
-- Remove timezone info from datetime fields
-- Respect rules of @base_models.
-
diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdc
index 8c15b54b2..575544a6f 100644
--- a/.cursor/rules/tdd.mdc
+++ b/.cursor/rules/tdd.mdc
@@ -18,10 +18,10 @@ This document outlines our test-driven development (TDD) process and the tools a
- Keep it simple - don't write more than needed
3. **Run Linting and Type Checking**
-[standards.mdc](standards.mdc)
+[coding_standards.mdc](coding_standards.mdc)
4. **Refactor if needed**
-If the code needs refactoring, with the best practices [best_practices.mdc](best_practices.mdc)
+If the code needs refactoring, with the best practices [coding_standards.mdc](coding_standards.mdc)
5. **Validate tests**
diff --git a/AGENTS.md b/AGENTS.md
index 3077b4310..9725b0307 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -30,7 +30,7 @@ This python >=3.10 code is in the `pipelex` directory.
## Testing
- Always test with `make codex-tests` -> it runs pytest on our `tests/` directory using proper settings
-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood)
+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood) in the python environment.
## PR Instructions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 11c3885b6..fe5d5ce7e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,10 @@
# Changelog
+## [v0.9.1] - 2025-09-05
+
+### Fixed
+- Fixed many inconsistencies in the documentation.
+
## [v0.9.0] - 2025-09-02
### Refacto
diff --git a/CLAUDE.md b/CLAUDE.md
index 9da6c7969..f2d28e1e2 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -82,7 +82,7 @@ class TestFooBar:
- Always test with `make t` -> it runs pytest using proper settings
- If some pytest tests fail, run pytest on the failed ones with the required verbosity to diagnose the issue
-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood)
+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all -c pipelex/libraries` under the hood) in the python environment.
## PR Instructions
diff --git a/README.md b/README.md
index 782f4314b..287554677 100644
--- a/README.md
+++ b/README.md
@@ -73,7 +73,7 @@ PurchaseDocumentText = "Transcript of a receipt, invoice, or order confirmation"
[pipe.extract_buyer]
type = "PipeLLM"
-description = "Extract buyer from purchase document"
+definition = "Extract buyer from purchase document"
inputs = { purchase_document_text = "PurchaseDocumentText" }
output = "Buyer"
llm = "llm_to_extract_info"
@@ -124,9 +124,9 @@ uv pip install pipelex
The package supports the following additional features:
-- `anthropic`: Anthropic/Claude support
+- `anthropic`: Anthropic/Claude support+
- `google`: Google models (Vertex) support
-- `mistralai`: Mistral AI support
+- `mistralai`: Mistral AI support√
- `bedrock`: AWS Bedrock support
- `fal`: Image generation with Black Forest Labs "FAL" service
@@ -174,7 +174,7 @@ WritingStyle = "A style of writing"
[pipe]
[pipe.analyze_tweet]
type = "PipeLLM"
-description = "Analyze the draft tweet and identify areas for improvement"
+definition = "Analyze the draft tweet and identify areas for improvement"
inputs = { draft_tweet = "DraftTweet" }
output = "TweetAnalysis"
llm = "llm_for_writing_analysis"
@@ -203,7 +203,7 @@ without providing an actual rewrite
[pipe.optimize_tweet]
type = "PipeLLM"
-description = "Optimize the tweet based on the analysis"
+definition = "Optimize the tweet based on the analysis"
inputs = { draft_tweet = "DraftTweet", tweet_analysis = "TweetAnalysis", writing_style = "WritingStyle" }
output = "OptimizedTweet"
llm = "llm_for_social_post_writing"
@@ -238,7 +238,7 @@ Keep the core meaning of the original tweet.
[pipe.optimize_tweet_sequence]
type = "PipeSequence"
-description = "Analyze and optimize a tech tweet in sequence"
+definition = "Analyze and optimize a tech tweet in sequence"
inputs = { draft_tweet = "DraftTweet", writing_style = "WritingStyle" }
output = "OptimizedTweet"
steps = [
diff --git a/docs/pages/advanced-customization/index.md b/docs/pages/advanced-customization/index.md
index 33fb3e03c..f68371a1d 100644
--- a/docs/pages/advanced-customization/index.md
+++ b/docs/pages/advanced-customization/index.md
@@ -14,6 +14,7 @@ There are two main ways to inject custom implementations:
from pipelex import Pipelex
pipelex = Pipelex(
+ config_dir_path="./pipelex_libraries",
template_provider=MyTemplateProvider(),
llm_model_provider=MyLLMProvider(),
inference_manager=MyInferenceManager(),
@@ -76,13 +77,13 @@ Pipelex supports injection of the following components:
1. **Template Provider** (`TemplateLibrary`)
- - Protocol: `TemplateLibraryProtocol`
+ - Protocol: `TemplateProviderAbstract`
- Default: `TemplateLibrary`
- [Details](template-provider-injection.md)
2. **LLM Model Provider** (`LLMModelLibrary`)
- - Protocol: `LLMModelLibraryProtocol`
+ - Protocol: `LLMModelProviderAbstract`
- Default: `LLMModelLibrary`
- [Details](llm-model-provider-injection.md)
@@ -112,7 +113,7 @@ Pipelex supports injection of the following components:
7. **Secrets Provider** (`EnvSecretsProvider`)
- - Protocol: `SecretsProviderProtocol`
+ - Protocol: `SecretsProviderAbstract`
- Default: `EnvSecretsProvider`
- [Details](secrets-provider-injection.md)
diff --git a/docs/pages/advanced-customization/storage-provider-ingestion.md b/docs/pages/advanced-customization/storage-provider-injection.md
similarity index 100%
rename from docs/pages/advanced-customization/storage-provider-ingestion.md
rename to docs/pages/advanced-customization/storage-provider-injection.md
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
index 0ccc8f508..37321a0ac 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
@@ -65,7 +65,7 @@ Here's how to use these configurations in your pipelines:
```plx
[pipe.generate_response]
type = "PipeLLM"
-description = "Generate a creative response"
+definition = "Generate a creative response"
inputs = { question = "Question" }
output = "Response"
llm = {
@@ -81,7 +81,7 @@ Generate a creative response to this question:
[pipe.extract_weather_data]
type = "PipeLLM"
-description = "Extract structured weather data from text"
+definition = "Extract structured weather data from text"
inputs = { text = "Text" }
output = "WeatherData"
llm = "llm_to_extract" # Using a preset
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md
index 212b84431..8b8b0234b 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md
@@ -55,7 +55,7 @@ Group concepts that naturally belong together in the same domain. A domain acts
```plx
# pipelex_libraries/pipelines/finance.plx
domain = "finance"
-description = "Financial document processing"
+definition = "Financial document processing"
[concept]
Invoice = "A commercial document issued by a seller to a buyer"
@@ -221,7 +221,7 @@ For example, a pipe that processes `Document` can also process `Contract` or `Em
```plx
[pipe.extract_key_points]
type = "PipeLLM"
-description = "Extract main points from any document"
+definition = "Extract main points from any document"
inputs = { doc = "Document" } # Can accept Document, Contract, or EmploymentContract
output = "KeyPoints"
```
@@ -308,21 +308,21 @@ Native concepts can be used directly in your pipeline definitions without any ad
```plx
[pipe.analyze_document]
type = "PipeLLM"
-description = "Analyze a PDF document"
+definition = "Analyze a PDF document"
inputs = { document = "PDF" }
output = "Text"
prompt_template = "Analyze this document and provide a summary"
[pipe.process_image]
type = "PipeLLM"
-description = "Describe an image"
+definition = "Describe an image"
inputs = { photo = "Image" }
output = "Text"
prompt_template = "Describe what you see in this image"
[pipe.extract_from_page]
type = "PipeLLM"
-description = "Extract information from a document page"
+definition = "Extract information from a document page"
inputs = { page_content = "Page" }
output = "ExtractedInfo"
prompt_template = "Extract key information from this page content"
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
index 1d2523434..6e5217159 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
@@ -37,7 +37,7 @@ Tagline = "A catchy marketing tagline"
# 2. Define the pipe that does the work
[pipe.generate_tagline]
type = "PipeLLM"
-description = "Generate a catchy tagline for a product"
+definition = "Generate a catchy tagline for a product"
inputs = { description = "ProductDescription" }
output = "Tagline"
prompt_template = """
@@ -72,7 +72,7 @@ Tagline = "A catchy marketing tagline"
# 2. Define operator pipes
[pipe.extract_keywords]
type = "PipeLLM"
-description = "Extract keywords from a product description"
+definition = "Extract keywords from a product description"
inputs = { description = "ProductDescription" }
output = "Keyword"
multiple_output = true
@@ -86,7 +86,7 @@ Focus on features, benefits, and unique selling points.
[pipe.generate_tagline_from_keywords]
type = "PipeLLM"
-description = "Generate a tagline from keywords"
+definition = "Generate a tagline from keywords"
inputs = { keywords = "Keyword" }
output = "Tagline"
prompt_template = """
@@ -100,7 +100,7 @@ The tagline should be memorable, concise (under 10 words), and highlight the mai
# 3. This controller pipe defines the two-step pipeline
[pipe.description_to_tagline]
type = "PipeSequence"
-description = "From product description to tagline"
+definition = "From product description to tagline"
inputs = { description = "ProductDescription" }
output = "Tagline"
steps = [
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md
index cd405ee91..55c4cfc63 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md
@@ -18,7 +18,7 @@ Answer = "A response to a question"
[pipe]
[pipe.answer_question]
type = "PipeLLM"
-description = "Answer a question"
+definition = "Answer a question"
inputs = { question = "tutorial.Question" }
output = "tutorial.Answer"
prompt_template = """
@@ -64,7 +64,7 @@ A domain consists of:
[pipe]
[pipe.summarize_invoice]
type = "PipeLLM"
- description = "Summarize an invoice to extract key information"
+ definition = "Summarize an invoice to extract key information"
inputs = { invoice = "finance.Invoice" }
output = "finance.InvoiceSummary"
```
@@ -115,8 +115,8 @@ When using a domain in your code, you refer to concepts with `domain.ConceptName
from pipelex.core.stuffs.stuff_factory import StuffFactory
# The concept_code combines domain and concept names
-invoice_stuff = StuffFactory.make_stuff(
- concept_str="finance.Invoice", # domain.ConceptName
+invoice_stuff = StuffFactory.make_from_concept_string(
+ concept_string="finance.Invoice", # domain.ConceptName
name="invoice_123",
content=invoice_data # dictionary or Invoice object
)
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md
index 6b0220916..432ce9ab5 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md
@@ -17,7 +17,7 @@ class StructureConfig(ConfigModel):
```toml
[pipelex.structure_config]
-is_default_text_then_structure = true
+is_default_text_then_structure = false # Set to true for better results: generates text before structuring
```
## Processing Flow
@@ -36,7 +36,7 @@ The `is_default_text_then_structure` flag determines how LLM pipes generate stru
```
Your job is to extract and structure information from a text.
Here is the text:
- {preliminary_text}
+ {{ preliminary_text|tag("text") }}
Now generate the JSON in the required format.
Do not create information that is not in the text.
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md
index 7b471ecbc..486f6ec48 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeBatch.md
@@ -23,7 +23,7 @@ This is the ideal controller for processing collections of documents, images, or
| Parameter | Type | Description | Required |
| ------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `type` | string | The type of the pipe: `PipeBatch` | Yes |
-| `description` | string | A description of the batch operation. | Yes |
+| `definition` | string | A description of the batch operation. | Yes |
| `inputs` | dictionary | The input concept(s) for the batch operation, as a dictionary mapping input names to concept codes. | Yes |
| `output` | string | The output concept produced by the batch operation. | Yes |
| `branch_pipe_code` | string | The name of the single pipe to execute for each item in the input list. | Yes |
@@ -44,7 +44,7 @@ Suppose you have a list of articles and you want to generate a summary for each
# The pipe that knows how to summarize one article
[pipe.summarize_one_article]
type = "PipeLLM"
-description = "Summarize a single article"
+definition = "Summarize a single article"
inputs = { article = "ArticleText" }
output = "ArticleSummary"
prompt_template = "Please provide a one-sentence summary of the following article:\n\n@article_text"
@@ -52,19 +52,18 @@ prompt_template = "Please provide a one-sentence summary of the following articl
# The PipeBatch definition
[pipe.summarize_all_articles]
type = "PipeBatch"
-description = "Summarize a batch of articles in parallel"
+definition = "Summarize a batch of articles in parallel"
inputs = { articles = "ArticleList" } # This is the list to iterate over
output = "SummaryList" # This will be the list of summaries
branch_pipe_code = "summarize_one_article"
-[pipe.summarize_all_articles.batch_params]
-input_item_stuff_name = "ArticleText" # Name of an item within the branch
+input_item_name = "ArticleText" # Name of an item within the branch
```
How this works:
1. The `summarize_all_articles` pipe receives an `ArticleList`. Let's say it contains 10 articles.
2. `PipeBatch` creates 10 parallel branches.
-3. In branch #1, it takes the first article from `ArticleList`, puts it into the branch's isolated working memory, and gives it the name `ArticleText` (as specified by `input_item_stuff_name`).
+3. In branch #1, it takes the first article from `ArticleList`, puts it into the branch's isolated working memory, and gives it the name `ArticleText` (as specified by `input_item_name`).
4. The `summarize_one_article` pipe is then executed in branch #1. It looks for an input named `ArticleText`, finds the injected article, and produces a summary.
5. Steps 3 and 4 happen simultaneously for all 10 articles in their respective branches.
6. Once all `summarize_one_article` pipes are done, `PipeBatch` collects the 10 `ArticleSummary` outputs and bundles them into a single `SummaryList`. This list is the final result.
\ No newline at end of file
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md
index 0edc60df9..741729e5d 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeCondition.md
@@ -20,7 +20,7 @@ The `PipeCondition` controller adds branching logic to your pipelines. It evalua
| Parameter | Type | Description | Required |
| ------------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `type` | string | The type of the pipe: `PipeCondition` | Yes |
-| `description` | string | A description of the condition operation. | Yes |
+| `definition` | string | A description of the condition operation. | Yes |
| `inputs` | dictionary | The input concept(s) for the condition, as a dictionary mapping input names to concept codes. | Yes |
| `output` | string | The output concept produced by the selected pipe. | Yes |
| `expression` | string | A simple Jinja2 expression. `{{ ... }}` are automatically added. Good for simple variable access like `"my_var.category"`. | Yes (or `expression_template`) |
@@ -46,7 +46,7 @@ CategoryInput = "Input with a category field"
# Define the PipeCondition first
[pipe.route_by_category]
type = "PipeCondition"
-description = "Route based on category field"
+definition = "Route based on category field"
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression = "input_data.category"
@@ -59,7 +59,7 @@ large = "process_large"
# Define the pipes that PipeCondition can route to
[pipe.process_small]
type = "PipeLLM"
-description = "Handle small category"
+definition = "Handle small category"
output = "native.Text"
prompt_template = """
Output this only: "small"
@@ -67,7 +67,7 @@ Output this only: "small"
[pipe.process_medium]
type = "PipeLLM"
-description = "Handle medium category"
+definition = "Handle medium category"
output = "native.Text"
prompt_template = """
Output this only: "medium"
@@ -75,7 +75,7 @@ Output this only: "medium"
[pipe.process_large]
type = "PipeLLM"
-description = "Handle large category"
+definition = "Handle large category"
output = "native.Text"
prompt_template = """
Output this only: "large"
@@ -94,7 +94,7 @@ How this works:
```plx
[pipe.route_with_fallback]
type = "PipeCondition"
-description = "Route with default handling"
+definition = "Route with default handling"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
expression = "classification.type"
@@ -106,7 +106,7 @@ receipt = "process_receipt"
[pipe.process_invoice]
type = "PipeLLM"
-description = "Process invoice documents"
+definition = "Process invoice documents"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt_template = """
@@ -115,7 +115,7 @@ Process this invoice document...
[pipe.process_receipt]
type = "PipeLLM"
-description = "Process receipt documents"
+definition = "Process receipt documents"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt_template = """
@@ -124,7 +124,7 @@ Process this receipt document...
[pipe.process_unknown]
type = "PipeLLM"
-description = "Handle unknown document types"
+definition = "Handle unknown document types"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt_template = """
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md
index c0108dcd2..cd7b8fd7a 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeParallel.md
@@ -23,11 +23,11 @@ You must use `add_each_output`, `combined_output`, or both.
| Parameter | Type | Description | Required |
| ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `type` | string | The type of the pipe: `PipeParallel` | Yes |
-| `description` | string | A description of the parallel operation. | Yes |
+| `definition` | string | A description of the parallel operation. | Yes |
| `inputs` | dictionary | The input concept(s) for the parallel operation, as a dictionary mapping input names to concept codes. | Yes |
| `output` | string | The output concept produced by the parallel operation. | Yes |
| `parallels` | array of tables| An array defining the pipes to run in parallel. Each table is a sub-pipe definition. | Yes |
-| `add_each_output` | boolean | If `true`, adds the output of each parallel pipe to the working memory individually. Defaults to `false`. | No |
+| `add_each_output` | boolean | If `true`, adds the output of each parallel pipe to the working memory individually. Defaults to `true`. | No |
| `combined_output` | string | The name of a concept to use for a single, combined output object. The structure of this concept must have fields that match the `result` names from the `parallels` array. | No |
### Parallel Step Configuration
@@ -49,23 +49,23 @@ structure = "ProductAnalysis" # A Pydantic model with 'features' and 'sentiment'
[pipe.extract_features]
type = "PipeLLM"
-description = "Extract features from text"
+definition = "Extract features from text"
inputs = { description = "ProductDescription" }
output = "ProductFeatures"
[pipe.analyze_sentiment]
type = "PipeLLM"
-description = "Analyze sentiment of text"
+definition = "Analyze sentiment of text"
inputs = { description = "ProductDescription" }
output = "ProductSentiment"
# The PipeParallel definition
[pipe.analyze_product_in_parallel]
type = "PipeParallel"
-description = "Extract features and sentiment at the same time"
+definition = "Extract features and sentiment at the same time"
inputs = { description = "ProductDescription" }
output = "ProductAnalysis" # This name is for the combined output
-add_each_output = false
+add_each_output = true
combined_output = "ProductAnalysis"
parallels = [
{ pipe = "extract_features", result = "features" },
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
index 062b53964..288647577 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
@@ -19,7 +19,7 @@ A `PipeSequence` defines a list of `steps`. Each step calls another pipe and giv
| Parameter | Type | Description | Required |
| ---------- | --------------- | -------------------------------------------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeSequence` | Yes |
-| `description` | string | A description of the sequence operation. | Yes |
+| `definition` | string | A description of the sequence operation. | Yes |
| `inputs` | dictionary | The input concept(s) for the *first* pipe in the sequence, as a dictionary mapping input names to concept codes. | No |
| `output` | string | The output concept produced by the *last* pipe in the sequence. | Yes |
| `steps` | array of tables | An ordered list of the pipes to execute. Each table in the array defines a single step. | Yes |
@@ -42,25 +42,25 @@ Let's imagine a pipeline that first extracts text from an image, then summarizes
```plx
[pipe.extract_text_from_image]
type = "PipeOcr"
-description = "Extract text from an image"
+definition = "Extract text from an image"
output = "Text"
[pipe.summarize_text]
type = "PipeLLM"
-description = "Summarize text"
+definition = "Summarize text"
inputs = { text = "Text" }
output = "Text"
[pipe.translate_to_french]
type = "PipeLLM"
-description = "Translate text to French"
+definition = "Translate text to French"
inputs = { text = "Text" }
output = "Text"
[pipe.image_to_french_summary]
type = "PipeSequence"
-description = "Extract, summarize, and translate text from an image"
+definition = "Extract, summarize, and translate text from an image"
inputs = { image = "source.Image" }
output = "target.FrenchText"
steps = [
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md
index 6281a9a0d..aadb71b8e 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeFunc.md
@@ -27,8 +27,8 @@ Here is an example of a function and its registration:
```python
# in a file like my_custom_functions.py
-from pipelex.core.working_memory import WorkingMemory
-from pipelex.core.stuff_content import TextContent
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.stuff_content import TextContent
from pipelex.tools.func_registry import func_registry
def concatenate_texts(working_memory: WorkingMemory) -> TextContent:
@@ -60,7 +60,7 @@ Once the function is registered, you can use it in your `.plx` file.
| Parameter | Type | Description | Required |
| --------------- | ------ | --------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeFunc` | Yes |
-| `description` | string | A description of the function operation. | Yes |
+| `definition` | string | A description of the function operation. | Yes |
| `function_name` | string | The unique name used to register the Python function (e.g., "combine_two_texts"). | Yes |
| `output` | string | The concept to associate with the function's return value. | Yes |
@@ -71,7 +71,7 @@ This PLX snippet shows how to use the `combine_two_texts` function defined above
```plx
[pipe.combine_them]
type = "PipeFunc"
-description = "Combine two text inputs using a custom Python function"
+definition = "Combine two text inputs using a custom Python function"
function_name = "combine_two_texts"
output = "ConcatenatedText"
```
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md
index be5ee2f2e..a6ad8bb3c 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeImgGen.md
@@ -21,11 +21,11 @@ The pipe can be configured to generate a single image or a list of images.
| Parameter | Type | Description | Required |
| ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeImgGen` | Yes |
-| `description` | string | A description of the image generation operation. | Yes |
+| `definition` | string | A description of the image generation operation. | Yes |
| `inputs` | dictionary | The input concept(s) for the image generation operation, as a dictionary mapping input names to concept codes. | Yes |
| `output` | string | The output concept produced by the image generation operation. | Yes |
-| `imgg_prompt` | string | A static text prompt for image generation. Use this *or* `input`. | No |
-| `output_multiplicity` | integer | The number of images to generate. If omitted, a single image is generated. | No |
+| `imgg_prompt` | string | A static text prompt for image generation. Use this *or* `input`. | No |
+| `nb_output` | integer | The number of images to generate. If omitted, a single image is generated. | No |
| `imgg_handle` | string | The handle for the image generation model to use (e.g., `"dall-e-3"`). Defaults to the model specified in the global config. | No |
| `aspect_ratio` | string | The desired aspect ratio of the image (e.g., `"16:9"`, `"1:1"`). | No |
| `quality` | string | The quality of the generated image (e.g., `"standard"`, `"hd"`). | No |
@@ -41,9 +41,9 @@ This pipe generates one image of a futuristic car without requiring any input.
```plx
[pipe.generate_car_image]
type = "PipeImgGen"
-description = "Generate a futuristic car image"
+definition = "Generate a futuristic car image"
output = "Image"
-imgg_prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."
+img_gen_prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."
imgg_handle = "dall-e-3"
aspect_ratio = "16:9"
quality = "hd"
@@ -59,7 +59,7 @@ ImagePrompt = "A text prompt for generating an image"
[pipe.generate_logo_variations]
type = "PipeImgGen"
-description = "Generate three logo variations from a prompt"
+definition = "Generate three logo variations from a prompt"
inputs = { prompt = "images.ImgGenPrompt" }
output = "Image"
nb_output = 3
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md
index 16ac09305..56e16d569 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeJinja2.md
@@ -23,7 +23,7 @@ The Jinja2 template has access to all the "stuffs" currently in the working memo
| Parameter | Type | Description | Required |
| --------------- | --------------- | --------------------------------------------------------------------------------------------------------- | --------------------------- |
| `type` | string | The type of the pipe: `PipeJinja2` | Yes |
-| `description` | string | A description of the Jinja2 operation. | Yes |
+| `definition` | string | A description of the Jinja2 operation. | Yes |
| `output` | string | The concept for the output. Defaults to `native.Text`. | No |
| `jinja2_name` | string | The name of a pre-loaded template file. | Yes (or `jinja2`) |
| `jinja2` | string | An inline Jinja2 template string. | Yes (or `jinja2_name`) |
@@ -54,7 +54,7 @@ Report generated on: {{ report_date }}
```plx
[pipe.generate_weekly_report]
type = "PipeJinja2"
-description = "Generate a formatted weekly report for a user"
+definition = "Generate a formatted weekly report for a user"
output = "WeeklyReportText"
jinja2_name = "weekly_report.md"
extra_context = { report_date = "2023-10-27" }
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
index 4b50efedb..f74e06469 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
@@ -24,7 +24,7 @@ Images must be declared in the `inputs` section of your pipe definition. The ima
```plx
[pipe.describe_image]
type = "PipeLLM"
-description = "Describe an image"
+definition = "Describe an image"
inputs = { image = "Image" }
output = "VisualDescription"
prompt_template = """
@@ -41,7 +41,7 @@ You can use any concept that refines `Image` as an input, and choose descriptive
```plx
[pipe.analyze_wedding]
type = "PipeLLM"
-description = "Analyze wedding photo"
+definition = "Analyze wedding photo"
inputs = { wedding_photo = "images.Photo" }
output = "PhotoAnalysis"
prompt_template = """
@@ -56,7 +56,7 @@ When working with structured content that contains image fields (like `PageConte
```plx
[pipe.analyze_page_view]
type = "PipeLLM"
-description = "Analyze the visual layout of a page"
+definition = "Analyze the visual layout of a page"
inputs = { "page_content.page_view" = "Image" }
output = "LayoutAnalysis"
prompt_template = """
@@ -77,7 +77,7 @@ You can include multiple images in a single prompt by listing them in the inputs
```plx
[pipe.compare_images]
type = "PipeLLM"
-description = "Compare two images"
+definition = "Compare two images"
inputs = {
first_image = "Image",
second_image = "Image"
@@ -95,7 +95,7 @@ You can mix any stuff and image inputs in the same pipe:
```plx
[pipe.analyze_document_with_context]
type = "PipeLLM"
-description = "Analyze a document page with additional context"
+definition = "Analyze a document page with additional context"
inputs = {
context = "Text",
document.page_view = "Image"
@@ -117,8 +117,8 @@ Analyze the document page shown in the image and explain how it relates to the p
| Parameter | Type | Description | Required |
| --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeLLM` | Yes |
-| `description` | string | A description of the LLM operation. | Yes |
-| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image"`). | Yes |
+| `definition` | string | A description of the LLM operation. | Yes |
+| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image_argurment"`)
| `output` | string | The output concept produced by the LLM operation. | Yes |
| `llm` | string or table | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`). | No |
| `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`). | No |
@@ -149,7 +149,7 @@ This pipe takes no input and writes a poem.
```plx
[pipe.write_poem]
type = "PipeLLM"
-description = "Write a short poem"
+definition = "Write a short poem"
output = "Text"
llm = "llm_for_creative_writing"
prompt = """
@@ -164,7 +164,7 @@ This pipe summarizes an input text, using a `prompt_template` to inject the inpu
```plx
[pipe.summarize_text]
type = "PipeLLM"
-description = "Summarize a text"
+definition = "Summarize a text"
inputs = { text = "TextToSummarize" }
output = "TextSummary"
prompt_template = """
@@ -183,7 +183,7 @@ This pipe takes an image of a table and uses a VLM to extract the content as an
```plx
[pipe.extract_table_from_image]
type = "PipeLLM"
-description = "Extract table data from an image"
+definition = "Extract table data from an image"
inputs = { image = "TableScreenshot" }
output = "TableData"
prompt_template = """
@@ -201,7 +201,7 @@ structure = "Expense" # Assumes a Pydantic model 'Expense' is defined
[pipe.process_expense_report]
type = "PipeLLM"
-description = "Process an expense report"
+definition = "Process an expense report"
inputs = { report = "ExpenseReport" }
output = "ProcessedExpenseReport"
prompt_template = """
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
index 643037dbc..29d3eda81 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
@@ -26,13 +26,13 @@ The `PageContent` object has the following structure:
| Parameter | Type | Description | Required |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type` | string | The type of the pipe: `PipeOcr` | Yes |
-| `description` | string | A description of the OCR operation. | Yes |
+| `definition` | string | A description of the OCR operation. | Yes |
| `inputs` | Fixed | The input for the PipeOcr is the key `ocr_input` and the value is either of concept `Image` or `Pdf`. | Yes |
| `output` | string | The output concept produced by the OCR operation. | Yes |
-| `should_include_images` | boolean | If `true`, any images found within the document pages will be extracted and included in the output. Defaults to `false`. | No |
-| `should_include_page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No |
+| `page_images` | boolean | If `true`, any images found within the document pages will be extracted and included in the output. Defaults to `false`. | No |
+| `page_views` | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`. | No |
| `page_views_dpi` | integer | The resolution (in Dots Per Inch) for the generated page views when processing a PDF. Defaults to `150`. | No |
-| `should_caption_images` | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.* | No |
+| `page_image_captions` | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.* | No |
### Example: Processing a PDF
@@ -48,13 +48,13 @@ refines = "Page"
[pipe.extract_text_from_document]
type = "PipeOcr"
-description = "Extract text from a scanned document"
+definition = "Extract text from a scanned document"
inputs = { ocr_input = "ScannedDocument" }
-output = "ExtractedText"
-should_include_page_views = true
+output = "Page"
+page_views = true
page_views_dpi = 200
```
-The output of the PipeOcr has to be a concept compatible with the native `Page` concept. A concept is compatible with the `Page` concept if it refines the `Page` concept.
+The output of the PipeOcr must be exactly the native `Page` concept.
To use this pipe, you would first need to load a PDF into the `ScannedDocument` concept. After the pipe runs, the `ExtractedPages` concept will contain a list of `PageContent` objects, where each object has the extracted text and a 200 DPI image of the corresponding page.
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
index 5dca68640..7a0099869 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
@@ -128,7 +128,7 @@ OppositePhotoPrompt = "Prompt for generating opposite version of photo"
[pipe.gen_photopposite]
type = "PipeSequence"
-description = "Generate opposite version of photo's main feature"
+definition = "Generate opposite version of photo's main feature"
output = "Image"
steps = [
{ pipe = "analyze_photo", result = "analysis" },
@@ -138,7 +138,7 @@ steps = [
[pipe.analyze_photo]
type = "PipeLLM"
-description = "Analyze photo content and identify main feature"
+definition = "Analyze photo content and identify main feature"
inputs = { photo = "Image" }
output = "PhotoAnalysis"
prompt_template = """
@@ -151,7 +151,7 @@ Focus on the dominant element, color, mood, or characteristic that defines this
[pipe.create_opposite_concept]
type = "PipeLLM"
-description = "Create concept for opposite version"
+definition = "Create concept for opposite version"
inputs = { analysis = "PhotoAnalysis" }
output = "OppositePhotoPrompt"
prompt_template = """
@@ -164,7 +164,7 @@ Describe what the opposite would look like, focusing on reversing the main featu
[pipe.render_opposite]
type = "PipeImgGen"
-description = "Generate the opposite photo"
+definition = "Generate the opposite photo"
inputs = { opposite_concept = "OppositePhotoPrompt" }
output = "Image"
```
@@ -271,13 +271,13 @@ Generated blueprints automatically integrate with:
```bash
# Validate your entire setup
-pipelex validate all -c pipelex/libraries
+pipelex validate all -c path/to/your/pipelex/config/folder
# Test a specific pipe
-pipelex validate pipe your_pipe_name -c pipelex/libraries
+pipelex validate pipe your_pipe_name -c path/to/your/pipelex/config/folder
# Check blueprint structure
-pipelex validate blueprint your_blueprint.plx -c pipelex/libraries --no-fix
+pipelex validate blueprint your_blueprint.plx -c path/to/your/pipelex/config/folder --no-fix
```
## Next Steps
diff --git a/docs/pages/configuration/config-advanced/feature-config.md b/docs/pages/configuration/config-advanced/feature-config.md
index 1997cab6a..574e1d46e 100644
--- a/docs/pages/configuration/config-advanced/feature-config.md
+++ b/docs/pages/configuration/config-advanced/feature-config.md
@@ -32,24 +32,24 @@ The feature flags directly affect which implementation is used for certain compo
### Pipeline Tracking
```toml
-is_pipeline_tracking_enabled = false
+is_pipeline_tracking_enabled = true
```
- Controls whether pipeline execution tracking is enabled
- When enabled, tracks the flow and execution of pipelines using by default mermaid chart:
- View and edit charts at [Mermaid Live Editor](https://mermaid.live)
- Useful for debugging and monitoring pipeline behavior
-- Default: `false`
+- Default: `true`
### Activity Tracking
```toml
-is_activity_tracking_enabled = false
+is_activity_tracking_enabled = true
```
- Controls whether activity tracking is enabled
- When enabled, tracks detailed information about system activities
-- Default: `false`
+- Default: `true`
### Reporting
@@ -68,8 +68,8 @@ is_reporting_enabled = true
# Enable pipeline tracking for debugging
is_pipeline_tracking_enabled = true
-# Disable activity tracking for performance
-is_activity_tracking_enabled = false
+# Enable activity tracking
+is_activity_tracking_enabled = true
# Enable reporting for cost monitoring
is_reporting_enabled = true
diff --git a/docs/pages/configuration/config-pipeline-validation/dry-run-config.md b/docs/pages/configuration/config-pipeline-validation/dry-run-config.md
index 6cb4998d8..435245640 100644
--- a/docs/pages/configuration/config-pipeline-validation/dry-run-config.md
+++ b/docs/pages/configuration/config-pipeline-validation/dry-run-config.md
@@ -8,12 +8,20 @@ The `DryRunConfig` class controls how Pipelex behaves during dry runs.
class DryRunConfig(ConfigModel):
apply_to_jinja2_rendering: bool
text_gen_truncate_length: int
+ nb_list_items: int
+ nb_ocr_pages: int
+ image_urls: List[str]
+ allowed_to_fail_pipes: List[str] = Field(default_factory=list)
```
### Fields
- `apply_to_jinja2_rendering`: When true, simulates Jinja2 template rendering during dry runs
- `text_gen_truncate_length`: Maximum length of generated text during dry runs
+- `nb_list_items`: Number of items to generate for list content during dry runs
+- `nb_ocr_pages`: Number of pages to simulate for OCR operations during dry runs
+- `image_urls`: List of image URLs to use for dry run testing (must be non-empty)
+- `allowed_to_fail_pipes`: List of pipe names that are allowed to fail during dry runs (optional)
## Example Configuration
@@ -21,6 +29,10 @@ class DryRunConfig(ConfigModel):
[pipelex.dry_run_config]
apply_to_jinja2_rendering = true
text_gen_truncate_length = 100
+nb_list_items = 3
+nb_ocr_pages = 2
+image_urls = ["https://example.com/image1.jpg", "https://example.com/image2.jpg"]
+allowed_to_fail_pipes = ["optional_pipe", "experimental_pipe"]
```
## Dry Run Behavior
diff --git a/docs/pages/configuration/config-pipeline-validation/static-validation-config.md b/docs/pages/configuration/config-pipeline-validation/static-validation-config.md
index 371ca1c72..a5023edb4 100644
--- a/docs/pages/configuration/config-pipeline-validation/static-validation-config.md
+++ b/docs/pages/configuration/config-pipeline-validation/static-validation-config.md
@@ -35,9 +35,11 @@ Each validation error type can be configured to have one of three reactions:
default_reaction = "raise"
[pipelex.static_validation_config.reactions]
-missing_input_variable = "log"
-extraneous_input_variable = "log"
-inadequate_input_concept = "log"
+# Enable one of these to tolerate some static validation errors, like you would for pyright or some linters
+# missing_input_variable = "log"
+# extraneous_input_variable = "log"
+# inadequate_input_concept = "log"
+# too_many_candidate_inputs = "log"
```
## Validation Process
diff --git a/docs/pages/configuration/config-practical/logging-config.md b/docs/pages/configuration/config-practical/logging-config.md
index c8b23553d..a7636fe5f 100644
--- a/docs/pages/configuration/config-practical/logging-config.md
+++ b/docs/pages/configuration/config-practical/logging-config.md
@@ -143,6 +143,7 @@ keywords_to_hilight = []
```toml
[pipelex.log_config]
default_log_level = "INFO"
+log_mode = "rich"
is_console_logging_enabled = true
json_logs_indent = 4
presentation_line_width = 120
diff --git a/docs/pages/configuration/config-practical/pipe-run-config.md b/docs/pages/configuration/config-practical/pipe-run-config.md
index 13acc214f..1cc9a7226 100644
--- a/docs/pages/configuration/config-practical/pipe-run-config.md
+++ b/docs/pages/configuration/config-practical/pipe-run-config.md
@@ -17,7 +17,7 @@ class PipeRunConfig(ConfigModel):
```toml
[pipelex.pipe_run_config]
-pipe_stack_limit = 100
+pipe_stack_limit = 20
```
## Stack Limit
diff --git a/docs/pages/configuration/config-practical/tracker-config.md b/docs/pages/configuration/config-practical/tracker-config.md
index 902eebf02..ecd9e0039 100644
--- a/docs/pages/configuration/config-practical/tracker-config.md
+++ b/docs/pages/configuration/config-practical/tracker-config.md
@@ -73,18 +73,18 @@ The pipeline tracker visualizes:
```toml
[tracker]
is_debug_mode = false
-is_include_text_preview = true
-is_include_interactivity = true
-theme = "auto"
-layout = "auto"
+is_include_text_preview = false
+is_include_interactivity = false
+theme = "base"
+layout = "dagre"
wrapping_width = "auto"
nb_items_limit = "unlimited"
sub_graph_colors = ["#1f77b4", "#ff7f0e", "#2ca02c"]
-pipe_edge_style = "solid"
-branch_edge_style = "dashed"
-aggregate_edge_style = "dotted"
-condition_edge_style = "dashdot"
-choice_edge_style = "solid"
+pipe_edge_style = "---->"
+branch_edge_style = "-...-"
+aggregate_edge_style = "-...-"
+condition_edge_style = "-----"
+choice_edge_style = "-----"
```
## Property Accessors
diff --git a/docs/pages/configuration/config-technical/aws-config.md b/docs/pages/configuration/config-technical/aws-config.md
index 4f9826247..e845b81ee 100644
--- a/docs/pages/configuration/config-technical/aws-config.md
+++ b/docs/pages/configuration/config-technical/aws-config.md
@@ -35,7 +35,7 @@ When using `api_key_method = "secret_provider"`, Pipelex will:
- `AWS_REGION`
⚠️ **Important**: To use the secret provider method, you must:
-1. Configure a secret provider in your project using the `SecretsProviderProtocol`: See more in the [Secrets](../../advanced-customization/secrets-provider-injection.md) documentation.
+1. Configure a secret provider in your project using the `SecretsProviderAbstract`: See more in the [Secrets](../../advanced-customization/secrets-provider-injection.md) documentation.
2. Store your AWS credentials in your secret provider
3. Ensure your secret provider is properly authenticated
diff --git a/docs/pages/configuration/config-technical/cogt-config.md b/docs/pages/configuration/config-technical/cogt-config.md
index d8e6e3d89..4c1cac795 100644
--- a/docs/pages/configuration/config-technical/cogt-config.md
+++ b/docs/pages/configuration/config-technical/cogt-config.md
@@ -5,12 +5,12 @@ The Cogt configuration manages all cognitive tools in Pipelex, including LLM (La
## Overview
```toml
-[pipelex.cogt]
+[cogt]
# Main Cogt configuration sections
-[pipelex.cogt.inference_manager_config]
-[pipelex.cogt.llm_config]
-[pipelex.cogt.imgg_config]
-[pipelex.cogt.ocr_config]
+[cogt.inference_manager_config]
+[cogt.llm_config]
+[cogt.imgg_config]
+[cogt.ocr_config]
```
## Inference Manager Configuration
@@ -18,7 +18,7 @@ The Cogt configuration manages all cognitive tools in Pipelex, including LLM (La
Controls automatic setup of various cognitive tools:
```toml
-[pipelex.cogt.inference_manager_config]
+[cogt.inference_manager_config]
is_auto_setup_preset_llm = true
is_auto_setup_preset_imgg = true
is_auto_setup_preset_ocr = true
@@ -29,22 +29,22 @@ is_auto_setup_preset_ocr = true
Configuration for all Language Model interactions:
```toml
-[pipelex.cogt.llm_config]
-default_max_images = 4 # Maximum number of images in prompts
+[cogt.llm_config]
+default_max_images = 100 # Maximum number of images in prompts
# Platform preferences for different LLMs
-[pipelex.cogt.llm_config.preferred_platforms]
+[cogt.llm_config.preferred_platforms]
gpt-4 = "openai"
claude-3-opus = "anthropic"
# Job configuration
-[pipelex.cogt.llm_config.llm_job_config]
-is_streaming_enabled = true
+[cogt.llm_config.llm_job_config]
+is_streaming_enabled = false
max_retries = 3 # Between 1 and 10
# Instructor settings
-[pipelex.cogt.llm_config.instructor_config]
-is_openai_structured_output_enabled = true
+[cogt.llm_config.instructor_config]
+is_openai_structured_output_enabled = false
```
### LLM Job Parameters
@@ -60,25 +60,25 @@ When configuring LLM jobs, you can set:
Configuration for image generation capabilities:
```toml
-[pipelex.cogt.imgg_config]
-default_imgg_handle = "stable_diffusion"
-imgg_handles = ["stable_diffusion", "dall_e"]
+[cogt.imgg_config]
+default_imgg_handle = "fal-ai/flux-pro"
+imgg_handles = ["fal-ai/flux-pro", "fal-ai/fast-lightning-sdxl"]
-[pipelex.cogt.imgg_config.imgg_job_config]
-is_sync_mode = true
+[cogt.imgg_config.imgg_job_config]
+is_sync_mode = false
# Default parameters for image generation
-[pipelex.cogt.imgg_config.imgg_param_defaults]
+[cogt.imgg_config.imgg_param_defaults]
aspect_ratio = "square" # Options: square, landscape_4_3, landscape_3_2, landscape_16_9, landscape_21_9,
- # portrait_4_3, portrait_2_3, portrait_9_16, portrait_9_21
+ # portrait_3_4, portrait_2_3, portrait_9_16, portrait_9_21
background = "auto" # Options: transparent, opaque, auto
-quality = "high" # Options: low, medium, high
-nb_steps = 50 # Number of diffusion steps
-guidance_scale = 7.5 # Controls adherence to prompt
+quality = "low" # Options: low, medium, high
+# nb_steps = 1 # Number of diffusion steps (28 is good for Flux, [1,2,4,8] for SDXL Lightning)
+guidance_scale = 3.5 # Controls adherence to prompt
is_moderated = true # Enable content moderation
-safety_tolerance = 3 # Safety level (1-6)
+safety_tolerance = 5 # Safety level (1-6)
is_raw = false # Raw output mode
-output_format = "png" # Options: png, jpg, webp
+output_format = "jpg" # Options: png, jpg, webp
seed = "auto" # "auto" or specific integer
```
@@ -113,9 +113,10 @@ Image generation jobs support these parameters:
Configuration for Optical Character Recognition:
```toml
-[pipelex.cogt.ocr_config]
-ocr_handles = ["tesseract", "azure_ocr"]
-page_output_text_file_name = "page_text.txt"
+[cogt.ocr_config]
+ocr_handles = ["mistral/mistral-ocr-latest"]
+page_output_text_file_name = "page_text.md"
+default_page_views_dpi = 72
```
## Validation Rules
@@ -155,43 +156,43 @@ page_output_text_file_name = "page_text.txt"
## Example Complete Configuration
```toml
-[pipelex.cogt]
-[pipelex.cogt.inference_manager_config]
+[cogt]
+[cogt.inference_manager_config]
is_auto_setup_preset_llm = true
is_auto_setup_preset_imgg = true
is_auto_setup_preset_ocr = true
-[pipelex.cogt.llm_config]
-default_max_images = 4
+[cogt.llm_config]
+default_max_images = 100
preferred_platforms = { "gpt-4" = "openai", "claude-3-opus" = "anthropic" }
-[pipelex.cogt.llm_config.llm_job_config]
-is_streaming_enabled = true
+[cogt.llm_config.llm_job_config]
+is_streaming_enabled = false
max_retries = 3
-[pipelex.cogt.llm_config.instructor_config]
-is_openai_structured_output_enabled = true
+[cogt.llm_config.instructor_config]
+is_openai_structured_output_enabled = false
-[pipelex.cogt.imgg_config]
-default_imgg_handle = "stable_diffusion"
-imgg_handles = ["stable_diffusion", "dall_e"]
+[cogt.imgg_config]
+default_imgg_handle = "fal-ai/flux-pro/v1.1-ultra"
+imgg_handles = ["fal-ai/flux-pro", "fal-ai/fast-lightning-sdxl"]
-[pipelex.cogt.imgg_config.imgg_job_config]
-is_sync_mode = true
+[cogt.imgg_config.imgg_job_config]
+is_sync_mode = false
-[pipelex.cogt.imgg_config.imgg_param_defaults]
+[cogt.imgg_config.imgg_param_defaults]
aspect_ratio = "square"
background = "auto"
-quality = "high"
-nb_steps = 50
-guidance_scale = 7.5
+quality = "low"
+# nb_steps = 1
+guidance_scale = 3.5
is_moderated = true
-safety_tolerance = 3
+safety_tolerance = 5
is_raw = false
-output_format = "png"
+output_format = "jpg"
seed = "auto"
-[pipelex.cogt.ocr_config]
-ocr_handles = ["tesseract", "azure_ocr"]
-page_output_text_file_name = "page_text.txt"
+[cogt.ocr_config]
+ocr_handles = ["mistral/mistral-ocr-latest"]
+page_output_text_file_name = "page_text.md"
```
diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.md
index f1e7615a9..7b7bd9a2c 100644
--- a/docs/pages/configuration/config-technical/library-config.md
+++ b/docs/pages/configuration/config-technical/library-config.md
@@ -41,22 +41,6 @@ pipelex_libraries/ # Exported library root
- Validates pipe configurations
- Links pipes with their respective domains
-## Configuration Options
-
-### Path Configuration
-
-All paths are configurable through class variables in `LibraryConfig`:
-
-```python
-package_name = "pipelex"
-internal_library_root = "libraries"
-exported_library_root = "pipelex_libraries"
-exported_pipelines_path = "pipelex_libraries/pipelines"
-exported_llm_integrations_path = "pipelex_libraries/llm_integrations"
-exported_llm_deck_path = "pipelex_libraries/llm_deck"
-exported_templates_path = "pipelex_libraries/templates"
-```
-
### Library Initialization
Use the CLI command to initialize libraries:
@@ -122,7 +106,7 @@ The library system includes specific error types:
2. **Validation**:
- - Run `pipelex validate all -c pipelex/libraries` after making changes
+ - Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes
- Check for domain consistency
- Verify concept relationships
diff --git a/docs/pages/configuration/index.md b/docs/pages/configuration/index.md
index 73d353d06..2c0cf18e2 100644
--- a/docs/pages/configuration/index.md
+++ b/docs/pages/configuration/index.md
@@ -51,7 +51,7 @@ Each subsequent configuration file in this sequence can override settings from t
- Base config: `pipelex.toml`
- Local overrides: `pipelex_local.toml`
- Environment overrides: `pipelex_dev.toml`, `pipelex_staging.toml`, `pipelex_prod.toml`, etc.
-- Run mode overrides: `pipelex_normal.toml`, `pipelex_unit_test.toml`, etc.
+- Run mode overrides: `pipelex_normal.toml`, `tests/pipelex_unit_test.toml`, etc.
- Super user overrides: `pipelex_super.toml`
NB: The run_mode unit_test is used for testing purposes.
@@ -69,5 +69,5 @@ NB: The run_mode unit_test is used for testing purposes.
1. **Version Control**: Include your base `pipelex.toml` in version control
2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings
3. **Documentation**: Comment any custom settings for team reference
-4. **Validation**: Run `pipelex validate all -c pipelex/libraries` after making configuration changes
+4. **Validation**: Run `pipelex validate all -c path/to/your/pipelex/config/folder` after making configuration changes
5. **Gitignore**: Add local and sensitive override files to `.gitignore`
diff --git a/docs/pages/cookbook-examples/extract-dpe.md b/docs/pages/cookbook-examples/extract-dpe.md
index 685ac210b..5e11cf882 100644
--- a/docs/pages/cookbook-examples/extract-dpe.md
+++ b/docs/pages/cookbook-examples/extract-dpe.md
@@ -14,7 +14,7 @@ The pipeline `power_extractor_dpe` is designed to recognize and extract the key
async def extract_dpe(pdf_url: str) -> Dpe:
working_memory = WorkingMemoryFactory.make_from_pdf(
pdf_url=pdf_url,
- concept_str="PDF",
+ concept_string="PDF",
name="pdf",
)
pipe_output = await execute_pipeline(
@@ -61,7 +61,7 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informa
```plx
[pipe.write_markdown_from_page_content_dpe]
type = "PipeLLM"
-description = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"
+definition = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"
inputs = { page_content = "Page" }
output = "Dpe" # The output is structured as a Dpe object
llm = "llm_for_img_to_text"
diff --git a/docs/pages/cookbook-examples/extract-gantt.md b/docs/pages/cookbook-examples/extract-gantt.md
index 2bbe8ac83..450ca3b63 100644
--- a/docs/pages/cookbook-examples/extract-gantt.md
+++ b/docs/pages/cookbook-examples/extract-gantt.md
@@ -15,7 +15,7 @@ async def extract_gantt(image_url: str) -> GanttChart:
# Create Working Memory
working_memory = WorkingMemoryFactory.make_from_image(
image_url=image_url,
- concept_str="gantt.GanttImage",
+ concept_string="gantt.GanttImage",
name="gantt_chart_image",
)
@@ -60,7 +60,7 @@ The `extract_gantt_by_steps` pipeline is a sequence of smaller, focused pipes. T
```plx
[pipe.extract_gantt_by_steps]
type = "PipeSequence"
-description = "Extract all details from a gantt chart"
+definition = "Extract all details from a gantt chart"
inputs = { gantt_chart_image = "GanttChartImage" }
output = "GanttChart"
steps = [
@@ -77,7 +77,7 @@ steps = [
# This is the pipe that extracts the details for a single task
[pipe.extract_details_of_task]
type = "PipeLLM"
-description = "Extract the precise dates of the task, start_date and end_date"
+definition = "Extract the precise dates of the task, start_date and end_date"
inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }
output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails object
structuring_method = "preliminary_text"
diff --git a/docs/pages/cookbook-examples/extract-generic.md b/docs/pages/cookbook-examples/extract-generic.md
index cf2f5a642..a08e5e586 100644
--- a/docs/pages/cookbook-examples/extract-generic.md
+++ b/docs/pages/cookbook-examples/extract-generic.md
@@ -14,7 +14,7 @@ The `power_extractor` pipeline is at the heart of this example. After its execut
async def extract_generic(pdf_url: str) -> TextAndImagesContent:
working_memory = WorkingMemoryFactory.make_from_pdf(
pdf_url=pdf_url,
- concept_str="PDF",
+ concept_string="PDF",
name="pdf",
)
pipe_output = await execute_pipeline(
@@ -31,9 +31,9 @@ The `merge_markdown_and_images` function is a great example of how you can add y
```python
def merge_markdown_and_images(working_memory: WorkingMemory) -> TextAndImagesContent:
# Pages extracted from the PDF by PipeOCR
- page_contents_list = working_memory.get_stuff_as_list(item_type=PageContent, name="page_contents")
+ page_contents_list = working_memory.get_stuff_as_list(name="page_contents", item_type=PageContent)
# Markdown text extracted from the Pages by PipeLLM
- page_markdown_list = working_memory.get_stuff_as_list(item_type=TextContent, name="markdowns")
+ page_markdown_list = working_memory.get_stuff_as_list(name="markdowns", item_type=TextContent)
# ... (check for length equality)
diff --git a/docs/pages/cookbook-examples/extract-proof-of-purchase.md b/docs/pages/cookbook-examples/extract-proof-of-purchase.md
index 792a4d27c..d952d908d 100644
--- a/docs/pages/cookbook-examples/extract-proof-of-purchase.md
+++ b/docs/pages/cookbook-examples/extract-proof-of-purchase.md
@@ -14,7 +14,7 @@ The pipeline `power_extractor_proof_of_purchase` is specifically designed to han
async def extract_proof_of_purchase(pdf_url: str) -> ProofOfPurchase:
working_memory = WorkingMemoryFactory.make_from_pdf(
pdf_url=pdf_url,
- concept_str="PDF",
+ concept_string="PDF",
name="pdf",
)
pipe_output = await execute_pipeline(
@@ -57,7 +57,7 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d
```plx
[pipe.write_markdown_from_page_content_proof_of_purchase]
type = "PipeLLM"
-description = "Write markdown from page content"
+definition = "Write markdown from page content"
inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the page
output = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase object
llm = "llm_for_img_to_text"
diff --git a/docs/pages/cookbook-examples/extract-table.md b/docs/pages/cookbook-examples/extract-table.md
index 54b58a591..b212e0d70 100644
--- a/docs/pages/cookbook-examples/extract-table.md
+++ b/docs/pages/cookbook-examples/extract-table.md
@@ -14,7 +14,7 @@ The pipeline `extract_html_table_and_review` takes an image of a table, processe
async def extract_table(table_screenshot: str) -> HtmlTable:
working_memory = WorkingMemoryFactory.make_from_image(
image_url=table_screenshot,
- concept_str="tables.TableScreenshot",
+ concept_string="tables.TableScreenshot",
name="table_screenshot",
)
pipe_output = await execute_pipeline(
@@ -62,7 +62,7 @@ The pipeline uses a two-step "extract and review" pattern. The first pipe does t
```plx
[pipe.extract_html_table_and_review]
type = "PipeSequence"
-description = "Get an HTML table and review it"
+definition = "Get an HTML table and review it"
inputs = { table_screenshot = "TableScreenshot" }
output = "HtmlTable"
steps = [
@@ -74,7 +74,7 @@ steps = [
[pipe.review_html_table]
type = "PipeLLM"
-description = "Review an HTML table"
+definition = "Review an HTML table"
inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }
output = "HtmlTable"
prompt_template = """
diff --git a/docs/pages/cookbook-examples/hello-world.md b/docs/pages/cookbook-examples/hello-world.md
index 2882a2289..98602b9a3 100644
--- a/docs/pages/cookbook-examples/hello-world.md
+++ b/docs/pages/cookbook-examples/hello-world.md
@@ -17,7 +17,7 @@ The `hello_world` function demonstrates the simplest possible Pipelex pipeline.
```python
import asyncio
-from pipelex import pretty_print
+from pipelex.tools.misc.pretty import pretty_print
from pipelex.pipelex import Pipelex
from pipelex.pipeline.execute import execute_pipeline
@@ -52,7 +52,7 @@ definition = "Discovering Pipelex"
[pipe]
[pipe.hello_world]
type = "PipeLLM"
-description = "Write text about Hello World."
+definition = "Write text about Hello World."
output = "Text"
llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }
prompt = """
diff --git a/docs/pages/cookbook-examples/invoice-extractor.md b/docs/pages/cookbook-examples/invoice-extractor.md
index 5824c5362..78086b372 100644
--- a/docs/pages/cookbook-examples/invoice-extractor.md
+++ b/docs/pages/cookbook-examples/invoice-extractor.md
@@ -65,7 +65,7 @@ The entire workflow is defined in a PLX file. This declarative approach makes th
# The main pipeline, a sequence of steps
[pipe.process_invoice]
type = "PipeSequence"
-description = "Process relevant information from an invoice"
+definition = "Process relevant information from an invoice"
inputs = { invoice_pdf = "PDF" }
output = "Invoice"
steps = [
@@ -78,7 +78,7 @@ steps = [
# A sub-pipeline that uses an LLM to extract the data
[pipe.extract_invoice_data]
type = "PipeLLM"
-description = "Extract invoice information from an invoice text transcript"
+definition = "Extract invoice information from an invoice text transcript"
inputs = { "invoice_page.page_view" = "Page", invoice_details = "InvoiceDetails" }
output = "Invoice"
# The output is constrained to the "Invoice" model
diff --git a/docs/pages/cookbook-examples/simple-ocr.md b/docs/pages/cookbook-examples/simple-ocr.md
index aa6350944..c2844f100 100644
--- a/docs/pages/cookbook-examples/simple-ocr.md
+++ b/docs/pages/cookbook-examples/simple-ocr.md
@@ -16,7 +16,7 @@ The core of this example is a simple function that creates a "working memory" fr
async def simple_ocr(pdf_url: str):
working_memory = WorkingMemoryFactory.make_from_pdf(
pdf_url=pdf_url,
- concept_str="PDF",
+ concept_string="PDF",
name="pdf",
)
pipe_output = await execute_pipeline(
diff --git a/docs/pages/cookbook-examples/write-screenplay.md b/docs/pages/cookbook-examples/write-screenplay.md
index 6cc8db9b0..6b324ca1f 100644
--- a/docs/pages/cookbook-examples/write-screenplay.md
+++ b/docs/pages/cookbook-examples/write-screenplay.md
@@ -15,9 +15,9 @@ async def generate_screenplay(pitch: str):
"""Generate a screenplay from a pitch using the pipeline."""
# Create Stuff object for the pitch
- pitch_stuff = StuffFactory.make_from_str(
- str_value=pitch,
- concept_str="screenplay.Pitch",
+ pitch_stuff = StuffFactory.make_from_concept_string(
+ concept_string="screenplay.Pitch",
+ content=TextContent(text=pitch),
name="pitch",
)
@@ -82,7 +82,7 @@ The `generate_screenplay` pipeline is a master `PipeSequence` that orchestrates
```plx
[pipe.generate_screenplay]
type = "PipeSequence"
-description = "Generate a complete screenplay from a pitch"
+definition = "Generate a complete screenplay from a pitch"
inputs = { pitch = "Pitch" }
output = "FormattedScreenplay"
steps = [
@@ -101,7 +101,7 @@ steps = [
# This is the nested sequence that creates all the scenes for a single chapter.
[pipe.create_scenes_sequence]
type = "PipeSequence"
-description = "Create all scenes for a chapter sequentially"
+definition = "Create all scenes for a chapter sequentially"
inputs = { chapter = "Chapter", characters = "CharacterList", detailed_pitch = "DetailedPitch" }
output = "Chapter"
steps = [
diff --git a/docs/pages/cookbook-examples/write-tweet.md b/docs/pages/cookbook-examples/write-tweet.md
index 224d84f0f..81c22cd1c 100644
--- a/docs/pages/cookbook-examples/write-tweet.md
+++ b/docs/pages/cookbook-examples/write-tweet.md
@@ -13,13 +13,13 @@ The `optimize_tweet` function is the core of this example. It takes two strings,
```python
async def optimize_tweet(draft_tweet_str: str, writing_style_str: str) -> OptimizedTweet:
# Create the draft tweet stuff
- draft_tweet = StuffFactory.make_stuff(
- concept_str="tech_tweet.DraftTweet",
+ draft_tweet = StuffFactory.make_from_concept_string(
+ concept_string="tech_tweet.DraftTweet",
content=TextContent(text=draft_tweet_str),
name="draft_tweet",
)
- writing_style = StuffFactory.make_stuff(
- concept_str="tech_tweet.WritingStyle",
+ writing_style = StuffFactory.make_from_concept_string(
+ concept_string="tech_tweet.WritingStyle",
content=TextContent(text=writing_style_str),
name="writing_style",
)
@@ -62,7 +62,7 @@ This pipeline uses a two-step "analyze and optimize" sequence. The first pipe an
```plx
[pipe.optimize_tweet_sequence]
type = "PipeSequence"
-description = "Analyze and optimize a tech tweet in sequence"
+definition = "Analyze and optimize a tech tweet in sequence"
inputs = { draft_tweet = "DraftTweet", writing_style = "WritingStyle" }
output = "OptimizedTweet"
steps = [
@@ -75,7 +75,7 @@ steps = [
# This is the pipe that analyzes the draft tweet.
[pipe.analyze_tweet]
type = "PipeLLM"
-description = "Analyze the draft tweet and identify areas for improvement"
+definition = "Analyze the draft tweet and identify areas for improvement"
inputs = { draft_tweet = "DraftTweet" }
output = "TweetAnalysis"
system_prompt = """
diff --git a/docs/pages/migration/index.md b/docs/pages/migration/index.md
index 92c6b2a81..252ea9112 100644
--- a/docs/pages/migration/index.md
+++ b/docs/pages/migration/index.md
@@ -40,15 +40,15 @@ Use the built-in migration command to automatically update your PLX files:
```bash
# Preview changes without applying them
-pipelex migrate --dry-run
+pipelex migrate run --dry-run
# Apply the migration to all PLX files
-pipelex migrate
+pipelex migrate run
```
The migration command will:
-- Find all `.plx` files in your configured pipelines directory
+- Find all `.toml` files in your configured pipelines directory
- Replace `Concept =` with `definition =` in concept definitions
- Preserve all other formatting and comments
- Create backups of modified files (with `.backup` extension)
@@ -59,7 +59,7 @@ If you prefer to migrate manually or need to handle special cases:
1. **Locate your PLX files**: Find all pipeline library files (typically in your configured pipelines directory)
2. **Update concept definitions**: Change `Concept =` to `definition =` in all `[concept.ConceptName]` sections
-3. **Validate syntax**: Run `pipelex validate all -c pipelex/libraries` to ensure your files are correctly formatted
+3. **Validate syntax**: Run `pipelex validate all -c path/to/your/pipelex/config/folder -c ./pipelex_libraries` to ensure your files are correctly formatted
### Examples
@@ -103,10 +103,10 @@ After migration, validate your pipeline files:
```bash
# Validate all pipeline files
-pipelex validate all
+pipelex validate all -c path/to/your/pipelex/config/folder
# Run a specific pipeline to test
-pipelex run your-pipeline-name
+# Pipeline execution is done through Python code, not CLI (coming soon)
```
### Troubleshooting
@@ -169,15 +169,15 @@ Use the built-in migration command to automatically update your PLX files:
```bash
# Preview changes without applying them
-pipelex migrate --dry-run
+pipelex migrate run --dry-run
# Apply the migration to all PLX files
-pipelex migrate
+pipelex migrate run
```
The migration command will:
-- Find all `.plx` files in your configured pipelines directory
+- Find all `.toml` files in your configured pipelines directory
- Convert `PipeClassName = "description"` to `type = "PipeClassName"` and `definition = "description"`
- Preserve all other formatting and comments
- Create backups of modified files (with `.backup` extension)
@@ -192,7 +192,7 @@ If you prefer to migrate manually:
- Replace `PipeClassName = "description"` with:
- `type = "PipeClassName"`
- `definition = "description"`
-3. **Validate syntax**: Run `pipelex validate all -c pipelex/libraries` to ensure your files are correctly formatted
+3. **Validate syntax**: Run `pipelex validate all -c path/to/your/pipelex/config/folder -c ./pipelex_libraries` to ensure your files are correctly formatted
### Examples
@@ -254,10 +254,10 @@ After migration, validate your pipeline files:
```bash
# Validate all pipeline files
-pipelex validate all
+pipelex validate all -c path/to/your/pipelex/config/folder
# Run a specific pipeline to test
-pipelex run your-pipeline-name
+# Pipeline execution is done through Python code, not CLI (coming soon)
```
---
diff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.md
index 1f5ef9420..8cb6c5632 100644
--- a/docs/pages/quick-start/index.md
+++ b/docs/pages/quick-start/index.md
@@ -23,7 +23,7 @@ domain = "characters"
[pipe]
[pipe.create_character]
type = "PipeLLM"
-description = "Creates a character."
+definition = "Creates a character."
output = "Text"
prompt_template = """You are a book writer. Your task is to create a character.
Think of it and then output the character description."""
@@ -69,7 +69,7 @@ python character.py
```plx
[pipe.create_character]
type = "PipeLLM"
-description = "Create a character."
+definition = "Create a character."
output = "Text"
llm = { llm_handle = "gpt-4o-mini", temperature = 0.9, max_tokens = "auto" }
prompt_template = """You are a book writer. Your task is to create a character.
@@ -81,7 +81,7 @@ Think of it and then output the character description."""
```plx
[pipe.create_character]
type = "PipeLLM"
-description = "Create a character."
+definition = "Create a character."
output = "Text"
llm = "llm_for_creative_writing"
prompt_template = """You are a book writer. Your task is to create a character.
@@ -134,7 +134,7 @@ Character = "A character is a fiction story" # <- Define here your output concep
[pipe]
[pipe.create_character]
type = "PipeLLM"
-description = "Create a character. Get a structured result."
+definition = "Create a character. Get a structured result."
output = "Character" # <- This is the output concept for your pipe
prompt_template = """You are a book writer. Your task is to create a character.
Think of it and then output the character description."""
@@ -191,7 +191,7 @@ CharacterMetadata = "Metadata regarding a character."
[pipe]
[pipe.extract_character_1]
type = "PipeLLM"
-description = "Get character information from a description."
+definition = "Get character information from a description."
inputs = { character = "Character" } # <- These are the inputs of your pipe, usable in the prompt_template
output = "CharacterMetadata"
prompt_template = """
@@ -235,8 +235,8 @@ async def process_existing_character():
He moves with quiet purpose and speaks with a calm, thoughtful cadence that suggests he's always listening for more than just what's said.""",
)
# Wrap it into a stuff object
- character_stuff = StuffFactory.make_stuff(
- concept_code="character.Character", # <- `character` is the domain, `Character` is the concept name
+ character_stuff = StuffFactory.make_from_concept_string(
+ concept_string="character.Character", # <- `character` is the domain, `Character` is the concept name
name="character",
content=character,
)
@@ -245,7 +245,7 @@ async def process_existing_character():
stuff=character_stuff,
)
# Run the pipe identified by its pipe_code (it's the name of the pipe)
- pipe_output, pipeline_run_id = await execute_pipeline(
+ pipe_output = await execute_pipeline(
pipe_code="extract_character_1",
working_memory=working_memory,
)
diff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.md
index 9ad9e9bf3..37b27f9f7 100644
--- a/docs/pages/tools/cli.md
+++ b/docs/pages/tools/cli.md
@@ -37,14 +37,14 @@ pipelex show pipe PIPE_CODE [-c/--config-folder-path PATH]
Migrate TOML files to the new syntax.
```bash
-pipelex migrate run [-c/--config-folder-path PATH] [--dry-run] [--backups/--no-backups]
+pipelex migrate run [-p/--path PATH] [--dry-run] [--backups/--no-backups]
```
## Usage Tips
-1. Always run `pipelex validate all -c pipelex/libraries` after making changes to your configuration or pipelines
+1. Always run `pipelex validate all -c path/to/your/pipelex/config/folder` after making changes to your configuration or pipelines
2. Use `pipelex show config` to debug configuration issues
3. When initializing a new project:
- Start with `pipelex init config`
- Then run `pipelex init libraries`
- - Finally, validate your setup with `pipelex validate all -c pipelex/libraries`
+ - Finally, validate your setup with `pipelex validate all -c path/to/your/pipelex/config/folder`
diff --git a/mkdocs.yml b/mkdocs.yml
index c56c4d6c3..0bb589d3a 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -154,6 +154,7 @@ nav:
- Reporting Delegate: pages/advanced-customization/reporting-delegate-injection.md
- Pipeline Tracker: pages/advanced-customization/pipeline-tracker-injection.md
- Activity Manager: pages/advanced-customization/activity-manager-injection.md
+ - Storage Provider: pages/advanced-customization/storage-provider-injection.md
- Template Provider: pages/advanced-customization/template-provider-injection.md
- Content Generator: pages/advanced-customization/content-generator-injection.md
- LLM Model Provider: pages/advanced-customization/llm-model-provider-injection.md
diff --git a/pipelex/cogt/imgg/imgg_job_components.py b/pipelex/cogt/imgg/imgg_job_components.py
index 3eb32e539..1987672f3 100644
--- a/pipelex/cogt/imgg/imgg_job_components.py
+++ b/pipelex/cogt/imgg/imgg_job_components.py
@@ -12,7 +12,7 @@ class AspectRatio(StrEnum):
LANDSCAPE_3_2 = "landscape_3_2"
LANDSCAPE_16_9 = "landscape_16_9"
LANDSCAPE_21_9 = "landscape_21_9"
- PORTRAIT_3_4 = "portrait_4_3"
+ PORTRAIT_3_4 = "portrait_3_4"
PORTRAIT_2_3 = "portrait_2_3"
PORTRAIT_9_16 = "portrait_9_16"
PORTRAIT_9_21 = "portrait_9_21"
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index b9ca86d27..5d686394c 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -163,7 +163,7 @@ structure_from_preliminary_text_user = "structure_from_preliminary_text_user"
default_reaction = "raise"
[pipelex.static_validation_config.reactions]
-# enable one of these to tolerate some static validation errors, like you would for pyright or some linters
+# default_reaction is on raise
# missing_input_variable = "log"
# extraneous_input_variable = "log"
# inadequate_input_concept = "log"
diff --git a/pipelex/tools/config/manager.py b/pipelex/tools/config/manager.py
index 76e8ff580..454ff0bc9 100644
--- a/pipelex/tools/config/manager.py
+++ b/pipelex/tools/config/manager.py
@@ -104,7 +104,6 @@ class ConfigManager:
- pipelex_local.toml (before env)
- pipelex_{environment}.toml
- pipelex_{run_mode}.toml
- - pipelex_user.toml (after env)
Returns:
Dict[str, Any]: The merged configuration dictionary
diff --git a/pyproject.toml b/pyproject.toml
index 5a67e0e50..bd920bbdf 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "pipelex"
-version = "0.9.0"
+version = "0.9.1"
description = "Pipelex is an open-source dev tool based on a simple declarative language that lets you define replicable, structured, composable LLM pipelines."
authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]
maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]
@@ -67,7 +67,7 @@ docs = [
dev = [
"boto3-stubs>=1.35.24",
- "cocode>=0.0.15",
+ "cocode==0.1.2",
"mypy>=1.11.2",
"pyright==1.1.398",
"pandas-stubs>=2.2.3.241126",
diff --git a/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
index 561a64696..086ae324c 100644
--- a/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
+++ b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
@@ -131,7 +131,7 @@ refines = "Text"
"""Test that migration doesn't affect non-concept Concept = lines."""
# These should NOT be changed
test_content = """# This is a comment about Concept = something
-description = "This pipe works with Concept = something"
+definition = "This pipe works with Concept = something"
prompt_template = "Define the Concept = whatever"
"""
diff --git a/tests/unit/pipelex/tools/misc/test_toml_utils.py b/tests/unit/pipelex/tools/misc/test_toml_utils.py
index d9ac548eb..8124dbbdb 100644
--- a/tests/unit/pipelex/tools/misc/test_toml_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_toml_utils.py
@@ -60,7 +60,7 @@ definition = "Test"
[pipe.test_pipe]
type = "PipeLLM"
-description = "Test"
+definition = "Test"
prompt_template = """
Output this only: "test"
"""
diff --git a/uv.lock b/uv.lock
index bde626364..c2f48854d 100644
--- a/uv.lock
+++ b/uv.lock
@@ -477,15 +477,15 @@ wheels = [
[[package]]
name = "cocode"
-version = "0.0.15"
+version = "0.1.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pipelex", extra = ["anthropic", "bedrock", "google"] },
{ name = "pygithub" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/ba/71/11df50dcd296239d228c4fb89a51be1aeef91d5432d71a8a0da0d77cfad5/cocode-0.0.15.tar.gz", hash = "sha256:1a7cebf33dcc2fa8ae58ea990cb6ff0e96e9e37fecdf69c3a5e59651b9080d85", size = 43564, upload-time = "2025-08-29T13:52:33.395Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/08/a8/890526fa2f3cab311cb7a6d56fedd56c6829254fe13ca58435883f14c21f/cocode-0.1.2.tar.gz", hash = "sha256:63b7f9c51bd18c55f485051d714577a5b755498a20e6a9c69fa9ed734fb47801", size = 43905, upload-time = "2025-09-03T09:14:33.233Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b8/fe/9d74caefb69c0a597c7f4eb78afb8bc4357f6ef1d8b49a69f223722b459e/cocode-0.0.15-py3-none-any.whl", hash = "sha256:84f91b81bd6c9a9d2a2c75b0962b5f6b778de0dbb1fd6671d2f2fb35630f6d52", size = 60138, upload-time = "2025-08-29T13:52:32.214Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/56/e37fd29be59dca658a22aef6328f475850ff4b8cc988e60e04ccdc9fb2b4/cocode-0.1.2-py3-none-any.whl", hash = "sha256:32aba620170da4e1855394fdf37274bf443a98fba98f0d5da1d89a06f06aa8c6", size = 60516, upload-time = "2025-09-03T09:14:32.072Z" },
]
[[package]]
@@ -1865,7 +1865,7 @@ wheels = [
[[package]]
name = "pipelex"
-version = "0.9.0"
+version = "0.9.1"
source = { editable = "." }
dependencies = [
{ name = "aiofiles" },
@@ -1952,7 +1952,7 @@ requires-dist = [
{ name = "backports-strenum", marker = "python_full_version < '3.11'", specifier = ">=1.3.0" },
{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },
{ name = "boto3-stubs", marker = "extra == 'dev'", specifier = ">=1.35.24" },
- { name = "cocode", marker = "extra == 'dev'", specifier = ">=0.0.15" },
+ { name = "cocode", marker = "extra == 'dev'", specifier = "==0.1.2" },
{ name = "fal-client", marker = "extra == 'fal'", specifier = ">=0.4.1" },
{ name = "filetype", specifier = ">=1.2.0" },
{ name = "google-auth-oauthlib", marker = "extra == 'google'", specifier = ">=1.2.1" },