Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Fix/remove images field from pipe llm blueprint (#105)

Thomas Hebrard336cb02b8398
diff --git a/.cursor/rules/best_practices.mdc b/.cursor/rules/best_practices.mdc
new file mode 100644
index 000000000..66bc8dc36
--- /dev/null
+++ b/.cursor/rules/best_practices.mdc
@@ -0,0 +1,77 @@
+---
+description:
+globs:
+alwaysApply: false
+---
+# Best Practices Guide
+
+This document outlines the core best practices and patterns used in our codebase.
+
+## Type Hints
+
+1. **Always Use Type Hints**
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+
+2. **StrEnum**
+ - Import StrEnum from pipelex.types
+ ```python
+ from pipelex.types import StrEnum
+
+ class ModelType(StrEnum):
+ GPT4 = "gpt-4"
+ GPT35 = "gpt-3.5-turbo"
+ ```
+
+## Factory Pattern
+
+1. **Use Factory Pattern for Object Creation**
+ - Create factories when dealing with multiple implementations
+
+## Documentation
+
+1. **Docstring Format**
+ - Quick description of the function/class
+ - List args and their types
+ - Document return values
+ - Example:
+ ```python
+ def process_image(image_path: str, size: Tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
+ pass
+ ```
+
+2. **Class Documentation**
+ - Document class purpose and behavior
+ - Include examples if complex
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
+
+## Custom Exceptions
+
+1. **Graceful Error Handling**
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to custom ones
+ ```python
+ try:
+ from fal_client import AsyncClient as FalAsyncClient
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "fal-client", "fal",
+ "The fal-client SDK is required to use FAL models."
+ ) from exc
+ ```
diff --git a/.cursor/rules/pipes.mdc b/.cursor/rules/pipes.mdc
index a32b3d3bc..f13c986e9 100644
--- a/.cursor/rules/pipes.mdc
+++ b/.cursor/rules/pipes.mdc
@@ -5,20 +5,9 @@ alwaysApply: false
---
This rule explains how to build pipes.
-# File Naming & Structure
-
-## The pipelines/ directory
-
-- Pipelines and structures are defined in the `pipelex/libraries/pipelines` directory
-
-pipelex/libraries
-└── pipelines
-
## Pipeline file naming
-- Pipeline TOML files should be placed in the `pipelex/libraries/pipelines/` directory or a subdirectory of it
- The file name should be descriptive, in snake_case and end with `.toml`
-- The file should contain both concepts and pipes definitions
## Pipeline file structure
@@ -167,7 +156,6 @@ Your summary should not be longer than 2 sentences.
PipeLLM = "Convert table screenshot to HTML"
inputs = { table_screenshot = "TableScreenshot" }
output = "HtmlTable"
-images = ["table_screenshot"]
system_prompt = """
You are a vision-based table extractor.
"""
diff --git a/.cursor/rules/pytest.mdc b/.cursor/rules/pytest.mdc
index b44f68fae..2b22e5bea 100644
--- a/.cursor/rules/pytest.mdc
+++ b/.cursor/rules/pytest.mdc
@@ -17,6 +17,8 @@ These rules apply when writing unit tests.
- More precisely, for `pipelex` and `pipelex.cogt` the async tests are placed inside subdirectories named `cogt_asynch` and `pipelex_asynch`
- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put test inside Test classes.
+- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`
## Markers
diff --git a/.cursor/rules/standards.mdc b/.cursor/rules/standards.mdc
new file mode 100644
index 000000000..4c86f0e1e
--- /dev/null
+++ b/.cursor/rules/standards.mdc
@@ -0,0 +1,60 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+# Coding Standards
+
+This document outlines the coding standards and quality control procedures that must be followed when contributing to this project.
+
+## Code Quality Checks
+
+### Linting and Type Checking
+
+Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:
+
+```bash
+make check
+```
+
+This command runs multiple code quality tools:
+- Pyright: Static type checking
+- Ruff: Fast Python linter
+- Mypy: Static type checker
+
+Always fix any issues reported by these tools before proceeding.
+
+### Running Tests
+
+We have several make commands for running tests:
+
+1. `make tp`: Runs all tests with these markers:
+ ```
+ (dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)
+ ```
+ Use this for quick test runs that don't require LLM or image generation.
+
+2. `make ti`: Runs all tests with these markers:
+ ```
+ inference and not imgg
+ ```
+ Use this for testing LLM functionality without image generation.
+
+3. To run specific tests:
+ ```bash
+ make tp TEST=TestClassName
+ # or
+ make tp TEST=test_function_name
+ ```
+ It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.
+
+## Important Project Directories
+
+### Pipelines Directory
+- All pipeline definitions go in `pipelex/libraries/pipelines/`
+
+### Tests Directory
+- All tests are located in the `tests/` directory
+
+### Documentation Directory
+- All documentation is located in the `docs/` directory
diff --git a/.cursor/rules/structures.mdc b/.cursor/rules/structures.mdc
index 9671eef39..d8a620a91 100644
--- a/.cursor/rules/structures.mdc
+++ b/.cursor/rules/structures.mdc
@@ -59,3 +59,4 @@ Structure classes defined within `pipelex_libraries/pipelines/` are automaticall
- Use Pydantic validators for data cleaning/validation
- Remove timezone info from datetime fields
- Respect rules of @base_models.
+
diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdc
new file mode 100644
index 000000000..5abebf380
--- /dev/null
+++ b/.cursor/rules/tdd.mdc
@@ -0,0 +1,28 @@
+---
+description:
+globs:
+alwaysApply: false
+---
+# Test-Driven Development Guide
+
+This document outlines our test-driven development (TDD) process and the tools available for testing.
+
+## TDD Cycle
+
+1. **Write a Test First**
+[pytest.mdc](mdc:.cursor/rules/pytest.mdc)
+
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
+
+3. **Run Linting and Type Checking**
+[standards.mdc](mdc:.cursor/rules/standards.mdc)
+
+4. **Refactor if needed**
+If the code needs refactoring, with the best practices [best_practices.mdc](mdc:.cursor/rules/best_practices.mdc)
+
+5. **Validate tests**
+
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
index a6991b3b8..6690f823a 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
@@ -13,6 +13,96 @@ For structured data output, `PipeLLM` employs two main strategies:
a. First, the LLM generates a free-form text based on the initial prompt.
b. Second, another LLM call is made with a specific prompt designed to extract and structure the information from the generated text into the target Pydantic model.
+## Working with Images (Vision Language Models)
+
+`PipeLLM` supports Vision Language Models (VLMs) that can process both text and images. To use images in your prompts:
+
+### Basic Image Input
+
+Images must be declared in the `inputs` section of your pipe definition. The image will be automatically passed to the VLM along with your text prompt.
+
+```toml
+[pipe.describe_image]
+PipeLLM = "Describe an image"
+inputs = { image = "Image" }
+output = "VisualDescription"
+prompt_template = """
+Describe the provided image in great detail.
+"""
+```
+
+**Important**: Do NOT reference image variables in your prompt template using `@image` or `$image`. Images are automatically passed to vision-enabled LLMs and should not be treated as text variables.
+
+**Flexible Image Inputs**
+
+You can use any concept that refines `Image` as an input, and choose descriptive variable names that fit your use case:
+
+```toml
+[pipe.analyze_wedding]
+PipeLLM = "Analyze wedding photo"
+inputs = { wedding_photo = "images.Photo" }
+output = "PhotoAnalysis"
+prompt_template = """
+Analyze this wedding photo and describe the key moments captured.
+"""
+```
+
+### Images as Sub-attributes of Structured Content
+
+When working with structured content that contains image fields (like `PageContent` which has a `page_view` field), you need to specify the full path to the image attribute in the `inputs` section:
+
+```toml
+[pipe.analyze_page_view]
+PipeLLM = "Analyze the visual layout of a page"
+inputs = { "page_content.page_view" = "Image" }
+output = "LayoutAnalysis"
+prompt_template = """
+Analyze the visual layout and design elements of this page.
+Focus on typography, spacing, and overall composition.
+"""
+```
+
+In this example:
+- `page_content` is the input variable containing a `PageContent` object
+- `page_view` is the `ImageContent` field within the `PageContent` structure
+- The dot notation `page_content.page_view` tells Pipelex to extract the image from that specific field
+
+### Multiple Images
+
+You can include multiple images in a single prompt by listing them in the inputs:
+
+```toml
+[pipe.compare_images]
+PipeLLM = "Compare two images"
+inputs = {
+ first_image = "Image",
+ second_image = "Image"
+}
+output = "ImageComparison"
+prompt_template = """
+Compare these two images and describe their similarities and differences.
+"""
+```
+
+### Combining Text and Image Inputs
+
+You can mix any stuff and image inputs in the same pipe:
+
+```toml
+[pipe.analyze_document_with_context]
+PipeLLM = "Analyze a document page with additional context"
+inputs = {
+ context = "Text",
+ document.page_view = "Image"
+}
+output = "DocumentAnalysis"
+prompt_template = """
+Given this context: $context
+
+Analyze the document page shown in the image and explain how it relates to the provided context.
+"""
+```
+
## Configuration
`PipeLLM` is configured in your pipeline's `.toml` file.
@@ -22,17 +112,18 @@ For structured data output, `PipeLLM` employs two main strategies:
| Parameter | Type | Description | Required |
| --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `PipeLLM` | string | A descriptive name for the LLM operation. | Yes |
-| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. | Yes |
+| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image"`). | Yes |
| `output` | string | The output concept produced by the LLM operation. | Yes |
| `llm` | string or table | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`). | No |
| `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`). | No |
| `prompt` | string | A simple, static user prompt. Use this when you don't need to inject any variables. | No |
-| `prompt_template` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). | No |
-| `images` | list of strings | For Vision Language Models (VLMs), specifies which input variables are images. | No |
+| `prompt_template` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). **Note**: Do not use `@` or `$` for image variables. | No |
+| `images` | list of strings | **Deprecated**: Use the `inputs` section to declare image inputs instead. | No |
| `structuring_method` | string | The method for generating structured output. Can be `direct` or `preliminary_text`. Defaults to the global configuration. | No |
| `prompt_template_to_structure` | string | The prompt template for the second step in `preliminary_text` mode. | No |
| `output_multiplicity` | string or integer | Defines the number of outputs. Use `"list"` for a variable-length list, or an integer (e.g., `3`) for a fixed-size list. | No |
+## Examples
### Simple Text Generation Example
@@ -75,7 +166,6 @@ This pipe takes an image of a table and uses a VLM to extract the content as an
PipeLLM = "Extract table data from an image"
inputs = { image = "TableScreenshot" }
output = "TableData"
-images = ["image"]
prompt_template = """
Extract the table data from this image and format it as a structured table.
"""
@@ -105,4 +195,4 @@ Analyze this expense report and extract the following information:
"""
```
-In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.
+In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.
\ No newline at end of file
diff --git a/docs/pages/cookbook-examples/extract-dpe.md b/docs/pages/cookbook-examples/extract-dpe.md
index 44309c8b5..150779c06 100644
--- a/docs/pages/cookbook-examples/extract-dpe.md
+++ b/docs/pages/cookbook-examples/extract-dpe.md
@@ -63,7 +63,6 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informa
PipeLLM = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"
inputs = { page_content = "Page" }
output = "Dpe" # The output is structured as a Dpe object
-images = ["page_content.page_view"]
llm = "llm_for_img_to_text"
structuring_method = "preliminary_text"
system_prompt = """You are a multimodal LLM, expert in converting images into perfect markdown."""
diff --git a/docs/pages/cookbook-examples/extract-gantt.md b/docs/pages/cookbook-examples/extract-gantt.md
index 9b0290e6f..c8877f1f0 100644
--- a/docs/pages/cookbook-examples/extract-gantt.md
+++ b/docs/pages/cookbook-examples/extract-gantt.md
@@ -79,7 +79,6 @@ PipeLLM = "Extract the precise dates of the task, start_date and end_date"
inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }
output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails object
structuring_method = "preliminary_text"
-images = ["gantt_chart_image"]
llm = "llm_to_extract_diagram"
prompt_template = """
I am sharing an image of a Gantt chart.
diff --git a/docs/pages/cookbook-examples/extract-proof-of-purchase.md b/docs/pages/cookbook-examples/extract-proof-of-purchase.md
index 0bea51594..13dd9b28e 100644
--- a/docs/pages/cookbook-examples/extract-proof-of-purchase.md
+++ b/docs/pages/cookbook-examples/extract-proof-of-purchase.md
@@ -57,9 +57,8 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d
```toml
[pipe.write_markdown_from_page_content_proof_of_purchase]
PipeLLM = "Write markdown from page content"
-inputs = { page_content = "Page" }
+inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the page
output = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase object
-images = ["page_content.page_view"] # The LLM receives the image of the page
llm = "llm_for_img_to_text"
structuring_method = "preliminary_text"
system_prompt = """You are a multimodal LLM, expert at converting images into perfect markdown."""
diff --git a/docs/pages/cookbook-examples/extract-table.md b/docs/pages/cookbook-examples/extract-table.md
index a74bf4e68..4d414b82a 100644
--- a/docs/pages/cookbook-examples/extract-table.md
+++ b/docs/pages/cookbook-examples/extract-table.md
@@ -75,7 +75,6 @@ steps = [
PipeLLM = "Review an HTML table"
inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }
output = "HtmlTable"
-images = ["table_screenshot"]
prompt_template = """
Your role is to correct an html_table to make sure that it matches the one in the provided image.
diff --git a/docs/pages/cookbook-examples/invoice-extractor.md b/docs/pages/cookbook-examples/invoice-extractor.md
index c4e1267b5..f9b42960e 100644
--- a/docs/pages/cookbook-examples/invoice-extractor.md
+++ b/docs/pages/cookbook-examples/invoice-extractor.md
@@ -77,11 +77,10 @@ steps = [
# A sub-pipeline that uses an LLM to extract the data
[pipe.extract_invoice_data]
PipeLLM = "Extract invoice information from an invoice text transcript"
-inputs = { invoice_page = "Page", invoice_details = "InvoiceDetails" }
+inputs = { "invoice_page.page_view" = "Page", invoice_details = "InvoiceDetails" }
output = "Invoice"
# The output is constrained to the "Invoice" model
llm = "llm_to_extract_invoice"
-images = ["invoice_page.page_view"]
prompt_template = """
Extract invoice information from this invoice:
diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.py
index a57b68b39..b016261d6 100644
--- a/pipelex/cli/_cli.py
+++ b/pipelex/cli/_cli.py
@@ -1,12 +1,9 @@
import os
import shutil
-from itertools import groupby
-from typing import Annotated, Dict, Optional
+from typing import Annotated, Optional
import typer
from click import Command, Context
-from rich import box
-from rich.table import Table
from typer.core import TyperGroup
from typing_extensions import override
@@ -99,63 +96,8 @@ def list_pipes() -> None:
"""List all available pipes."""
Pipelex.make()
- def _format_concept_code(concept_code: Optional[str], current_domain: str) -> str:
- """Format concept code by removing domain prefix if it matches current domain."""
- if not concept_code:
- return ""
- parts = concept_code.split(".")
- if len(parts) == 2 and parts[0] == current_domain:
- return parts[1]
- return concept_code
-
try:
- pipe_provider = get_pipe_provider()
- pipes = pipe_provider.get_pipes()
-
- # Sort pipes by domain and code
- ordered_items = sorted(pipes, key=lambda x: (x.domain or "", x.code or ""))
-
- # Create dictionary for return value
- pipes_dict: Dict[str, Dict[str, Dict[str, str]]] = {}
-
- # Group by domain and create separate tables
- for domain, domain_pipes in groupby(ordered_items, key=lambda x: x.domain):
- table = Table(
- title=f"[bold magenta]domain = {domain}[/]",
- show_header=True,
- show_lines=True,
- header_style="bold cyan",
- box=box.SQUARE_DOUBLE_HEAD,
- border_style="blue",
- )
-
- table.add_column("Code", style="green")
- table.add_column("Definition", style="white")
- table.add_column("Input", style="yellow")
- table.add_column("Output", style="yellow")
-
- pipes_dict[domain] = {}
-
- for pipe in domain_pipes:
- inputs = pipe.inputs
- formatted_inputs = [f"{name}: {_format_concept_code(concept_code, domain)}" for name, concept_code in inputs.items]
- formatted_inputs_str = ", ".join(formatted_inputs)
- output_code = _format_concept_code(pipe.output_concept_code, domain)
-
- table.add_row(
- pipe.code,
- pipe.definition or "",
- formatted_inputs_str,
- output_code,
- )
-
- pipes_dict[domain][pipe.code] = {
- "definition": pipe.definition or "",
- "inputs": formatted_inputs_str,
- "output": pipe.output_concept_code,
- }
-
- pretty_print(table)
+ get_pipe_provider().pretty_list_pipes()
except Exception as e:
raise PipelexCLIError(f"Failed to list pipes: {e}")
diff --git a/pipelex/core/concept_library.py b/pipelex/core/concept_library.py
index 2005a8347..b4e8701a0 100644
--- a/pipelex/core/concept_library.py
+++ b/pipelex/core/concept_library.py
@@ -1,4 +1,4 @@
-from typing import Dict, List, Optional
+from typing import Any, Dict, List, Optional, Type
from pydantic import Field, RootModel
from typing_extensions import override
@@ -8,7 +8,9 @@ from pipelex.core.concept import Concept
from pipelex.core.concept_factory import ConceptFactory
from pipelex.core.concept_native import NativeConcept
from pipelex.core.concept_provider_abstract import ConceptProviderAbstract
+from pipelex.core.stuff_content import ImageContent
from pipelex.exceptions import ConceptLibraryConceptNotFoundError, ConceptLibraryError
+from pipelex.hub import get_class_registry
ConceptLibraryRoot = Dict[str, Concept]
@@ -126,3 +128,22 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptProviderAbstract):
@override
def teardown(self) -> None:
self.root = {}
+
+ @override
+ def get_class(self, concept_code: str) -> Optional[Type[Any]]:
+ return get_class_registry().get_class(concept_code)
+
+ @override
+ def is_image_concept(self, concept_code: str) -> bool:
+ """
+ Check if the concept is an image concept.
+ It is an image concept if its structure class is a subclass of ImageContent
+ or if it refines the native Image concept.
+ """
+ concept = self.get_concept(concept_code=concept_code)
+ if not concept:
+ return False
+ pydantic_model = self.get_class(concept_code=concept.structure_class_name)
+ is_image_class = bool(pydantic_model and issubclass(pydantic_model, ImageContent))
+ refines_image = self.is_compatible_by_concept_code(tested_concept_code=concept.code, wanted_concept_code=NativeConcept.IMAGE.code)
+ return is_image_class or refines_image
diff --git a/pipelex/core/concept_provider_abstract.py b/pipelex/core/concept_provider_abstract.py
index adf8a30f9..09bd24655 100644
--- a/pipelex/core/concept_provider_abstract.py
+++ b/pipelex/core/concept_provider_abstract.py
@@ -1,5 +1,5 @@
from abc import ABC, abstractmethod
-from typing import ClassVar, Dict, List, Optional
+from typing import Any, ClassVar, Dict, List, Optional, Type
from pipelex.core.concept import Concept
@@ -42,3 +42,11 @@ class ConceptProviderAbstract(ABC):
@abstractmethod
def teardown(self) -> None:
pass
+
+ @abstractmethod
+ def get_class(self, concept_code: str) -> Optional[Type[Any]]:
+ pass
+
+ @abstractmethod
+ def is_image_concept(self, concept_code: str) -> bool:
+ pass
diff --git a/pipelex/core/pipe_library.py b/pipelex/core/pipe_library.py
index 13820fcfc..f8faa607b 100644
--- a/pipelex/core/pipe_library.py
+++ b/pipelex/core/pipe_library.py
@@ -1,8 +1,12 @@
+from itertools import groupby
from typing import Dict, List, Optional
from pydantic import Field, RootModel
+from rich import box
+from rich.table import Table
from typing_extensions import override
+from pipelex import pretty_print
from pipelex.core.pipe_abstract import PipeAbstract
from pipelex.core.pipe_provider_abstract import PipeProviderAbstract
from pipelex.exceptions import ConceptError, ConceptLibraryConceptNotFoundError, PipeLibraryError, PipeLibraryPipeNotFoundError
@@ -64,3 +68,61 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeProviderAbstract):
@override
def teardown(self) -> None:
self.root = {}
+
+ @override
+ def pretty_list_pipes(self) -> None:
+ def _format_concept_code(concept_code: Optional[str], current_domain: str) -> str:
+ """Format concept code by removing domain prefix if it matches current domain."""
+ if not concept_code:
+ return ""
+ parts = concept_code.split(".")
+ if len(parts) == 2 and parts[0] == current_domain:
+ return parts[1]
+ return concept_code
+
+ pipes = self.get_pipes()
+
+ # Sort pipes by domain and code
+ ordered_items = sorted(pipes, key=lambda x: (x.domain or "", x.code or ""))
+
+ # Create dictionary for return value
+ pipes_dict: Dict[str, Dict[str, Dict[str, str]]] = {}
+
+ # Group by domain and create separate tables
+ for domain, domain_pipes in groupby(ordered_items, key=lambda x: x.domain):
+ table = Table(
+ title=f"[bold magenta]domain = {domain}[/]",
+ show_header=True,
+ show_lines=True,
+ header_style="bold cyan",
+ box=box.SQUARE_DOUBLE_HEAD,
+ border_style="blue",
+ )
+
+ table.add_column("Code", style="green")
+ table.add_column("Definition", style="white")
+ table.add_column("Input", style="yellow")
+ table.add_column("Output", style="yellow")
+
+ pipes_dict[domain] = {}
+
+ for pipe in domain_pipes:
+ inputs = pipe.inputs
+ formatted_inputs = [f"{name}: {_format_concept_code(concept_code, domain)}" for name, concept_code in inputs.items]
+ formatted_inputs_str = ", ".join(formatted_inputs)
+ output_code = _format_concept_code(pipe.output_concept_code, domain)
+
+ table.add_row(
+ pipe.code,
+ pipe.definition or "",
+ formatted_inputs_str,
+ output_code,
+ )
+
+ pipes_dict[domain][pipe.code] = {
+ "definition": pipe.definition or "",
+ "inputs": formatted_inputs_str,
+ "output": pipe.output_concept_code,
+ }
+
+ pretty_print(table)
diff --git a/pipelex/core/pipe_provider_abstract.py b/pipelex/core/pipe_provider_abstract.py
index c24214148..535e17fdf 100644
--- a/pipelex/core/pipe_provider_abstract.py
+++ b/pipelex/core/pipe_provider_abstract.py
@@ -26,3 +26,7 @@ class PipeProviderAbstract(ABC):
@abstractmethod
def teardown(self) -> None:
pass
+
+ @abstractmethod
+ def pretty_list_pipes(self) -> None:
+ pass
diff --git a/pipelex/libraries/pipelines/images.toml b/pipelex/libraries/pipelines/images.toml
index 0bb7cd01b..b2391fdbf 100644
--- a/pipelex/libraries/pipelines/images.toml
+++ b/pipelex/libraries/pipelines/images.toml
@@ -1,5 +1,3 @@
-
-
domain = "images"
definition = "Generic image-related domain"
@@ -26,7 +24,6 @@ PipeLLM = "Describe an image"
inputs = { image = "Image" }
output = "VisualDescription"
system_prompt = "You are a very good observer."
-images = ["image"]
llm = "llm_to_describe_img"
prompt_template = """
Describe the provided image in great detail.
@@ -37,7 +34,6 @@ PipeLLM = "Describe a photo"
inputs = { photo = "Photo" }
output = "VisualDescription"
system_prompt = "You are a very good observer."
-images = ["photo"]
llm = "llm_to_describe_img"
prompt_template = """
Describe the provided photo and how it was shot: scene, lighting, camera, etc.
diff --git a/pipelex/pipe_operators/pipe_llm.py b/pipelex/pipe_operators/pipe_llm.py
index 15d58ce41..d1f540cef 100644
--- a/pipelex/pipe_operators/pipe_llm.py
+++ b/pipelex/pipe_operators/pipe_llm.py
@@ -69,7 +69,14 @@ class PipeLLM(PipeOperator):
output_multiplicity: Optional[PipeOutputMultiplicity] = None
def needed_inputs(self) -> PipeInputSpec:
- return self.pipe_llm_prompt.needed_inputs()
+ pipe_llm_prompt_needed_inputs = self.pipe_llm_prompt.needed_inputs()
+ # The images are not tagged in the prompt_template. Therefore if an image is provided in the
+ # inputs, it becomes a needed input.
+ concept_provider = get_concept_provider()
+ for input_name, concept_code in self.inputs.root.items():
+ if concept_provider.is_image_concept(concept_code=concept_code):
+ pipe_llm_prompt_needed_inputs.add_requirement(variable_name=input_name, concept_code=NativeConcept.IMAGE.code)
+ return pipe_llm_prompt_needed_inputs
@model_validator(mode="after")
def validate_inputs(self) -> Self:
@@ -158,6 +165,14 @@ class PipeLLM(PipeOperator):
log.error(extraneous_input_var_error.desc())
case StaticValidationReaction.RAISE:
raise extraneous_input_var_error
+ else:
+ # Check if this input is an image concept but is being used as a variable in the prompt
+ if concept_provider.is_image_concept(concept_code=input_name):
+ raise PipeDefinitionError(
+ f"Image-based input '{input_name}' of concept '{input_name}' "
+ f"cannot be used as a variable in a prompt for Pipe '{self.code}'. "
+ f"Image variables are automatically passed to vision-enabled LLMs."
+ )
@model_validator(mode="after")
def validate_output_concept_consistency(self) -> Self:
diff --git a/pipelex/pipe_operators/pipe_llm_factory.py b/pipelex/pipe_operators/pipe_llm_factory.py
index e78058732..95cca4c55 100644
--- a/pipelex/pipe_operators/pipe_llm_factory.py
+++ b/pipelex/pipe_operators/pipe_llm_factory.py
@@ -8,7 +8,7 @@ from pipelex.core.pipe_blueprint import PipeBlueprint, PipeSpecificFactoryProtoc
from pipelex.core.pipe_input_spec import PipeInputSpec
from pipelex.core.pipe_run_params import make_output_multiplicity
from pipelex.exceptions import PipeDefinitionError
-from pipelex.hub import get_optional_domain
+from pipelex.hub import get_concept_provider, get_optional_domain
from pipelex.pipe_operators.pipe_jinja2 import PipeJinja2
from pipelex.pipe_operators.pipe_jinja2_factory import PipeJinja2Factory
from pipelex.pipe_operators.pipe_llm import PipeLLM, StructuringMethod
@@ -28,8 +28,6 @@ class PipeLLMBlueprint(PipeBlueprint):
prompt_name: Optional[str] = None
prompt: Optional[str] = None
- images: Optional[List[str]] = None
-
llm: Optional[LLMSettingOrPresetId] = None
llm_to_structure: Optional[LLMSettingOrPresetId] = None
llm_to_structure_direct: Optional[LLMSettingOrPresetId] = None
@@ -43,7 +41,6 @@ class PipeLLMBlueprint(PipeBlueprint):
nb_output: Optional[int] = None
multiple_output: Optional[bool] = None
- # TODO: chack that the listed images are listed in the inputs
@model_validator(mode="after")
def validate_multiple_output(self) -> Self:
if excess_attributes_list := has_more_than_one_among_attributes_from_lists(
@@ -112,6 +109,15 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):
jinja2_name=pipe_code,
)
+ user_images: List[str] = []
+ if pipe_blueprint.inputs:
+ for stuff_name, concept_code in (pipe_blueprint.inputs).items():
+ concept = get_concept_provider().get_required_concept(concept_code=concept_code)
+ if get_concept_provider().is_image_concept(concept_code=concept.code):
+ user_images.append(stuff_name)
+ else:
+ # Implicit text concept
+ pass
pipe_llm_prompt = PipeLLMPrompt(
code="adhoc_for_pipe_llm_prompt",
domain=domain_code,
@@ -122,7 +128,7 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):
user_pipe_jinja2=user_pipe_jinja2,
user_prompt_verbatim_name=pipe_blueprint.prompt_name,
user_text=pipe_blueprint.prompt,
- user_images=pipe_blueprint.images,
+ user_images=user_images or None,
)
llm_settings = LLMSettingChoices(
diff --git a/pipelex/pipe_operators/pipe_llm_prompt.py b/pipelex/pipe_operators/pipe_llm_prompt.py
index d538b9a42..e7d107878 100644
--- a/pipelex/pipe_operators/pipe_llm_prompt.py
+++ b/pipelex/pipe_operators/pipe_llm_prompt.py
@@ -106,10 +106,6 @@ class PipeLLMPrompt(PipeOperator):
continue
pipe_input_spec.add_requirement(variable_name=conceptless_required_variable, concept_code=NativeConcept.ANYTHING.code)
- if self.user_images:
- for user_image in self.user_images:
- pipe_input_spec.add_requirement(variable_name=user_image, concept_code=NativeConcept.IMAGE.code)
-
return pipe_input_spec
@override
diff --git a/tests/pipelex/pipelex_asynch/test_image_inputs.py b/tests/pipelex/pipelex_asynch/test_image_inputs.py
new file mode 100644
index 000000000..6fd39e055
--- /dev/null
+++ b/tests/pipelex/pipelex_asynch/test_image_inputs.py
@@ -0,0 +1,95 @@
+from typing import cast
+
+import pytest
+from pytest import FixtureRequest
+
+from pipelex import pretty_print
+from pipelex.core.pipe_output import PipeOutput
+from pipelex.core.pipe_run_params import PipeRunMode
+from pipelex.core.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.core.stuff_content import ImageContent, PageContent, TextAndImagesContent, TextContent
+from pipelex.core.stuff_factory import StuffFactory
+from pipelex.core.working_memory_factory import WorkingMemoryFactory
+from pipelex.hub import get_pipe_router, get_report_delegate
+from pipelex.pipeline.job_metadata import JobMetadata
+from tests.test_data import ImageTestCases
+from tests.test_pipelines.misc_tests.tests import Article
+
+
[email protected](loop_scope="class")
+class TestImageInputs:
+ """Test class for verifying image input functionality in pipes."""
+
+ async def test_extract_article_from_image(
+ self,
+ request: FixtureRequest,
+ pipe_run_mode: PipeRunMode,
+ ) -> None:
+ """Test that an image is indeed given to the LLM, and that it can extract extact whats on the image."""
+ working_memory = WorkingMemoryFactory.make_from_image(image_url=ImageTestCases.IMAGE_FILE_PATH_PNG)
+
+ # Run the pipe
+ pipe_output: PipeOutput = await get_pipe_router().run_pipe_code(
+ pipe_code="extract_article_from_image",
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
+ working_memory=working_memory,
+ job_metadata=JobMetadata(
+ top_job_id=cast(str, request.node.originalname), # type: ignore
+ ),
+ )
+
+ # Log output and generate report
+ pretty_print(pipe_output, title="Pipe output")
+ get_report_delegate().generate_report()
+
+ # Verify output
+ if pipe_run_mode != PipeRunMode.DRY:
+ article = pipe_output.main_stuff_as(content_type=Article)
+ pretty_print(article, title="Article")
+ assert article.title == "The Solar System: An Overview"
+ assert pipe_output is not None
+ assert pipe_output.working_memory is not None
+ assert pipe_output.main_stuff is not None
+
+ async def test_describe_page(self, request: FixtureRequest, pipe_run_mode: PipeRunMode) -> None:
+ """
+ Test that a pipe can accept a PageContent input, give to the LLM the image via subattributes,
+ But also accepts basic objects
+ """
+ # Create the page content
+ image_content = ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG)
+ text_and_images = TextAndImagesContent(text=TextContent(text="This is a test page"), images=[])
+ page_content = PageContent(text_and_images=text_and_images, page_view=image_content)
+
+ # Create stuff from page content
+ stuff = StuffFactory.make_stuff(concept_str="Page", content=page_content, name="page")
+
+ # Create working memory
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
+
+ # Run the pipe
+ pipe_output: PipeOutput = await get_pipe_router().run_pipe_code(
+ pipe_code="describe_page",
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
+ working_memory=working_memory,
+ job_metadata=JobMetadata(
+ top_job_id=cast(str, request.node.originalname), # type: ignore
+ ),
+ )
+
+ # Log output and generate report
+ pretty_print(pipe_output, title="Pipe output")
+ get_report_delegate().generate_report()
+
+ # Verify output
+ if pipe_run_mode != PipeRunMode.DRY:
+ article = pipe_output.main_stuff_as(content_type=Article)
+ pretty_print(article, title="Article")
+ assert article.title == "The Solar System: An Overview"
+ assert article.description == "This is a test page"
+ assert pipe_output is not None
+ assert pipe_output.working_memory is not None
+ assert pipe_output.main_stuff is not None
diff --git a/tests/pipelex/pipelex_asynch/test_pipe_llm.py b/tests/pipelex/pipelex_asynch/test_pipe_llm.py
index 86535a7ca..6dcfe0d47 100644
--- a/tests/pipelex/pipelex_asynch/test_pipe_llm.py
+++ b/tests/pipelex/pipelex_asynch/test_pipe_llm.py
@@ -62,7 +62,6 @@ class TestPipeLLM:
if not stuff_name:
pytest.fail(f"Cannot use nameless stuff in this test: {stuff}")
working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)
-
pipe_job = PipeJobFactory.make_pipe_job(
working_memory=working_memory,
pipe=PipeLLM(
@@ -75,11 +74,11 @@ class TestPipeLLM:
domain="generic",
system_prompt=PipeTestCases.SYSTEM_PROMPT,
user_text=PipeTestCases.MULTI_IMG_DESC_PROMPT,
- user_images=attribute_paths,
),
),
pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
)
+
pipe_llm_output: PipeLLMOutput = await get_pipe_router().run_pipe_job(
pipe_job=pipe_job,
)
diff --git a/tests/pipelex_unit_test.toml b/tests/pipelex_unit_test.toml
index 7034400ea..42c5c632a 100644
--- a/tests/pipelex_unit_test.toml
+++ b/tests/pipelex_unit_test.toml
@@ -1,5 +1,3 @@
-
-
[cogt.llm_config.preferred_platforms]
# These override the defaults set for any llm integration
# Use at your convenience:
diff --git a/tests/test_pipelines/misc_tests/test_errors.toml b/tests/test_pipelines/misc_tests/test_errors.toml
deleted file mode 100644
index 216df442d..000000000
--- a/tests/test_pipelines/misc_tests/test_errors.toml
+++ /dev/null
@@ -1,45 +0,0 @@
-
-
-domain = "test_errors"
-definition = "This library is intended for testing errors"
-
-# [pipe.jinja2_syntax_error]
-# PipeLLM = "Jinja2 syntax error (on purpose, for testing): bad curly braces"
-# inputs = { text = "Text" }
-# output = "Text"
-# prompt_template = """
-# {{ a123|tag }
-# """
-
-# [pipe.jinja2_dummy_filter1]
-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"
-# inputs = { text = "Text" }
-# output = "Text"
-# prompt_template = """
-# {{ foobar|dummy_filter1 }}
-# """
-
-# [pipe.jinja2_dummy_filter2]
-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"
-# inputs = { text = "Text" }
-# output = "Text"
-# prompt_template = """
-# {{ foobar|dummy_filter2() }}
-# """
-
-# [pipe.jinja2_dummy_filter3]
-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"
-# inputs = { text = "Text" }
-# output = "Text"
-# prompt_template = """
-# {{ foobar|dummy_filter3("param") }}
-# """
-
-
-# [pipe.jinja2_dummy_filter4]
-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"
-# inputs = { text = "Text" }
-# output = "Text"
-# prompt_template = """
-# {{ foobar|dummy_filter4(param1, param2) }}
-# """
diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.toml b/tests/test_pipelines/misc_tests/test_image_inputs.toml
new file mode 100644
index 000000000..f75082116
--- /dev/null
+++ b/tests/test_pipelines/misc_tests/test_image_inputs.toml
@@ -0,0 +1,32 @@
+domain = "test_image_inputs"
+definition = "Test domain for verifying image input functionality"
+
+[concept]
+Article = "Article"
+
+[pipe]
+[pipe.extract_article_from_image]
+PipeLLM = "Describe a page content"
+inputs = { image = "Image" }
+output = "Article"
+system_prompt = """
+You are an expert at describing page contents.
+"""
+prompt_template = """
+Extract the text and the date of the article in the image.
+"""
+
+[pipe.describe_page]
+PipeLLM = "Describe a page"
+inputs = { "page.page_view" = "Image", page = "Page" }
+output = "Article"
+system_prompt = """
+You are an expert at describing page contents.
+"""
+prompt_template = """
+Extract the date and title of the article.
+
+Also, add this as the description of the article:
+
[email protected]_and_images.text.text
+"""
\ No newline at end of file
diff --git a/tests/test_pipelines/misc_tests/tests.py b/tests/test_pipelines/misc_tests/tests.py
index e648f6c2b..cc36b393a 100644
--- a/tests/test_pipelines/misc_tests/tests.py
+++ b/tests/test_pipelines/misc_tests/tests.py
@@ -1,5 +1,11 @@
-from pipelex.core.stuff_content import ListContent, TextContent
+from pipelex.core.stuff_content import ListContent, StructuredContent, TextContent
class RandomListContent(ListContent[TextContent]):
pass
+
+
+class Article(StructuredContent):
+ title: str
+ description: str
+ date: str
diff --git a/tests/test_pipelines/misc_tests/tests.toml b/tests/test_pipelines/misc_tests/tests.toml
index c7e5083ae..bf53640a2 100644
--- a/tests/test_pipelines/misc_tests/tests.toml
+++ b/tests/test_pipelines/misc_tests/tests.toml
@@ -23,7 +23,6 @@ output = "Text"
prompt_template = """
Describe the using rap lyrics, including puns and references to the image.
"""
-images = ["image"]
[pipe.create_characters]
PipeLLM = "Create Characters"