Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Fix/remove images field from pipe llm blueprint (#105)
Thomas Hebrard336cb02b8398
diff --git a/.cursor/rules/best_practices.mdc b/.cursor/rules/best_practices.mdcnew file mode 100644index 000000000..66bc8dc36--- /dev/null+++ b/.cursor/rules/best_practices.mdc@@ -0,0 +1,77 @@+---+description:+globs:+alwaysApply: false+---+# Best Practices Guide++This document outlines the core best practices and patterns used in our codebase.++## Type Hints++1. **Always Use Type Hints**+ - Every function parameter must be typed+ - Every function return must be typed+ - Use type hints for all variables where type is not obvious++2. **StrEnum**+ - Import StrEnum from pipelex.types+ ```python+ from pipelex.types import StrEnum++ class ModelType(StrEnum):+ GPT4 = "gpt-4"+ GPT35 = "gpt-3.5-turbo"+ ```++## Factory Pattern++1. **Use Factory Pattern for Object Creation**+ - Create factories when dealing with multiple implementations++## Documentation++1. **Docstring Format**+ - Quick description of the function/class+ - List args and their types+ - Document return values+ - Example:+ ```python+ def process_image(image_path: str, size: Tuple[int, int]) -> bytes:+ """Process and resize an image.++ Args:+ image_path: Path to the source image+ size: Tuple of (width, height) for resizing++ Returns:+ Processed image as bytes+ """+ pass+ ```++2. **Class Documentation**+ - Document class purpose and behavior+ - Include examples if complex+ ```python+ class ImageProcessor:+ """Handles image processing operations.++ Provides methods for resizing, converting, and optimizing images.+ """+ ```++## Custom Exceptions++1. **Graceful Error Handling**+ - Use try/except blocks with specific exceptions+ - Convert third-party exceptions to custom ones+ ```python+ try:+ from fal_client import AsyncClient as FalAsyncClient+ except ImportError as exc:+ raise MissingDependencyError(+ "fal-client", "fal",+ "The fal-client SDK is required to use FAL models."+ ) from exc+ ```diff --git a/.cursor/rules/pipes.mdc b/.cursor/rules/pipes.mdcindex a32b3d3bc..f13c986e9 100644--- a/.cursor/rules/pipes.mdc+++ b/.cursor/rules/pipes.mdc@@ -5,20 +5,9 @@ alwaysApply: false---This rule explains how to build pipes.-# File Naming & Structure--## The pipelines/ directory--- Pipelines and structures are defined in the `pipelex/libraries/pipelines` directory--pipelex/libraries-└── pipelines-## Pipeline file naming-- Pipeline TOML files should be placed in the `pipelex/libraries/pipelines/` directory or a subdirectory of it- The file name should be descriptive, in snake_case and end with `.toml`-- The file should contain both concepts and pipes definitions## Pipeline file structure@@ -167,7 +156,6 @@ Your summary should not be longer than 2 sentences.PipeLLM = "Convert table screenshot to HTML"inputs = { table_screenshot = "TableScreenshot" }output = "HtmlTable"-images = ["table_screenshot"]system_prompt = """You are a vision-based table extractor."""diff --git a/.cursor/rules/pytest.mdc b/.cursor/rules/pytest.mdcindex b44f68fae..2b22e5bea 100644--- a/.cursor/rules/pytest.mdc+++ b/.cursor/rules/pytest.mdc@@ -17,6 +17,8 @@ These rules apply when writing unit tests.- More precisely, for `pipelex` and `pipelex.cogt` the async tests are placed inside subdirectories named `cogt_asynch` and `pipelex_asynch`- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `tests.pipelex.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.+- Always put test inside Test classes.+- The pipelex pipelines should be stored in `tests/test_pipelines` as well as the related structured Output classes that inherit from `StructuredContent`## Markersdiff --git a/.cursor/rules/standards.mdc b/.cursor/rules/standards.mdcnew file mode 100644index 000000000..4c86f0e1e--- /dev/null+++ b/.cursor/rules/standards.mdc@@ -0,0 +1,60 @@+---+description:+globs:+alwaysApply: true+---+# Coding Standards++This document outlines the coding standards and quality control procedures that must be followed when contributing to this project.++## Code Quality Checks++### Linting and Type Checking++Before finalizing a task, you must run the following command to check for linting issues, type errors, and code quality problems:++```bash+make check+```++This command runs multiple code quality tools:+- Pyright: Static type checking+- Ruff: Fast Python linter+- Mypy: Static type checker++Always fix any issues reported by these tools before proceeding.++### Running Tests++We have several make commands for running tests:++1. `make tp`: Runs all tests with these markers:+ ```+ (dry_runnable or not (inference or llm or imgg or ocr)) and not (needs_output or pipelex_api)+ ```+ Use this for quick test runs that don't require LLM or image generation.++2. `make ti`: Runs all tests with these markers:+ ```+ inference and not imgg+ ```+ Use this for testing LLM functionality without image generation.++3. To run specific tests:+ ```bash+ make tp TEST=TestClassName+ # or+ make tp TEST=test_function_name+ ```+ It matches names, so `TEST=test_function_name` is going to run all test with the function name that STARTS with `test_function_name`.++## Important Project Directories++### Pipelines Directory+- All pipeline definitions go in `pipelex/libraries/pipelines/`++### Tests Directory+- All tests are located in the `tests/` directory++### Documentation Directory+- All documentation is located in the `docs/` directorydiff --git a/.cursor/rules/structures.mdc b/.cursor/rules/structures.mdcindex 9671eef39..d8a620a91 100644--- a/.cursor/rules/structures.mdc+++ b/.cursor/rules/structures.mdc@@ -59,3 +59,4 @@ Structure classes defined within `pipelex_libraries/pipelines/` are automaticall- Use Pydantic validators for data cleaning/validation- Remove timezone info from datetime fields- Respect rules of @base_models.+diff --git a/.cursor/rules/tdd.mdc b/.cursor/rules/tdd.mdcnew file mode 100644index 000000000..5abebf380--- /dev/null+++ b/.cursor/rules/tdd.mdc@@ -0,0 +1,28 @@+---+description:+globs:+alwaysApply: false+---+# Test-Driven Development Guide++This document outlines our test-driven development (TDD) process and the tools available for testing.++## TDD Cycle++1. **Write a Test First**+[pytest.mdc](mdc:.cursor/rules/pytest.mdc)++2. **Write the Code**+ - Implement the minimum amount of code needed to pass the test+ - Follow the project's coding standards+ - Keep it simple - don't write more than needed++3. **Run Linting and Type Checking**+[standards.mdc](mdc:.cursor/rules/standards.mdc)++4. **Refactor if needed**+If the code needs refactoring, with the best practices [best_practices.mdc](mdc:.cursor/rules/best_practices.mdc)++5. **Validate tests**++Remember: The key to TDD is writing the test first and letting it drive your implementation. Always run the full test suite and quality checks before considering a feature complete.diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.mdindex a6991b3b8..6690f823a 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md@@ -13,6 +13,96 @@ For structured data output, `PipeLLM` employs two main strategies:a. First, the LLM generates a free-form text based on the initial prompt.b. Second, another LLM call is made with a specific prompt designed to extract and structure the information from the generated text into the target Pydantic model.+## Working with Images (Vision Language Models)++`PipeLLM` supports Vision Language Models (VLMs) that can process both text and images. To use images in your prompts:++### Basic Image Input++Images must be declared in the `inputs` section of your pipe definition. The image will be automatically passed to the VLM along with your text prompt.++```toml+[pipe.describe_image]+PipeLLM = "Describe an image"+inputs = { image = "Image" }+output = "VisualDescription"+prompt_template = """+Describe the provided image in great detail.+"""+```++**Important**: Do NOT reference image variables in your prompt template using `@image` or `$image`. Images are automatically passed to vision-enabled LLMs and should not be treated as text variables.++**Flexible Image Inputs**++You can use any concept that refines `Image` as an input, and choose descriptive variable names that fit your use case:++```toml+[pipe.analyze_wedding]+PipeLLM = "Analyze wedding photo"+inputs = { wedding_photo = "images.Photo" }+output = "PhotoAnalysis"+prompt_template = """+Analyze this wedding photo and describe the key moments captured.+"""+```++### Images as Sub-attributes of Structured Content++When working with structured content that contains image fields (like `PageContent` which has a `page_view` field), you need to specify the full path to the image attribute in the `inputs` section:++```toml+[pipe.analyze_page_view]+PipeLLM = "Analyze the visual layout of a page"+inputs = { "page_content.page_view" = "Image" }+output = "LayoutAnalysis"+prompt_template = """+Analyze the visual layout and design elements of this page.+Focus on typography, spacing, and overall composition.+"""+```++In this example:+- `page_content` is the input variable containing a `PageContent` object+- `page_view` is the `ImageContent` field within the `PageContent` structure+- The dot notation `page_content.page_view` tells Pipelex to extract the image from that specific field++### Multiple Images++You can include multiple images in a single prompt by listing them in the inputs:++```toml+[pipe.compare_images]+PipeLLM = "Compare two images"+inputs = {+ first_image = "Image",+ second_image = "Image"+}+output = "ImageComparison"+prompt_template = """+Compare these two images and describe their similarities and differences.+"""+```++### Combining Text and Image Inputs++You can mix any stuff and image inputs in the same pipe:++```toml+[pipe.analyze_document_with_context]+PipeLLM = "Analyze a document page with additional context"+inputs = {+ context = "Text",+ document.page_view = "Image"+}+output = "DocumentAnalysis"+prompt_template = """+Given this context: $context++Analyze the document page shown in the image and explain how it relates to the provided context.+"""+```+## Configuration`PipeLLM` is configured in your pipeline's `.toml` file.@@ -22,17 +112,18 @@ For structured data output, `PipeLLM` employs two main strategies:| Parameter | Type | Description | Required || --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- || `PipeLLM` | string | A descriptive name for the LLM operation. | Yes |-| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. | Yes |+| `inputs` | dictionary | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image"`). | Yes || `output` | string | The output concept produced by the LLM operation. | Yes || `llm` | string or table | Specifies the LLM preset(s) to use. Can be a single preset or a table mapping different presets for different generation modes (e.g., `main`, `object_direct`). | No || `system_prompt` | string | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Can be inline text or a reference to a template file (`"file:path/to/prompt.md"`). | No || `prompt` | string | A simple, static user prompt. Use this when you don't need to inject any variables. | No |-| `prompt_template` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). | No |-| `images` | list of strings | For Vision Language Models (VLMs), specifies which input variables are images. | No |+| `prompt_template` | string | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`) and `@` to insert the content of an entire input (e.g., `@text_to_summarize`). **Note**: Do not use `@` or `$` for image variables. | No |+| `images` | list of strings | **Deprecated**: Use the `inputs` section to declare image inputs instead. | No || `structuring_method` | string | The method for generating structured output. Can be `direct` or `preliminary_text`. Defaults to the global configuration. | No || `prompt_template_to_structure` | string | The prompt template for the second step in `preliminary_text` mode. | No || `output_multiplicity` | string or integer | Defines the number of outputs. Use `"list"` for a variable-length list, or an integer (e.g., `3`) for a fixed-size list. | No |+## Examples### Simple Text Generation Example@@ -75,7 +166,6 @@ This pipe takes an image of a table and uses a VLM to extract the content as anPipeLLM = "Extract table data from an image"inputs = { image = "TableScreenshot" }output = "TableData"-images = ["image"]prompt_template = """Extract the table data from this image and format it as a structured table."""@@ -105,4 +195,4 @@ Analyze this expense report and extract the following information:"""```-In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.+In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.\ No newline at end of filediff --git a/docs/pages/cookbook-examples/extract-dpe.md b/docs/pages/cookbook-examples/extract-dpe.mdindex 44309c8b5..150779c06 100644--- a/docs/pages/cookbook-examples/extract-dpe.md+++ b/docs/pages/cookbook-examples/extract-dpe.md@@ -63,7 +63,6 @@ The pipeline uses a `PipeLLM` with a very specific prompt to extract the informaPipeLLM = "Write markdown from page content of a 'Diagnostic de Performance Energetique'"inputs = { page_content = "Page" }output = "Dpe" # The output is structured as a Dpe object-images = ["page_content.page_view"]llm = "llm_for_img_to_text"structuring_method = "preliminary_text"system_prompt = """You are a multimodal LLM, expert in converting images into perfect markdown."""diff --git a/docs/pages/cookbook-examples/extract-gantt.md b/docs/pages/cookbook-examples/extract-gantt.mdindex 9b0290e6f..c8877f1f0 100644--- a/docs/pages/cookbook-examples/extract-gantt.md+++ b/docs/pages/cookbook-examples/extract-gantt.md@@ -79,7 +79,6 @@ PipeLLM = "Extract the precise dates of the task, start_date and end_date"inputs = { gantt_chart_image = "GanttChartImage", gantt_timescale = "GanttTimescaleDescription", gantt_task_name = "GanttTaskName" }output = "GanttTaskDetails" # The output is structured as a GanttTaskDetails objectstructuring_method = "preliminary_text"-images = ["gantt_chart_image"]llm = "llm_to_extract_diagram"prompt_template = """I am sharing an image of a Gantt chart.diff --git a/docs/pages/cookbook-examples/extract-proof-of-purchase.md b/docs/pages/cookbook-examples/extract-proof-of-purchase.mdindex 0bea51594..13dd9b28e 100644--- a/docs/pages/cookbook-examples/extract-proof-of-purchase.md+++ b/docs/pages/cookbook-examples/extract-proof-of-purchase.md@@ -57,9 +57,8 @@ The pipeline uses a powerful `PipeLLM` to extract the structured data from the d```toml[pipe.write_markdown_from_page_content_proof_of_purchase]PipeLLM = "Write markdown from page content"-inputs = { page_content = "Page" }+inputs = { "page_content.page_view" = "Page" } # The LLM receives the image of the pageoutput = "ProofOfPurchase" # The LLM is forced to output a ProofOfPurchase object-images = ["page_content.page_view"] # The LLM receives the image of the pagellm = "llm_for_img_to_text"structuring_method = "preliminary_text"system_prompt = """You are a multimodal LLM, expert at converting images into perfect markdown."""diff --git a/docs/pages/cookbook-examples/extract-table.md b/docs/pages/cookbook-examples/extract-table.mdindex a74bf4e68..4d414b82a 100644--- a/docs/pages/cookbook-examples/extract-table.md+++ b/docs/pages/cookbook-examples/extract-table.md@@ -75,7 +75,6 @@ steps = [PipeLLM = "Review an HTML table"inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }output = "HtmlTable"-images = ["table_screenshot"]prompt_template = """Your role is to correct an html_table to make sure that it matches the one in the provided image.diff --git a/docs/pages/cookbook-examples/invoice-extractor.md b/docs/pages/cookbook-examples/invoice-extractor.mdindex c4e1267b5..f9b42960e 100644--- a/docs/pages/cookbook-examples/invoice-extractor.md+++ b/docs/pages/cookbook-examples/invoice-extractor.md@@ -77,11 +77,10 @@ steps = [# A sub-pipeline that uses an LLM to extract the data[pipe.extract_invoice_data]PipeLLM = "Extract invoice information from an invoice text transcript"-inputs = { invoice_page = "Page", invoice_details = "InvoiceDetails" }+inputs = { "invoice_page.page_view" = "Page", invoice_details = "InvoiceDetails" }output = "Invoice"# The output is constrained to the "Invoice" modelllm = "llm_to_extract_invoice"-images = ["invoice_page.page_view"]prompt_template = """Extract invoice information from this invoice:diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex a57b68b39..b016261d6 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -1,12 +1,9 @@import osimport shutil-from itertools import groupby-from typing import Annotated, Dict, Optional+from typing import Annotated, Optionalimport typerfrom click import Command, Context-from rich import box-from rich.table import Tablefrom typer.core import TyperGroupfrom typing_extensions import override@@ -99,63 +96,8 @@ def list_pipes() -> None:"""List all available pipes."""Pipelex.make()- def _format_concept_code(concept_code: Optional[str], current_domain: str) -> str:- """Format concept code by removing domain prefix if it matches current domain."""- if not concept_code:- return ""- parts = concept_code.split(".")- if len(parts) == 2 and parts[0] == current_domain:- return parts[1]- return concept_code-try:- pipe_provider = get_pipe_provider()- pipes = pipe_provider.get_pipes()-- # Sort pipes by domain and code- ordered_items = sorted(pipes, key=lambda x: (x.domain or "", x.code or ""))-- # Create dictionary for return value- pipes_dict: Dict[str, Dict[str, Dict[str, str]]] = {}-- # Group by domain and create separate tables- for domain, domain_pipes in groupby(ordered_items, key=lambda x: x.domain):- table = Table(- title=f"[bold magenta]domain = {domain}[/]",- show_header=True,- show_lines=True,- header_style="bold cyan",- box=box.SQUARE_DOUBLE_HEAD,- border_style="blue",- )-- table.add_column("Code", style="green")- table.add_column("Definition", style="white")- table.add_column("Input", style="yellow")- table.add_column("Output", style="yellow")-- pipes_dict[domain] = {}-- for pipe in domain_pipes:- inputs = pipe.inputs- formatted_inputs = [f"{name}: {_format_concept_code(concept_code, domain)}" for name, concept_code in inputs.items]- formatted_inputs_str = ", ".join(formatted_inputs)- output_code = _format_concept_code(pipe.output_concept_code, domain)-- table.add_row(- pipe.code,- pipe.definition or "",- formatted_inputs_str,- output_code,- )-- pipes_dict[domain][pipe.code] = {- "definition": pipe.definition or "",- "inputs": formatted_inputs_str,- "output": pipe.output_concept_code,- }-- pretty_print(table)+ get_pipe_provider().pretty_list_pipes()except Exception as e:raise PipelexCLIError(f"Failed to list pipes: {e}")diff --git a/pipelex/core/concept_library.py b/pipelex/core/concept_library.pyindex 2005a8347..b4e8701a0 100644--- a/pipelex/core/concept_library.py+++ b/pipelex/core/concept_library.py@@ -1,4 +1,4 @@-from typing import Dict, List, Optional+from typing import Any, Dict, List, Optional, Typefrom pydantic import Field, RootModelfrom typing_extensions import override@@ -8,7 +8,9 @@ from pipelex.core.concept import Conceptfrom pipelex.core.concept_factory import ConceptFactoryfrom pipelex.core.concept_native import NativeConceptfrom pipelex.core.concept_provider_abstract import ConceptProviderAbstract+from pipelex.core.stuff_content import ImageContentfrom pipelex.exceptions import ConceptLibraryConceptNotFoundError, ConceptLibraryError+from pipelex.hub import get_class_registryConceptLibraryRoot = Dict[str, Concept]@@ -126,3 +128,22 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptProviderAbstract):@overridedef teardown(self) -> None:self.root = {}++ @override+ def get_class(self, concept_code: str) -> Optional[Type[Any]]:+ return get_class_registry().get_class(concept_code)++ @override+ def is_image_concept(self, concept_code: str) -> bool:+ """+ Check if the concept is an image concept.+ It is an image concept if its structure class is a subclass of ImageContent+ or if it refines the native Image concept.+ """+ concept = self.get_concept(concept_code=concept_code)+ if not concept:+ return False+ pydantic_model = self.get_class(concept_code=concept.structure_class_name)+ is_image_class = bool(pydantic_model and issubclass(pydantic_model, ImageContent))+ refines_image = self.is_compatible_by_concept_code(tested_concept_code=concept.code, wanted_concept_code=NativeConcept.IMAGE.code)+ return is_image_class or refines_imagediff --git a/pipelex/core/concept_provider_abstract.py b/pipelex/core/concept_provider_abstract.pyindex adf8a30f9..09bd24655 100644--- a/pipelex/core/concept_provider_abstract.py+++ b/pipelex/core/concept_provider_abstract.py@@ -1,5 +1,5 @@from abc import ABC, abstractmethod-from typing import ClassVar, Dict, List, Optional+from typing import Any, ClassVar, Dict, List, Optional, Typefrom pipelex.core.concept import Concept@@ -42,3 +42,11 @@ class ConceptProviderAbstract(ABC):@abstractmethoddef teardown(self) -> None:pass++ @abstractmethod+ def get_class(self, concept_code: str) -> Optional[Type[Any]]:+ pass++ @abstractmethod+ def is_image_concept(self, concept_code: str) -> bool:+ passdiff --git a/pipelex/core/pipe_library.py b/pipelex/core/pipe_library.pyindex 13820fcfc..f8faa607b 100644--- a/pipelex/core/pipe_library.py+++ b/pipelex/core/pipe_library.py@@ -1,8 +1,12 @@+from itertools import groupbyfrom typing import Dict, List, Optionalfrom pydantic import Field, RootModel+from rich import box+from rich.table import Tablefrom typing_extensions import override+from pipelex import pretty_printfrom pipelex.core.pipe_abstract import PipeAbstractfrom pipelex.core.pipe_provider_abstract import PipeProviderAbstractfrom pipelex.exceptions import ConceptError, ConceptLibraryConceptNotFoundError, PipeLibraryError, PipeLibraryPipeNotFoundError@@ -64,3 +68,61 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeProviderAbstract):@overridedef teardown(self) -> None:self.root = {}++ @override+ def pretty_list_pipes(self) -> None:+ def _format_concept_code(concept_code: Optional[str], current_domain: str) -> str:+ """Format concept code by removing domain prefix if it matches current domain."""+ if not concept_code:+ return ""+ parts = concept_code.split(".")+ if len(parts) == 2 and parts[0] == current_domain:+ return parts[1]+ return concept_code++ pipes = self.get_pipes()++ # Sort pipes by domain and code+ ordered_items = sorted(pipes, key=lambda x: (x.domain or "", x.code or ""))++ # Create dictionary for return value+ pipes_dict: Dict[str, Dict[str, Dict[str, str]]] = {}++ # Group by domain and create separate tables+ for domain, domain_pipes in groupby(ordered_items, key=lambda x: x.domain):+ table = Table(+ title=f"[bold magenta]domain = {domain}[/]",+ show_header=True,+ show_lines=True,+ header_style="bold cyan",+ box=box.SQUARE_DOUBLE_HEAD,+ border_style="blue",+ )++ table.add_column("Code", style="green")+ table.add_column("Definition", style="white")+ table.add_column("Input", style="yellow")+ table.add_column("Output", style="yellow")++ pipes_dict[domain] = {}++ for pipe in domain_pipes:+ inputs = pipe.inputs+ formatted_inputs = [f"{name}: {_format_concept_code(concept_code, domain)}" for name, concept_code in inputs.items]+ formatted_inputs_str = ", ".join(formatted_inputs)+ output_code = _format_concept_code(pipe.output_concept_code, domain)++ table.add_row(+ pipe.code,+ pipe.definition or "",+ formatted_inputs_str,+ output_code,+ )++ pipes_dict[domain][pipe.code] = {+ "definition": pipe.definition or "",+ "inputs": formatted_inputs_str,+ "output": pipe.output_concept_code,+ }++ pretty_print(table)diff --git a/pipelex/core/pipe_provider_abstract.py b/pipelex/core/pipe_provider_abstract.pyindex c24214148..535e17fdf 100644--- a/pipelex/core/pipe_provider_abstract.py+++ b/pipelex/core/pipe_provider_abstract.py@@ -26,3 +26,7 @@ class PipeProviderAbstract(ABC):@abstractmethoddef teardown(self) -> None:pass++ @abstractmethod+ def pretty_list_pipes(self) -> None:+ passdiff --git a/pipelex/libraries/pipelines/images.toml b/pipelex/libraries/pipelines/images.tomlindex 0bb7cd01b..b2391fdbf 100644--- a/pipelex/libraries/pipelines/images.toml+++ b/pipelex/libraries/pipelines/images.toml@@ -1,5 +1,3 @@--domain = "images"definition = "Generic image-related domain"@@ -26,7 +24,6 @@ PipeLLM = "Describe an image"inputs = { image = "Image" }output = "VisualDescription"system_prompt = "You are a very good observer."-images = ["image"]llm = "llm_to_describe_img"prompt_template = """Describe the provided image in great detail.@@ -37,7 +34,6 @@ PipeLLM = "Describe a photo"inputs = { photo = "Photo" }output = "VisualDescription"system_prompt = "You are a very good observer."-images = ["photo"]llm = "llm_to_describe_img"prompt_template = """Describe the provided photo and how it was shot: scene, lighting, camera, etc.diff --git a/pipelex/pipe_operators/pipe_llm.py b/pipelex/pipe_operators/pipe_llm.pyindex 15d58ce41..d1f540cef 100644--- a/pipelex/pipe_operators/pipe_llm.py+++ b/pipelex/pipe_operators/pipe_llm.py@@ -69,7 +69,14 @@ class PipeLLM(PipeOperator):output_multiplicity: Optional[PipeOutputMultiplicity] = Nonedef needed_inputs(self) -> PipeInputSpec:- return self.pipe_llm_prompt.needed_inputs()+ pipe_llm_prompt_needed_inputs = self.pipe_llm_prompt.needed_inputs()+ # The images are not tagged in the prompt_template. Therefore if an image is provided in the+ # inputs, it becomes a needed input.+ concept_provider = get_concept_provider()+ for input_name, concept_code in self.inputs.root.items():+ if concept_provider.is_image_concept(concept_code=concept_code):+ pipe_llm_prompt_needed_inputs.add_requirement(variable_name=input_name, concept_code=NativeConcept.IMAGE.code)+ return pipe_llm_prompt_needed_inputs@model_validator(mode="after")def validate_inputs(self) -> Self:@@ -158,6 +165,14 @@ class PipeLLM(PipeOperator):log.error(extraneous_input_var_error.desc())case StaticValidationReaction.RAISE:raise extraneous_input_var_error+ else:+ # Check if this input is an image concept but is being used as a variable in the prompt+ if concept_provider.is_image_concept(concept_code=input_name):+ raise PipeDefinitionError(+ f"Image-based input '{input_name}' of concept '{input_name}' "+ f"cannot be used as a variable in a prompt for Pipe '{self.code}'. "+ f"Image variables are automatically passed to vision-enabled LLMs."+ )@model_validator(mode="after")def validate_output_concept_consistency(self) -> Self:diff --git a/pipelex/pipe_operators/pipe_llm_factory.py b/pipelex/pipe_operators/pipe_llm_factory.pyindex e78058732..95cca4c55 100644--- a/pipelex/pipe_operators/pipe_llm_factory.py+++ b/pipelex/pipe_operators/pipe_llm_factory.py@@ -8,7 +8,7 @@ from pipelex.core.pipe_blueprint import PipeBlueprint, PipeSpecificFactoryProtocfrom pipelex.core.pipe_input_spec import PipeInputSpecfrom pipelex.core.pipe_run_params import make_output_multiplicityfrom pipelex.exceptions import PipeDefinitionError-from pipelex.hub import get_optional_domain+from pipelex.hub import get_concept_provider, get_optional_domainfrom pipelex.pipe_operators.pipe_jinja2 import PipeJinja2from pipelex.pipe_operators.pipe_jinja2_factory import PipeJinja2Factoryfrom pipelex.pipe_operators.pipe_llm import PipeLLM, StructuringMethod@@ -28,8 +28,6 @@ class PipeLLMBlueprint(PipeBlueprint):prompt_name: Optional[str] = Noneprompt: Optional[str] = None- images: Optional[List[str]] = None-llm: Optional[LLMSettingOrPresetId] = Nonellm_to_structure: Optional[LLMSettingOrPresetId] = Nonellm_to_structure_direct: Optional[LLMSettingOrPresetId] = None@@ -43,7 +41,6 @@ class PipeLLMBlueprint(PipeBlueprint):nb_output: Optional[int] = Nonemultiple_output: Optional[bool] = None- # TODO: chack that the listed images are listed in the inputs@model_validator(mode="after")def validate_multiple_output(self) -> Self:if excess_attributes_list := has_more_than_one_among_attributes_from_lists(@@ -112,6 +109,15 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):jinja2_name=pipe_code,)+ user_images: List[str] = []+ if pipe_blueprint.inputs:+ for stuff_name, concept_code in (pipe_blueprint.inputs).items():+ concept = get_concept_provider().get_required_concept(concept_code=concept_code)+ if get_concept_provider().is_image_concept(concept_code=concept.code):+ user_images.append(stuff_name)+ else:+ # Implicit text concept+ passpipe_llm_prompt = PipeLLMPrompt(code="adhoc_for_pipe_llm_prompt",domain=domain_code,@@ -122,7 +128,7 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):user_pipe_jinja2=user_pipe_jinja2,user_prompt_verbatim_name=pipe_blueprint.prompt_name,user_text=pipe_blueprint.prompt,- user_images=pipe_blueprint.images,+ user_images=user_images or None,)llm_settings = LLMSettingChoices(diff --git a/pipelex/pipe_operators/pipe_llm_prompt.py b/pipelex/pipe_operators/pipe_llm_prompt.pyindex d538b9a42..e7d107878 100644--- a/pipelex/pipe_operators/pipe_llm_prompt.py+++ b/pipelex/pipe_operators/pipe_llm_prompt.py@@ -106,10 +106,6 @@ class PipeLLMPrompt(PipeOperator):continuepipe_input_spec.add_requirement(variable_name=conceptless_required_variable, concept_code=NativeConcept.ANYTHING.code)- if self.user_images:- for user_image in self.user_images:- pipe_input_spec.add_requirement(variable_name=user_image, concept_code=NativeConcept.IMAGE.code)-return pipe_input_spec@overridediff --git a/tests/pipelex/pipelex_asynch/test_image_inputs.py b/tests/pipelex/pipelex_asynch/test_image_inputs.pynew file mode 100644index 000000000..6fd39e055--- /dev/null+++ b/tests/pipelex/pipelex_asynch/test_image_inputs.py@@ -0,0 +1,95 @@+from typing import cast++import pytest+from pytest import FixtureRequest++from pipelex import pretty_print+from pipelex.core.pipe_output import PipeOutput+from pipelex.core.pipe_run_params import PipeRunMode+from pipelex.core.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.core.stuff_content import ImageContent, PageContent, TextAndImagesContent, TextContent+from pipelex.core.stuff_factory import StuffFactory+from pipelex.core.working_memory_factory import WorkingMemoryFactory+from pipelex.hub import get_pipe_router, get_report_delegate+from pipelex.pipeline.job_metadata import JobMetadata+from tests.test_data import ImageTestCases+from tests.test_pipelines.misc_tests.tests import Article++[email protected]_runnable[email protected](loop_scope="class")+class TestImageInputs:+ """Test class for verifying image input functionality in pipes."""++ async def test_extract_article_from_image(+ self,+ request: FixtureRequest,+ pipe_run_mode: PipeRunMode,+ ) -> None:+ """Test that an image is indeed given to the LLM, and that it can extract extact whats on the image."""+ working_memory = WorkingMemoryFactory.make_from_image(image_url=ImageTestCases.IMAGE_FILE_PATH_PNG)++ # Run the pipe+ pipe_output: PipeOutput = await get_pipe_router().run_pipe_code(+ pipe_code="extract_article_from_image",+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),+ working_memory=working_memory,+ job_metadata=JobMetadata(+ top_job_id=cast(str, request.node.originalname), # type: ignore+ ),+ )++ # Log output and generate report+ pretty_print(pipe_output, title="Pipe output")+ get_report_delegate().generate_report()++ # Verify output+ if pipe_run_mode != PipeRunMode.DRY:+ article = pipe_output.main_stuff_as(content_type=Article)+ pretty_print(article, title="Article")+ assert article.title == "The Solar System: An Overview"+ assert pipe_output is not None+ assert pipe_output.working_memory is not None+ assert pipe_output.main_stuff is not None++ async def test_describe_page(self, request: FixtureRequest, pipe_run_mode: PipeRunMode) -> None:+ """+ Test that a pipe can accept a PageContent input, give to the LLM the image via subattributes,+ But also accepts basic objects+ """+ # Create the page content+ image_content = ImageContent(url=ImageTestCases.IMAGE_FILE_PATH_PNG)+ text_and_images = TextAndImagesContent(text=TextContent(text="This is a test page"), images=[])+ page_content = PageContent(text_and_images=text_and_images, page_view=image_content)++ # Create stuff from page content+ stuff = StuffFactory.make_stuff(concept_str="Page", content=page_content, name="page")++ # Create working memory+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)++ # Run the pipe+ pipe_output: PipeOutput = await get_pipe_router().run_pipe_code(+ pipe_code="describe_page",+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),+ working_memory=working_memory,+ job_metadata=JobMetadata(+ top_job_id=cast(str, request.node.originalname), # type: ignore+ ),+ )++ # Log output and generate report+ pretty_print(pipe_output, title="Pipe output")+ get_report_delegate().generate_report()++ # Verify output+ if pipe_run_mode != PipeRunMode.DRY:+ article = pipe_output.main_stuff_as(content_type=Article)+ pretty_print(article, title="Article")+ assert article.title == "The Solar System: An Overview"+ assert article.description == "This is a test page"+ assert pipe_output is not None+ assert pipe_output.working_memory is not None+ assert pipe_output.main_stuff is not Nonediff --git a/tests/pipelex/pipelex_asynch/test_pipe_llm.py b/tests/pipelex/pipelex_asynch/test_pipe_llm.pyindex 86535a7ca..6dcfe0d47 100644--- a/tests/pipelex/pipelex_asynch/test_pipe_llm.py+++ b/tests/pipelex/pipelex_asynch/test_pipe_llm.py@@ -62,7 +62,6 @@ class TestPipeLLM:if not stuff_name:pytest.fail(f"Cannot use nameless stuff in this test: {stuff}")working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=stuff)-pipe_job = PipeJobFactory.make_pipe_job(working_memory=working_memory,pipe=PipeLLM(@@ -75,11 +74,11 @@ class TestPipeLLM:domain="generic",system_prompt=PipeTestCases.SYSTEM_PROMPT,user_text=PipeTestCases.MULTI_IMG_DESC_PROMPT,- user_images=attribute_paths,),),pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),)+pipe_llm_output: PipeLLMOutput = await get_pipe_router().run_pipe_job(pipe_job=pipe_job,)diff --git a/tests/pipelex_unit_test.toml b/tests/pipelex_unit_test.tomlindex 7034400ea..42c5c632a 100644--- a/tests/pipelex_unit_test.toml+++ b/tests/pipelex_unit_test.toml@@ -1,5 +1,3 @@--[cogt.llm_config.preferred_platforms]# These override the defaults set for any llm integration# Use at your convenience:diff --git a/tests/test_pipelines/misc_tests/test_errors.toml b/tests/test_pipelines/misc_tests/test_errors.tomldeleted file mode 100644index 216df442d..000000000--- a/tests/test_pipelines/misc_tests/test_errors.toml+++ /dev/null@@ -1,45 +0,0 @@---domain = "test_errors"-definition = "This library is intended for testing errors"--# [pipe.jinja2_syntax_error]-# PipeLLM = "Jinja2 syntax error (on purpose, for testing): bad curly braces"-# inputs = { text = "Text" }-# output = "Text"-# prompt_template = """-# {{ a123|tag }-# """--# [pipe.jinja2_dummy_filter1]-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"-# inputs = { text = "Text" }-# output = "Text"-# prompt_template = """-# {{ foobar|dummy_filter1 }}-# """--# [pipe.jinja2_dummy_filter2]-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"-# inputs = { text = "Text" }-# output = "Text"-# prompt_template = """-# {{ foobar|dummy_filter2() }}-# """--# [pipe.jinja2_dummy_filter3]-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"-# inputs = { text = "Text" }-# output = "Text"-# prompt_template = """-# {{ foobar|dummy_filter3("param") }}-# """---# [pipe.jinja2_dummy_filter4]-# PipeLLM = "Jinja2 undefined filter (on purpose, for testing)"-# inputs = { text = "Text" }-# output = "Text"-# prompt_template = """-# {{ foobar|dummy_filter4(param1, param2) }}-# """diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.toml b/tests/test_pipelines/misc_tests/test_image_inputs.tomlnew file mode 100644index 000000000..f75082116--- /dev/null+++ b/tests/test_pipelines/misc_tests/test_image_inputs.toml@@ -0,0 +1,32 @@+domain = "test_image_inputs"+definition = "Test domain for verifying image input functionality"++[concept]+Article = "Article"++[pipe]+[pipe.extract_article_from_image]+PipeLLM = "Describe a page content"+inputs = { image = "Image" }+output = "Article"+system_prompt = """+You are an expert at describing page contents.+"""+prompt_template = """+Extract the text and the date of the article in the image.+"""++[pipe.describe_page]+PipeLLM = "Describe a page"+inputs = { "page.page_view" = "Image", page = "Page" }+output = "Article"+system_prompt = """+You are an expert at describing page contents.+"""+prompt_template = """+Extract the date and title of the article.++Also, add this as the description of the article:+[email protected]_and_images.text.text+"""\ No newline at end of filediff --git a/tests/test_pipelines/misc_tests/tests.py b/tests/test_pipelines/misc_tests/tests.pyindex e648f6c2b..cc36b393a 100644--- a/tests/test_pipelines/misc_tests/tests.py+++ b/tests/test_pipelines/misc_tests/tests.py@@ -1,5 +1,11 @@-from pipelex.core.stuff_content import ListContent, TextContent+from pipelex.core.stuff_content import ListContent, StructuredContent, TextContentclass RandomListContent(ListContent[TextContent]):pass+++class Article(StructuredContent):+ title: str+ description: str+ date: strdiff --git a/tests/test_pipelines/misc_tests/tests.toml b/tests/test_pipelines/misc_tests/tests.tomlindex c7e5083ae..bf53640a2 100644--- a/tests/test_pipelines/misc_tests/tests.toml+++ b/tests/test_pipelines/misc_tests/tests.toml@@ -23,7 +23,6 @@ output = "Text"prompt_template = """Describe the using rap lyrics, including puns and references to the image."""-images = ["image"][pipe.create_characters]PipeLLM = "Create Characters"