Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Feature/enable native concepts in input memory (#206)
Louis Choquelf512fb5cdf7c
diff --git a/.cursor/rules/pipelex.mdc b/.cursor/rules/pipelex.mdcindex 92ce9fddb..ae078a3f4 100644--- a/.cursor/rules/pipelex.mdc+++ b/.cursor/rules/pipelex.mdc@@ -393,17 +393,14 @@ Be sure to make the proper choice of prefix for each insertion.**DO NOT write "$topic" alone in an isolated line.****DO write things like "Write an essay about $topic" included in an actual sentence.**-# Example to start a pipe when its done+# Example to execute a pipeline-Here is an example of how to start a pipe:```pythonimport asynciofrom pipelex import pretty_print-from pipelex.core.working_memory_factory import WorkingMemoryFactoryfrom pipelex.hub import get_pipeline_tracker, get_report_delegatefrom pipelex.pipelex import Pipelex-from pipelex.core.stuff_factory import StuffFactoryfrom pipelex.pipeline.execute import execute_pipelinefrom pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart@@ -413,18 +410,15 @@ IMAGE_URL = "assets/gantt/gantt_tree_house.png"async def extract_gantt(image_url: str) -> GanttChart:- # Create Working Memory- stuff = StuffFactory.make_from_image(- image_url=image_url,- concept_str="gantt.GanttImage",- name="gantt_chart_image",- )- working_memory = WorkingMemoryFactory.make_from_multiple_stuff([stuff])-# Run the pipepipe_output = await execute_pipeline(pipe_code="extract_gantt_by_steps",- working_memory=working_memory,+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },)# Output the resultreturn pipe_output.main_stuff_as(content_type=GanttChart)@@ -443,12 +437,72 @@ pretty_print(gantt_chart, title="Gantt Chart")get_pipeline_tracker().output_flowchart()```-# Error Handling & Validation+The input memory is a dictionary of key-value pairs, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:+```python+StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str+ImplicitMemory = Dict[str, StuffContentOrData]+```+As you can seen, we made it so different ways can be used to define that stuff using structured content or data.-## Validation Error Handling+So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:-- Use `StuffContentValidationError` for handling content validation failures-- Use `format_pydantic_validation_error` to format validation errors consistently+```python+# Here we have a single input and it's a Text.+# If you assign a string, by default it will be considered as a TextContent.+ pipe_output = await execute_pipeline(+ pipe_code="master_advisory_orchestrator",+ input_memory={+ "user_input": problem_description,+ },+ )-ALWAYS RUN `pipelex validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+# Here we have a single input and it's a PDF.+# Because PDFContent is a native concept, we can use it directly as a value,+# the system knows what content it corresponds to:+ pipe_output = await execute_pipeline(+ pipe_code="power_extractor_dpe",+ input_memory={+ "ocr_input": PDFContent(url=pdf_url),+ },+ )++# Here we have a single input and it's an Image.+# Because ImageContent is a native concept, we can use it directly as a value:+ pipe_output = await execute_pipeline(+ pipe_code="fashion_variation_pipeline",+ input_memory={+ "fashion_photo": ImageContent(url=image_url),+ },+ )++# Here we have a single input, it's an image but+# its actually a more specific concept gantt.GanttImage which refines Image,+# so we must provide it using a dict with the concept and the content:+ pipe_output = await execute_pipeline(+ pipe_code="extract_gantt_by_steps",+ input_memory={+ "gantt_chart_image": {+ "concept": "gantt.GanttImage",+ "content": ImageContent(url=image_url),+ }+ },+ )+# Here is a more complex example with multiple inputs assigned using different ways:+ pipe_output = await execute_pipeline(+ pipe_code="retrieve_then_answer",+ dynamic_output_concept_code="contracts.Fees",+ input_memory={+ "text": load_text_from_path(path=text_path),+ "question": {+ "concept": "answer.Question",+ "content": question,+ },+ "client_instructions": client_instructions,+ },+ )+```++ALWAYS RUN `pipelex validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.+Then, create an example file to run the pipeline in the `examples` folder.+But don't write documentation unless asked explicitly to.diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.ymlindex 15850e9dd..46fe56115 100644--- a/.github/ISSUE_TEMPLATE/config.yml+++ b/.github/ISSUE_TEMPLATE/config.yml@@ -1,5 +1,8 @@blank_issues_enabled: falsecontact_links:+ - name: 💡 Pipeline requests+ url: https://github.com/Pipelex/pipelex-cookbook/discussions/categories/ideas+ about: "Please start an Idea Discussion in the Cookbook."- name: 💬 Ask a questionurl: https://github.com/Pipelex/pipelex/discussionsabout: "Please start a Discussion instead of filing a blank issue."diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ca055f4d6..b87e790c8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -403,7 +403,7 @@ is_reporting_enabled = true**Major Model Additions** - Claude 4 (Opus & Sonnet), Grok-3, and GPT-4 image generation are now in the house.### Pipeline Base Library update-- **New pipe** - `extract_page_contents_and_views_from_pdf` transferred from cookbook to base library (congrats on the promotion!). This pipe extracts text, linked images, **AND** page_view images (rendered pages) - it's very useful if you want to use Vision in follow-up pipes+- **New pipe** - `ocr_page_contents_and_views_from_pdf` transferred from cookbook to base library (congrats on the promotion!). This pipe extracts text, linked images, **AND** page_view images (rendered pages) - it's very useful if you want to use Vision in follow-up pipes### Addeddiff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.mdindex 5d7f7bf0d..dfd1e9300 100644--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md@@ -121,30 +121,27 @@ To run the `description_to_tagline` pipeline we defined above, you would call it```pythonimport asyncio-from pipelex.core.working_memory import WorkingMemory-from pipelex.core.working_memory_factory import WorkingMemoryFactory+from pipelex.pipelex import Pipelexfrom pipelex.pipeline.execute import execute_pipelineasync def main():# First, initialize Pipelex (this loads all pipeline definitions)Pipelex.make()-- # Create working memory with the pipeline's input- working_memory = WorkingMemoryFactory.make_from_text(- text="EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",- concept_code="ProductDescription",- name="description"- )# Execute the pipeline and wait for the resultpipe_output = await execute_pipeline(pipe_code="description_to_tagline",- working_memory=working_memory,+ input_memory={+ "description": {+ "concept": "ProductDescription",+ "content": "EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",+ },+ },)# Get the final output- tagline = pipe_output.main_stuff_as(content_type=str)+ tagline = pipe_output.main_stuff_as_strprint(f"Generated tagline: {tagline}")if __name__ == "__main__":diff --git a/docs/pages/cookbook-examples/simple-ocr.md b/docs/pages/cookbook-examples/simple-ocr.mdindex c5441ddab..aa6350944 100644--- a/docs/pages/cookbook-examples/simple-ocr.md+++ b/docs/pages/cookbook-examples/simple-ocr.md@@ -10,7 +10,7 @@ This is a fundamental building block for many document processing workflows.## The Pipeline Explained-The core of this example is a simple function that creates a "working memory" from a PDF and then executes a pre-defined pipeline called `extract_page_contents_from_pdf`.+The core of this example is a simple function that creates a "working memory" from a PDF and then executes a pre-defined pipeline called `ocr_page_contents_from_pdf`.```pythonasync def simple_ocr(pdf_url: str):@@ -20,7 +20,7 @@ async def simple_ocr(pdf_url: str):name="pdf",)pipe_output = await execute_pipeline(- pipe_code="extract_page_contents_from_pdf",+ pipe_code="ocr_page_contents_from_pdf",working_memory=working_memory,)page_content_list: ListContent[PageContent] = pipe_output.main_stuff_as_list(item_type=PageContent)diff --git a/pipelex/core/concept_factory.py b/pipelex/core/concept_factory.pyindex 232026343..fffcf5b43 100644--- a/pipelex/core/concept_factory.py+++ b/pipelex/core/concept_factory.py@@ -5,7 +5,7 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationErrorfrom pipelex.core.concept import Conceptfrom pipelex.core.concept_code_factory import ConceptCodeFactory-from pipelex.core.concept_native import NativeConcept+from pipelex.core.concept_native import NativeConcept, NativeConceptClassfrom pipelex.core.domain import SpecialDomainfrom pipelex.core.stuff_content import TextContentfrom pipelex.exceptions import ConceptFactoryError, StructureClassError@@ -179,6 +179,11 @@ class ConceptFactory:structure_class_name=native_concept.content_class_name,)+ @classmethod+ def make_native_concept_from_native_concept_class(cls, native_concept_class: NativeConceptClass) -> Concept:+ native_concept = native_concept_class.native_concept+ return cls.make_native_concept(native_concept=native_concept)+@classmethoddef list_native_concepts(cls) -> List[Concept]:concepts: List[Concept] = []diff --git a/pipelex/core/concept_native.py b/pipelex/core/concept_native.pyindex d0c6f6e83..efc50c3e2 100644--- a/pipelex/core/concept_native.py+++ b/pipelex/core/concept_native.py@@ -15,6 +15,30 @@ class NativeConceptClass(StrEnum):LLM_PROMPT = "LLMPromptContent"PAGE = "PageContent"+ @classmethod+ def class_names(cls) -> List[str]:+ return [code.value for code in cls]++ @property+ def native_concept(self) -> "NativeConcept":+ match self:+ case NativeConceptClass.DYNAMIC:+ return NativeConcept.DYNAMIC+ case NativeConceptClass.TEXT:+ return NativeConcept.TEXT+ case NativeConceptClass.IMAGE:+ return NativeConcept.IMAGE+ case NativeConceptClass.PDF:+ return NativeConcept.PDF+ case NativeConceptClass.TEXT_AND_IMAGES:+ return NativeConcept.TEXT_AND_IMAGES+ case NativeConceptClass.NUMBER:+ return NativeConcept.NUMBER+ case NativeConceptClass.LLM_PROMPT:+ return NativeConcept.LLM_PROMPT+ case NativeConceptClass.PAGE:+ return NativeConcept.PAGE+# Exceptionally, we use an Enum here (and not our usual StrEnum) to avoid confusion with# the concept_code which must have the form "native.ConceptName"diff --git a/pipelex/core/stuff_factory.py b/pipelex/core/stuff_factory.pyindex 908848d72..f5e663611 100644--- a/pipelex/core/stuff_factory.py+++ b/pipelex/core/stuff_factory.py@@ -6,7 +6,8 @@ from pydantic import BaseModel, ValidationErrorfrom pipelex.client.protocol import StuffContentOrDatafrom pipelex.core.concept import Conceptfrom pipelex.core.concept_code_factory import ConceptCodeFactory-from pipelex.core.concept_native import NativeConcept+from pipelex.core.concept_factory import ConceptFactory+from pipelex.core.concept_native import NativeConcept, NativeConceptClassfrom pipelex.core.stuff import Stufffrom pipelex.core.stuff_content import (ListContent,@@ -216,6 +217,15 @@ class StuffFactory:elif isinstance(stuff_content_or_data, StuffContent):content = stuff_content_or_dataconcept_name = type(content).__name__+ if concept_name in NativeConceptClass.class_names():+ native_concept_class = NativeConceptClass(concept_name)+ concept = ConceptFactory.make_native_concept_from_native_concept_class(native_concept_class=native_concept_class)+ return cls.make_stuff(+ concept_str=concept.code,+ content=content,+ name=name,+ code=code,+ )try:return cls.make_stuff_using_concept_name_and_search_domains(concept_name=concept_name,@@ -259,16 +269,24 @@ class StuffFactory:content_value = stuff_content_dict["content"]except KeyError as exc:raise StuffFactoryError(f"Stuff content data dict is badly formed: {exc}") from exc- content = StuffContentFactory.make_stuffcontent_from_concept_code_with_fallback(- concept_code=concept_code,- value=content_value,- )- return StuffFactory.make_stuff(- concept_str=concept_code,- name=name,- content=content,- code=code,- )+ if isinstance(content_value, StuffContent):+ return StuffFactory.make_stuff(+ concept_str=concept_code,+ name=name,+ content=content_value,+ code=code,+ )+ else:+ content = StuffContentFactory.make_stuffcontent_from_concept_code_with_fallback(+ concept_code=concept_code,+ value=content_value,+ )+ return StuffFactory.make_stuff(+ concept_str=concept_code,+ name=name,+ content=content,+ code=code,+ )class StuffContentFactoryError(PipelexError):diff --git a/pipelex/libraries/pipelines/documents.toml b/pipelex/libraries/pipelines/documents.tomlindex cc52a4c66..77fdfd536 100644--- a/pipelex/libraries/pipelines/documents.toml+++ b/pipelex/libraries/pipelines/documents.toml@@ -10,15 +10,15 @@ TextAndImagesContent = "A content that comprises text and images where the text# PipeOcr requires to have a single input# It can be named however you want# but it must be either an image or a pdf or a concept which refines one of them-[pipe.extract_page_contents_from_pdf]+[pipe.ocr_page_contents_from_pdf]PipeOcr = "Extract page contents from a PDF document"inputs = { ocr_input = "PDF" }output = "Page"page_images = truepage_views = false-[pipe.extract_page_contents_and_views_from_pdf]-PipeOcr = "Extract page contents from a PDF document as well aspage views"+[pipe.ocr_page_contents_and_views_from_pdf]+PipeOcr = "Extract page contents from a PDF document as well as full page views"inputs = { ocr_input = "PDF" }output = "Page"page_images = truediff --git a/tests/cases/source_code.py b/tests/cases/source_code.pynew file mode 100644index 000000000..463b191db--- /dev/null+++ b/tests/cases/source_code.py@@ -0,0 +1,33 @@+"""Source code manipulation functions for testing."""++from typing import List++from pipelex.core.stuff_content import TextContent+from pipelex.core.working_memory import WorkingMemory+++def wrap_lines(working_memory: WorkingMemory) -> TextContent:+ """+ Wraps each line of source code text with HTML span tags.++ Args:+ working_memory: The working memory containing the source text++ Returns:+ TextContent with each line wrapped in span tags with class="line"+ """+ # Get the source text from working memory as string+ source_text = working_memory.get_stuff_as_str("source_text")++ # Split into lines and wrap each line+ lines = source_text.split("\n")+ wrapped_lines: List[str] = []++ for line in lines:+ wrapped_line = f'<span class="line">{line}</span>'+ wrapped_lines.append(wrapped_line)++ # Join back into a single string+ wrapped_text = "\n".join(wrapped_lines)++ return TextContent(text=wrapped_text)diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.pynew file mode 100644index 000000000..d610235db--- /dev/null+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.py@@ -0,0 +1 @@+# Empty file for Python package structurediff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.pynew file mode 100644index 000000000..d36d6219f--- /dev/null+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.py@@ -0,0 +1,102 @@+import pytest++from pipelex import log, pretty_print+from pipelex.core.concept_native import NativeConcept+from pipelex.core.pipe_run_params import PipeRunMode+from pipelex.core.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.core.stuff_content import TextContent+from pipelex.core.stuff_factory import StuffFactory+from pipelex.core.working_memory_factory import WorkingMemoryFactory+from pipelex.hub import get_pipe_router, get_report_delegate+from pipelex.pipe_operators.pipe_func import PipeFunc, PipeFuncOutput+from pipelex.pipe_works.pipe_job_factory import PipeJobFactory+from pipelex.tools.func_registry import func_registry+from tests.cases.source_code import wrap_lines++[email protected]_runnable[email protected](loop_scope="class")+class TestPipeFunc:+ @classmethod+ def setup_class(cls):+ """Register test functions before running tests."""+ func_registry.register_function(wrap_lines)++ @classmethod+ def teardown_class(cls):+ """Clean up registered functions after tests."""+ if func_registry.has_function("wrap_lines"):+ func_registry.unregister_function_by_name("wrap_lines")++ async def test_wrap_lines_pipe_func(+ self,+ pipe_run_mode: PipeRunMode,+ ):+ # Sample source code to test with+ sample_code = """def hello_world():+ print("Hello, World!")+ return "Hello!"++if __name__ == "__main__":+ hello_world()"""++ # Create stuff with the source code+ source_text_stuff = StuffFactory.make_stuff(+ name="source_text",+ concept_str=NativeConcept.TEXT.code,+ content=TextContent(text=sample_code),+ )++ # Create working memory with the source text+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=source_text_stuff)++ # Create the PipeFunc job+ pipe_job = PipeJobFactory.make_pipe_job(+ pipe=PipeFunc(+ code="wrap_lines",+ domain="source_code",+ output_concept_code=NativeConcept.TEXT.code,+ function_name="wrap_lines",+ ),+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),+ working_memory=working_memory,+ )++ # Execute the pipe+ pipe_func_output: PipeFuncOutput = await get_pipe_router().run_pipe_job(+ pipe_job=pipe_job,+ )++ # Log and verify results+ log.verbose(pipe_func_output, title="pipe_func_output")+ wrapped_text = pipe_func_output.main_stuff_as_text+ pretty_print(wrapped_text, title="wrapped_text")+ get_report_delegate().generate_report()++ # Verify the output contains wrapped lines+ assert pipe_func_output is not None+ assert pipe_func_output.working_memory is not None+ assert pipe_func_output.main_stuff is not None++ # Get the actual text string for verification+ wrapped_text_str = wrapped_text.text++ # In DRY mode, we get mock content, so we only check basic structure+ if pipe_run_mode == PipeRunMode.DRY:+ # In dry mode, just verify we have text content and it's non-empty+ assert isinstance(wrapped_text_str, str)+ assert len(wrapped_text_str) > 0+ else:+ # In LIVE mode, check the actual wrapping functionality+ # Check that each line is wrapped in span tags+ lines = wrapped_text_str.split("\n")+ for line in lines:+ assert line.startswith('<span class="line">'), f"Line should start with span tag: {line}"+ assert line.endswith("</span>"), f"Line should end with span tag: {line}"++ # Verify we have the expected number of lines (original had 6 lines)+ assert len(lines) == 6, f"Expected 6 lines, got {len(lines)}"++ # Verify specific content is preserved within spans+ assert '<span class="line">def hello_world():</span>' in wrapped_text_str+ assert '<span class="line"> print("Hello, World!")</span>' in wrapped_text_strdiff --git a/tests/integration/pipelex/test_data.py b/tests/integration/pipelex/test_data.pyindex 4b15024a0..eba365325 100644--- a/tests/integration/pipelex/test_data.py+++ b/tests/integration/pipelex/test_data.py@@ -155,7 +155,7 @@ class PipeTestCases:("Extract page contents from PDF",SIMPLE_STUFF_PDF,- "extract_page_contents_from_pdf",+ "ocr_page_contents_from_pdf",),]FAILURE_PIPES: ClassVar[List[Tuple[str, Type[Exception], str]]] = [diff --git a/uv.lock b/uv.lockindex c56e16196..6462e726b 100644--- a/uv.lock+++ b/uv.lock@@ -1748,7 +1748,7 @@ wheels = [[[package]]name = "pipelex"-version = "0.6.7"+version = "0.6.9"source = { editable = "." }dependencies = [{ name = "aiofiles" },