Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Feature/enable native concepts in input memory (#206)

Louis Choquelf512fb5cdf7c
diff --git a/.cursor/rules/pipelex.mdc b/.cursor/rules/pipelex.mdc
index 92ce9fddb..ae078a3f4 100644
--- a/.cursor/rules/pipelex.mdc
+++ b/.cursor/rules/pipelex.mdc
@@ -393,17 +393,14 @@ Be sure to make the proper choice of prefix for each insertion.
**DO NOT write "$topic" alone in an isolated line.**
**DO write things like "Write an essay about $topic" included in an actual sentence.**
-# Example to start a pipe when its done
+# Example to execute a pipeline
-Here is an example of how to start a pipe:
```python
import asyncio
from pipelex import pretty_print
-from pipelex.core.working_memory_factory import WorkingMemoryFactory
from pipelex.hub import get_pipeline_tracker, get_report_delegate
from pipelex.pipelex import Pipelex
-from pipelex.core.stuff_factory import StuffFactory
from pipelex.pipeline.execute import execute_pipeline
from pipelex_libraries.pipelines.examples.extract_gantt.gantt import GanttChart
@@ -413,18 +410,15 @@ IMAGE_URL = "assets/gantt/gantt_tree_house.png"
async def extract_gantt(image_url: str) -> GanttChart:
- # Create Working Memory
- stuff = StuffFactory.make_from_image(
- image_url=image_url,
- concept_str="gantt.GanttImage",
- name="gantt_chart_image",
- )
- working_memory = WorkingMemoryFactory.make_from_multiple_stuff([stuff])
-
# Run the pipe
pipe_output = await execute_pipeline(
pipe_code="extract_gantt_by_steps",
- working_memory=working_memory,
+ input_memory={
+ "gantt_chart_image": {
+ "concept": "gantt.GanttImage",
+ "content": ImageContent(url=image_url),
+ }
+ },
)
# Output the result
return pipe_output.main_stuff_as(content_type=GanttChart)
@@ -443,12 +437,72 @@ pretty_print(gantt_chart, title="Gantt Chart")
get_pipeline_tracker().output_flowchart()
```
-# Error Handling & Validation
+The input memory is a dictionary of key-value pairs, where the key is the name of the input variable and the value provides details to make it a stuff object. The relevant definitions are:
+```python
+StuffContentOrData = Dict[str, Any] | StuffContent | List[Any] | str
+ImplicitMemory = Dict[str, StuffContentOrData]
+```
+As you can seen, we made it so different ways can be used to define that stuff using structured content or data.
-## Validation Error Handling
+So here are a few concrete examples of calls to execute_pipeline with various ways to set up the input memory:
-- Use `StuffContentValidationError` for handling content validation failures
-- Use `format_pydantic_validation_error` to format validation errors consistently
+```python
+# Here we have a single input and it's a Text.
+# If you assign a string, by default it will be considered as a TextContent.
+ pipe_output = await execute_pipeline(
+ pipe_code="master_advisory_orchestrator",
+ input_memory={
+ "user_input": problem_description,
+ },
+ )
-ALWAYS RUN `pipelex validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
+# Here we have a single input and it's a PDF.
+# Because PDFContent is a native concept, we can use it directly as a value,
+# the system knows what content it corresponds to:
+ pipe_output = await execute_pipeline(
+ pipe_code="power_extractor_dpe",
+ input_memory={
+ "ocr_input": PDFContent(url=pdf_url),
+ },
+ )
+
+# Here we have a single input and it's an Image.
+# Because ImageContent is a native concept, we can use it directly as a value:
+ pipe_output = await execute_pipeline(
+ pipe_code="fashion_variation_pipeline",
+ input_memory={
+ "fashion_photo": ImageContent(url=image_url),
+ },
+ )
+
+# Here we have a single input, it's an image but
+# its actually a more specific concept gantt.GanttImage which refines Image,
+# so we must provide it using a dict with the concept and the content:
+ pipe_output = await execute_pipeline(
+ pipe_code="extract_gantt_by_steps",
+ input_memory={
+ "gantt_chart_image": {
+ "concept": "gantt.GanttImage",
+ "content": ImageContent(url=image_url),
+ }
+ },
+ )
+# Here is a more complex example with multiple inputs assigned using different ways:
+ pipe_output = await execute_pipeline(
+ pipe_code="retrieve_then_answer",
+ dynamic_output_concept_code="contracts.Fees",
+ input_memory={
+ "text": load_text_from_path(path=text_path),
+ "question": {
+ "concept": "answer.Question",
+ "content": question,
+ },
+ "client_instructions": client_instructions,
+ },
+ )
+```
+
+ALWAYS RUN `pipelex validate` when you are finished writing pipelines: This checks for errors. If there are errors, iterate until it works.
+Then, create an example file to run the pipeline in the `examples` folder.
+But don't write documentation unless asked explicitly to.
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
index 15850e9dd..46fe56115 100644
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -1,5 +1,8 @@
blank_issues_enabled: false
contact_links:
+ - name: 💡 Pipeline requests
+ url: https://github.com/Pipelex/pipelex-cookbook/discussions/categories/ideas
+ about: "Please start an Idea Discussion in the Cookbook."
- name: 💬 Ask a question
url: https://github.com/Pipelex/pipelex/discussions
about: "Please start a Discussion instead of filing a blank issue."
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ca055f4d6..b87e790c8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -403,7 +403,7 @@ is_reporting_enabled = true
**Major Model Additions** - Claude 4 (Opus & Sonnet), Grok-3, and GPT-4 image generation are now in the house.
### Pipeline Base Library update
-- **New pipe** - `extract_page_contents_and_views_from_pdf` transferred from cookbook to base library (congrats on the promotion!). This pipe extracts text, linked images, **AND** page_view images (rendered pages) - it's very useful if you want to use Vision in follow-up pipes
+- **New pipe** - `ocr_page_contents_and_views_from_pdf` transferred from cookbook to base library (congrats on the promotion!). This pipe extracts text, linked images, **AND** page_view images (rendered pages) - it's very useful if you want to use Vision in follow-up pipes
### Added
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
index 5d7f7bf0d..dfd1e9300 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
@@ -121,30 +121,27 @@ To run the `description_to_tagline` pipeline we defined above, you would call it
```python
import asyncio
-from pipelex.core.working_memory import WorkingMemory
-from pipelex.core.working_memory_factory import WorkingMemoryFactory
+
from pipelex.pipelex import Pipelex
from pipelex.pipeline.execute import execute_pipeline
async def main():
# First, initialize Pipelex (this loads all pipeline definitions)
Pipelex.make()
-
- # Create working memory with the pipeline's input
- working_memory = WorkingMemoryFactory.make_from_text(
- text="EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",
- concept_code="ProductDescription",
- name="description"
- )
# Execute the pipeline and wait for the result
pipe_output = await execute_pipeline(
pipe_code="description_to_tagline",
- working_memory=working_memory,
+ input_memory={
+ "description": {
+ "concept": "ProductDescription",
+ "content": "EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",
+ },
+ },
)
# Get the final output
- tagline = pipe_output.main_stuff_as(content_type=str)
+ tagline = pipe_output.main_stuff_as_str
print(f"Generated tagline: {tagline}")
if __name__ == "__main__":
diff --git a/docs/pages/cookbook-examples/simple-ocr.md b/docs/pages/cookbook-examples/simple-ocr.md
index c5441ddab..aa6350944 100644
--- a/docs/pages/cookbook-examples/simple-ocr.md
+++ b/docs/pages/cookbook-examples/simple-ocr.md
@@ -10,7 +10,7 @@ This is a fundamental building block for many document processing workflows.
## The Pipeline Explained
-The core of this example is a simple function that creates a "working memory" from a PDF and then executes a pre-defined pipeline called `extract_page_contents_from_pdf`.
+The core of this example is a simple function that creates a "working memory" from a PDF and then executes a pre-defined pipeline called `ocr_page_contents_from_pdf`.
```python
async def simple_ocr(pdf_url: str):
@@ -20,7 +20,7 @@ async def simple_ocr(pdf_url: str):
name="pdf",
)
pipe_output = await execute_pipeline(
- pipe_code="extract_page_contents_from_pdf",
+ pipe_code="ocr_page_contents_from_pdf",
working_memory=working_memory,
)
page_content_list: ListContent[PageContent] = pipe_output.main_stuff_as_list(item_type=PageContent)
diff --git a/pipelex/core/concept_factory.py b/pipelex/core/concept_factory.py
index 232026343..fffcf5b43 100644
--- a/pipelex/core/concept_factory.py
+++ b/pipelex/core/concept_factory.py
@@ -5,7 +5,7 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError
from pipelex.core.concept import Concept
from pipelex.core.concept_code_factory import ConceptCodeFactory
-from pipelex.core.concept_native import NativeConcept
+from pipelex.core.concept_native import NativeConcept, NativeConceptClass
from pipelex.core.domain import SpecialDomain
from pipelex.core.stuff_content import TextContent
from pipelex.exceptions import ConceptFactoryError, StructureClassError
@@ -179,6 +179,11 @@ class ConceptFactory:
structure_class_name=native_concept.content_class_name,
)
+ @classmethod
+ def make_native_concept_from_native_concept_class(cls, native_concept_class: NativeConceptClass) -> Concept:
+ native_concept = native_concept_class.native_concept
+ return cls.make_native_concept(native_concept=native_concept)
+
@classmethod
def list_native_concepts(cls) -> List[Concept]:
concepts: List[Concept] = []
diff --git a/pipelex/core/concept_native.py b/pipelex/core/concept_native.py
index d0c6f6e83..efc50c3e2 100644
--- a/pipelex/core/concept_native.py
+++ b/pipelex/core/concept_native.py
@@ -15,6 +15,30 @@ class NativeConceptClass(StrEnum):
LLM_PROMPT = "LLMPromptContent"
PAGE = "PageContent"
+ @classmethod
+ def class_names(cls) -> List[str]:
+ return [code.value for code in cls]
+
+ @property
+ def native_concept(self) -> "NativeConcept":
+ match self:
+ case NativeConceptClass.DYNAMIC:
+ return NativeConcept.DYNAMIC
+ case NativeConceptClass.TEXT:
+ return NativeConcept.TEXT
+ case NativeConceptClass.IMAGE:
+ return NativeConcept.IMAGE
+ case NativeConceptClass.PDF:
+ return NativeConcept.PDF
+ case NativeConceptClass.TEXT_AND_IMAGES:
+ return NativeConcept.TEXT_AND_IMAGES
+ case NativeConceptClass.NUMBER:
+ return NativeConcept.NUMBER
+ case NativeConceptClass.LLM_PROMPT:
+ return NativeConcept.LLM_PROMPT
+ case NativeConceptClass.PAGE:
+ return NativeConcept.PAGE
+
# Exceptionally, we use an Enum here (and not our usual StrEnum) to avoid confusion with
# the concept_code which must have the form "native.ConceptName"
diff --git a/pipelex/core/stuff_factory.py b/pipelex/core/stuff_factory.py
index 908848d72..f5e663611 100644
--- a/pipelex/core/stuff_factory.py
+++ b/pipelex/core/stuff_factory.py
@@ -6,7 +6,8 @@ from pydantic import BaseModel, ValidationError
from pipelex.client.protocol import StuffContentOrData
from pipelex.core.concept import Concept
from pipelex.core.concept_code_factory import ConceptCodeFactory
-from pipelex.core.concept_native import NativeConcept
+from pipelex.core.concept_factory import ConceptFactory
+from pipelex.core.concept_native import NativeConcept, NativeConceptClass
from pipelex.core.stuff import Stuff
from pipelex.core.stuff_content import (
ListContent,
@@ -216,6 +217,15 @@ class StuffFactory:
elif isinstance(stuff_content_or_data, StuffContent):
content = stuff_content_or_data
concept_name = type(content).__name__
+ if concept_name in NativeConceptClass.class_names():
+ native_concept_class = NativeConceptClass(concept_name)
+ concept = ConceptFactory.make_native_concept_from_native_concept_class(native_concept_class=native_concept_class)
+ return cls.make_stuff(
+ concept_str=concept.code,
+ content=content,
+ name=name,
+ code=code,
+ )
try:
return cls.make_stuff_using_concept_name_and_search_domains(
concept_name=concept_name,
@@ -259,16 +269,24 @@ class StuffFactory:
content_value = stuff_content_dict["content"]
except KeyError as exc:
raise StuffFactoryError(f"Stuff content data dict is badly formed: {exc}") from exc
- content = StuffContentFactory.make_stuffcontent_from_concept_code_with_fallback(
- concept_code=concept_code,
- value=content_value,
- )
- return StuffFactory.make_stuff(
- concept_str=concept_code,
- name=name,
- content=content,
- code=code,
- )
+ if isinstance(content_value, StuffContent):
+ return StuffFactory.make_stuff(
+ concept_str=concept_code,
+ name=name,
+ content=content_value,
+ code=code,
+ )
+ else:
+ content = StuffContentFactory.make_stuffcontent_from_concept_code_with_fallback(
+ concept_code=concept_code,
+ value=content_value,
+ )
+ return StuffFactory.make_stuff(
+ concept_str=concept_code,
+ name=name,
+ content=content,
+ code=code,
+ )
class StuffContentFactoryError(PipelexError):
diff --git a/pipelex/libraries/pipelines/documents.toml b/pipelex/libraries/pipelines/documents.toml
index cc52a4c66..77fdfd536 100644
--- a/pipelex/libraries/pipelines/documents.toml
+++ b/pipelex/libraries/pipelines/documents.toml
@@ -10,15 +10,15 @@ TextAndImagesContent = "A content that comprises text and images where the text
# PipeOcr requires to have a single input
# It can be named however you want
# but it must be either an image or a pdf or a concept which refines one of them
-[pipe.extract_page_contents_from_pdf]
+[pipe.ocr_page_contents_from_pdf]
PipeOcr = "Extract page contents from a PDF document"
inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
page_views = false
-[pipe.extract_page_contents_and_views_from_pdf]
-PipeOcr = "Extract page contents from a PDF document as well aspage views"
+[pipe.ocr_page_contents_and_views_from_pdf]
+PipeOcr = "Extract page contents from a PDF document as well as full page views"
inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
diff --git a/tests/cases/source_code.py b/tests/cases/source_code.py
new file mode 100644
index 000000000..463b191db
--- /dev/null
+++ b/tests/cases/source_code.py
@@ -0,0 +1,33 @@
+"""Source code manipulation functions for testing."""
+
+from typing import List
+
+from pipelex.core.stuff_content import TextContent
+from pipelex.core.working_memory import WorkingMemory
+
+
+def wrap_lines(working_memory: WorkingMemory) -> TextContent:
+ """
+ Wraps each line of source code text with HTML span tags.
+
+ Args:
+ working_memory: The working memory containing the source text
+
+ Returns:
+ TextContent with each line wrapped in span tags with class="line"
+ """
+ # Get the source text from working memory as string
+ source_text = working_memory.get_stuff_as_str("source_text")
+
+ # Split into lines and wrap each line
+ lines = source_text.split("\n")
+ wrapped_lines: List[str] = []
+
+ for line in lines:
+ wrapped_line = f'<span class="line">{line}</span>'
+ wrapped_lines.append(wrapped_line)
+
+ # Join back into a single string
+ wrapped_text = "\n".join(wrapped_lines)
+
+ return TextContent(text=wrapped_text)
diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.py
new file mode 100644
index 000000000..d610235db
--- /dev/null
+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/__init__.py
@@ -0,0 +1 @@
+# Empty file for Python package structure
diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.py
new file mode 100644
index 000000000..d36d6219f
--- /dev/null
+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_func/test_pipe_func.py
@@ -0,0 +1,102 @@
+import pytest
+
+from pipelex import log, pretty_print
+from pipelex.core.concept_native import NativeConcept
+from pipelex.core.pipe_run_params import PipeRunMode
+from pipelex.core.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.core.stuff_content import TextContent
+from pipelex.core.stuff_factory import StuffFactory
+from pipelex.core.working_memory_factory import WorkingMemoryFactory
+from pipelex.hub import get_pipe_router, get_report_delegate
+from pipelex.pipe_operators.pipe_func import PipeFunc, PipeFuncOutput
+from pipelex.pipe_works.pipe_job_factory import PipeJobFactory
+from pipelex.tools.func_registry import func_registry
+from tests.cases.source_code import wrap_lines
+
+
[email protected](loop_scope="class")
+class TestPipeFunc:
+ @classmethod
+ def setup_class(cls):
+ """Register test functions before running tests."""
+ func_registry.register_function(wrap_lines)
+
+ @classmethod
+ def teardown_class(cls):
+ """Clean up registered functions after tests."""
+ if func_registry.has_function("wrap_lines"):
+ func_registry.unregister_function_by_name("wrap_lines")
+
+ async def test_wrap_lines_pipe_func(
+ self,
+ pipe_run_mode: PipeRunMode,
+ ):
+ # Sample source code to test with
+ sample_code = """def hello_world():
+ print("Hello, World!")
+ return "Hello!"
+
+if __name__ == "__main__":
+ hello_world()"""
+
+ # Create stuff with the source code
+ source_text_stuff = StuffFactory.make_stuff(
+ name="source_text",
+ concept_str=NativeConcept.TEXT.code,
+ content=TextContent(text=sample_code),
+ )
+
+ # Create working memory with the source text
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=source_text_stuff)
+
+ # Create the PipeFunc job
+ pipe_job = PipeJobFactory.make_pipe_job(
+ pipe=PipeFunc(
+ code="wrap_lines",
+ domain="source_code",
+ output_concept_code=NativeConcept.TEXT.code,
+ function_name="wrap_lines",
+ ),
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
+ working_memory=working_memory,
+ )
+
+ # Execute the pipe
+ pipe_func_output: PipeFuncOutput = await get_pipe_router().run_pipe_job(
+ pipe_job=pipe_job,
+ )
+
+ # Log and verify results
+ log.verbose(pipe_func_output, title="pipe_func_output")
+ wrapped_text = pipe_func_output.main_stuff_as_text
+ pretty_print(wrapped_text, title="wrapped_text")
+ get_report_delegate().generate_report()
+
+ # Verify the output contains wrapped lines
+ assert pipe_func_output is not None
+ assert pipe_func_output.working_memory is not None
+ assert pipe_func_output.main_stuff is not None
+
+ # Get the actual text string for verification
+ wrapped_text_str = wrapped_text.text
+
+ # In DRY mode, we get mock content, so we only check basic structure
+ if pipe_run_mode == PipeRunMode.DRY:
+ # In dry mode, just verify we have text content and it's non-empty
+ assert isinstance(wrapped_text_str, str)
+ assert len(wrapped_text_str) > 0
+ else:
+ # In LIVE mode, check the actual wrapping functionality
+ # Check that each line is wrapped in span tags
+ lines = wrapped_text_str.split("\n")
+ for line in lines:
+ assert line.startswith('<span class="line">'), f"Line should start with span tag: {line}"
+ assert line.endswith("</span>"), f"Line should end with span tag: {line}"
+
+ # Verify we have the expected number of lines (original had 6 lines)
+ assert len(lines) == 6, f"Expected 6 lines, got {len(lines)}"
+
+ # Verify specific content is preserved within spans
+ assert '<span class="line">def hello_world():</span>' in wrapped_text_str
+ assert '<span class="line"> print("Hello, World!")</span>' in wrapped_text_str
diff --git a/tests/integration/pipelex/test_data.py b/tests/integration/pipelex/test_data.py
index 4b15024a0..eba365325 100644
--- a/tests/integration/pipelex/test_data.py
+++ b/tests/integration/pipelex/test_data.py
@@ -155,7 +155,7 @@ class PipeTestCases:
(
"Extract page contents from PDF",
SIMPLE_STUFF_PDF,
- "extract_page_contents_from_pdf",
+ "ocr_page_contents_from_pdf",
),
]
FAILURE_PIPES: ClassVar[List[Tuple[str, Type[Exception], str]]] = [
diff --git a/uv.lock b/uv.lock
index c56e16196..6462e726b 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1748,7 +1748,7 @@ wheels = [
[[package]]
name = "pipelex"
-version = "0.6.7"
+version = "0.6.9"
source = { editable = "." }
dependencies = [
{ name = "aiofiles" },