Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Fix/execute setup double code (#478)

Thomas Hebrard4557fcecfb3f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8157b9967..1d27209e0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,7 @@
- Refactored the pipe builder validation loop.
- Reorganized the unit tests, and added new ones.
- Reorganized the config files.
+- Refactored methods `execute_pipeline` and `start_pipeline`.
- Moved `dev_cli` to `cli.dev_cli`.
## [v0.15.7] - 2025-11-18
diff --git a/docs/home/6-build-reliable-ai-workflows/libraries.md b/docs/home/6-build-reliable-ai-workflows/libraries.md
new file mode 100644
index 000000000..87d980035
--- /dev/null
+++ b/docs/home/6-build-reliable-ai-workflows/libraries.md
@@ -0,0 +1,214 @@
+# Libraries
+
+A **Library** in Pipelex is a complete collection of domains, concepts, and pipes that can be loaded and used together. It represents the full set of Pipelex definitions available for execution within a specific context, typically for a single pipeline run.
+
+## Library Structure
+
+A Library is composed of three core components:
+
+- **DomainLibrary**: Manages all domain definitions
+- **ConceptLibrary**: Manages all concept definitions across domains
+- **PipeLibrary**: Manages all pipe definitions
+
+These three components together form what we call a **Pipelex Bundle** (the content you define in `.plx` files). Learn more about bundle structure and syntax in the [Pipelex Bundle Specification](./pipelex-bundle-specification.md).
+
+## Understanding Library Scope
+
+When you execute pipelines using `execute_pipeline` or `start_pipeline`, a library is created to hold all the necessary definitions. This library:
+
+- Contains the pipes and concepts available for execution
+- Provides isolation between different pipeline runs when using different library IDs
+- Can be loaded from local directories or from PLX content strings
+
+## Uniqueness Rules
+
+Libraries enforce specific uniqueness constraints to maintain consistency:
+
+| Component | Uniqueness Scope | Example |
+|-----------|-----------------|---------|
+| **Domains** | Unique per library | Each library can have one `marketing` domain |
+| **Pipes** | Unique per library* | Each library can have one `generate_tagline` pipe |
+| **Concepts** | Unique per domain | Each domain can have one `ProductDescription` concept |
+| **Domain.Concept** | Unique per library | `marketing.ProductDescription` is unique within a library |
+
+!!! info "Future Change"
+ *Pipe uniqueness will soon be scoped per domain instead of per library, allowing different domains within the same library to have pipes with the same code.
+
+## Local vs Remote Libraries
+
+### Local Libraries (Current)
+
+Currently, all libraries are **local**, meaning they are loaded from:
+
+- Directories on your filesystem (using `library_dirs` parameter)
+- PLX content strings (using `plx_content` parameter)
+- The current working directory (default behavior)
+
+```python
+# Loading from local directories
+pipe_output = await execute_pipeline(
+ pipe_code="generate_tagline",
+ library_dirs=["./pipelines", "./shared_pipes"],
+ inputs={...},
+)
+```
+
+### Remote Libraries (Coming Soon)
+
+In the future, you'll be able to import and use remote libraries, enabling:
+
+- Sharing pipe definitions across teams and projects
+- Versioned library dependencies
+- Centralized library management
+
+## Library Lifecycle
+
+### 1. Library Creation
+
+When you call `execute_pipeline` or `start_pipeline`, a library is created with a unique `library_id`:
+
+```python
+# Explicit library ID
+pipe_output = await execute_pipeline(
+ pipe_code="my_pipe",
+ library_id="my_custom_library", # Custom ID
+ inputs={...},
+)
+
+# Automatic library ID (defaults to pipeline_run_id)
+pipe_output = await execute_pipeline(
+ pipe_code="my_pipe",
+ # library_id not specified - uses pipeline_run_id
+ inputs={...},
+)
+```
+
+### 2. Library Loading
+
+The library is populated based on the parameters you provide:
+
+**Option A: Loading from directories**
+
+```python
+# Loads all .plx files from specified directories
+pipe_output = await execute_pipeline(
+ pipe_code="my_pipe",
+ library_dirs=["./pipelines"],
+ inputs={...},
+)
+```
+
+**Option B: Loading from PLX content**
+
+```python
+# Loads only the provided PLX content
+plx_content = """
+domain = "marketing"
+
+[concept]
+ProductDescription = "A product description"
+
+[pipe.my_pipe]
+type = "PipeLLM"
+inputs = { desc = "ProductDescription" }
+output = "Tagline"
+prompt = "Generate a tagline for: @desc"
+"""
+
+pipe_output = await execute_pipeline(
+ plx_content=plx_content,
+ pipe_code="my_pipe",
+ inputs={...},
+)
+```
+
+### 3. Library Validation
+
+After loading, the library validates:
+
+- All pipe input/output concepts exist in the concept library
+- All pipe dependencies (for pipe controllers) exist in the pipe library
+- Domain and concept definitions are consistent
+
+## Working with Multiple Libraries
+
+You can manage multiple libraries simultaneously by using different `library_id` values:
+
+```python
+# Library for marketing pipelines
+marketing_output = await execute_pipeline(
+ pipe_code="generate_tagline",
+ library_id="marketing_lib",
+ library_dirs=["./marketing_pipes"],
+ inputs={...},
+)
+
+# Library for analytics pipelines
+analytics_output = await execute_pipeline(
+ pipe_code="analyze_data",
+ library_id="analytics_lib",
+ library_dirs=["./analytics_pipes"],
+ inputs={...},
+)
+```
+
+## Best Practices
+
+### 1. Use Explicit Library IDs for Long-Running Applications
+
+```python
+# Good: Explicit ID for maintaining state
+pipe_output = await execute_pipeline(
+ pipe_code="my_pipe",
+ library_id="app_library",
+ inputs={...},
+)
+```
+
+### 2. Use PLX Content for Dynamic Pipelines
+
+When generating or modifying pipelines dynamically, use `plx_content`:
+
+```python
+# Generate PLX content dynamically
+plx_content = generate_custom_pipeline(user_requirements)
+
+pipe_output = await execute_pipeline(
+ plx_content=plx_content,
+ inputs={...},
+)
+```
+
+A temporary library will be created holding the Pipelex bundle, and the library_id will be pipeline_run_id.
+
+### 3. Reuse Library IDs for Related Executions
+
+If multiple pipeline runs should share the same library context:
+
+```python
+library_id = "shared_context"
+
+# First execution
+output1 = await execute_pipeline(
+ pipe_code="pipe1",
+ library_id=library_id,
+ library_dirs=["./pipes"],
+ inputs={...},
+)
+
+# Second execution with same library
+output2 = await execute_pipeline(
+ pipe_code="pipe2",
+ library_id=library_id,
+ library_dirs=["./pipes"],
+ inputs={...},
+)
+```
+
+## Related Documentation
+
+- [Executing Pipelines](pipes/executing-pipelines.md) - Learn how to execute pipelines with different library configurations
+- [Pipelex Bundle Specification](./pipelex-bundle-specification.md) - Understand the structure of PLX files
+- [Domains](./domain.md) - Learn about organizing pipes into domains
+- [Concepts](./concepts/define_your_concepts.md) - Understand how concepts work within libraries
+
diff --git a/docs/home/6-build-reliable-ai-workflows/pipelex-bundle-specification.md b/docs/home/6-build-reliable-ai-workflows/pipelex-bundle-specification.md
index 868ce65da..f57263f18 100644
--- a/docs/home/6-build-reliable-ai-workflows/pipelex-bundle-specification.md
+++ b/docs/home/6-build-reliable-ai-workflows/pipelex-bundle-specification.md
@@ -201,6 +201,8 @@ Reconcile this payment with the invoice...
One of Pipelex's strengths is how bundles can reference and build upon each other, creating modular, reusable pipeline architectures.
+When you execute pipelines, multiple bundles are loaded together into a **Library** - a runtime collection that provides access to all domains, concepts, and pipes from the loaded bundles. This library system enables bundle composition and ensures proper isolation between different pipeline runs. Learn more about how libraries work in [Libraries](libraries.md).
+
### Calling Pipes from Other Bundles
A pipe in one bundle can call a pipe from another bundle using the full pipe identifier:
diff --git a/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md b/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
index afe0d2df9..21f1850a1 100644
--- a/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
+++ b/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
@@ -26,9 +26,39 @@ This function executes the specified pipe and waits for it to complete, returnin
There are 2 ways of using the `execute_pipeline` method:
+### Understanding Libraries
+
+When executing pipelines programmatically, Pipelex loads the pipe **libraries** to execute the pipes. A library is a collection of pipes loaded from one or more directories.
+
+#### Library Parameters
+
+When using `execute_pipeline` or `start_pipeline`, you can control library behavior with these parameters:
+
+- **`library_id`**: A unique identifier for the library instance. If not specified, it defaults to the `pipeline_run_id` (a unique ID generated for each pipeline execution).
+
+- **`library_dirs`**: A list of directory paths to load pipe definitions from. **These directories must contain both your `.plx` files AND any Python files defining `StructuredContent` classes** (e.g., `*_struct.py` files). If not specified, Pipelex will load from the current working directory (the directory from which your Python script is executed).
+
+- **`plx_content`**: When provided, Pipelex will load only this PLX content into the library, bypassing directory scanning. This is useful for dynamic pipeline execution without file-based definitions.
+
+#### Library Loading Behavior
+
+The loading behavior depends on which parameters you provide:
+
+1. **Using `pipe_code` only** (Option 2.1 below): Pipelex loads all pipe definitions from `library_dirs` (or current directory if not specified), then looks up the pipe by its code.
+
+2. **Using `plx_content`** (Option 2.2 below): Pipelex loads only the provided PLX content into the library, creating an isolated execution environment.
+
+!!! info "Python Structure Classes"
+ If your concepts use Python `StructuredContent` classes instead of inline structures, those Python files must be in the directories specified by `library_dirs`. Pipelex auto-discovers and registers these classes during library loading. Learn more about [Python StructuredContent Classes](../concepts/python-classes.md).
+
+!!! info "Learn More About Libraries"
+ For a comprehensive understanding of libraries, including their structure, uniqueness rules, lifecycle, and best practices, see [Libraries](../libraries.md).
+
### Option 2.1: Using the pipe code
-This requires your pipelex file to be in the current directory, that means it will be loaded in the library. You can use the `pipelex show pipes` command to list all the pipes available in your project.
+This approach loads pipe definitions from directories and executes a specific pipe by its code.
+
+**Basic usage** (loads from current working directory):
```python
from pipelex.pipelex import Pipelex
@@ -49,6 +79,42 @@ pipe_output = await execute_pipeline(
)
```
+**Loading from specific directories**:
+
+```python
+# Load pipes from specific directories
+pipe_output = await execute_pipeline(
+ pipe_code="description_to_tagline",
+ library_dirs=["./pipelines", "./shared_pipes"],
+ inputs={
+ "description": {
+ "concept": "ProductDescription",
+ "content": "...",
+ },
+ },
+)
+```
+
+**Using a custom library ID**:
+
+```python
+# Use a custom library ID for managing multiple library instances
+pipe_output = await execute_pipeline(
+ pipe_code="description_to_tagline",
+ library_id="my_marketing_library",
+ library_dirs=["./marketing_pipes"],
+ inputs={
+ "description": {
+ "concept": "ProductDescription",
+ "content": "...",
+ },
+ },
+)
+```
+
+!!! tip "Listing available pipes"
+ Use the `pipelex show pipes` command to list all the pipes available in your project.
+
### Option 2.2: Using the pipelex bundle content
You can directly pass to the `execute_pipeline` method the content of your pipelex file and the necessary inputs.
diff --git a/mkdocs.yml b/mkdocs.yml
index 75b90f250..3cbcb9358 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -128,6 +128,7 @@ nav:
- Refining Concepts: home/6-build-reliable-ai-workflows/concepts/refining-concepts.md
- Design and Run Pipelines:
- Overview: home/6-build-reliable-ai-workflows/pipes/index.md
+ - Libraries: home/6-build-reliable-ai-workflows/libraries.md
- Executing Pipelines: home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
- Providing Inputs to Pipelines: home/6-build-reliable-ai-workflows/pipes/provide-inputs.md
- Working Memory: home/6-build-reliable-ai-workflows/pipes/working-memory.md
diff --git a/pipelex/builder/builder_loop.py b/pipelex/builder/builder_loop.py
index 9b9644f04..cefaf90a9 100644
--- a/pipelex/builder/builder_loop.py
+++ b/pipelex/builder/builder_loop.py
@@ -29,9 +29,10 @@ class BuilderLoop:
is_save_second_iteration_enabled: bool = True,
is_save_working_memory_enabled: bool = True,
) -> PipelexBundleSpec:
+ # TODO: Doesn't make sense to be able to put a builder_pipe code but hardcoding the Path to the builder pipe.
pipe_output = await execute_pipeline(
pipe_code=builder_pipe,
- library_path=str(Path(builder.__file__).parent),
+ library_dirs=[str(Path(builder.__file__).parent)],
inputs=inputs,
)
diff --git a/pipelex/pipeline/execute.py b/pipelex/pipeline/execute.py
index 9637f38e4..0769dc675 100644
--- a/pipelex/pipeline/execute.py
+++ b/pipelex/pipeline/execute.py
@@ -1,42 +1,23 @@
-from pathlib import Path
-from typing import TYPE_CHECKING
-
from pipelex.client.protocol import PipelineInputs
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import (
get_library_manager,
get_pipe_router,
- get_pipeline_manager,
- get_report_delegate,
- get_required_pipe,
get_telemetry_manager,
- set_current_library,
teardown_current_library,
)
from pipelex.pipe_run.exceptions import PipeRouterError
-from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-from pipelex.pipe_run.pipe_run_params import (
- FORCE_DRY_RUN_MODE_ENV_KEY,
- VariableMultiplicity,
-)
-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError
-from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.pipeline.validate_bundle import validate_bundle
-from pipelex.system.environment import get_optional_env
+from pipelex.pipe_run.pipe_run_params import VariableMultiplicity
+from pipelex.pipeline.exceptions import PipelineExecutionError
+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
from pipelex.system.telemetry.events import EventName, EventProperty, Outcome
-if TYPE_CHECKING:
- from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
- from pipelex.core.pipes.pipe_abstract import PipeAbstract
-
async def execute_pipeline(
library_id: str | None = None,
- library_path: str | None = None,
+ library_dirs: list[str] | None = None,
pipe_code: str | None = None,
plx_content: str | None = None,
inputs: PipelineInputs | WorkingMemory | None = None,
@@ -48,128 +29,65 @@ async def execute_pipeline(
) -> PipeOutput:
"""Execute a pipeline and wait for its completion.
- This function executes a pipe and returns its output along with the pipeline run ID.
- Unlike *start_pipeline*, this function waits for the pipe execution to complete
- before returning, and it returns the output in addition to the pipeline run ID.
+ This function executes a pipe and returns its output. Unlike ``start_pipeline``,
+ this function waits for the pipe execution to complete before returning.
Parameters
----------
library_id:
- The library ID to use for the pipeline execution. If not provided, the library_id will be set to the pipeline run ID.
- library_path:
- Path to the library directory to load.
+ Unique identifier for the library instance. If not provided, defaults to the
+ auto-generated ``pipeline_run_id``. Use a custom ID when you need to manage
+ multiple library instances or maintain library state across executions.
+ library_dirs:
+ List of directory paths to load pipe definitions from. If not provided, loads
+ from the current working directory (the directory from which the Python script
+ is executed). Ignored when ``plx_content`` is provided.
pipe_code:
- The code identifying the pipeline to execute.
+ Code identifying the pipe to execute. Required when ``plx_content`` is not
+ provided. When both ``plx_content`` and ``pipe_code`` are provided, the
+ specified pipe from the PLX content will be executed (overriding any
+ ``main_pipe`` defined in the plx_content).
plx_content:
- Content of the pipeline bundle to execute.
+ Complete PLX file content as a string. When provided, only this content is
+ loaded into the library, creating an isolated execution environment. The pipe
+ to execute is determined by ``pipe_code`` (if provided) or the ``main_pipe``
+ property in the PLX content.
inputs:
- Inputs passed to the pipeline.
+ Inputs passed to the pipeline. Can be either a ``PipelineInputs`` dictionary
+ or a ``WorkingMemory`` instance.
output_name:
Name of the output slot to write to.
output_multiplicity:
- Output multiplicity.
+ Output multiplicity specification.
dynamic_output_concept_code:
Override the dynamic output concept code.
pipe_run_mode:
- Pipe run mode: if specified, it must be ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``.
- If not specified, the pipe run mode is inferred from the environment variable
- ``PIPELEX_FORCE_DRY_RUN_MODE``. If the environment variable is not set,
- the pipe run mode is ``PipeRunMode.LIVE``.
+ Pipe run mode: ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``. If not specified,
+ inferred from the environment variable ``PIPELEX_FORCE_DRY_RUN_MODE``. Defaults
+ to ``PipeRunMode.LIVE`` if the environment variable is not set.
search_domains:
- List of domains to search for pipes.
+ List of domains to search for pipes. The executed pipe's domain is automatically
+ added if not already present.
Returns:
-------
- Tuple[PipeOutput, str]
- A tuple containing the pipe output and the pipeline run ID.
+ PipeOutput
+ The pipe output from the execution.
"""
- if not plx_content and not pipe_code:
- msg = "Either pipe_code or plx_content must be provided to the API execute_pipeline."
- raise ValueError(msg)
-
- pipeline = get_pipeline_manager().add_new_pipeline()
- pipeline_run_id = pipeline.pipeline_run_id
-
- if not library_id:
- library_id = pipeline_run_id
-
- library_manager = get_library_manager()
- set_current_library(library_id=library_id)
- library_manager.open_library(library_id=library_id)
-
- pipe: PipeAbstract | None = None
- blueprint: PipelexBundleBlueprint | None = None
-
- if plx_content:
- validate_bundle_result = await validate_bundle(plx_content=plx_content)
- library_manager.load_from_blueprints(library_id=library_id, blueprints=validate_bundle_result.blueprints)
- # For now, we only support one blueprint when given a plx_content. So blueprints is of length 1.
- blueprint = validate_bundle_result.blueprints[0]
- if pipe_code:
- pipe = get_required_pipe(pipe_code=pipe_code)
- elif blueprint.main_pipe:
- pipe = get_required_pipe(pipe_code=blueprint.main_pipe)
- else:
- msg = "No pipe code or main pipe in the PLX content provided to the API execute_pipeline."
- raise PipeExecutionError(message=msg)
- elif pipe_code:
- if library_path:
- library_manager.load_libraries(library_id=library_id, library_dirs=[Path(library_path)])
- else:
- library_manager.load_libraries(library_id=library_id)
- pipe = get_required_pipe(pipe_code=pipe_code)
- else:
- msg = "Either provide pipe_code or plx_content to the API execute_pipeline. 'pipe_code' must be provided when 'plx_content' is None"
- raise PipeExecutionError(message=msg)
-
- search_domains = search_domains or []
- if pipe.domain not in search_domains:
- search_domains.insert(0, pipe.domain)
-
- working_memory: WorkingMemory | None = None
-
- if inputs:
- if isinstance(inputs, WorkingMemory):
- working_memory = inputs
- else:
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
- pipeline_inputs=inputs,
- search_domains=search_domains,
- )
-
- if pipe_run_mode is None:
- if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):
- pipe_run_mode = PipeRunMode(run_mode_from_env)
- else:
- pipe_run_mode = PipeRunMode.LIVE
-
- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
-
- job_metadata = JobMetadata(
- pipeline_run_id=pipeline.pipeline_run_id,
- )
-
- pipe_run_params = PipeRunParamsFactory.make_run_params(
+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
+ library_id=library_id,
+ library_dirs=library_dirs,
+ pipe_code=pipe_code,
+ plx_content=plx_content,
+ inputs=inputs,
+ output_name=output_name,
output_multiplicity=output_multiplicity,
dynamic_output_concept_code=dynamic_output_concept_code,
pipe_run_mode=pipe_run_mode,
+ search_domains=search_domains,
)
- pipe_job = PipeJobFactory.make_pipe_job(
- pipe=pipe,
- pipe_run_params=pipe_run_params,
- job_metadata=job_metadata,
- working_memory=working_memory,
- output_name=output_name,
- )
-
- properties = {
- EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
- EventProperty.PIPE_TYPE: pipe.pipe_type,
- }
- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_EXECUTE, properties=properties)
-
try:
pipe_output = await get_pipe_router().run(pipe_job)
except PipeRouterError as exc:
@@ -185,8 +103,8 @@ async def execute_pipeline(
library.teardown()
teardown_current_library()
properties = {
- EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
- EventProperty.PIPE_TYPE: pipe.pipe_type,
+ EventProperty.PIPELINE_RUN_ID: pipeline_run_id,
+ EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,
EventProperty.PIPELINE_EXECUTE_OUTCOME: Outcome.SUCCESS,
}
get_telemetry_manager().track_event(event_name=EventName.PIPELINE_COMPLETE, properties=properties)
diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.py
new file mode 100644
index 000000000..94b19cee5
--- /dev/null
+++ b/pipelex/pipeline/pipeline_run_setup.py
@@ -0,0 +1,182 @@
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from pipelex.client.protocol import PipelineInputs
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.hub import (
+ get_library_manager,
+ get_pipeline_manager,
+ get_report_delegate,
+ get_required_pipe,
+ get_telemetry_manager,
+ set_current_library,
+)
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params import (
+ FORCE_DRY_RUN_MODE_ENV_KEY,
+ VariableMultiplicity,
+)
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.exceptions import PipeExecutionError
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.system.environment import get_optional_env
+from pipelex.system.telemetry.events import EventName, EventProperty
+
+if TYPE_CHECKING:
+ from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+ from pipelex.core.pipes.pipe_abstract import PipeAbstract
+
+
+async def pipeline_run_setup(
+ library_id: str | None = None,
+ library_dirs: list[str] | None = None,
+ pipe_code: str | None = None,
+ plx_content: str | None = None,
+ inputs: PipelineInputs | WorkingMemory | None = None,
+ output_name: str | None = None,
+ output_multiplicity: VariableMultiplicity | None = None,
+ dynamic_output_concept_code: str | None = None,
+ pipe_run_mode: PipeRunMode | None = None,
+ search_domains: list[str] | None = None,
+) -> tuple[PipeJob, str, str]:
+ """Set up a pipeline for execution.
+
+ This function handles all the common setup logic for both ``execute_pipeline``
+ and ``start_pipeline``, including library setup, pipe loading, working memory
+ initialization, and pipe job creation.
+
+ Parameters
+ ----------
+ library_id:
+ Unique identifier for the library instance. If not provided, defaults to the
+ auto-generated ``pipeline_run_id``. Use a custom ID when you need to manage
+ multiple library instances or maintain library state across executions.
+ library_dirs:
+ List of directory paths to load pipe definitions from. If not provided, loads
+ from the current working directory (the directory from which the Python script
+ is executed). Ignored when ``plx_content`` is provided.
+ pipe_code:
+ Code identifying the pipe to execute. Required when ``plx_content`` is not
+ provided. When both ``plx_content`` and ``pipe_code`` are provided, the
+ specified pipe from the PLX content will be executed (overriding any
+ ``main_pipe`` defined in the content).
+ plx_content:
+ Complete PLX file content as a string. When provided, only this content is
+ loaded into the library, creating an isolated execution environment. The pipe
+ to execute is determined by ``pipe_code`` (if provided) or the ``main_pipe``
+ property in the PLX content.
+ inputs:
+ Inputs passed to the pipeline. Can be either a ``PipelineInputs`` dictionary
+ or a ``WorkingMemory`` instance.
+ output_name:
+ Name of the output slot to write to.
+ output_multiplicity:
+ Output multiplicity specification.
+ dynamic_output_concept_code:
+ Override the dynamic output concept code.
+ pipe_run_mode:
+ Pipe run mode: ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``. If not specified,
+ inferred from the environment variable ``PIPELEX_FORCE_DRY_RUN_MODE``. Defaults
+ to ``PipeRunMode.LIVE`` if the environment variable is not set.
+ search_domains:
+ List of domains to search for pipes. The executed pipe's domain is automatically
+ added if not already present.
+
+ Returns:
+ -------
+ tuple[PipeJob, str, str]
+ A tuple containing the pipe job ready for execution, the pipeline run ID,
+ and the library ID.
+
+ """
+ if not plx_content and not pipe_code:
+ msg = "Either pipe_code or plx_content must be provided to the pipeline API."
+ raise ValueError(msg)
+
+ pipeline = get_pipeline_manager().add_new_pipeline()
+ pipeline_run_id = pipeline.pipeline_run_id
+
+ if not library_id:
+ library_id = pipeline_run_id
+
+ library_manager = get_library_manager()
+ set_current_library(library_id=library_id)
+ library_manager.open_library(library_id=library_id)
+
+ pipe: PipeAbstract | None = None
+ blueprint: PipelexBundleBlueprint | None = None
+
+ if plx_content:
+ validate_bundle_result = await validate_bundle(plx_content=plx_content)
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=validate_bundle_result.blueprints)
+ # For now, we only support one blueprint when given a plx_content. So blueprints is of length 1.
+ blueprint = validate_bundle_result.blueprints[0]
+ if pipe_code:
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ elif blueprint.main_pipe:
+ pipe = get_required_pipe(pipe_code=blueprint.main_pipe)
+ else:
+ msg = "No pipe code or main pipe in the PLX content provided to the pipeline API."
+ raise PipeExecutionError(message=msg)
+ elif pipe_code:
+ if library_dirs:
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path(library_dir) for library_dir in library_dirs])
+ else:
+ library_manager.load_libraries(library_id=library_id, library_dirs=[Path.cwd()])
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ else:
+ msg = "Either provide pipe_code or plx_content to the pipeline API. 'pipe_code' must be provided when 'plx_content' is None"
+ raise PipeExecutionError(message=msg)
+
+ search_domains = search_domains or []
+ if pipe.domain not in search_domains:
+ search_domains.insert(0, pipe.domain)
+
+ working_memory: WorkingMemory | None = None
+
+ if inputs:
+ if isinstance(inputs, WorkingMemory):
+ working_memory = inputs
+ else:
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=inputs,
+ search_domains=search_domains,
+ )
+
+ if pipe_run_mode is None:
+ if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):
+ pipe_run_mode = PipeRunMode(run_mode_from_env)
+ else:
+ pipe_run_mode = PipeRunMode.LIVE
+
+ get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
+
+ job_metadata = JobMetadata(
+ pipeline_run_id=pipeline.pipeline_run_id,
+ )
+
+ pipe_run_params = PipeRunParamsFactory.make_run_params(
+ output_multiplicity=output_multiplicity,
+ dynamic_output_concept_code=dynamic_output_concept_code,
+ pipe_run_mode=pipe_run_mode,
+ )
+
+ pipe_job = PipeJobFactory.make_pipe_job(
+ pipe=pipe,
+ pipe_run_params=pipe_run_params,
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ output_name=output_name,
+ )
+
+ properties = {
+ EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
+ EventProperty.PIPE_TYPE: pipe.pipe_type,
+ }
+ get_telemetry_manager().track_event(event_name=EventName.PIPELINE_EXECUTE, properties=properties)
+
+ return pipe_job, pipeline_run_id, library_id
diff --git a/pipelex/pipeline/start.py b/pipelex/pipeline/start.py
index 655b2616a..044d35d92 100644
--- a/pipelex/pipeline/start.py
+++ b/pipelex/pipeline/start.py
@@ -1,41 +1,17 @@
import asyncio
-from pathlib import Path
-from typing import TYPE_CHECKING
from pipelex.client.protocol import PipelineInputs
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_output import PipeOutput
-from pipelex.hub import (
- get_library_manager,
- get_pipe_router,
- get_pipeline_manager,
- get_report_delegate,
- get_required_pipe,
- get_telemetry_manager,
- set_current_library,
-)
-from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
+from pipelex.hub import get_pipe_router
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
-from pipelex.pipe_run.pipe_run_params import (
- FORCE_DRY_RUN_MODE_ENV_KEY,
- VariableMultiplicity,
-)
-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
-from pipelex.pipeline.exceptions import PipeExecutionError
-from pipelex.pipeline.job_metadata import JobMetadata
-from pipelex.pipeline.validate_bundle import validate_bundle
-from pipelex.system.environment import get_optional_env
-from pipelex.system.telemetry.events import EventName, EventProperty
-
-if TYPE_CHECKING:
- from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
- from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.pipe_run.pipe_run_params import VariableMultiplicity
+from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
async def start_pipeline(
library_id: str | None = None,
- library_path: str | None = None,
+ library_dirs: list[str] | None = None,
pipe_code: str | None = None,
plx_content: str | None = None,
inputs: PipelineInputs | WorkingMemory | None = None,
@@ -47,131 +23,68 @@ async def start_pipeline(
) -> tuple[str, asyncio.Task[PipeOutput]]:
"""Start a pipeline in the background.
- This function mirrors *execute_pipeline* but returns immediately with the
+ This function mirrors ``execute_pipeline`` but returns immediately with the
``pipeline_run_id`` and a task instead of waiting for the pipe run to complete.
- The actual execution is scheduled on the current event-loop using
- :pyfunc:`asyncio.create_task`.
+ The actual execution is scheduled on the current event loop using
+ ``asyncio.create_task``.
Parameters
----------
library_id:
- The library ID to use for the pipeline execution. If not provided, the library_id will be set to the pipeline run ID.
- library_path:
- Path to the library directory to load.
+ Unique identifier for the library instance. If not provided, defaults to the
+ auto-generated ``pipeline_run_id``. Use a custom ID when you need to manage
+ multiple library instances or maintain library state across executions.
+ library_dirs:
+ List of directory paths to load pipe definitions from. If not provided, loads
+ from the current working directory (the directory from which the Python script
+ is executed). Ignored when ``plx_content`` is provided.
pipe_code:
- The code identifying the pipeline to execute.
+ Code identifying the pipe to execute. Required when ``plx_content`` is not
+ provided. When both ``plx_content`` and ``pipe_code`` are provided, the
+ specified pipe from the PLX content will be executed (overriding any
+ ``main_pipe`` defined in the content).
plx_content:
- Content of the pipeline bundle to execute.
+ Complete PLX file content as a string. When provided, only this content is
+ loaded into the library, creating an isolated execution environment. The pipe
+ to execute is determined by ``pipe_code`` (if provided) or the ``main_pipe``
+ property in the PLX content.
inputs:
- Inputs passed to the pipeline.
+ Inputs passed to the pipeline. Can be either a ``PipelineInputs`` dictionary
+ or a ``WorkingMemory`` instance.
output_name:
Name of the output slot to write to.
output_multiplicity:
- Output multiplicity.
+ Output multiplicity specification.
dynamic_output_concept_code:
Override the dynamic output concept code.
pipe_run_mode:
- Pipe run mode: if specified, it must be ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``.
- If not specified, the pipe run mode is inferred from the environment variable
- ``PIPELEX_FORCE_DRY_RUN_MODE``. If the environment variable is not set,
- the pipe run mode is ``PipeRunMode.LIVE``.
+ Pipe run mode: ``PipeRunMode.LIVE`` or ``PipeRunMode.DRY``. If not specified,
+ inferred from the environment variable ``PIPELEX_FORCE_DRY_RUN_MODE``. Defaults
+ to ``PipeRunMode.LIVE`` if the environment variable is not set.
search_domains:
- List of domains to search for pipes.
+ List of domains to search for pipes. The executed pipe's domain is automatically
+ added if not already present.
Returns:
-------
- Tuple[str, asyncio.Task[PipeOutput]]
+ tuple[str, asyncio.Task[PipeOutput]]
The ``pipeline_run_id`` of the newly started pipeline and a task that
can be awaited to get the pipe output.
"""
- if not plx_content and not pipe_code:
- msg = "Either pipe_code or plx_content must be provided to the API start_pipeline."
- raise ValueError(msg)
-
- pipeline = get_pipeline_manager().add_new_pipeline()
- pipeline_run_id = pipeline.pipeline_run_id
-
- if not library_id:
- library_id = pipeline_run_id
-
- library_manager = get_library_manager()
- set_current_library(library_id=library_id)
- library_manager.open_library(library_id=library_id)
-
- pipe: PipeAbstract | None = None
- blueprint: PipelexBundleBlueprint | None = None
-
- if plx_content:
- validate_bundle_result = await validate_bundle(plx_content=plx_content)
- library_manager.load_from_blueprints(library_id=library_id, blueprints=validate_bundle_result.blueprints)
- # For now, we only support one blueprint when given a plx_content. So blueprints is of length 1.
- blueprint = validate_bundle_result.blueprints[0]
- if pipe_code:
- pipe = get_required_pipe(pipe_code=pipe_code)
- elif blueprint.main_pipe:
- pipe = get_required_pipe(pipe_code=blueprint.main_pipe)
- else:
- msg = "No pipe code or main pipe in the PLX content provided to the API start_pipeline."
- raise PipeExecutionError(message=msg)
- elif pipe_code:
- if library_path:
- library_manager.load_libraries(library_id=library_id, library_dirs=[Path(library_path)])
- else:
- library_manager.load_libraries(library_id=library_id)
- pipe = get_required_pipe(pipe_code=pipe_code)
- else:
- msg = "Either provide pipe_code or plx_content to the API start_pipeline. 'pipe_code' must be provided when 'plx_content' is None"
- raise PipeExecutionError(message=msg)
-
- search_domains = search_domains or []
- if pipe.domain not in search_domains:
- search_domains.insert(0, pipe.domain)
-
- working_memory: WorkingMemory | None = None
-
- if inputs:
- if isinstance(inputs, WorkingMemory):
- working_memory = inputs
- else:
- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
- pipeline_inputs=inputs,
- search_domains=search_domains,
- )
-
- if pipe_run_mode is None:
- if run_mode_from_env := get_optional_env(key=FORCE_DRY_RUN_MODE_ENV_KEY):
- pipe_run_mode = PipeRunMode(run_mode_from_env)
- else:
- pipe_run_mode = PipeRunMode.LIVE
-
- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
-
- job_metadata = JobMetadata(
- pipeline_run_id=pipeline.pipeline_run_id,
- )
-
- pipe_run_params = PipeRunParamsFactory.make_run_params(
+ pipe_job, pipeline_run_id, library_id = await pipeline_run_setup(
+ library_id=library_id,
+ library_dirs=library_dirs,
+ pipe_code=pipe_code,
+ plx_content=plx_content,
+ inputs=inputs,
+ output_name=output_name,
output_multiplicity=output_multiplicity,
dynamic_output_concept_code=dynamic_output_concept_code,
pipe_run_mode=pipe_run_mode,
+ search_domains=search_domains,
)
- pipe_job = PipeJobFactory.make_pipe_job(
- pipe=pipe,
- pipe_run_params=pipe_run_params,
- job_metadata=job_metadata,
- working_memory=working_memory,
- output_name=output_name,
- )
-
- properties = {
- EventProperty.PIPELINE_RUN_ID: job_metadata.pipeline_run_id,
- EventProperty.PIPE_TYPE: pipe.pipe_type,
- }
- get_telemetry_manager().track_event(event_name=EventName.PIPELINE_EXECUTE, properties=properties)
-
- # Launch execution without awaiting the result.
task: asyncio.Task[PipeOutput] = asyncio.create_task(get_pipe_router().run(pipe_job))
- return pipeline.pipeline_run_id, task
+ return pipeline_run_id, task
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
index 14c36d7ab..220e31750 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_multiplicity.py
@@ -33,7 +33,7 @@ class TestPipeSequenceMultiplicity:
# Execute the pipeline
pipe_output = await execute_pipeline(
- library_path="tests/integration/pipelex/pipes/controller/pipe_sequence/",
+ library_dirs=["tests/integration/pipelex/pipes/controller/pipe_sequence/"],
pipe_code="creative_ideation_sequence",
inputs=working_memory,
pipe_run_mode=pipe_run_mode,
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
index 390215b0f..ebe356118 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/test_pipe_sequence_with_input_memory.py
@@ -95,7 +95,7 @@ class TestPipeSequenceWithInputMemory:
"""Test the optimize_tweet_sequence pipeline using inputs parameter."""
# Execute the pipeline using inputs
pipe_output = await execute_pipeline(
- library_path="tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/",
+ library_dirs=["tests/e2e/pipelex/pipes/pipe_controller/pipe_sequence/"],
pipe_code="optimize_tweet_sequence",
inputs={
"draft_tweet": {
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py b/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
index 79c174a0d..86fc2fd29 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
+++ b/tests/e2e/pipelex/pipes/pipe_operators/test_pipe_llm_vision.py
@@ -1,7 +1,5 @@
"""E2E test for PipeLLM with vision capabilities."""
-from pathlib import Path
-
import pytest
from pipelex import pretty_print, pretty_print_md
@@ -22,7 +20,7 @@ class TestPipeLLMVision:
# Execute the pipeline with an image
pipe_output = await execute_pipeline(
pipe_code="describe_image_e2e",
- library_path=str(Path("tests/e2e/pipelex/pipes/pipe_operators")),
+ library_dirs=["tests/e2e/pipelex/pipes/pipe_operators"],
inputs={
"image": ImageContent(url=LLMVisionTestCases.URL_CLOUDFRONT_ALAN_TURING),
},
@@ -57,7 +55,7 @@ class TestPipeLLMVision:
# Execute the pipeline with an image
pipe_output = await execute_pipeline(
pipe_code=pipe_code,
- library_path=str(Path("tests/e2e/pipelex/pipes/pipe_operators")),
+ library_dirs=["tests/e2e/pipelex/pipes/pipe_operators"],
inputs={
"image_a": ImageContent(url=LLMVisionTestCases.URL_CLOUDFRONT_ALAN_TURING),
"image_b": ImageContent(url=PipeTestCases.URL_IMG_FASHION_PHOTO_1),