Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Docs/onboarding tidy cli (#369)

Louis Choquel144f89e62dc3
diff --git a/.cursor/rules/python_standards.mdc b/.cursor/rules/python_standards.mdc
index 13d3c5a92..6e5615be1 100644
--- a/.cursor/rules/python_standards.mdc
+++ b/.cursor/rules/python_standards.mdc
@@ -13,6 +13,7 @@ This document outlines the core coding standards, best practices, and quality co
- Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
- When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
- Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
## Enums and tests
diff --git a/.cursor/rules/write_pipelex.mdc b/.cursor/rules/write_pipelex.mdc
index 6e3339c1e..a4f75c153 100644
--- a/.cursor/rules/write_pipelex.mdc
+++ b/.cursor/rules/write_pipelex.mdc
@@ -89,6 +89,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+# Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+# Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+# Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
## Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
index 4d612f7da..2c65362a4 100644
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -83,6 +83,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+#### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+## Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+## Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+## Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
### Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/.pipelex/inference/backends.toml b/.pipelex/inference/backends.toml
index 35e9de016..97206e935 100644
--- a/.pipelex/inference/backends.toml
+++ b/.pipelex/inference/backends.toml
@@ -10,11 +10,11 @@ api_key = "${AZURE_API_KEY}"
api_version = "${AZURE_API_VERSION}"
[bedrock]
-enabled = true
+enabled = false
aws_region = "${AWS_REGION}"
[google]
-enabled = true
+enabled = false
api_key = "${GOOGLE_API_KEY}"
[vertexai]
diff --git a/.windsurfrules.md b/.windsurfrules.md
index 4d612f7da..2c65362a4 100644
--- a/.windsurfrules.md
+++ b/.windsurfrules.md
@@ -83,6 +83,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+#### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+## Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+## Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+## Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
### Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/AGENTS.md b/AGENTS.md
index 4d612f7da..2c65362a4 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -83,6 +83,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+#### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+## Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+## Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+## Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
### Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/BLACKBOX_RULES.md b/BLACKBOX_RULES.md
index 4d612f7da..2c65362a4 100644
--- a/BLACKBOX_RULES.md
+++ b/BLACKBOX_RULES.md
@@ -83,6 +83,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+#### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+## Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+## Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+## Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
### Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 43c1a705d..adb803736 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,24 @@
# Changelog
+## Unreleased
+
+### Added
+ - **New `pipelex run` Command**: A top-level command to execute pipelines directly from the CLI. It can run a pipe existing in the package or a pipe from any `.plx` bundle file, with options to provide inputs from a JSON file and save the output.
+ - **New `pipelex build runner` Command**: A new command that generates a Python script to run a specified pipe. The script includes all necessary imports and placeholder inputs, significantly speeding up the integration of pipes into Python applications.
+ - **New `pipelex show backends` Command**: A new command to display all configured inference backends, their status (enabled/disabled), and the active model routing profile and its rules.
+ - **Revamped Documentation**: The documentation has been overhauled with a new "Getting Started" guide on the main page, making it easier for new users to install Pipelex, configure API keys, and run their first pipeline. New sections on executing pipelines and input/output multiplicity have also been added.
+
+### Changed
+ - **Bundle `main_pipe` Attribute**: Pipelex bundles (`.plx` files) now support a `main_pipe` attribute to designate the primary entry point of the bundle. This is used by the new `pipelex run` and `build build runner` commands to simplify execution.
+ - **CLI Reorganization**: The main command-line interface has been restructured for better usability, with improved help texts and a more logical command order.
+ - **`PipeExtract` Operator**: The `PipeExtract` operator's output is now consistently validated to be the `Page` concept, simplifying its usage for document processing.
+ - **Python Coding Standards**: Updated internal coding standards to recommend declaring variables with a type but no default value to better leverage linters for bug detection.
+
+### Removed
+ - Removed seldom used `prompt_template_to_structure` and `system_prompt_to_structure` configurations at the pipe and domain level.
+ - Removed **Project Name** discovery from Configuration.
+ - Removed temporary design document for the new inference backend system, as the feature is now fully implemented and documented.
+
## [v0.12.0] - 2025-10-15
### Highlights - Moving fast and breaking things
diff --git a/CLAUDE.md b/CLAUDE.md
index 4d612f7da..2c65362a4 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -83,6 +83,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+#### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+## Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+## Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+## Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
### Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/Makefile b/Makefile
index 16085ff07..dc2e0aed8 100644
--- a/Makefile
+++ b/Makefile
@@ -455,7 +455,7 @@ check-TODOs: env
docs: env
$(call PRINT_TITLE,"Serving documentation with mkdocs")
- $(VENV_MKDOCS) serve
+ $(VENV_MKDOCS) serve --watch docs
docs-check: env
$(call PRINT_TITLE,"Checking documentation build with mkdocs")
diff --git a/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md b/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md
deleted file mode 100644
index 4ae4ecd66..000000000
--- a/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md
+++ /dev/null
@@ -1,116 +0,0 @@
-# New Inference Backend Configuration System
-
-This document provides an overview of the new system for configuring LLM inference backends, models, and routing within Pipelex. The system is designed to be highly flexible and is managed through a series of TOML configuration files.
-
-The core components of this system reside in the following directories:
-- `pipelex/cogt/model_backends/`
-- `pipelex/cogt/model_routing/`
-- `pipelex/cogt/models/`
-
-## Core Concepts
-
-The new system is built around a few key concepts: Inference Backends, Model Specs, Routing Profiles, and the Model Deck. This unified system now supports all types of AI models: LLMs for text generation, OCR models for text extraction, and Image Generation models.
-
-### 1. Inference Backends
-
-An **Inference Backend** represents a provider of AI services, such as OpenAI, Anthropic, Google Vertex AI, FAL, etc. Each backend is defined by its name, API endpoint, and authentication details (e.g., API key). Backends can provide different types of models including LLMs, OCR models, and Image Generation models.
-
-- **Implementation**: `pipelex.cogt.model_backends.backend.InferenceBackend`
-- **Configuration**: Backends are defined in a central TOML file (e.g., `backends.toml`), specified by `cogt.inference_config.backends_library_path` in the main configuration.
-
-### 2. Model Specs
-
-An **Inference Model Spec** (`InferenceModelSpec`) provides detailed information about a specific model available through a backend. This includes:
-
-- `model_type`: The type of model (`llm`, `text_extractor`, or `img_gen`).
-- `model_id`: The identifier used by the backend's API.
-- `sdk`: The Pipelex SDK to use for communication (e.g., `openai`, `anthropic`, `fal`, `pypdfium2`).
-- Capabilities: Supported `inputs` (e.g., `text`, `images`, `pdf`) and `outputs` (e.g., `text`, `structured`, `pages`, `image`).
-- Pricing: `costs` for input and output tokens or operations.
-- Constraints: `max_tokens`, `max_prompt_images`, and other model-specific limitations.
-
-- **Implementation**: `pipelex.cogt.model_backends.model_spec.InferenceModelSpec`
-- **Configuration**: Model specs for each backend are defined in their own dedicated TOML file (e.g., `openai.toml`, `anthropic.toml`). The path to these files is determined by `cogt.inference_config.model_specs_path(backend_name)`.
-
-### 3. Routing Profiles
-
-A **Routing Profile** (`RoutingProfile`) defines the rules for selecting which backend should be used for a given model name. This allows for flexible and environment-specific model routing. A routing profile consists of:
-
-- A `default` backend to use if no other rule matches.
-- A set of `routes` that map a model name (or a pattern) to a specific backend.
-
-The routing logic supports exact matches, as well as prefix, suffix, and "contains" wildcard (`*`) matching.
-
-- **Implementation**: `pipelex.cogt.model_routing.routing_profile.RoutingProfile`
-- **Configuration**: Multiple routing profiles can be defined in a single TOML file (e.g., `routing.toml`), specified by `cogt.inference_config.routing_profile_library_path`. One profile is designated as `active`.
-
-### 4. The Model Deck
-
-The **Model Deck** (`ModelDeck`) is the final, unified collection of all configured and available models in the system. It is constructed at runtime by the `ModelManager` and serves as the single source of truth for the rest of the application. The Model Deck contains:
-
-- **Inference Models**: A resolved map of model names (`model_handle`) to their corresponding `InferenceModelSpec`.
-- **Aliases**: User-friendly names that map to one or more model names, creating a fallback chain (e.g., `best-claude` -> `claude-3-opus-20240229`, `base-img-gen` -> `flux-pro/v1.1`).
-- **LLM Presets**: Pre-defined configurations (`LLMSetting`) for specific tasks, combining an `llm_handle` with parameters like `temperature` and `max_tokens` (e.g., `llm_to_reason`).
-- **OCR Presets**: Pre-defined configurations (`OcrSetting`) for OCR tasks, combining an `ocr_handle` with parameters like `max_nb_images` and `image_min_size` (e.g., `base_ocr_mistral`).
-- **Image Generation Presets**: Pre-defined configurations (`ImgGenSetting`) for image generation tasks, combining an `img_gen_handle` with parameters like `quality`, `guidance_scale`, and `safety_tolerance` (e.g., `base_img_gen`, `high_quality_img_gen`).
-- **Default Choices**: Default model presets to use for text generation (`for_text`), structured data extraction (`for_object`), OCR operations (`ocr.choice_default`), and image generation (`img_gen.choice_default`).
-
-- **Implementation**: `pipelex.cogt.models.model_deck.ModelDeck`
-- **Configuration**: Aliases, presets, and defaults are loaded from one or more `model_deck.toml` files, specified by `cogt.inference_config.get_model_deck_paths()`.
-
-### 5. OCR Integration
-
-**OCR (Optical Character Recognition)** models are now fully integrated into the inference backend system. OCR models extract text and images from PDFs and images, producing structured page data.
-
-- **Model Type**: `text_extractor`
-- **Common SDKs**: `mistral` (for Mistral OCR models), `pypdfium2` (for local PDF text extraction)
-- **Inputs**: `pdf`, `image`
-- **Outputs**: `pages` (structured page data with text and images)
-- **Usage**: OCR models are referenced by their `ocr_handle` in `PipeExtract` operations and can use OCR presets for common configurations
-
-Example OCR model configuration:
-```toml
-[mistral-ocr]
-model_type = "text_extractor"
-model_id = "mistral-ocr-latest"
-inputs = ["pdf", "image"]
-outputs = ["pages"]
-costs = { input = 0.4, output = 2.0 }
-```
-
-### 6. Image Generation Integration
-
-**Image Generation** models are integrated into the same system, allowing for consistent configuration and routing of image generation requests.
-
-- **Model Type**: `img_gen`
-- **Common SDKs**: `fal` (for FAL models), `openai` (for DALL-E models)
-- **Inputs**: `text` (prompts)
-- **Outputs**: `image`
-- **Usage**: Image generation models are referenced by their `img_gen_handle` in `PipeImgGen` operations and support presets for different quality levels and use cases
-
-Example Image Generation model configuration:
-```toml
-["flux-pro/v1.1"]
-model_type = "img_gen"
-model_id = "fal-ai/flux-pro/v1.1"
-inputs = ["text"]
-outputs = ["image"]
-costs = { input = 0.05, output = 0.0 }
-```
-
-## How It All Works Together: The Loading Process
-
-The entire system is orchestrated by the `ModelManager` during application startup. Here is a step-by-step breakdown of the process:
-
-1. **Load Backends**: The `InferenceBackendLibrary` reads the `backends.toml` file to get a list of all defined backends. For each enabled backend, it proceeds to load its associated models.
-2. **Load Model Specs**: For each backend, the library reads the corresponding `<backend_name>.toml` file to load the `InferenceModelSpec` for every model that backend provides. This includes LLM models, OCR models (`text_extractor` type), and Image Generation models (`img_gen` type).
-3. **Load Routing Profiles**: The `RoutingProfileLibrary` reads the `routing.toml` file to load all defined routing profiles and identifies the `active` profile.
-4. **Build the Deck**: The `ModelManager` begins to build the `ModelDeck`.
- a. It gets the list of all models from all loaded backends (LLMs, OCR models, and Image Generation models).
- b. For each model, it consults the `active` `RoutingProfile` to determine which backend should be used.
- c. It retrieves the `InferenceModelSpec` for that model from the chosen backend.
- d. This resolved model spec is added to the `ModelDeck`'s list of `inference_models`.
-5. **Load Deck Configuration**: The `ModelManager` reads the `base_deck.toml` and other model deck files from the `deck` directory and loads the `aliases`, `llm_presets`, `ocr_presets`, `img_gen_presets`, and default choices into the `ModelDeck`.
-6. **Finalization**: The `ModelDeck` is now fully constructed and ready to be used by the rest of Pipelex to get model specifications and settings for LLMs, OCR operations, and Image Generation via `get_model_deck()`.
-
-This layered configuration system provides a powerful way to manage models from various providers, route them according to specific needs, and define reusable presets for common tasks, all through a set of clear and manageable TOML files. The unified approach ensures that LLMs, OCR models, and Image Generation models are all managed consistently within the same infrastructure.
diff --git a/README.md b/README.md
index fabf80da6..ec71e96cb 100644
--- a/README.md
+++ b/README.md
@@ -40,10 +40,23 @@ Stop reinventing AI workflows from scratch. With Pipelex, your proven methods be
</p>
<h2 align="center">πŸš€ See Pipelex in Action</h2>
- <h3 align="center">From Whiteboard to AI Workflow in less than 5 minutes with no hands</h3>
- <p align="center">
- <a href="https://go.pipelex.com/demo"><img src="https://go.pipelex.com/demo-thumbnail" alt="Pipelex Demo" width="600" style="max-width: 100%; height: auto;"></a>
- </p>
+
+ <table align="center">
+ <tr>
+ <td align="center" width="50%">
+ <h3>From Whiteboard to AI Workflow in less than 5 minutes with no hands (2025-07)</h3>
+ <a href="https://go.pipelex.com/demo">
+ <img src="https://go.pipelex.com/demo-thumbnail" alt="Pipelex Demo" width="100%" style="max-width: 500px; height: auto;">
+ </a>
+ </td>
+ <td align="center" width="50%">
+ <h3>The AI workflow that writes an AI workflow in 64 seconds (2025-09)</h3>
+ <a href="https://go.pipelex.com/Demo-Live">
+ <img src="https://d2cinlfp2qnig1.cloudfront.net/banners/pipelex_play_video_demo_live.jpg" alt="Pipelex Live Demo" width="100%" style="max-width: 500px; height: auto;">
+ </a>
+ </td>
+ </tr>
+ </table>
</div>
diff --git a/docs/index.md b/docs/index.md
index 2d8356b3a..1b84137e4 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -24,9 +24,15 @@ $env:PIPELEX_INFERENCE_API_KEY="###"
# Windows CMD
set PIPELEX_INFERENCE_API_KEY=###
+
+# Note: of course, Pipelex automatically loads environment variables from `.env` files, that works too.
```
-**Where to get an API key:** The `PIPELEX_INFERENCE_API_KEY` key provides access to all the AI models, you can get a free key from [our Discord](https://go.pipelex.com/discord). You can also use another AI routing service like [BlackBox AI](https://docs.blackbox.ai/), or bring your own API keys (OpenAI, Anthropic, Google, etc.), or run local AI (no key needed). See [Configure AI Providers](pages/setup/configure-ai-providers.md) for details. If you are using non-standard APIs, that's OK too, don't hesitate to join our [Discord](https://go.pipelex.com/discord) for guidance.
+**Where to get an API key:**
+
+- The `PIPELEX_INFERENCE_API_KEY` key provides access to all the AI models. To get your key, join our Discord community: [https://go.pipelex.com/discord](https://go.pipelex.com/discord), then request your **free API key** (no credit card required, limited time offer) in the [πŸ”‘γƒ»free-api-key](https://discord.com/channels/1369447918955921449/1418228010431025233) channel.
+- You can also use other AI routing services like [BlackBox AI](https://docs.blackbox.ai/), or you can bring your own API keys (OpenAI, Anthropic, Google, Mistral,etc.), or run local AI (no key needed).
+See [Configure AI Providers](pages/setup/configure-ai-providers.md) for details. If you are using non-standard APIs, that's OK too, don't hesitate to join our [Discord](https://go.pipelex.com/discord) for guidance.
## Generate your first pipe
@@ -61,7 +67,9 @@ pipelex run bundle.plx --inputs input.json
pipelex run <pipe_code> --output results/output.json
```
-The `--inputs` file should be a JSON dictionary where keys are input variable names and values are the input data. For native types like Text, use strings directly. For structured types, provide objects matching the expected structure.
+The `--inputs` file should be a JSON dictionary where keys are input variable names and values are the input data. For native concepts like Text, you can use strings directly. For structured types, provide objects matching the expected structure.
+
+Learn more: [Executing Pipelines with Inputs](pages/build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md)
**Python:**
@@ -80,7 +88,7 @@ asyncio.run(run_pipeline())
## Easily iterate on your pipe
-Now, thanks to our Pipelex language, and its high level of abstraction, you can directly edit the pipeline. It's pretty easy even for non-technical users. Better yet, you can get assisted in making changes with the help of your favorite AI coding assistant. To that end, we have prepared comprehensive guides for the most popular AI coding assistants and you can install them with one call:
+Now, thanks to our Pipelex language, you can easily edit the pipeline, even if you're not a coder. Better yet, you can get assisted in making changes with the help of your favorite AI coding assistant. To that end, we have prepared comprehensive guides for the most popular AI coding assistants and you can install them with one call:
```bash
pipelex kit rules
@@ -93,6 +101,16 @@ Now refine your pipeline with natural language:
- "Include confidence scores between 0 and 100 in the match analysis"
- "Write a recap email at the end"
+## IDE Support
+
+By the way, we **highly** recommend installing our own extension for PLX files into your IDE of choice. You can find it in the [Open VSX Registry](https://open-vsx.org/extension/Pipelex/pipelex). It's coming soon to the VS Code marketplace too and if you are using Cursor, Windsurf or another VS Code fork, you can search for it directly in your extensions tab.
+
+## Examples
+
+Visit the
+[![GitHub](https://img.shields.io/badge/Cookbook-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/): you can clone it, fork it, play with it
+
+
---
## What is Pipelex?
@@ -125,4 +143,3 @@ Each step can be structured and validated, so you benefit from the reliability o
[![Cookbook](https://img.shields.io/badge/Cookbook-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/)
-
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
index 49f14c4fa..5d2cb6b6b 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
@@ -83,7 +83,12 @@ Please extract the most relevant keywords from the following product description
Focus on features, benefits, and unique selling points.
"""
+```
+
+!!! note "Multiple Outputs"
+ The `multiple_output = true` setting allows the LLM to generate as many keywords as it finds relevant. For a comprehensive guide on controlling how many items pipes produce or accept, see [Understanding Multiplicity](understanding-multiplicity.md).
+```plx
[pipe.generate_tagline_from_keywords]
type = "PipeLLM"
description = "Generate a tagline from keywords"
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md
new file mode 100644
index 000000000..b15e9b17b
--- /dev/null
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md
@@ -0,0 +1,367 @@
+# Executing Pipelines with Inputs
+
+When you run a Pipelex pipeline from Python, you often need to provide input data. The `input_memory` parameter gives you a flexible and intuitive way to pass data to your pipelines.
+
+This guide explains the different formats you can use and when to use each one.
+
+## Understanding input_memory
+
+The `input_memory` parameter accepts a dictionary where:
+
+- **Keys** are the input variable names (matching the `inputs` defined in your pipeline)
+- **Values** can be provided in multiple formats for convenience
+
+Under the hood, `input_memory` uses this type definition:
+
+```python
+StuffContentOrData = dict[str, Any] | StuffContent | list[Any] | str
+ImplicitMemory = dict[str, StuffContentOrData]
+```
+
+Pipelex is smart about detecting what you mean and automatically wraps your data in the appropriate content types.
+
+!!! tip "Working with Lists"
+ When providing multiple items as input (lists) or expecting multiple outputs, understanding multiplicity is essential. See [Understanding Multiplicity](understanding-multiplicity.md) for a comprehensive guide on how Pipelex handles single items versus collections.
+
+## Quick Comparison
+
+Here's when to use each format:
+
+| Format | Use Case | Example |
+|--------|----------|---------|
+| **Simple String** | Text inputs, quick prototypes | `"input": "Hello world"` |
+| **Native Content** | PDF, Image, or other native types | `"doc": PDFContent(url="file.pdf")` |
+| **Explicit Format** | Custom concepts, refined concepts | `"data": {"concept": "domain.ConceptCode", "content": value}` |
+| **Mixed Formats** | Multiple inputs of different types | Combine any of the above |
+
+## Format 1: Simple String Inputs
+
+The simplest way to provide text input is with a plain string. Pipelex automatically converts it to `TextContent`.
+
+### Example
+
+```python
+pipe_output = await execute_pipeline(
+ pipe_code="summarize_story",
+ input_memory={
+ "story": """
+ Once upon a time there was a brave knight who saved the kingdom from a dragon.
+ [...]
+ # Imagine this is a long text for the purpose of this example
+ [...]
+ and they lived happily ever after.
+ """,
+ },
+)
+```
+
+**When to use:**
+
+- Your pipeline expects a `Text` concept
+- You're passing simple string data
+- You want the most concise syntax
+
+## Format 2: Native Content Types
+
+For native concepts like `PDF`, `Image`, `Text`, `Number`, and `Page`, you can use their content classes directly.
+
+### PDF Example
+
+```python
+from pipelex.core.stuffs.pdf_content import PDFContent
+
+pipe_output = await execute_pipeline(
+ pipe_code="process_invoice",
+ input_memory={
+ "document": PDFContent(url="invoice.pdf"),
+ },
+)
+```
+
+### Image Example
+
+```python
+from pipelex.core.stuffs.image_content import ImageContent
+
+pipe_output = await execute_pipeline(
+ pipe_code="analyze_photo",
+ input_memory={
+ "photo": ImageContent(url="photo.jpg"),
+ },
+)
+```
+
+**Why this works:**
+
+Native content types are automatically recognized by Pipelex. The system knows that `PDFContent` corresponds to the `PDF` concept, `ImageContent` to `Image`, and so on.
+
+**When to use:**
+
+- Your pipeline expects a native concept (`PDF`, `Image`, `Text`, `Number`, `Page`)
+- You want type safety and IDE autocomplete
+- You're loading from files or URLs
+
+## Format 3: Explicit Concept Format
+
+For custom concepts or concepts that refine native types, use the explicit format with `concept` and `content` keys.
+
+### Example with Refined Concept
+
+```python
+from pipelex.core.stuffs.image_content import ImageContent
+
+pipe_output = await execute_pipeline(
+ pipe_code="extract_gantt",
+ input_memory={
+ "gantt_chart_image": {
+ "concept": "gantt.GanttChartImage", # Custom concept that refines Image
+ "content": ImageContent(url="gantt.png"),
+ }
+ },
+)
+```
+
+### Example with Custom Structured Type
+
+```python
+pipe_output = await execute_pipeline(
+ pipe_code="answer_question",
+ input_memory={
+ "question": {
+ "concept": "qa.Question",
+ "content": "What is the capital of France?",
+ },
+ },
+)
+```
+
+**When to use:**
+
+- Your pipeline expects a custom concept (not a native one)
+- Your concept refines a native type but adds semantic meaning
+- You need to be explicit about which concept you're providing
+
+!!! note "Why Explicit Format is Sometimes Required"
+ If you have a concept like `gantt.GanttChartImage` that refines the native `Image` concept, Pipelex can't automatically know you mean a `GanttChartImage` when you pass an `ImageContent`. The explicit format removes this ambiguity.
+
+## Format 4: Multiple Inputs with Mixed Formats
+
+You can mix and match formats freely in the same `input_memory` dictionary.
+
+### Complex Example
+
+```python
+from pipelex.core.stuffs.text_content import load_text_from_path
+
+pipe_output = await execute_pipeline(
+ pipe_code="analyze_contract",
+ input_memory={
+ # Format 1: Simple string
+ "client_instructions": "Focus on payment terms",
+
+ # Format 1: String loaded from file (still just a string)
+ "contract_text": load_text_from_path("contract.txt"),
+
+ # Format 3: Explicit concept
+ "question": {
+ "concept": "legal.Question",
+ "content": "What are the fees?",
+ },
+ },
+)
+```
+
+**When to use:**
+
+- Your pipeline has multiple inputs
+- Different inputs have different types or requirements
+- You want to use the most convenient format for each input
+
+## Guidelines: Choosing the Right Format
+
+Follow these simple rules:
+
+### Use Simple Strings When
+
+- βœ… The input is text data
+- βœ… The pipeline expects a `Text` concept
+- βœ… You want minimal syntax
+
+### Use Native Content Types When
+
+- βœ… Working with `PDF`, `Image`, `Number`, or `Page` concepts
+- βœ… You want type checking and IDE support
+- βœ… Loading from files or URLs
+
+### Use Explicit Format When
+
+- βœ… Your concept is not a native type
+- βœ… Your concept refines a native type (adds semantic meaning)
+- βœ… You need to specify exactly which concept you're providing
+- βœ… Pipelex gives you a type error without it
+
+## Common Patterns
+
+### Pattern 1: Document Processing
+
+```python
+# Most document processing pipelines expect a PDF input
+pipe_output = await execute_pipeline(
+ pipe_code="extract_data",
+ input_memory={
+ "document": PDFContent(url="document.pdf"),
+ },
+)
+```
+
+### Pattern 2: Image Analysis
+
+```python
+# Image processing with native Image concept
+pipe_output = await execute_pipeline(
+ pipe_code="describe_image",
+ input_memory={
+ "image": ImageContent(url="photo.jpg"),
+ },
+)
+```
+
+### Pattern 3: Text Generation
+
+```python
+# Simple text input for generation tasks
+pipe_output = await execute_pipeline(
+ pipe_code="write_story",
+ input_memory={
+ "topic": "A robot learning to love",
+ },
+)
+```
+
+### Pattern 4: Specialized Types
+
+```python
+# Custom concept that needs explicit specification
+pipe_output = await execute_pipeline(
+ pipe_code="process_tweet",
+ input_memory={
+ "draft_tweet": {
+ "concept": "social.DraftTweet",
+ "content": "Check out this amazing framework!",
+ },
+ },
+)
+```
+
+## Working with Structured Content
+
+If you have structured data defined as a `StructuredContent` class, you can pass instances directly when using the explicit format.
+
+### Example
+
+```python
+from my_project.models import Character
+
+# Create a structured object
+character = Character(
+ name="Alice",
+ age=30,
+ description="A brave explorer",
+)
+
+# Pass it to the pipeline
+pipe_output = await execute_pipeline(
+ pipe_code="analyze_character",
+ input_memory={
+ "character": {
+ "concept": "story.Character",
+ "content": character,
+ },
+ },
+)
+```
+
+## Alternative: Using working_memory Directly
+
+For advanced use cases, you can construct the working memory explicitly instead of using `input_memory`:
+
+```python
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+
+# Create a stuff object
+character_stuff = StuffFactory.make_from_concept_string(
+ concept_string="story.Character",
+ name="character",
+ content=character_data,
+)
+
+# Create working memory
+working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=character_stuff,
+)
+
+# Execute with working memory
+pipe_output = await execute_pipeline(
+ pipe_code="analyze_character",
+ working_memory=working_memory,
+)
+```
+
+**When to use `working_memory` directly:**
+
+- You need fine-grained control over stuff naming
+- You're building complex multi-stuff scenarios
+- You're working with intermediate pipeline states
+
+**For most cases, `input_memory` is simpler and recommended.**
+
+## Troubleshooting
+
+### Error: "Concept not found"
+
+If you see an error about a concept not being found, you may need to use the explicit format:
+
+```python
+# ❌ Won't work if MyType is a custom concept
+input_memory={"data": my_value}
+
+# βœ… Use explicit format
+input_memory={
+ "data": {
+ "concept": "domain.MyType",
+ "content": my_value,
+ }
+}
+```
+
+### Error: "Type mismatch"
+
+Make sure the content type matches what your concept expects:
+
+```python
+# ❌ Wrong: passing a string when concept expects structured data
+input_memory={
+ "character": {
+ "concept": "story.Character",
+ "content": "Alice", # Should be a Character instance
+ }
+}
+
+# βœ… Correct: pass the right type
+input_memory={
+ "character": {
+ "concept": "story.Character",
+ "content": Character(name="Alice", age=30, ...),
+ }
+}
+```
+
+## Related Documentation
+
+- [Design and Run Pipelines](design_and_run_pipelines.md) - Learn about pipeline execution basics
+- [Define Your Concepts](define_your_concepts.md) - Understand concepts and their role
+- [Structuring Concepts](structuring-concepts.md) - Create structured data types
+- [Cookbook Examples](../cookbook-examples/index.md) - See real-world input patterns
+
+
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
index cf9a2866c..880f5636a 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
@@ -139,6 +139,9 @@ Analyze the document page shown in the image and explain how it relates to the p
3. **Variable Multiple Outputs**: Use `multiple_output = true` when you need a variable-length list where the LLM determines how many outputs to generate based on the content and context.
+!!! info "Learn More About Multiplicity"
+ For a comprehensive guide on output multiplicity, input multiplicity, and the philosophy behind how Pipelex handles single items versus collections, see [Understanding Multiplicity](../understanding-multiplicity.md).
+
## Examples
### Simple Text Generation Example
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md
new file mode 100644
index 000000000..1f336fdbc
--- /dev/null
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md
@@ -0,0 +1,423 @@
+# Understanding Multiplicity
+
+Multiplicity in Pipelex defines how many items a particular stuff can comprise in a particular context. This applies to any of the pipe input variables and also to the output of the pipe. This idea is fundamental to building flexible AI workflows that can handle both single items and collections.
+
+This guide explains the philosophy behind multiplicity in Pipelex and how to use it effectively in your pipelines.
+
+## The Philosophy: Why Concepts Are Always Singular
+
+In Pipelex, concepts are always defined in the singular form. You define a `Keyword`, not `Keywords`. You define an `Invoice`, not `Invoices`. This isn't just a naming conventionβ€”it's a fundamental design principle.
+
+### Concepts Represent Semantic Entities
+
+A concept represents a semantic entityβ€”a meaningful piece of knowledge with clear boundaries. When you define a concept, you're describing what something *is*, not how many of them you might have:
+
+```plx
+[concept]
+Keyword = "A significant word or term extracted from text"
+ProductIdea = "A concept for a new product or service"
+Document = "A written or printed record containing information"
+```
+
+Each of these definitions describes a single, coherent entity. The essence of what makes something a "Keyword" doesn't change whether you have one keyword or a hundred.
+
+### Lists Are Circumstantial, Not Essential
+
+The number of items you're working with is a circumstantial detail of your workflow, not part of the concept's identity:
+
+- A pipe that extracts keywords from text might find 3 keywords or 30β€”but each is still a `Keyword`
+- A pipe that generates product ideas might produce 5 ideas or 10β€”but each remains a `ProductIdea`
+- A pipe that processes documents might handle 1 document or a batch of 100β€”but each is still a `Document`
+
+By keeping concepts singular and using multiplicity to express quantity, Pipelex maintains a clean semantic model that's both flexible and intuitive.
+
+## Output Multiplicity
+
+Output multiplicity controls how many items a pipe produces. This is most commonly used with `PipeLLM` to generate multiple outputs from a single execution.
+
+### The Two Modes of Output Multiplicity
+
+Pipelex supports two ways to specify output multiplicity:
+
+1. **Fixed multiplicity** (`nb_output`): Generate an exact number of items
+2. **Variable multiplicity** (`multiple_output`): Let the LLM decide how many items to generate
+
+### Fixed Output Multiplicity
+
+Use `nb_output` when you need a specific, predetermined number of outputs:
+
+```plx
+[concept]
+Headline = "A catchy title for content"
+
+[pipe.generate_headline_options]
+type = "PipeLLM"
+description = "Generate headline alternatives"
+inputs = { article_text = "Text" }
+output = "Headline"
+nb_output = 5
+prompt = """
+Read this article:
+
+@article_text
+
+Generate $_nb_output different headline options for this article.
+Make each one unique and compelling.
+"""
+```
+
+In this example, the pipe will always produce exactly 5 headlines. The variable `$_nb_output` is automatically available in your prompt, making it easy to communicate the requirement to the LLM.
+
+**Common use cases for fixed multiplicity:**
+
+- Generate N alternative versions for A/B testing
+- Create a fixed set of options for user selection
+- Produce a specific number of variations for comparison
+- Match external requirements (e.g., "always provide 3 recommendations")
+
+### Variable Output Multiplicity
+
+Use `multiple_output = true` when the number of outputs should depend on the content:
+
+```plx
+[concept]
+LineItem = "A single line item from an invoice"
+
+[concept.LineItem.structure]
+description = "Description of the item or service"
+quantity = { type = "number", description = "Quantity purchased" }
+unit_price = { type = "number", description = "Price per unit" }
+total = { type = "number", description = "Total price for this line" }
+
+[pipe.extract_line_items]
+type = "PipeLLM"
+description = "Extract all line items from an invoice"
+inputs = { invoice_text = "Text" }
+output = "LineItem"
+multiple_output = true
+prompt = """
+Extract all line items from this invoice:
+
+@invoice_text
+
+For each line item, extract the description, quantity, unit price, and total amount.
+"""
+```
+
+Here, the pipe will extract however many line items appear in the invoice. A simple invoice might have 2 line items, while a detailed purchase order might have 50.
+
+**Common use cases for variable multiplicity:**
+
+- Extract entities from text (unknown count in advance)
+- Generate as many alternatives as needed
+- List all items that match criteria
+- Identify all occurrences of a pattern
+
+### Single Output (Default)
+
+When you don't specify `nb_output` or `multiple_output`, a pipe produces a single output:
+
+```plx
+[concept]
+Summary = "A concise overview of content"
+
+[pipe.summarize]
+type = "PipeLLM"
+description = "Create a summary of the document"
+inputs = { document = "Text" }
+output = "Summary"
+prompt = """
+Summarize this document concisely:
+
+@document
+"""
+```
+
+This is the default behavior and represents the most common case.
+
+## Input Multiplicity
+
+Input multiplicity specifies whether a pipe expects a single item or multiple items as input. This allows you to design pipes that explicitly require lists or explicitly require single items.
+
+### Syntax for Input Multiplicity
+
+Input multiplicity is specified using an expanded syntax in the `inputs` dictionary:
+
+```plx
+# Standard syntax (single item, the default)
+inputs = { document = "Text" }
+
+# Equivalent with explicit multiplicity = false (not needed, but shown for clarity)
+inputs = { document = { concept = "Text", multiplicity = false } }
+
+# Variable list (indeterminate number)
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+# Fixed count (exactly N items)
+inputs = { comparison_items = { concept = "Text", multiplicity = 2 } }
+```
+
+### The Three Input Multiplicity Modes
+
+**1. Single item (default)**
+
+When you use the standard syntax, the pipe expects exactly one item. This is the default behavior:
+
+```plx
+[concept]
+Report = "A detailed analytical document"
+
+[pipe.analyze_report]
+type = "PipeLLM"
+description = "Analyze a single report"
+inputs = { report = "Report" }
+output = "Analysis"
+prompt = """
+Analyze this report in detail:
+
+@report
+"""
+```
+
+!!! note "No Need for `multiplicity = false`"
+ You don't need to specify `multiplicity = false` explicitly, it's the default. Only use the expanded syntax when you need `multiplicity = true` or a fixed integer count.
+
+**2. Variable list (`multiplicity = true`)**
+
+When you set `multiplicity = true`, the pipe expects a list with an indeterminate number of items:
+
+```plx
+[concept]
+Document = "A written or printed record"
+Summary = "A concise overview of multiple documents"
+
+[pipe.summarize_all_documents]
+type = "PipeLLM"
+description = "Create a unified summary of multiple documents"
+inputs = { documents = { concept = "Document", multiplicity = true } }
+output = "Summary"
+prompt = """
+Analyze all of these documents:
+
+@documents
+
+Create a single unified summary that captures the key points across all documents.
+"""
+```
+
+**3. Fixed count (`multiplicity = N`)**
+
+When you set `multiplicity` to a specific integer, the pipe expects exactly that many items:
+
+```plx
+[concept]
+Image = "A visual image file"
+Comparison = "A detailed comparison analysis"
+
+[pipe.compare_two_images]
+type = "PipeLLM"
+description = "Compare exactly two images side by side"
+inputs = { images = { concept = "Image", multiplicity = 2 } }
+output = "Comparison"
+prompt = """
+Compare these two images in detail:
+
+@images
+
+Describe their similarities, differences, and relative strengths.
+"""
+```
+
+## Practical Use Cases
+
+### Use Case 1: Batch Processing with Variable Input
+
+Process an unknown number of invoices, extracting structured data from each:
+
+```plx
+[concept]
+InvoiceImage = "An image of an invoice document"
+InvoiceData = "Structured invoice information"
+
+[pipe.extract_single_invoice]
+type = "PipeLLM"
+description = "Extract data from one invoice"
+inputs = { invoice_image = "InvoiceImage" }
+output = "InvoiceData"
+prompt = """
+Extract all fields from this invoice:
+
+@invoice_image
+"""
+
+[pipe.process_invoice_batch]
+type = "PipeSequence"
+description = "Process multiple invoices"
+inputs = { invoice_images = { concept = "InvoiceImage", multiplicity = true } }
+output = "InvoiceData"
+steps = [
+ { pipe = "extract_single_invoice", batch_over = "invoice_images", batch_as = "invoice_image", result = "all_invoice_data" }
+]
+```
+
+### Use Case 2: Generate Fixed Alternatives for Testing
+
+Create exactly 3 subject line variations for A/B testing:
+
+```plx
+[concept]
+EmailContent = "The body text of an email"
+SubjectLine = "An email subject line"
+
+[pipe.generate_subject_lines]
+type = "PipeLLM"
+description = "Generate 3 subject line options"
+inputs = { email_body = "EmailContent" }
+output = "SubjectLine"
+nb_output = 3
+prompt = """
+Email content:
+@email_body
+
+Generate $_nb_output compelling subject lines for this email.
+Each should use a different persuasion technique.
+"""
+```
+
+### Use Case 3: Comparative Analysis
+
+Compare exactly two products side by side:
+
+```plx
+[concept]
+ProductDescription = "A description of a product's features"
+Comparison = "A comparative analysis of products"
+
+[pipe.compare_products]
+type = "PipeLLM"
+description = "Compare two products"
+inputs = { products = { concept = "ProductDescription", multiplicity = 2 } }
+output = "Comparison"
+prompt = """
+Compare these two products:
+
+@products
+
+Provide a balanced comparison of features, benefits, and potential drawbacks.
+"""
+```
+
+### Use Case 4: Entity Extraction with Unknown Count
+
+Extract all company names mentioned in a document:
+
+```plx
+[concept]
+Article = "A news or information article"
+CompanyName = "The name of a company or organization"
+
+[pipe.extract_companies]
+type = "PipeLLM"
+description = "Extract all company names from an article"
+inputs = { article = "Article" }
+output = "CompanyName"
+multiple_output = true
+prompt = """
+Read this article:
+
+@article
+
+Extract all company and organization names mentioned in the article.
+Only include entities that are explicitly named.
+"""
+```
+
+## Best Practices
+
+### When to Use Variable Output (`multiple_output = true`)
+
+Use variable output multiplicity when:
+
+- The number of outputs depends on the content being analyzed
+- You're extracting or identifying items (entities, keywords, issues)
+- The count isn't known until after processing
+- You want the LLM to use its judgment about completeness
+
+### When to Use Fixed Output (`nb_output = N`)
+
+Use fixed output multiplicity when:
+
+- You need a specific number for downstream processes
+- You're generating alternatives for comparison or selection
+- External requirements dictate a fixed count
+- You want consistent batch sizes for processing
+
+### When to Use Variable Input (`multiplicity = true`)
+
+Use variable input multiplicity when:
+
+- The pipe should handle batches of unknown size
+- You're aggregating or summarizing multiple items
+- The workflow involves collecting items before processing
+- You want maximum flexibility in how the pipe is called
+
+### When to Use Fixed Input (`multiplicity = N`)
+
+Use fixed input multiplicity when:
+
+- The operation inherently requires a specific count (e.g., comparison of 2 items)
+- You want to enforce validation at the pipeline level
+- The prompt logic depends on an exact number of inputs
+- You're preventing misuse by requiring the right number of items
+
+## How Multiplicity Interacts with ListContent
+
+When a pipe produces multiple outputs, Pipelex automatically wraps them in a `ListContent` container. This container maintains the type information:
+
+```python
+from pipelex.pipeline.execute import execute_pipeline
+
+# Execute a pipe with multiple outputs
+pipe_output = await execute_pipeline(
+ pipe_code="extract_line_items",
+ input_memory={"invoice_text": "Your invoice text here..."}
+)
+
+# Get the list of line items
+line_items_list = pipe_output.main_stuff_as_items(item_type=LineItem)
+# Result: [LineItem(...), LineItem(...), LineItem(...)]
+
+# Or work with the ListContent directly
+line_items_container = pipe_output.main_stuff_as_list(item_type=LineItem)
+# Result: ListContent[LineItem] with additional metadata
+```
+
+Similarly, when providing multiple inputs, you can use lists:
+
+```python
+from pipelex.core.stuffs.text_content import TextContent
+
+pipe_output = await execute_pipeline(
+ pipe_code="summarize_all_documents",
+ input_memory={
+ "documents": [
+ TextContent(text="Document 1 content..."),
+ TextContent(text="Document 2 content..."),
+ TextContent(text="Document 3 content..."),
+ ]
+ }
+)
+```
+
+## Summary
+
+Multiplicity in Pipelex gives you precise control over how many items flow through your pipelines:
+
+- **Concepts stay singular** to maintain clean semantics
+- **Output multiplicity** (`nb_output`, `multiple_output`) controls generation
+- **Input multiplicity** (in the `inputs` definition) enforces expectations
+- **Variable multiplicity** (`true`) for unknown counts
+- **Fixed multiplicity** (integer) for exact requirements
+
+By understanding and using multiplicity effectively, you can build pipelines that handle both single items and collections with clarity and type safety.
+
diff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.md
index 557131cfe..95669e2f9 100644
--- a/docs/pages/tools/cli.md
+++ b/docs/pages/tools/cli.md
@@ -2,41 +2,394 @@
The Pipelex CLI provides a command-line interface for managing and interacting with your Pipelex projects. This document outlines all available commands and their usage.
-## Available Commands
+## Overview
-### Init group
+The Pipelex CLI is organized into several command groups:
-Initialize project configuration.
+- **init** - Initialize Pipelex configuration
+- **kit** - Manage agent rules and migration instructions (see [Kit Commands](kit.md))
+- **build** - Generate pipelines from natural language (see [Pipe Builder](pipe-builder.md))
+- **validate** - Validate configuration and pipelines
+- **run** - Execute pipelines
+- **show** - Inspect configuration, pipes, and AI models
+
+## Init Commands
+
+Initialize project configuration files in your project's `.pipelex` directory.
+
+### Initialize Configuration
+
+```bash
+pipelex init config [OPTIONS]
+```
+
+Creates the `.pipelex` directory structure and copies default configuration files from the Pipelex package template.
+
+**Options:**
+
+- `--reset`, `-r` - Overwrite existing configuration files (use with caution)
+
+**Examples:**
+
+```bash
+# Initialize configuration for the first time
+pipelex init config
+
+# Reinitialize configuration, overwriting existing files
+pipelex init config --reset
+```
+
+**What gets initialized:**
+
+This command creates the `.pipelex/` directory with:
+
+- The main pipelex.toml configuration file for logging, reporting, tracking, etc.
+- `inference/` - AI backend and routing configuration
+- `deck/` - AI model aliases and presets
+
+**Related Configuration:**
+
+- [Configure AI Providers](../setup/configure-ai-providers.md)
+- [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md)
+
+## Validate Commands
+
+Validate your pipeline definitions and configuration for correctness.
+
+### Validate All Pipes
```bash
-pipelex init config [--reset/-r]
+pipelex validate all
```
-### Validate group
+Performs comprehensive validation:
+
+1. Validates all library configurations
+2. Runs static validation on all discovered pipes
+3. Performs dry-run execution to check pipeline logic
-Validate configuration and pipelines.
+This is the recommended validation to run before committing changes or deploying pipelines.
+
+**Examples:**
```bash
+# Validate everything
pipelex validate all
+```
+
+### Validate Single Pipe
+
+```bash
pipelex validate pipe PIPE_CODE
```
-### Show group
+Validates and dry-runs a specific pipe, useful for iterative development.
+
+**Arguments:**
+
+- `PIPE_CODE` - The pipe code to validate (e.g., `analyze_cv_matching` or `write_weekly_report`)
+
+**Examples:**
+
+```bash
+# Validate a specific pipe
+pipelex validate pipe analyze_cv_matching
+pipelex validate pipe write_weekly_report
+```
+
+**What validation checks:**
+
+- Syntax correctness of `.plx` files
+- Concept and pipe definitions are valid
+- Input/output connections are correct
+- All referenced pipes and concepts exist
+- Dry-run execution succeeds without errors, which implies the logic is correct and the pipe can be run
+
+**Related Configuration:**
+
+- [Static Validation Configuration](../configuration/config-pipeline-validation/static-validation-config.md)
+- [Dry Run Configuration](../configuration/config-pipeline-validation/dry-run-config.md)
+
+## Show Commands
+
+Inspect your Pipelex configuration, pipelines, and available AI models.
+
+### Show Configuration
+
+```bash
+pipelex show config
+```
+
+Displays the current Pipelex configuration loaded from all sources (default config overriden by user config).
-Inspect configuration and pipes.
+**Examples:**
```bash
+# Display current configuration
pipelex show config
+```
+
+**Note:** This shows the main Pipelex configuration but not the inference backend details. Use `pipelex show backends` for backend configuration.
+
+### List All Pipes
+
+```bash
pipelex show pipes
+```
+
+Lists all pipes discovered in your project and imported packages, showing their pipe codes and basic information.
+
+**Examples:**
+
+```bash
+# List all available pipes
+pipelex show pipes
+```
+
+This includes:
+
+- Internal Pipelex pipes (like the pipe builder)
+- Pipes from your project's `.plx` files
+- Pipes that are part of imported packages
+
+### Show Pipe Definition
+
+```bash
pipelex show pipe PIPE_CODE
```
+Displays the complete definition of a specific pipe including inputs, outputs, prompts, model settings, and all configuration.
+
+**Arguments:**
+
+- `PIPE_CODE` - The pipe code to inspect
+
+**Examples:**
+
+```bash
+# Show pipe definition
+pipelex show pipe analyze_cv_matching
+pipelex show pipe write_weekly_report
+```
+
+### List AI Models
+
+```bash
+pipelex show models BACKEND_NAME [OPTIONS]
+```
+
+Lists all available models from a configured backend provider by querying the provider's API.
+
+**Arguments:**
+
+- `BACKEND_NAME` - The backend to query (e.g., `openai`, `anthropic`, `mistral`)
+
+**Options:**
+
+- `--flat`, `-f` - Output in flat CSV format for easy copying into other configuration files
+
+**Examples:**
+
+```bash
+# List models
+pipelex show models openai
+pipelex show models mistral
+
+# List models in flat format
+pipelex show models anthropic --flat
+```
+
+**Use case:** When configuring new models in your deck, use this command to see what models are available from each provider.
+
+**Related Configuration:**
+
+- [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md)
+
+### Show Backends
+
+```bash
+pipelex show backends [OPTIONS]
+```
+
+Displays all configured inference backends and the active routing profile with its routing rules.
+
+**Options:**
+
+- `--all`, `-a` - Show all backends including disabled ones (by default, only enabled backends are shown)
+
+**Examples:**
+
+```bash
+# Show enabled backends and routing profile
+pipelex show backends
+
+# Show all backends including disabled ones
+pipelex show backends --all
+```
+
+**What it displays:**
+
+- Table of configured backends with status, endpoint, and model count
+- Active routing profile name and description
+- Default backend for the profile
+- Routing rules mapping model patterns to backends
+
+**Related Configuration:**
+
+- [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md)
+- Backend configuration files: `.pipelex/inference/backends.toml`
+- Routing configuration: `.pipelex/inference/routing_profiles.toml`
+
+## Run Command
+
+Execute a pipeline with optional inputs and outputs.
+
+### Run a Pipeline
+
+```bash
+pipelex run [TARGET] [OPTIONS]
+```
+
+Executes a pipeline, either from a standalone bundle (.plx) file or from your project's pipe library.
+
+**Arguments:**
+
+- `TARGET` - Either a pipe code or a bundle file path, auto-detected according to presence of the .plx file extension
+
+**Options:**
+
+- `--pipe` - Pipe code to run (alternative to positional argument)
+- `--bundle` - Bundle file path (alternative to positional argument)
+- `--inputs`, `-i` - Path to JSON file containing input_memory
+- `--output`, `-o` - Path to save output JSON (defaults to `results/run_{pipe_code}.json`)
+- `--no-output` - Skip saving output to file
+- `--no-pretty-print` - Skip pretty printing the main output
+
+**Examples:**
+
+```bash
+# Run a pipe by code
+pipelex run hello_world
+
+# Run with inputs from JSON file
+pipelex run write_weekly_report --inputs weekly_report_data.json
+
+# Run a bundle file (uses its main_pipe)
+pipelex run my_bundle.plx
+
+# Run a specific pipe from a bundle
+pipelex run my_bundle.plx --pipe extract_invoice
+
+# Run with explicit options
+pipelex run --pipe hello_world --output my_output.json
+
+# Run without saving or pretty printing
+pipelex run my_pipe --no-output --no-pretty-print
+```
+
+**Input JSON Format:**
+
+The input JSON file should contain a dictionary where keys are input variable names:
+
+```json
+{
+ "input_variable": "simple string value",
+ "another_input": {
+ "concept": "domain.ConceptName",
+ "content": { "field": "value" }
+ }
+}
+```
+
+**Output Format:**
+
+The output JSON contains the complete working memory after pipeline execution, including all intermediate results and the final output.
+
+**Related Documentation:**
+
+- [Executing Pipelines with Inputs](../build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md)
+- [Design and Run Pipelines](../build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md)
+
+## Build Commands
+
+Generate pipelines and runner code from natural language descriptions. See the [Pipe Builder](pipe-builder.md) documentation for comprehensive details.
+
+### Build Pipe
+
+```bash
+pipelex build pipe "PROMPT" [OPTIONS]
+```
+
+Generates a complete pipeline from a natural language prompt with automatic validation and error correction.
+
+**Quick Example:**
+
+```bash
+pipelex build pipe "Analyze a CV and a Job offer and determine if they match" -o cv_matching_pipeline.plx
+```
+
+For complete documentation including all options and examples, see [Pipe Builder](pipe-builder.md).
+
+### Build Runner
+
+```bash
+pipelex build runner [TARGET] [OPTIONS]
+```
+
+Generates Python code to run a pipe with example inputs and all necessary imports.
+
+**Quick Example:**
+
+```bash
+pipelex build runner my_pipe -o run_my_pipe.py
+```
+
+For complete documentation, see the [Generate Runner Code section in Pipe Builder](pipe-builder.md#generate-runner-code).
+
+## Kit Commands
+
+Manage agent rules for AI coding assistants and sync migration instructions. See the [Kit Commands](kit.md) documentation for comprehensive details.
+
+### Quick Reference
+
+```bash
+# Install agent rules for AI assistants
+pipelex kit rules
+
+# Remove agent rules
+pipelex kit remove-rules
+
+# Install migration instructions
+pipelex kit migrations
+```
+
+For complete documentation including all options and examples, see [Kit Commands](kit.md).
+
## Usage Tips
-1. Always run `pipelex validate all` after making changes to your configuration or pipelines
-2. Use `pipelex show config` to debug configuration issues
-3. Use `pipelex show pipes` to see all discovered pipelines
-4. When initializing a new project:
+1. **Initial Setup**
- Run `pipelex init config` to create configuration files
- - Create your `.plx` pipeline files anywhere in your project
- - Validate your setup with `pipelex validate all`
+ - Configure your AI providers in `.pipelex/inference/backends.toml`
+ - Install agent rules with `pipelex kit rules` if using AI assistants
+
+2. **Development Workflow**
+ - Write or generate pipelines in `.plx` files
+ - Validate with `pipelex validate pipe your_pipe_code` during development
+ - Run `pipelex validate all` before committing changes, or pass a pipe code for faster iteration on a specific pipe
+
+3. **Running Pipelines**
+ - Use `pipelex show pipes` to see available pipes
+ - Use `pipelex show pipe pipe_code` to inspect pipe details
+ - Run with `pipelex run pipe_code`, add the required inputs using `--inputs`
+
+4. **Configuration Management**
+ - Use `pipelex show config` to verify current settings
+ - Use `pipelex show backends` to check inference backend setup
+ - Use `pipelex show models backend_name` to see available models
+
+## Related Documentation
+
+- [Pipe Builder](pipe-builder.md) - Generate pipelines from natural language
+- [Kit Commands](kit.md) - Agent rules and migration management
+- [Configure AI Providers](../setup/configure-ai-providers.md) - Set up LLM backends
+- [Design and Run Pipelines](../build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md) - Pipeline development guide
diff --git a/docs/pages/tools/pipe-builder.md b/docs/pages/tools/pipe-builder.md
index 1896659e8..9ab278ae5 100644
--- a/docs/pages/tools/pipe-builder.md
+++ b/docs/pages/tools/pipe-builder.md
@@ -52,6 +52,79 @@ The Pipe Builder follows this process:
4. **Validation**: Validates the generated pipeline structure
5. **Automatic Fixes**: Fixes common errors like missing inputs or incorrect pipe connections
+## Generate Runner Code
+
+After creating a pipeline, you can generate Python code to run it with the `build runner` command. This creates a ready-to-use Python script with all necessary imports and example input values.
+
+### Usage
+
+```bash
+pipelex build runner [TARGET] [OPTIONS]
+```
+
+**Arguments:**
+
+- `TARGET` - Either a pipe code or a bundle file path, auto-detected according to presence of the .plx file extension
+
+**Options:**
+
+- `--pipe` - Pipe code to use (alternative to positional argument)
+- `--bundle` - Bundle file path (alternative to positional argument)
+- `--output`, `-o` - Path to save the generated Python file (defaults to `results/run_{pipe_code}.py`)
+
+### Examples
+
+**Generate runner for a pipe:**
+
+```bash
+pipelex build runner my_pipe
+```
+
+**Generate runner from a bundle file:**
+
+```bash
+pipelex build runner my_bundle.plx
+```
+
+**Specify a pipe from a bundle:**
+
+```bash
+pipelex build runner --bundle my_bundle.plx --pipe my_pipe
+```
+
+**Custom output path:**
+
+```bash
+pipelex build runner my_pipe --output custom_runner.py
+```
+
+### What Gets Generated
+
+The generated Python file includes:
+
+1. **All necessary imports** - Imports for Pipelex execution, content types, and any custom structures
+2. **Input memory setup** - Example input values based on the pipe's input types
+3. **Pipeline execution** - Async function that executes the pipeline
+4. **Output handling** - Code to extract and display the results
+5. **Main execution block** - Pipelex initialization and asyncio setup
+
+### Input Type Handling
+
+- **Native concepts** (Text, Image, PDF, etc.) - Automatically generates appropriate content objects
+- **Custom concepts** - Recursively generates the structure with example values
+- **Structured content** - Creates example data matching the concept's fields
+
+### Next Steps
+
+After generating the runner file:
+
+1. Open the generated Python file
+2. Review and customize the example input values
+3. Run the script: `python results/run_{pipe_code}.py`
+4. Iterate and adjust as needed
+
+For the complete CLI reference, see [CLI Commands](cli.md).
+
## Example Use Cases
**Document Processing:**
diff --git a/generate_mascot_concept.json b/generate_mascot_concept.json
deleted file mode 100644
index a5758ecc0..000000000
--- a/generate_mascot_concept.json
+++ /dev/null
@@ -1,55 +0,0 @@
-{
- "root": {
- "elevator_pitch": {
- "stuff_code": "HkBMY",
- "stuff_name": "elevator_pitch",
- "concept": {
- "code": "Text",
- "domain": "native",
- "description": "A text",
- "structure_class_name": "TextContent",
- "refines": null
- },
- "content": {}
- },
- "mascot_characteristics": {
- "stuff_code": "gUStr",
- "stuff_name": "mascot_characteristics",
- "concept": {
- "code": "MascotCharacteristics",
- "domain": "startup_mascot_generation",
- "description": "Structured analysis of key attributes that should inform mascot design.",
- "structure_class_name": "MascotCharacteristics",
- "refines": null
- },
- "content": {}
- },
- "image_prompt": {
- "stuff_code": "BZoCP",
- "stuff_name": "image_prompt",
- "concept": {
- "code": "ImagePrompt",
- "domain": "startup_mascot_generation",
- "description": "A detailed textual description used to generate an image through AI image generation systems.",
- "structure_class_name": "TextContent",
- "refines": "native.Text"
- },
- "content": {}
- },
- "mascot_image": {
- "stuff_code": "Pe9HX",
- "stuff_name": "mascot_image",
- "concept": {
- "code": "MascotImage",
- "domain": "startup_mascot_generation",
- "description": "A visual representation of an animal character designed to represent a brand or organization.",
- "structure_class_name": "ImageContent",
- "refines": "native.Image"
- },
- "content": {}
- }
- },
- "aliases": {
- "main_stuff": "mascot_image"
- }
-}
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index d4012bdde..a3c58a396 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -112,6 +112,8 @@ nav:
- Define Your Concepts: pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md
- Structuring Concepts: pages/build-reliable-ai-workflows-with-pipelex/structuring-concepts.md
- Design and Run Pipelines: pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
+ - Executing Pipelines with Inputs: pages/build-reliable-ai-workflows-with-pipelex/executing-pipelines-with-inputs.md
+ - Understanding Multiplicity: pages/build-reliable-ai-workflows-with-pipelex/understanding-multiplicity.md
- Pipeline Creation: pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
- Pipe Operators:
- Overview: pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/index.md
diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.py
index ce224dfcd..3a992bbf6 100644
--- a/pipelex/cli/_cli.py
+++ b/pipelex/cli/_cli.py
@@ -3,12 +3,20 @@ from click import Command, Context
from typer.core import TyperGroup
from typing_extensions import override
-from pipelex.cli.commands import init_app, run_cmd, show_app, validate_app
from pipelex.cli.commands.build_cmd import build_app
+from pipelex.cli.commands.init_cmd import init_app
from pipelex.cli.commands.kit_cmd import kit_app
+from pipelex.cli.commands.run_cmd import run_cmd
+from pipelex.cli.commands.show_cmd import show_app
+from pipelex.cli.commands.validate_cmd import validate_app
class PipelexCLI(TyperGroup):
+ @override
+ def list_commands(self, ctx: Context) -> list[str]:
+ # List the commands in the proper order because natural ordering doesn't work between Typer groups and commands
+ return ["init", "kit", "build", "validate", "run", "show"]
+
@override
def get_command(self, ctx: Context, cmd_name: str) -> Command | None:
cmd = super().get_command(ctx, cmd_name)
@@ -32,9 +40,15 @@ app = typer.Typer(
)
-app.add_typer(init_app, name="init", help="Initialization commands")
-app.add_typer(validate_app, name="validate", help="Validation and dry-run commands")
-app.add_typer(show_app, name="show", help="Show and list commands")
-app.add_typer(build_app, name="build", help="Build artifacts like pipeline blueprints")
-app.add_typer(kit_app, name="kit", help="Manage kit assets")
-app.command(name="run", help="Execute a pipeline")(run_cmd)
+app.add_typer(init_app, name="init", help="Initialize Pipelex configuration in a `.pipelex` directory")
+app.add_typer(kit_app, name="kit", help="Manage kit assets: agent rules, migration rules")
+app.add_typer(
+ build_app, name="build", help="Generate AI workflows from natural language requirements: pipelines in .plx format and python code to run them"
+)
+app.add_typer(
+ validate_app,
+ name="validate",
+ help="Validate pipes: static validation for syntax and dependencies, dry-run execution for logic and consistency",
+)
+app.command(name="run", help="Run a pipe, optionally providing a specific bundle file (.plx)")(run_cmd)
+app.add_typer(show_app, name="show", help="Show configuration, pipes, and list AI models")
diff --git a/pipelex/cli/commands/__init__.py b/pipelex/cli/commands/__init__.py
index e92b3228a..e69de29bb 100644
--- a/pipelex/cli/commands/__init__.py
+++ b/pipelex/cli/commands/__init__.py
@@ -1,11 +0,0 @@
-"""Command groups for Pipelex CLI.
-
-This package organizes CLI commands into logical modules.
-"""
-
-from pipelex.cli.commands.init_cmd import init_app
-from pipelex.cli.commands.run_cmd import run_cmd
-from pipelex.cli.commands.show_cmd import show_app
-from pipelex.cli.commands.validate_cmd import validate_app
-
-__all__ = ["init_app", "run_cmd", "show_app", "validate_app"]
diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.py
index c2cc5f5cb..ebf3d875c 100644
--- a/pipelex/cli/commands/build_cmd.py
+++ b/pipelex/cli/commands/build_cmd.py
@@ -2,17 +2,20 @@ import asyncio
import time
from typing import Annotated
+import click
import typer
from pipelex import pretty_print
-from pipelex.builder.builder import PipelexBundleSpec
+from pipelex.builder.builder import PipelexBundleSpec, load_pipe_from_bundle
+from pipelex.builder.builder_errors import PipelexBundleError
from pipelex.builder.builder_loop import BuilderLoop
from pipelex.builder.runner_code import generate_runner_code
+from pipelex.exceptions import PipeInputError
from pipelex.hub import get_report_delegate, get_required_pipe
from pipelex.language.plx_factory import PlxFactory
from pipelex.pipelex import Pipelex
from pipelex.pipeline.execute import execute_pipeline
-from pipelex.tools.misc.file_utils import ensure_directory_for_file_path, save_text_to_path
+from pipelex.tools.misc.file_utils import ensure_directory_for_file_path, get_incremental_file_path, save_text_to_path
from pipelex.tools.misc.json_utils import save_as_json_to_path
build_app = typer.Typer(help="Build working pipelines from natural language requirements", no_args_is_help=True)
@@ -34,11 +37,11 @@ pipelex build pipe "Given an theme, write a Haiku"
"""
-@build_app.command("pipe", help="Generate a pipeline with one validation/fix loop corecting the deterministic issues")
+@build_app.command("pipe", help="Build a Pipelex bundle with one validation/fix loop correcting deterministic issues")
def build_pipe_cmd(
- brief: Annotated[
+ prompt: Annotated[
str,
- typer.Argument(help="Brief description of what the pipeline should do"),
+ typer.Argument(help="Prompt describing what the pipeline should do"),
],
output_path: Annotated[
str,
@@ -51,14 +54,14 @@ def build_pipe_cmd(
) -> None:
Pipelex.make()
typer.echo("=" * 70)
- typer.echo(typer.style("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN))
+ typer.secho("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN)
typer.echo("")
async def run_pipeline():
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW)
elif not output_path:
- typer.echo(typer.style("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED))
+ typer.secho("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED)
raise typer.Exit(1)
else:
ensure_directory_for_file_path(file_path=output_path)
@@ -66,23 +69,160 @@ def build_pipe_cmd(
builder_loop = BuilderLoop()
# Save to file unless explicitly disabled with --no-output
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW)
return
- pipelex_bundle_spec = await builder_loop.build_and_fix(pipe_code="pipe_builder", input_memory={"brief": brief})
+ pipelex_bundle_spec = await builder_loop.build_and_fix(pipe_code="pipe_builder", input_memory={"brief": prompt})
plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
save_text_to_path(text=plx_content, path=output_path)
- typer.echo(typer.style(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
start_time = time.time()
asyncio.run(run_pipeline())
end_time = time.time()
- typer.echo(typer.style(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN)
get_report_delegate().generate_report()
-@build_app.command("one-shot", help="Generate a pipeline in one shot without validation loop (fast but may need manual fixes)")
+@build_app.command("runner", help="Build the Python code to run a pipe with the necessary inputs")
+def prepare_runner_cmd(
+ target: Annotated[
+ str | None,
+ typer.Argument(help="Pipe code or bundle file path (auto-detected)"),
+ ] = None,
+ pipe: Annotated[
+ str | None,
+ typer.Option("--pipe", help="Pipe code to use, can be omitted if you specify a bundle (.plx) that declares a main pipe"),
+ ] = None,
+ bundle: Annotated[
+ str | None,
+ typer.Option("--bundle", help="Bundle file path (.plx) - uses its main_pipe unless you specify a pipe code"),
+ ] = None,
+ output_path: Annotated[
+ str | None,
+ typer.Option("--output", "-o", help="Path to save the generated Python file, defaults to 'results/run_{pipe_code}.py'"),
+ ] = None,
+) -> None:
+ """Prepare a Python runner file for a pipe.
+
+ The generated file will include:
+ - All necessary imports
+ - Example input values based on the pipe's input types
+
+ Native concept types (Text, Image, PDF, etc.) will be automatically handled.
+ Custom concept types will have their structure recursively generated.
+
+ Examples:
+ pipelex build runner my_pipe
+ pipelex build runner --bundle my_bundle.plx
+ pipelex build runner --bundle my_bundle.plx --pipe my_pipe
+ pipelex build runner my_bundle.plx
+ pipelex build runner my_pipe --output runner.py
+ """
+ # Validate mutual exclusivity
+ provided_options = sum([target is not None, pipe is not None, bundle is not None])
+ if provided_options == 0:
+ ctx: click.Context = click.get_current_context()
+ typer.echo(ctx.get_help())
+ raise typer.Exit(0)
+
+ # Let's analyze the options and determine what pipe code to use and if we need to load a bundle
+ pipe_code: str | None = None
+ bundle_path: str | None = None
+
+ # Determine source:
+ if target:
+ if target.endswith(".plx"):
+ bundle_path = target
+ if bundle:
+ typer.secho(
+ "Failed to run: cannot use option --bundle if you're already passing a bundle file (.plx) as positional argument",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+ else:
+ pipe_code = target
+ if pipe:
+ typer.secho(
+ "Failed to run: cannot use option --pipe if you're already passing a pipe code as positional argument",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+
+ if bundle:
+ assert not bundle_path, "bundle_path should be None at this stage if --bundle is provided"
+ bundle_path = bundle
+
+ if pipe:
+ assert not pipe_code, "pipe_code should be None at this stage if --pipe is provided"
+ pipe_code = pipe
+
+ if not pipe_code and not bundle_path:
+ typer.secho("Failed to run: no pipe code or bundle file specified", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1)
+
+ async def prepare_runner(pipe_code: str | None = None, bundle_path: str | None = None):
+ # Initialize Pipelex
+ Pipelex.make()
+
+ if bundle_path:
+ try:
+ main_pipe_code = await load_pipe_from_bundle(bundle_path)
+ if not pipe_code:
+ pipe_code = main_pipe_code
+ typer.echo(f"Using main pipe '{pipe_code}' from bundle '{bundle_path}'")
+ else:
+ typer.echo(f"Using pipe '{pipe_code}' from bundle '{bundle_path}'")
+ except FileNotFoundError as exc:
+ typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ except PipelexBundleError as exc:
+ typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ except PipeInputError as exc:
+ typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ elif not pipe_code:
+ typer.secho("Failed to run: no pipe code specified", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1)
+
+ # Get the pipe
+ try:
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ except Exception as exc:
+ typer.secho(f"❌ Error: Could not find pipe '{pipe_code}': {exc}", fg=typer.colors.RED)
+ raise typer.Exit(1) from exc
+
+ # Generate the code
+ try:
+ runner_code = generate_runner_code(pipe)
+ except Exception as exc:
+ typer.secho(f"❌ Error generating runner code: {exc}", fg=typer.colors.RED)
+ raise typer.Exit(1) from exc
+
+ # Determine output path
+ final_output_path = output_path or get_incremental_file_path(
+ base_path="results",
+ base_name=f"run_{pipe_code}",
+ extension="py",
+ )
+
+ # Save the file
+ try:
+ ensure_directory_for_file_path(file_path=final_output_path)
+ save_text_to_path(text=runner_code, path=final_output_path)
+ typer.secho(f"βœ… Generated runner file: {final_output_path}", fg=typer.colors.GREEN)
+ except Exception as exc:
+ typer.secho(f"❌ Error saving file: {exc}", fg=typer.colors.RED)
+ raise typer.Exit(1) from exc
+
+ asyncio.run(prepare_runner(pipe_code=pipe_code, bundle_path=bundle_path))
+
+
+@build_app.command("one-shot-pipe", help="Developer utility for contributors: deliver pipeline in one shot, without validation loop")
def build_one_shot_cmd(
brief: Annotated[
str,
@@ -99,14 +239,14 @@ def build_one_shot_cmd(
) -> None:
Pipelex.make()
typer.echo("=" * 70)
- typer.echo(typer.style("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN))
+ typer.secho("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN)
typer.echo("")
async def run_pipeline():
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW)
elif not output_path:
- typer.echo(typer.style("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED))
+ typer.secho("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED)
raise typer.Exit(1)
else:
ensure_directory_for_file_path(file_path=output_path)
@@ -119,23 +259,25 @@ def build_one_shot_cmd(
# Save to file unless explicitly disabled with --no-output
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW)
return
pipelex_bundle_spec = pipe_output.working_memory.get_stuff_as(name="pipelex_bundle_spec", content_type=PipelexBundleSpec)
plx_content = PlxFactory.make_plx_content(blueprint=pipelex_bundle_spec.to_blueprint())
save_text_to_path(text=plx_content, path=output_path)
- typer.echo(typer.style(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
start_time = time.time()
asyncio.run(run_pipeline())
end_time = time.time()
- typer.echo(typer.style(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN)
get_report_delegate().generate_report()
-@build_app.command("partial", help="Generate a partial pipeline specification and save it as JSON (for debugging)")
+@build_app.command(
+ "partial-pipe", help="Developer utility for contributors: deliver a partial pipeline specification (not an actual bundle) and save it as JSON"
+)
def build_partial_cmd(
brief: Annotated[
str,
@@ -152,14 +294,14 @@ def build_partial_cmd(
) -> None:
Pipelex.make()
typer.echo("=" * 70)
- typer.echo(typer.style("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN))
+ typer.secho("πŸ”₯ Starting pipe builder... πŸš€", fg=typer.colors.GREEN)
typer.echo("")
async def run_pipeline():
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline will not be saved to file (--no-output specified)", fg=typer.colors.YELLOW)
elif not output_path:
- typer.echo(typer.style("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED))
+ typer.secho("\nπŸ›‘ Cannot save a pipeline to an empty file name", fg=typer.colors.RED)
raise typer.Exit(1)
else:
ensure_directory_for_file_path(file_path=output_path)
@@ -170,63 +312,15 @@ def build_partial_cmd(
)
# Save to file unless explicitly disabled with --no-output
if no_output:
- typer.echo(typer.style("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW))
+ typer.secho("\n⚠️ Pipeline not saved to file (--no-output specified)", fg=typer.colors.YELLOW)
return
json_output = pipe_output.main_stuff.content.smart_dump()
save_as_json_to_path(object_to_save=json_output, path=output_path)
- typer.echo(typer.style(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline saved to: {output_path}", fg=typer.colors.GREEN)
start_time = time.time()
asyncio.run(run_pipeline())
end_time = time.time()
- typer.echo(typer.style(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN))
+ typer.secho(f"\nβœ… Pipeline built in {end_time - start_time:.2f} seconds", fg=typer.colors.GREEN)
get_report_delegate().generate_report()
-
-
-@build_app.command("prepare", help="Prepare a Python runner file for a pipe")
-def prepare_runner_cmd(
- pipe_code: Annotated[str, typer.Argument(help="The pipe code to prepare a runner for")],
- output_path: Annotated[
- str | None,
- typer.Option("--output", "-o", help="Path to save the generated Python file"),
- ] = None,
-) -> None:
- """Prepare a Python runner file for a pipe.
-
- The generated file will include:
- - All necessary imports
- - Example input values based on the pipe's input types
-
- Native concept types (Text, Image, PDF, etc.) will be automatically handled.
- Custom concept types will have their structure recursively generated.
- """
- # Initialize Pipelex
- Pipelex.make()
-
- # Get the pipe
- try:
- pipe = get_required_pipe(pipe_code=pipe_code)
- except Exception as e:
- typer.echo(typer.style(f"❌ Error: Could not find pipe '{pipe_code}': {e}", fg=typer.colors.RED))
- raise typer.Exit(1) from e
-
- # Generate the code
- try:
- runner_code = generate_runner_code(pipe)
- except Exception as e:
- typer.echo(typer.style(f"❌ Error generating runner code: {e}", fg=typer.colors.RED))
- raise typer.Exit(1) from e
-
- # Determine output path
- if not output_path:
- output_path = f"run_{pipe_code}.py"
-
- # Save the file
- try:
- ensure_directory_for_file_path(file_path=output_path)
- save_text_to_path(text=runner_code, path=output_path)
- typer.echo(typer.style(f"βœ… Generated runner file: {output_path}", fg=typer.colors.GREEN))
- except Exception as e:
- typer.echo(typer.style(f"❌ Error saving file: {e}", fg=typer.colors.RED))
- raise typer.Exit(1) from e
diff --git a/pipelex/cli/commands/init_cmd.py b/pipelex/cli/commands/init_cmd.py
index 75083607d..e949fce2f 100644
--- a/pipelex/cli/commands/init_cmd.py
+++ b/pipelex/cli/commands/init_cmd.py
@@ -62,10 +62,10 @@ def do_init_config(reset: bool = False) -> None:
# Typer group for init commands
-init_app = typer.Typer(help="Initialization commands", no_args_is_help=True)
+init_app = typer.Typer(no_args_is_help=True)
-@init_app.command("config")
+@init_app.command("config", help="Initialize Pipelex configuration in a `.pipelex` directory")
def init_config_cmd(
reset: Annotated[bool, typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten.")] = False,
) -> None:
diff --git a/pipelex/cli/commands/kit_cmd.py b/pipelex/cli/commands/kit_cmd.py
index 1f68b7c06..de382318f 100644
--- a/pipelex/cli/commands/kit_cmd.py
+++ b/pipelex/cli/commands/kit_cmd.py
@@ -11,10 +11,10 @@ from pipelex.kit.index_loader import load_index
from pipelex.kit.migrations_export import export_migration_instructions
from pipelex.kit.targets_update import build_merged_rules, remove_from_targets, update_targets
-kit_app = typer.Typer(help="Manage kit assets: export Cursor rules and merge agent docs", no_args_is_help=True)
+kit_app = typer.Typer(no_args_is_help=True)
-@kit_app.command("rules")
+@kit_app.command("rules", help="Export Pipelex Cursor rules and merge Pipelex marked sections into other agent rules files")
def agent_rules(
repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,
cursor: Annotated[bool, typer.Option("--cursor/--no-cursor", help="Export Cursor rules to .cursor/rules")] = True,
@@ -23,12 +23,6 @@ def agent_rules(
diff: Annotated[bool, typer.Option("--diff", help="Show unified diff of changes")] = False,
backup: Annotated[str | None, typer.Option("--backup", help="Backup suffix (e.g., '.bak')")] = None,
) -> None:
- """Sync kit assets: export Cursor rules and merge agent documentation.
-
- This command:
- 1. Exports agent markdown files to Cursor .mdc files with YAML front-matter
- 2. Builds merged agent documentation and updates target files using markers
- """
try:
if repo_root is None:
repo_root = Path()
@@ -55,7 +49,9 @@ def agent_rules(
raise PipelexCLIError(msg) from exc
-@kit_app.command("remove-rules")
+@kit_app.command(
+ "remove-rules", help="Remove agent rules: delete Pipelex Cursor rules and remove Pipelex marked sections from other agent rules files"
+)
def remove_rules(
repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,
cursor: Annotated[bool, typer.Option("--cursor/--no-cursor", help="Remove Cursor rules from .cursor/rules")] = True,
@@ -65,12 +61,6 @@ def remove_rules(
diff: Annotated[bool, typer.Option("--diff", help="Show unified diff of changes")] = False,
backup: Annotated[str | None, typer.Option("--backup", help="Backup suffix (e.g., '.bak')")] = None,
) -> None:
- """Remove agent rules: delete Cursor rules and remove marked sections from target files.
-
- This command:
- 1. Deletes agent markdown files from Cursor .mdc files in .cursor/rules
- 2. Removes marked sections from target files (or deletes entire files with --delete-files)
- """
try:
if repo_root is None:
repo_root = Path()
@@ -105,16 +95,11 @@ def remove_rules(
raise PipelexCLIError(msg) from exc
-@kit_app.command("migrations")
+@kit_app.command("migrations", help="Sync Pipelex migration instructions to the `.pipelex/migrations` directory")
def migration_instructions(
repo_root: Annotated[Path | None, typer.Option("--repo-root", dir_okay=True, writable=True, help="Repository root directory")] = None,
dry_run: Annotated[bool, typer.Option("--dry-run", help="Show what would be done without making changes")] = False,
) -> None:
- """Sync migration instructions from kit to .pipelex/migrations.
-
- This command copies migration documentation files from the pipelex.kit
- package to the user's .pipelex/migrations directory.
- """
try:
if repo_root is None:
repo_root = Path()
diff --git a/pipelex/cli/commands/run_cmd.py b/pipelex/cli/commands/run_cmd.py
index 9278ea1ce..b69f7f26f 100644
--- a/pipelex/cli/commands/run_cmd.py
+++ b/pipelex/cli/commands/run_cmd.py
@@ -3,6 +3,7 @@ from __future__ import annotations
import asyncio
from typing import Annotated
+import click
import typer
from pipelex import log, pretty_print_md
@@ -22,11 +23,11 @@ def run_cmd(
] = None,
pipe: Annotated[
str | None,
- typer.Option("--pipe", help="Explicitly specify pipe code to run"),
+ typer.Option("--pipe", help="Pipe code to run, can be omitted if you specify a bundle (.plx) that declares a main pipe"),
] = None,
bundle: Annotated[
str | None,
- typer.Option("--bundle", help="Bundle file path (.plx) - runs its main_pipe"),
+ typer.Option("--bundle", help="Bundle file path (.plx) - runs its main_pipe unless you specify a pipe code"),
] = None,
inputs: Annotated[
str | None,
@@ -34,7 +35,7 @@ def run_cmd(
] = None,
output: Annotated[
str | None,
- typer.Option("--output", "-o", help="Path to save output JSON (default: {pipe_code}.json)"),
+ typer.Option("--output", "-o", help="Path to save output JSON, default to '{pipe_code}.json'"),
] = None,
no_output: Annotated[
bool,
@@ -45,59 +46,74 @@ def run_cmd(
typer.Option("--no-pretty-print", help="Skip pretty printing the main_stuff"),
] = False,
) -> None:
- """Execute a pipeline by pipe code or bundle file.
+ """Execute a pipeline from a specific bundle file (or not), specifying its pipe code or not.
+ If the bundle is provided, it will run its main pipe unless you specify a pipe code.
+ If the pipe code is provided, you don't need to provide a bundle file if it's already part of the imported packages.
Examples:
pipelex run my_pipe
- pipelex run --pipe my_pipe --inputs data.json
pipelex run --bundle my_bundle.plx
+ pipelex run --bundle my_bundle.plx --pipe my_pipe
+ pipelex run --pipe my_pipe --inputs data.json
pipelex run my_bundle.plx --inputs data.json
pipelex run my_pipe --output results.json --no-pretty-print
"""
- # Initialize Pipelex
- Pipelex.make()
-
# Validate mutual exclusivity
provided_options = sum([target is not None, pipe is not None, bundle is not None])
if provided_options == 0:
- typer.secho("Failed to run: must provide a pipe code, bundle file, or target", fg=typer.colors.RED, err=True)
- raise typer.Exit(1)
- if provided_options > 1:
- typer.secho(
- "Failed to run: cannot use multiple options (--pipe, --bundle, or positional target) simultaneously",
- fg=typer.colors.RED,
- err=True,
- )
- raise typer.Exit(1)
-
- async def run_pipeline():
- # Initialize to satisfy linter (will be assigned before use)
- pipe_code: str = ""
- source_description: str = ""
- bundle_path: str | None = None
-
- # Determine source: bundle path or pipe code
- if bundle:
- bundle_path = bundle
- elif pipe:
- pipe_code = pipe
- source_description = f"pipe '{pipe_code}'"
- elif target:
- if target.endswith(".plx"):
- bundle_path = target
- else:
- pipe_code = target
- source_description = f"pipe '{pipe_code}'"
+ ctx: click.Context = click.get_current_context()
+ typer.echo(ctx.get_help())
+ raise typer.Exit(0)
+
+ # Let's analyze the options and determine what pipe code to use and if we need to load a bundle
+ pipe_code: str | None = None
+ bundle_path: str | None = None
+
+ # Determine source:
+ if target:
+ if target.endswith(".plx"):
+ bundle_path = target
+ if bundle:
+ typer.secho(
+ "Failed to run: cannot use option --bundle if you're already passing a bundle file (.plx) as positional argument",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
else:
- # Should never reach here due to validation above
- typer.secho("Failed to run: no pipe or bundle specified", fg=typer.colors.RED, err=True)
- raise typer.Exit(1)
+ pipe_code = target
+ if pipe:
+ typer.secho(
+ "Failed to run: cannot use option --pipe if you're already passing a pipe code as positional argument",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+
+ if bundle:
+ assert not bundle_path, "bundle_path should be None at this stage if --bundle is provided"
+ bundle_path = bundle
+
+ if pipe:
+ assert not pipe_code, "pipe_code should be None at this stage if --pipe is provided"
+ pipe_code = pipe
+
+ if not pipe_code and not bundle_path:
+ typer.secho("Failed to run: no pipe code or bundle file specified", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1)
- # Load bundle if needed
+ async def run_pipeline(pipe_code: str | None = None, bundle_path: str | None = None):
+ # Initialize Pipelex
+ Pipelex.make()
+ source_description: str
if bundle_path:
try:
- pipe_code = await load_pipe_from_bundle(bundle_path)
- source_description = f"bundle '{bundle_path}' β€’ main_pipe: '{pipe_code}'"
+ main_pipe_code = await load_pipe_from_bundle(bundle_path)
+ if not pipe_code:
+ pipe_code = main_pipe_code
+ source_description = f"bundle '{bundle_path}' β€’ main pipe: '{pipe_code}'"
+ else:
+ source_description = f"bundle '{bundle_path}' β€’ pipe: '{pipe_code}'"
except FileNotFoundError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
@@ -107,6 +123,11 @@ def run_cmd(
except PipeInputError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
+ elif pipe_code:
+ source_description = f"pipe '{pipe_code}'"
+ else:
+ typer.secho("Failed to run: no pipe code specified", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1)
try:
# Load inputs if provided
@@ -145,13 +166,13 @@ def run_cmd(
)
working_memory_dict = pipe_output.working_memory.model_dump()
save_as_json_to_path(object_to_save=working_memory_dict, path=output_path)
- typer.echo(typer.style(f"βœ… Working memory saved to: {output_path}", fg=typer.colors.GREEN))
+ typer.secho(f"βœ… Working memory saved to: {output_path}", fg=typer.colors.GREEN)
- typer.echo(typer.style("βœ… Pipeline execution completed successfully", fg=typer.colors.GREEN))
+ typer.secho("βœ… Pipeline execution completed successfully", fg=typer.colors.GREEN)
except Exception as exc:
log.error(f"Error executing pipeline: {exc}")
typer.secho(f"Failed to execute pipeline: {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- asyncio.run(run_pipeline())
+ asyncio.run(run_pipeline(pipe_code=pipe_code, bundle_path=bundle_path))
diff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.py
index 43cc0a99f..5222e42b6 100644
--- a/pipelex/cli/commands/show_cmd.py
+++ b/pipelex/cli/commands/show_cmd.py
@@ -1,17 +1,24 @@
from __future__ import annotations
import asyncio
-from typing import Annotated
+from typing import TYPE_CHECKING, Annotated, cast
import typer
+from rich import box
+from rich.console import Console
+from rich.table import Table
from pipelex import pretty_print
+from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
from pipelex.cogt.model_backends.model_lists import ModelLister
from pipelex.exceptions import PipelexCLIError, PipelexConfigError
-from pipelex.hub import get_pipe_library, get_required_pipe
+from pipelex.hub import get_models_manager, get_pipe_library, get_required_pipe
from pipelex.pipelex import Pipelex
from pipelex.system.configuration.config_loader import config_manager
+if TYPE_CHECKING:
+ from pipelex.cogt.models.model_manager import ModelManager
+
def do_show_config() -> None:
"""Show the pipelex configuration."""
@@ -41,38 +48,181 @@ def do_show_pipe(pipe_code: str) -> None:
pretty_print(pipe, title=f"Pipe '{pipe_code}'")
+def do_show_backends(show_all: bool = False) -> None:
+ """Display all backends and the active routing profile."""
+ Pipelex.make()
+
+ try:
+ models_manager = cast("ModelManager", get_models_manager())
+
+ # Load backends with or without disabled ones based on show_all flag
+ if show_all:
+ backend_library = InferenceBackendLibrary()
+ backend_library.load(include_disabled=True)
+ else:
+ backend_library = models_manager.inference_backend_library
+
+ routing_profile_library = models_manager.routing_profile_library
+ except Exception as exc:
+ msg = f"Error accessing backend or routing configuration: {exc}"
+ raise PipelexCLIError(msg) from exc
+
+ console = Console()
+
+ # Get all backends
+ all_backends = list(backend_library.root.values())
+ if not all_backends:
+ console.print("[yellow]No backends configured.[/yellow]")
+ return
+
+ # Filter backends based on show_all flag
+ backends_to_display = all_backends if show_all else [b for b in all_backends if b.enabled]
+
+ # Display backends table
+ table_title = "All Configured Backends" if show_all else "Enabled Backends"
+ backends_table = Table(
+ title=table_title,
+ show_header=True,
+ header_style="bold cyan",
+ box=box.SQUARE_DOUBLE_HEAD,
+ )
+ backends_table.add_column("Backend Name", style="green")
+ if show_all:
+ backends_table.add_column("Status", style="yellow")
+ backends_table.add_column("Endpoint", style="blue")
+ backends_table.add_column("Models", style="cyan", justify="right")
+
+ for backend in sorted(backends_to_display, key=lambda b: b.name):
+ endpoint = backend.endpoint if backend.endpoint else "[dim]N/A[/dim]"
+ model_count = str(len(backend.model_specs))
+
+ if show_all:
+ status = "[green]Enabled[/green]" if backend.enabled else "[red]Disabled[/red]"
+ backends_table.add_row(backend.name, status, endpoint, model_count)
+ else:
+ backends_table.add_row(backend.name, endpoint, model_count)
+
+ console.print("\n")
+ console.print(backends_table)
+ console.print("\n")
+
+ # Display routing profile information
+ try:
+ active_profile = routing_profile_library.active_profile
+
+ console.print(f"[bold cyan]Active Routing Profile:[/bold cyan] [green]{active_profile.name}[/green]")
+ if active_profile.description:
+ console.print(f"[dim]{active_profile.description}[/dim]")
+
+ if active_profile.default:
+ console.print(f"[bold]Default Backend:[/bold] [cyan]{active_profile.default}[/cyan]")
+
+ # Display routing rules
+ if active_profile.routes:
+ console.print("\n[bold]Routing Rules:[/bold]")
+ routes_table = Table(
+ show_header=True,
+ header_style="bold cyan",
+ box=box.SIMPLE,
+ show_edge=False,
+ )
+ routes_table.add_column("Pattern", style="green")
+ routes_table.add_column("β†’", style="dim", justify="center")
+ routes_table.add_column("Target Backend", style="cyan")
+
+ for pattern, target_backend in sorted(active_profile.routes.items()):
+ routes_table.add_row(pattern, "β†’", target_backend)
+
+ console.print(routes_table)
+ else:
+ console.print("[dim]No specific routing rules defined.[/dim]")
+
+ except Exception as exc:
+ console.print(f"[yellow]Warning: Could not load routing profile information: {exc}[/yellow]")
+
+ console.print("\n")
+
+ # Display helper messages
+ if not show_all:
+ enabled_count = len([b for b in all_backends if b.enabled])
+ disabled_count = len(all_backends) - enabled_count
+ if disabled_count > 0:
+ console.print(f"[dim]πŸ’‘ Showing {enabled_count} enabled backend(s). {disabled_count} disabled backend(s) hidden.[/dim]")
+ console.print("[dim] To see all backends: [bold]pipelex show backends --all[/bold][/dim]\n")
+
+ console.print("[dim]πŸ’‘ To enable more backends, edit: [bold].pipelex/inference/backends.toml[/bold][/dim]")
+ console.print("[dim]πŸ’‘ To list available models for a backend: [bold]pipelex show models <backend_name>[/bold][/dim]\n")
+
+
# Typer group for show commands
-show_app = typer.Typer(help="Show and list commands", no_args_is_help=True)
+show_app = typer.Typer(
+ no_args_is_help=True,
+)
-@show_app.command("config")
+@show_app.command("config", help="Display the main Pipelex configuration (not including inference backends)")
def show_config_cmd() -> None:
do_show_config()
-@show_app.command("pipes")
+@show_app.command("pipes", help="List all available pipes in the current project")
def list_pipes_cmd() -> None:
+ """List all pipes that have been loaded into the pipe library.
+
+ This includes pipes from your project's .plx files and any
+ pipes from imported packages.
+ """
do_list_pipes()
-@show_app.command("pipe")
+@show_app.command("pipe", help="Display the detailed definition of a specific pipe")
def show_pipe_cmd(
- pipe_code: Annotated[str, typer.Argument(help="Pipeline code to show definition for")],
+ pipe_code: Annotated[str, typer.Argument(help="Pipeline code to show definition for (e.g., 'my_domain.my_pipe')")],
) -> None:
+ """Show the complete definition of a pipe including its inputs, outputs,
+ prompt, and all configuration settings.
+
+ Example:
+ pipelex show pipe hello_world
+ """
do_show_pipe(pipe_code=pipe_code)
-@show_app.command("models")
+@show_app.command("models", help="List available AI models from a specific backend provider")
def show_models_cmd(
- backend_name: Annotated[str, typer.Argument(help="Backend name to list models for")],
+ backend_name: Annotated[str, typer.Argument(help="Backend name to list models for (e.g., 'openai', 'anthropic', 'google')")],
flat: Annotated[
bool,
- typer.Option("--flat", "-f", help="Output in flat CSV format for easy copy-pasting"),
+ typer.Option("--flat", "-f", help="Output in flat CSV format for easy copy-pasting into configuration files"),
] = False,
) -> None:
+ """List all available models from a configured backend provider.
+
+ This queries the backend's API to retrieve the current list of available models.
+ Use --flat for a simplified output that's easy to copy into config files.
+
+ Examples:
+ pipelex show models openai
+ pipelex show models anthropic --flat
+ """
asyncio.run(
ModelLister.list_models(
backend_name=backend_name,
flat=flat,
)
)
+
+
+@show_app.command("backends", help="Display backend configurations and active routing profile")
+def show_backends_cmd(
+ show_all_backends: Annotated[bool, typer.Option("--all", "-a", help="Show all backends including disabled ones")] = False,
+) -> None:
+ """Display all configured backends and the active routing profile with its routing rules.
+
+ By default, shows only enabled backends. Use --all to include disabled backends.
+
+ Examples:
+ pipelex show backends
+ pipelex show backends --all
+ """
+ do_show_backends(show_all=show_all_backends)
diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.py
index 6deb49c50..d96cce0dc 100644
--- a/pipelex/cli/commands/validate_cmd.py
+++ b/pipelex/cli/commands/validate_cmd.py
@@ -34,16 +34,16 @@ def do_dry_run_pipe(pipe_code: str) -> None:
# Typer group for validation commands
-validate_app = typer.Typer(help="Validation and dry-run commands", no_args_is_help=True)
+validate_app = typer.Typer(no_args_is_help=True)
-@validate_app.command("all")
-def validate_all_cmd() -> None:
- do_validate_all_libraries_and_dry_run()
-
-
-@validate_app.command("pipe")
+@validate_app.command("pipe", help="Validate and dry run a single pipe")
def dry_run_pipe_cmd(
pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],
) -> None:
do_dry_run_pipe(pipe_code=pipe_code)
+
+
+@validate_app.command("all", help="Validate and dry run all pipes")
+def validate_all_cmd() -> None:
+ do_validate_all_libraries_and_dry_run()
diff --git a/pipelex/cogt/model_backends/backend.py b/pipelex/cogt/model_backends/backend.py
index f3db3983d..9f019fad6 100644
--- a/pipelex/cogt/model_backends/backend.py
+++ b/pipelex/cogt/model_backends/backend.py
@@ -8,6 +8,7 @@ from pipelex.system.configuration.config_model import ConfigModel
class InferenceBackend(ConfigModel):
name: str
+ enabled: bool = True
endpoint: str | None = None
api_key: str | None = None
extra_config: dict[str, Any] = Field(default_factory=dict)
diff --git a/pipelex/cogt/model_backends/backend_factory.py b/pipelex/cogt/model_backends/backend_factory.py
index ba8e1150f..9e458f01e 100644
--- a/pipelex/cogt/model_backends/backend_factory.py
+++ b/pipelex/cogt/model_backends/backend_factory.py
@@ -34,6 +34,7 @@ class InferenceBackendFactory:
pass
return InferenceBackend(
name=name,
+ enabled=blueprint.enabled,
endpoint=endpoint,
api_key=api_key,
extra_config=extra_config,
diff --git a/pipelex/cogt/model_backends/backend_library.py b/pipelex/cogt/model_backends/backend_library.py
index f481fb515..b5a2c1d2c 100644
--- a/pipelex/cogt/model_backends/backend_library.py
+++ b/pipelex/cogt/model_backends/backend_library.py
@@ -36,7 +36,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
def make_empty(cls) -> Self:
return cls(root={})
- def load(self):
+ def load(self, include_disabled: bool = False):
backends_library_path = get_config().cogt.inference_config.backends_library_path
try:
backends_dict = load_toml_from_path(path=backends_library_path)
@@ -51,7 +51,8 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
standard_fields = InferenceBackendBlueprint.model_fields.keys()
extra_config: dict[str, Any] = {}
inference_backend_blueprint_dict_raw = backend_dict.copy()
- if not inference_backend_blueprint_dict_raw.get("enabled", True):
+ enabled = inference_backend_blueprint_dict_raw.get("enabled", True)
+ if not enabled and not include_disabled:
continue
if runtime_manager.is_ci_testing and backend_name == "vertexai":
continue
diff --git a/pipelex/core/pipes/input_requirements.py b/pipelex/core/pipes/input_requirements.py
index b3a813bff..fd202969f 100644
--- a/pipelex/core/pipes/input_requirements.py
+++ b/pipelex/core/pipes/input_requirements.py
@@ -109,7 +109,6 @@ class InputRequirements(RootModel[InputRequirementsRoot]):
the_requirements: list[NamedInputRequirement] = []
for requirement_expression, requirement in self.root.items():
required_variable_name = requirement_expression.split(".", 1)[0]
- # TODO: refactor this with a proper class like InputRequirement
the_requirements.append(
NamedInputRequirement(
variable_name=required_variable_name,
diff --git a/pipelex/core/pipes/pipe_library.py b/pipelex/core/pipes/pipe_library.py
index 47128d78d..bd51bbd74 100644
--- a/pipelex/core/pipes/pipe_library.py
+++ b/pipelex/core/pipes/pipe_library.py
@@ -59,7 +59,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
def get_required_pipe(self, pipe_code: str) -> PipeAbstract:
the_pipe = self.get_optional_pipe(pipe_code=pipe_code)
if not the_pipe:
- msg = f"Pipe '{pipe_code}' not found. Check for typos and make sure it is declared in a library listed in the config."
+ msg = f"Pipe '{pipe_code}' not found. Check for typos and make sure it is declared in plx file in an imported package."
raise PipeLibraryPipeNotFoundError(msg)
return the_pipe
diff --git a/pipelex/exceptions.py b/pipelex/exceptions.py
index 704fd178f..e80470f33 100644
--- a/pipelex/exceptions.py
+++ b/pipelex/exceptions.py
@@ -266,6 +266,7 @@ class PipeStackOverflowError(PipeExecutionError):
pass
+# TODO: create variants of DryRunError for each type of Pipe
class DryRunError(PipeExecutionError):
"""Raised when a dry run fails due to missing inputs or other validation issues."""
diff --git a/pipelex/kit/__init__.py b/pipelex/kit/__init__.py
index e69de29bb..8b1378917 100644
--- a/pipelex/kit/__init__.py
+++ b/pipelex/kit/__init__.py
@@ -0,0 +1 @@
+
diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.md
index 9bd022ede..bbbcd80ae 100644
--- a/pipelex/kit/agent_rules/python_standards.md
+++ b/pipelex/kit/agent_rules/python_standards.md
@@ -7,6 +7,7 @@ This document outlines the core coding standards, best practices, and quality co
- Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
- When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
- Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
## Enums and tests
diff --git a/pipelex/kit/agent_rules/write_pipelex.md b/pipelex/kit/agent_rules/write_pipelex.md
index 4237b9c18..dfb1ea6f5 100644
--- a/pipelex/kit/agent_rules/write_pipelex.md
+++ b/pipelex/kit/agent_rules/write_pipelex.md
@@ -82,6 +82,26 @@ inputs = {
- `output`: The name of the concept to output. The `ConceptName` should have the same name as the python class if you want structured output:
+### Input Multiplicity
+
+By default, inputs expect a single item. Use expanded syntax to specify multiple items:
+
+```plx
+# Single item (default) - use standard syntax
+inputs = { document = "Text" }
+
+# Variable list - indeterminate number of items
+inputs = { documents = { concept = "Text", multiplicity = true } }
+
+# Fixed count - exactly N items
+inputs = { comparison_items = { concept = "Image", multiplicity = 2 } }
+```
+
+**Key points:**
+- Don't use `multiplicity = false` - it's unnecessary (default behavior)
+- Use `multiplicity = true` for lists of unknown length
+- Use `multiplicity = N` (integer > 1) when operation requires exact count (e.g., comparing 2 items)
+
## Structuring Models
Once you've defined your concepts semantically (see "Concept Definitions" above), you need to specify their structure if they have fields.
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 554eaccb4..bfe5536a5 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -10,7 +10,6 @@ from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
-from pipelex.core.pipes.input_requirement_blueprint import InputRequirementBlueprint
from pipelex.core.pipes.input_requirements import InputRequirements
from pipelex.core.pipes.input_requirements_factory import InputRequirementsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
@@ -284,14 +283,6 @@ class PipeCondition(PipeController):
# TODO: restore pipe_layer feature
# pipe_run_params.push_pipe_code(pipe_code=pipe_code)
- # Convert PipeInput to blueprint format
- inputs_blueprint: dict[str, str | InputRequirementBlueprint] = {}
- for var_name, requirement in self.inputs.root.items():
- inputs_blueprint[var_name] = InputRequirementBlueprint(
- concept=requirement.concept.concept_string,
- multiplicity=requirement.multiplicity,
- )
-
evaluated_expression = await self._evaluate_expression(working_memory=working_memory)
# Select the outcome based on the evaluated expression
diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.py
index 601d92762..56a0f89d2 100644
--- a/pipelex/pipe_operators/func/pipe_func.py
+++ b/pipelex/pipe_operators/func/pipe_func.py
@@ -114,6 +114,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
msg = f"Function '{self.function_name}' return type {return_type} is not a subclass of StuffContent"
raise DryRunError(msg)
+ # TODO: Support PipeFunc returning with multiplicity. Create an equivalent of TypedNamedInputRequirement for outputs.
requirement = TypedNamedInputRequirement(
variable_name="mock_output",
concept=ConceptFactory.make(