Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix/update-doc-and-cli (#511)

Thomas Hebrard4b08cfa065d2
diff --git a/docs/home/2-get-started/pipe-builder.md b/docs/home/2-get-started/pipe-builder.md
index 94665b4d1..7603570dd 100644
--- a/docs/home/2-get-started/pipe-builder.md
+++ b/docs/home/2-get-started/pipe-builder.md
@@ -36,6 +36,9 @@ The pipe builder generates three files in a numbered directory (e.g., `results/p
!!! tip "Pipe Builder Requirements"
For now, the pipe builder requires access to **Claude 4.5 Sonnet**, either through Pipelex Inference, or using your own key through Anthropic, Amazon Bedrock or BlackboxAI. Don't hesitate to join our [Discord](https://go.pipelex.com/discord) to get a key, otherwise, you can also create the workflows yourself, following our [documentation guide](./write-workflows-manually.md).
+!!! info "Learn More"
+ Want to understand how the Pipe Builder works under the hood? See [Pipe Builder Deep Dive](../9-tools/pipe-builder.md) for the full explanation of its multi-step generation process.
+
## Run your pipeline
**Option 1: CLI**
diff --git a/docs/home/2-get-started/write-workflows-manually.md b/docs/home/2-get-started/write-workflows-manually.md
index 7f87cf868..029951b89 100644
--- a/docs/home/2-get-started/write-workflows-manually.md
+++ b/docs/home/2-get-started/write-workflows-manually.md
@@ -341,8 +341,8 @@ Now that you understand the basics, explore more:
**Explore Tools:**
- [Pipe Builder](../../home/9-tools/pipe-builder.md) - Generate pipelines from natural language
-- [Kit Commands](../../home/9-tools/kit.md) - Manage agent rules and migrations
-- [CLI Commands](../../home/9-tools/cli.md) - Command-line interface reference
+- [Kit Commands](../9-tools/cli/kit.md) - Manage agent rules and migrations
+- [CLI Commands](../9-tools/cli/index.md) - Command-line interface reference
**Configure:**
diff --git a/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md b/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
index 21f1850a1..eed130300 100644
--- a/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
+++ b/docs/home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md
@@ -18,7 +18,7 @@ Then you can execute the pipeline with the CLI.
pipelex run path/to/my_pipe.plx --inputs results/inputs.json
```
-See more about the CLI [here](../../../home/9-tools/cli.md).
+See more about the CLI [here](../../9-tools/cli/index.md).
## Option 2: Using the python method `execute_pipeline`
diff --git a/docs/home/6-build-reliable-ai-workflows/pipes/provide-inputs.md b/docs/home/6-build-reliable-ai-workflows/pipes/provide-inputs.md
index e09a7fa1a..bbc55eaf1 100644
--- a/docs/home/6-build-reliable-ai-workflows/pipes/provide-inputs.md
+++ b/docs/home/6-build-reliable-ai-workflows/pipes/provide-inputs.md
@@ -16,7 +16,7 @@ This creates a `results/inputs.json` file with the structure needed for your pip
pipelex run path/to/my_pipe.plx --inputs results/inputs.json
```
-See more about the options of the CLI [here](../../../home/9-tools/cli.md).
+See more about the options of the CLI [here](../../9-tools/cli/index.md).
!!! tip "Starting Point for Input Structure"
Use `pipelex build inputs` to quickly understand what inputs your pipeline expects and generate a template to fill in.
diff --git a/docs/home/9-tools/cli.md b/docs/home/9-tools/cli.md
deleted file mode 100644
index 2cd1177da..000000000
--- a/docs/home/9-tools/cli.md
+++ /dev/null
@@ -1,464 +0,0 @@
-# Pipelex CLI Documentation
-
-The Pipelex CLI provides a command-line interface for managing and interacting with your Pipelex projects. This document outlines all available commands and their usage.
-
-## Overview
-
-The Pipelex CLI is organized into several command groups:
-
-- **init** - Initialize Pipelex configuration
-- **kit** - Manage agent rules and migration instructions (see [Kit Commands](kit.md))
-- **build** - Generate pipelines from natural language (see [Pipe Builder](pipe-builder.md))
-- **validate** - Validate configuration and pipelines
-- **run** - Execute pipelines
-- **show** - Inspect configuration, pipes, and AI models
-
-## Init Commands
-
-Initialize project configuration files in your project's `.pipelex` directory.
-
-### Initialize Configuration
-
-```bash
-pipelex init config [OPTIONS]
-```
-
-Creates the `.pipelex` directory structure and copies default configuration files from the Pipelex package template.
-
-**Options:**
-
-- `--reset`, `-r` - Overwrite existing configuration files (use with caution)
-
-**Examples:**
-
-```bash
-# Initialize configuration for the first time
-pipelex init config
-
-# Reinitialize configuration, overwriting existing files
-pipelex init config --reset
-```
-
-**What gets initialized:**
-
-This command creates the `.pipelex/` directory with:
-
-- The main pipelex.toml configuration file for logging, reporting, tracking, etc.
-- `inference/` - AI backend and routing configuration
-- `deck/` - AI model aliases and presets
-
-**Related Configuration:**
-
-- [Configure AI Providers](../../home/5-setup/configure-ai-providers.md)
-- [Inference Backend Configuration](../../home/7-configuration/config-technical/inference-backend-config.md)
-
-## Validate Commands
-
-Validate your pipeline definitions and configuration for correctness.
-
-### Validate All Pipes
-
-```bash
-pipelex validate all
-```
-
-Performs comprehensive validation:
-
-1. Validates all library configurations
-2. Runs static validation on all discovered pipes
-3. Performs dry-run execution to check pipeline logic
-
-This is the recommended validation to run before committing changes or deploying pipelines.
-
-**Examples:**
-
-```bash
-# Validate everything
-pipelex validate all
-```
-
-### Validate Single Pipe
-
-```bash
-pipelex validate PIPE_CODE
-pipelex validate --pipe PIPE_CODE
-```
-
-Validates and dry-runs a specific pipe from your imported packages, useful for iterative development.
-
-**Arguments:**
-
-- `PIPE_CODE` - The pipe code to validate as a positional argument, or use `--pipe` option
-
-**Options:**
-
-- `--pipe PIPE_CODE` - Explicitly specify the pipe code to validate (alternative to positional argument)
-
-**Examples:**
-
-```bash
-# Validate a specific pipe (positional argument)
-pipelex validate analyze_cv_matching
-pipelex validate write_weekly_report
-
-# Validate a specific pipe (explicit option)
-pipelex validate --pipe analyze_cv_matching
-```
-
-### Validate Bundle
-
-```bash
-pipelex validate BUNDLE_FILE.plx
-pipelex validate --bundle BUNDLE_FILE.plx
-```
-
-Validates all pipes defined in a bundle file. The command automatically detects `.plx` files as bundles.
-
-**Arguments:**
-
-- `BUNDLE_FILE.plx` - Path to the bundle file (auto-detected by `.plx` extension)
-
-**Options:**
-
-- `--bundle BUNDLE_FILE.plx` - Explicitly specify the bundle file path
-
-**Examples:**
-
-```bash
-# Validate a bundle (auto-detected)
-pipelex validate my_pipeline.plx
-pipelex validate pipelines/invoice_processor.plx
-
-# Validate a bundle (explicit option)
-pipelex validate --bundle my_pipeline.plx
-```
-
-**Note:** When validating a bundle, ALL pipes in that bundle are validated, not just the main pipe.
-
-### Validate Specific Pipe in Bundle
-
-```bash
-pipelex validate --bundle BUNDLE_FILE.plx --pipe PIPE_CODE
-```
-
-Validates all pipes in a bundle, while ensuring a specific pipe exists in that bundle. The entire bundle is validated, not just the specified pipe.
-
-**Options:**
-
-- `--bundle BUNDLE_FILE.plx` - Path to the bundle file
-- `--pipe PIPE_CODE` - Pipe code that must exist in the bundle
-
-**Examples:**
-
-```bash
-# Validate bundle and ensure specific pipe exists in it
-pipelex validate --bundle my_pipeline.plx --pipe extract_invoice
-pipelex validate --bundle invoice_processor.plx --pipe validate_amounts
-```
-
-!!! important "Bundle Validation Behavior"
- The specified pipe must be defined in the bundle. This is useful when you want to validate a bundle and confirm a specific pipe is present and valid within it. However, the entire bundle will be validated regardless.
-
-### What Validation Checks
-
-All validation commands check:
-
-- Syntax correctness of `.plx` files
-- Concept and pipe definitions are valid
-- Input/output connections are correct
-- All referenced pipes and concepts exist
-- Dry-run execution succeeds without errors, which implies the logic is correct and the pipe can be run
-
-**Related Configuration:**
-
-- [Static Validation Configuration](../../home/7-configuration/config-pipeline-validation/static-validation-config.md)
-- [Dry Run Configuration](../../home/7-configuration/config-pipeline-validation/dry-run-config.md)
-
-## Show Commands
-
-Inspect your Pipelex configuration, pipelines, and available AI models.
-
-### Show Configuration
-
-```bash
-pipelex show config
-```
-
-Displays the current Pipelex configuration loaded from all sources (default config overriden by user config).
-
-**Examples:**
-
-```bash
-# Display current configuration
-pipelex show config
-```
-
-**Note:** This shows the main Pipelex configuration but not the inference backend details. Use `pipelex show backends` for backend configuration.
-
-### List All Pipes
-
-```bash
-pipelex show pipes
-```
-
-Lists all pipes discovered in your project and imported packages, showing their pipe codes and basic information.
-
-**Examples:**
-
-```bash
-# List all available pipes
-pipelex show pipes
-```
-
-This includes:
-
-- Internal Pipelex pipes (like the pipe builder)
-- Pipes from your project's `.plx` files
-- Pipes that are part of imported packages
-
-### Show Pipe Definition
-
-```bash
-pipelex show pipe PIPE_CODE
-```
-
-Displays the complete definition of a specific pipe including inputs, outputs, prompts, model settings, and all configuration.
-
-**Arguments:**
-
-- `PIPE_CODE` - The pipe code to inspect
-
-**Examples:**
-
-```bash
-# Show pipe definition
-pipelex show pipe analyze_cv_matching
-pipelex show pipe write_weekly_report
-```
-
-### List AI Models
-
-```bash
-pipelex show models BACKEND_NAME [OPTIONS]
-```
-
-Lists all available models from a configured backend provider by querying the provider's API.
-
-**Arguments:**
-
-- `BACKEND_NAME` - The backend to query (e.g., `openai`, `anthropic`, `mistral`)
-
-**Options:**
-
-- `--flat`, `-f` - Output in flat CSV format for easy copying into other configuration files
-
-**Examples:**
-
-```bash
-# List models
-pipelex show models openai
-pipelex show models mistral
-
-# List models in flat format
-pipelex show models anthropic --flat
-```
-
-**Use case:** When configuring new models in your deck, use this command to see what models are available from each provider.
-
-**Related Configuration:**
-
-- [Inference Backend Configuration](../../home/7-configuration/config-technical/inference-backend-config.md)
-
-### Show Backends
-
-```bash
-pipelex show backends [OPTIONS]
-```
-
-Displays all configured inference backends and the active routing profile with its routing rules.
-
-**Options:**
-
-- `--all`, `-a` - Show all backends including disabled ones (by default, only enabled backends are shown)
-
-**Examples:**
-
-```bash
-# Show enabled backends and routing profile
-pipelex show backends
-
-# Show all backends including disabled ones
-pipelex show backends --all
-```
-
-**What it displays:**
-
-- Table of configured backends with status, endpoint, and model count
-- Active routing profile name and description
-- Default backend for the profile
-- Routing rules mapping model patterns to backends
-
-**Related Configuration:**
-
-- [Inference Backend Configuration](../../home/7-configuration/config-technical/inference-backend-config.md)
-- Backend configuration files: `.pipelex/inference/backends.toml`
-- Routing configuration: `.pipelex/inference/routing_profiles.toml`
-
-## Run Command
-
-Execute a pipeline with optional inputs and outputs.
-
-### Run a Pipeline
-
-```bash
-pipelex run [TARGET] [OPTIONS]
-```
-
-Executes a pipeline, either from a standalone bundle (.plx) file or from your project's pipe library.
-
-**Arguments:**
-
-- `TARGET` - Either a pipe code or a bundle file path, auto-detected according to presence of the .plx file extension
-
-**Options:**
-
-- `--pipe` - Pipe code to run (alternative to positional argument)
-- `--bundle` - Bundle file path (alternative to positional argument)
-- `--inputs`, `-i` - Path to JSON file containing inputs
-- `--output`, `-o` - Path to save output JSON (defaults to `results/run_{pipe_code}.json`)
-- `--no-output` - Skip saving output to file
-- `--no-pretty-print` - Skip pretty printing the main output
-
-**Examples:**
-
-```bash
-# Run a pipe by code
-pipelex run hello_world
-
-# Run with inputs from JSON file
-pipelex run write_weekly_report --inputs weekly_report_data.json
-
-# Run a bundle file (uses its main_pipe)
-pipelex run my_bundle.plx
-
-# Run a specific pipe from a bundle
-pipelex run my_bundle.plx --pipe extract_invoice
-
-# Run with explicit options
-pipelex run --pipe hello_world --output my_output.json
-
-# Run without saving or pretty printing
-pipelex run my_pipe --no-output --no-pretty-print
-```
-
-**Input JSON Format:**
-
-The input JSON file should contain a dictionary where keys are input variable names:
-
-```json
-{
- "input_variable": "simple string value",
- "another_input": {
- "concept": "domain.ConceptName",
- "content": { "field": "value" }
- }
-}
-```
-
-**Output Format:**
-
-The output JSON contains the complete working memory after pipeline execution, including all intermediate results and the final output.
-
-**Related Documentation:**
-
-- [Executing Pipelines](../../home/6-build-reliable-ai-workflows/pipes/executing-pipelines.md)
-- [Providing Inputs to Pipelines](../../home/6-build-reliable-ai-workflows/pipes/provide-inputs.md)
-- [Design and Run Pipelines](../../home/6-build-reliable-ai-workflows/pipes/index.md)
-
-## Build Commands
-
-Generate pipelines and runner code from natural language descriptions. See the [Pipe Builder](pipe-builder.md) documentation for comprehensive details.
-
-### Build Pipe
-
-```bash
-pipelex build pipe "PROMPT" [OPTIONS]
-```
-
-Generates a complete pipeline from a natural language prompt with automatic validation and error correction.
-
-**Quick Example:**
-
-```bash
-pipelex build pipe "Analyze a CV and a Job offer and determine if they match" -o cv_matching_pipeline.plx
-```
-
-For complete documentation including all options and examples, see [Pipe Builder](pipe-builder.md).
-
-### Build Runner
-
-```bash
-pipelex build runner [TARGET] [OPTIONS]
-```
-
-Generates Python code to run a pipe with example inputs and all necessary imports.
-
-**Quick Example:**
-
-```bash
-pipelex build runner my_pipe -o run_my_pipe.py
-```
-
-For complete documentation, see the [Generate Runner Code section in Pipe Builder](pipe-builder.md#generate-runner-code).
-
-## Kit Commands
-
-Manage agent rules for AI coding assistants and sync migration instructions. See the [Kit Commands](kit.md) documentation for comprehensive details.
-
-### Quick Reference
-
-```bash
-# Install agent rules for AI assistants
-pipelex kit rules
-
-# Remove agent rules
-pipelex kit remove-rules
-
-# Install migration instructions
-pipelex kit migrations
-```
-
-For complete documentation including all options and examples, see [Kit Commands](kit.md).
-
-## Usage Tips
-
-1. **Initial Setup**
-
-- Run `pipelex init config` to create configuration files
-- Configure your AI providers in `.pipelex/inference/backends.toml`
-- Install agent rules with `pipelex kit rules` if using AI assistants
-
-2. **Development Workflow**
-
-- Write or generate pipelines in `.plx` files
-- Validate with `pipelex validate your_pipe_code` or `pipelex validate your_bundle.plx` during development
- - Run `pipelex validate all` before committing changes
-
-3. **Running Pipelines**
-
-- Use `pipelex show pipes` to see available pipes
-- Use `pipelex show pipe pipe_code` to inspect pipe details
-- Run with `pipelex run pipe_code`, add the required inputs using `--inputs`
-
-4. **Configuration Management**
-
-- Use `pipelex show config` to verify current settings
-- Use `pipelex show backends` to check inference backend setup
-- Use `pipelex show models backend_name` to see available models
-
-## Related Documentation
-
-- [Pipe Builder](pipe-builder.md) - Generate pipelines from natural language
-- [Kit Commands](kit.md) - Agent rules and migration management
-- [Configure AI Providers](../../home/5-setup/configure-ai-providers.md) - Set up LLM backends
-- [Design and Run Pipelines](../../home/6-build-reliable-ai-workflows/pipes/index.md) - Pipeline development guide
diff --git a/docs/home/9-tools/cli/build/index.md b/docs/home/9-tools/cli/build/index.md
new file mode 100644
index 000000000..10580d746
--- /dev/null
+++ b/docs/home/9-tools/cli/build/index.md
@@ -0,0 +1,11 @@
+# Build Commands
+
+Generate pipelines, runner code, and Python structures from natural language descriptions.
+
+## Available Commands
+
+| Command | Description |
+|---------|-------------|
+| [**build pipe**](pipe.md) | Generate a pipeline from natural language |
+| [**build runner**](runner.md) | Generate Python code to run a pipe |
+| [**build structures**](structures.md) | Generate the structures (Pydantic models) of your concepts |
diff --git a/docs/home/9-tools/cli/build/pipe.md b/docs/home/9-tools/cli/build/pipe.md
new file mode 100644
index 000000000..22828b769
--- /dev/null
+++ b/docs/home/9-tools/cli/build/pipe.md
@@ -0,0 +1,106 @@
+# Build Pipe
+
+!!! warning "Beta Feature"
+ The Pipe Builder is currently in beta and progressing fast. Expect frequent improvements and changes.
+
+!!! tip "Built with Pipelex"
+ The Pipe Builder is itself a Pipelex pipeline! This showcases the power of Pipelex: a tool that builds pipelines... using a pipeline.
+
+The Pipe Builder is an AI-powered tool that generates Pipelex pipelines from natural language descriptions. Describe what you want to achieve, and the builder translates your requirements into a working `.plx` file.
+
+!!! info "Deep Dive"
+ Want to understand how the Pipe Builder works under the hood? See [Pipe Builder Deep Dive](../../pipe-builder.md) for the full explanation of its multi-step generation process.
+
+## Usage
+
+```bash
+pipelex build pipe "Brief description of what the pipeline should do"
+```
+
+The resulting pipeline will be saved in a folder `pipeline_01/` (with increasing number) containing:
+
+| File | Description |
+|------|-------------|
+| `bundle.plx` | The pipeline definition |
+| `inputs.json` | Template for pipeline inputs |
+| `run_{pipe_code}.py` | Python script to run the pipeline |
+| `structures/` | Generated Pydantic models for your concepts |
+| `bundle_view.html` | HTML visualization of the build process and plan |
+| `bundle_view.svg` | SVG visualization of the build process and plan |
+| `__init__.py` | Python package init file |
+
+The HTML and SVG files provide a visual representation of the resulting workflow.
+
+**Example:**
+
+```bash
+pipelex build pipe "Given an expense report, apply company rules" -o results/expense_pipeline.plx
+```
+
+## Options
+
+- `--output`, `-o`: Path to save the generated `.plx` file
+- `--no-output`: Skip saving the file (useful for testing)
+
+## Example Use Cases
+
+**Document Processing:**
+
+```bash
+pipelex build pipe "Take a CV in a PDF file and a Job offer text, and analyze if they match"
+```
+
+**Data Transformation:**
+
+```bash
+pipelex build pipe "Extract structured data from invoice images"
+```
+
+**Multi-step Workflows:**
+
+```bash
+pipelex build pipe "Given an RFP PDF, build a compliance matrix"
+```
+
+## Tips for Best Results
+
+- You can be specific in your brief about inputs, outputs, data formats, or structures if you know what you need
+- If you're uncertain about the details, let the AI figure it out and see what it generates
+- Include any domain-specific requirements you're aware of upfront
+
+## Current Limitations
+
+The Pipe Builder is in active development and currently:
+
+- Can automatically fix input/output connection errors
+- May require manual adjustments for complex conditional logic or custom functions
+- Validation focuses on structural correctness, not business logic
+
+## Iterating on Generated Pipelines
+
+After generating a pipeline, you can continue refining it using any Software Engineering (SWE) agent. The generated `.plx` file can be iteratively improved through natural language instructions.
+
+Pipelex provides specialized agent rules that guide AI assistants in working with pipelines. Install them with:
+
+```bash
+pipelex kit rules
+```
+
+See [Kit Commands](../kit.md) for details on supported AI assistants.
+
+## Next Steps
+
+After generating your pipeline:
+
+1. **Validate it**: `pipelex validate your_pipe.plx` - See [Validate Commands](../validate.md)
+2. **Run it**: `pipelex run your_pipe.plx` - See [Run Command](../run.md)
+3. **Generate a runner**: `pipelex build runner your_pipe.plx` - See [Build Runner](runner.md)
+4. **Generate structures**: `pipelex build structures ./` - See [Build Structures](structures.md)
+
+## Related Documentation
+
+- [Pipe Builder Deep Dive](../../pipe-builder.md) - How the builder works under the hood
+- [Design and Run Pipelines](../../../6-build-reliable-ai-workflows/pipes/index.md)
+- [Pipe Operators](../../../6-build-reliable-ai-workflows/pipes/pipe-operators/index.md)
+- [Pipe Controllers](../../../6-build-reliable-ai-workflows/pipes/pipe-controllers/index.md)
+
diff --git a/docs/home/9-tools/cli/build/runner.md b/docs/home/9-tools/cli/build/runner.md
new file mode 100644
index 000000000..d3deca0d8
--- /dev/null
+++ b/docs/home/9-tools/cli/build/runner.md
@@ -0,0 +1,77 @@
+# Build Runner
+
+Generate Python code to run a pipe with all necessary imports and example inputs.
+
+## Usage
+
+```bash
+pipelex build runner <TARGET> [OPTIONS]
+```
+
+**Arguments:**
+
+- `TARGET` - Either a bundle file path (`.plx`) or a library directory
+
+**Options:**
+
+- `--pipe` - Pipe code to use (**mandatory** for library directory, **mandatory** for `.plx` without `main_pipe`)
+- `--output`, `-o` - Path to save the generated Python file (defaults to target's directory)
+
+## Examples
+
+**Generate runner from a bundle (uses main_pipe):**
+
+```bash
+pipelex build runner my_bundle.plx
+```
+
+**Specify which pipe to use from a bundle:**
+
+```bash
+pipelex build runner my_bundle.plx --pipe my_pipe
+```
+
+**Generate runner from a library directory (--pipe is mandatory):**
+
+```bash
+pipelex build runner ./my_library/ --pipe my_pipe
+```
+
+**Custom output path:**
+
+```bash
+pipelex build runner my_bundle.plx --output custom_runner.py
+```
+
+## What Gets Generated
+
+The generated Python file includes:
+
+1. **All necessary imports** - Imports for Pipelex execution, content types, and any custom structures
+2. **Generated Pydantic structures** - Models for your concepts (also generated separately)
+3. **Input memory setup** - Example input values based on the pipe's input types
+4. **Pipeline execution** - Async function that executes the pipeline
+5. **Output handling** - Code to extract and display the results
+6. **Main execution block** - Pipelex initialization and asyncio setup
+
+## Input Type Handling
+
+- **Native concepts** (Text, Image, PDF, etc.) - Automatically generates appropriate content objects
+- **Custom concepts** - Recursively generates the structure with example values
+- **Structured content** - Creates example data matching the concept's fields
+
+## Using the Generated Runner
+
+After generating the runner file:
+
+1. Open the generated Python file
+2. Review and customize the example input values
+3. Run the script: `python results/run_{pipe_code}.py`
+4. Iterate and adjust as needed
+
+## Related Documentation
+
+- [Build Pipe](pipe.md) - Generate pipelines from natural language
+- [Build Structures](structures.md) - Generate Pydantic models separately
+- [Run Command](../run.md) - Run pipes directly from CLI
+
diff --git a/docs/home/9-tools/cli/build/structures.md b/docs/home/9-tools/cli/build/structures.md
new file mode 100644
index 000000000..9a7bd919f
--- /dev/null
+++ b/docs/home/9-tools/cli/build/structures.md
@@ -0,0 +1,91 @@
+# Build Structures
+
+Generate Pydantic models from your concept definitions.
+
+## Usage
+
+```bash
+pipelex build structures <TARGET> [OPTIONS]
+```
+
+**Arguments:**
+
+- `TARGET` - Either a library directory containing `.plx` files, or a specific `.plx` file
+
+**Options:**
+
+- `-o`, `--output-dir` - Output directory for generated Python files (defaults to `structures/` in target's directory)
+
+## Examples
+
+**Generate structures from a library directory:**
+
+```bash
+pipelex build structures ./my_pipelines/
+```
+
+**Generate structures from a specific bundle file:**
+
+```bash
+pipelex build structures ./my_pipeline/bundle.plx
+```
+
+**Generate structures to a specific output directory:**
+
+```bash
+pipelex build structures ./my_pipelines/ -o ./generated/
+```
+
+## What Gets Generated
+
+For each concept in your library, the command generates:
+
+- **Pydantic model classes** - Full type definitions for your concepts
+- **Field definitions** - Proper types and constraints
+- **Validation** - Based on concept constraints
+
+## Why Use Generated Structures?
+
+Now you have your structures as Python code:
+
+- **Iterate on them** - Modify and extend the generated classes
+- **Add custom validation** - Add Pydantic validators for business logic
+- **Use as type hints** - Get IDE autocompletion and type checking in your code
+- **Integrate in your app** - Use the models directly in your application
+
+## Example Output
+
+For a concept defined in a `.plx` file like:
+
+```toml
+[concept.CandidateProfile]
+description = "A structured representation of a candidate's professional background."
+
+[concept.CandidateProfile.structure]
+skills = { type = "text", description = "Technical and soft skills", required = true }
+years_of_experience = { type = "number", description = "Total years of experience" }
+education = { type = "text", description = "Educational background", required = true }
+work_history = { type = "text", description = "Summary of previous roles", required = true }
+```
+
+You get a Pydantic model:
+
+```python
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pydantic import Field
+
+class CandidateProfile(StructuredContent):
+ """Generated CandidateProfile class"""
+
+ skills: str = Field(..., description="Technical and soft skills")
+ years_of_experience: float = Field(..., description="Total years of experience")
+ education: str = Field(..., description="Educational background")
+ work_history: str = Field(..., description="Summary of previous roles")
+```
+
+## Related Documentation
+
+- [Build Pipe](pipe.md) - Generate pipelines from natural language
+- [Build Runner](runner.md) - Generate Python runner scripts
+- [Concepts](../../../6-build-reliable-ai-workflows/concepts/define_your_concepts.md) - Understanding concept definitions
+
diff --git a/docs/home/9-tools/cli/index.md b/docs/home/9-tools/cli/index.md
new file mode 100644
index 000000000..745db767f
--- /dev/null
+++ b/docs/home/9-tools/cli/index.md
@@ -0,0 +1,50 @@
+# Pipelex CLI Documentation
+
+The Pipelex CLI provides a command-line interface for managing and interacting with your Pipelex projects. This document outlines all available commands and their usage.
+
+## Overview
+
+The Pipelex CLI is organized into several command groups:
+
+| Command | Description |
+|---------|-------------|
+| [**init**](init.md) | Initialize Pipelex configuration |
+| [**validate**](validate.md) | Validate configuration and pipelines |
+| [**show**](show.md) | Inspect configuration, pipes, and AI models |
+| [**run**](run.md) | Execute pipelines |
+| [**build**](build/index.md) | Generate pipelines, runners, and structures |
+| [**kit**](kit.md) | Manage agent rules and migration instructions |
+
+## Usage Tips
+
+1. **Initial Setup**
+
+ - Run `pipelex init` to create configuration files and select your backends
+ - Configure your AI providers in `.pipelex/inference/backends.toml`
+ - Install agent rules with `pipelex kit rules` if using AI assistants
+
+2. **Development Workflow**
+
+ - Write or generate pipelines in `.plx` files
+ - Validate with `pipelex validate your_pipe_code` or `pipelex validate your_bundle.plx` during development
+ - Run `pipelex validate all` before committing changes
+
+3. **Running Pipelines**
+
+ - Use `pipelex show pipes` to see available pipes
+ - Use `pipelex show pipe pipe_code` to inspect pipe details
+ - Run with `pipelex run pipe_code`, add the required inputs using `--inputs`
+
+4. **Configuration Management**
+
+ - Use `pipelex show config` to verify current settings
+ - Use `pipelex show backends` to check inference backend setup
+ - Use `pipelex show models backend_name` to see available models
+
+## Related Documentation
+
+- [Build Pipe](build/pipe.md) - Generate pipelines from natural language
+- [Kit Commands](kit.md) - Agent rules and migration management
+- [Configure AI Providers](../../5-setup/configure-ai-providers.md) - Set up LLM backends
+- [Design and Run Pipelines](../../6-build-reliable-ai-workflows/pipes/index.md) - Pipeline development guide
+
diff --git a/docs/home/9-tools/cli/init.md b/docs/home/9-tools/cli/init.md
new file mode 100644
index 000000000..d8eb8bf27
--- /dev/null
+++ b/docs/home/9-tools/cli/init.md
@@ -0,0 +1,66 @@
+# Init Commands
+
+Initialize project configuration files in your project's `.pipelex` directory.
+
+## Initialize Configuration
+
+```bash
+pipelex init [FOCUS]
+```
+
+Creates the `.pipelex` directory structure, copies default configuration files, and guides you through backend and telemetry setup.
+
+!!! note "Config updates not yet supported"
+ The `pipelex init` command always performs a full reset of the configuration. Incremental config updates will be supported in a future release.
+
+**Arguments:**
+
+- `FOCUS` - What to initialize (optional):
+ - `all` (default) - Initialize everything
+ - `config` - Only configuration files
+ - `inference` - Only inference backend setup
+ - `routing` - Only routing profile setup
+ - `telemetry` - Only telemetry configuration
+
+**Examples:**
+
+```bash
+# Initialize everything (recommended for first-time setup)
+pipelex init
+
+# Initialize only configuration files
+pipelex init config
+
+# Reconfigure inference backends
+pipelex init inference
+
+# Reconfigure telemetry settings
+pipelex init telemetry
+```
+
+## What Gets Initialized
+
+This command creates the `.pipelex/` directory with:
+
+- **pipelex.toml** - Main configuration file for logging, reporting, tracking, etc.
+- **inference/** - AI backend and routing configuration
+ - `backends.toml` - Backend provider settings
+ - `routing_profiles.toml` - Model routing rules
+ - `backends/` - Individual backend configuration files
+ - `deck/` - AI model aliases and presets
+- **telemetry.toml** - Telemetry and observability settings
+
+## Interactive Setup Flow
+
+When you run `pipelex init`, you'll be guided through:
+
+1. **Backend Selection** - Choose which AI providers to enable (OpenAI, Anthropic, Mistral, Pipelex Gateway, etc.)
+2. **Routing Configuration** - Set up how models are routed to backends
+3. **Telemetry Setup** - Configure observability and analytics
+
+## Related Configuration
+
+- [Configure AI Providers](../../5-setup/configure-ai-providers.md)
+- [Inference Backend Configuration](../../7-configuration/config-technical/inference-backend-config.md)
+- [Telemetry Configuration](../../7-configuration/config-practical/telemetry-config.md)
+
diff --git a/docs/home/9-tools/kit.md b/docs/home/9-tools/cli/kit.md
similarity index 96%
rename from docs/home/9-tools/kit.md
rename to docs/home/9-tools/cli/kit.md
index 7787ad8f9..d6a998738 100644
--- a/docs/home/9-tools/kit.md
+++ b/docs/home/9-tools/cli/kit.md
@@ -1,4 +1,4 @@
-# Pipelex Kit Commands
+# Kit Commands
The Pipelex Kit provides commands for managing agent rules and migration instructions. These commands help you integrate Pipelex guidelines into your AI coding assistants and keep track of breaking changes across versions.
@@ -131,6 +131,6 @@ These rules are configured in `pipelex/kit/index.toml`.
## Related Documentation
-- [Pipe Builder](pipe-builder.md) - Generate pipelines from natural language
-- [CLI Commands](cli.md) - General Pipelex CLI documentation
+- [Pipe Builder](../pipe-builder.md) - Generate pipelines from natural language
+- [CLI Overview](index.md) - General Pipelex CLI documentation
diff --git a/docs/home/9-tools/cli/run.md b/docs/home/9-tools/cli/run.md
new file mode 100644
index 000000000..c68610ce4
--- /dev/null
+++ b/docs/home/9-tools/cli/run.md
@@ -0,0 +1,71 @@
+# Run Command
+
+Execute a pipeline with optional inputs and outputs.
+
+## Run a Pipeline
+
+```bash
+pipelex run [TARGET] [OPTIONS]
+```
+
+Executes a pipeline, either from a standalone bundle (.plx) file or from your project's pipe library.
+
+**Arguments:**
+
+- `TARGET` - Either a pipe code or a bundle file path, auto-detected according to presence of the .plx file extension
+
+**Options:**
+
+- `--pipe` - Pipe code to run (alternative to positional argument)
+- `--bundle` - Bundle file path (alternative to positional argument)
+- `--inputs`, `-i` - Path to JSON file containing inputs
+- `--output`, `-o` - Path to save output JSON (defaults to `results/run_{pipe_code}.json`)
+- `--no-output` - Skip saving output to file
+- `--no-pretty-print` - Skip pretty printing the main output
+
+**Examples:**
+
+```bash
+# Run a pipe by code
+pipelex run hello_world
+
+# Run with inputs from JSON file
+pipelex run write_weekly_report --inputs weekly_report_data.json
+
+# Run a bundle file (uses its main_pipe)
+pipelex run my_bundle.plx
+
+# Run a specific pipe from a bundle
+pipelex run my_bundle.plx --pipe extract_invoice
+
+# Run with explicit options
+pipelex run --pipe hello_world --output my_output.json
+
+# Run without saving or pretty printing
+pipelex run my_pipe --no-output --no-pretty-print
+```
+
+## Input JSON Format
+
+The input JSON file should contain a dictionary where keys are input variable names:
+
+```json
+{
+ "input_variable": "simple string value",
+ "another_input": {
+ "concept": "domain.ConceptName",
+ "content": { "field": "value" }
+ }
+}
+```
+
+## Output Format
+
+The output JSON contains the complete working memory after pipeline execution, including all intermediate results and the final output.
+
+## Related Documentation
+
+- [Executing Pipelines](../../6-build-reliable-ai-workflows/pipes/executing-pipelines.md)
+- [Providing Inputs to Pipelines](../../6-build-reliable-ai-workflows/pipes/provide-inputs.md)
+- [Design and Run Pipelines](../../6-build-reliable-ai-workflows/pipes/index.md)
+
diff --git a/docs/home/9-tools/cli/show.md b/docs/home/9-tools/cli/show.md
new file mode 100644
index 000000000..35a5819f3
--- /dev/null
+++ b/docs/home/9-tools/cli/show.md
@@ -0,0 +1,127 @@
+# Show Commands
+
+Inspect your Pipelex configuration, pipelines, and available AI models.
+
+## Show Configuration
+
+```bash
+pipelex show config
+```
+
+Displays the current Pipelex configuration loaded from all sources (default config overridden by user config).
+
+**Examples:**
+
+```bash
+# Display current configuration
+pipelex show config
+```
+
+!!! note
+ This shows the main Pipelex configuration but not the inference backend details. Use `pipelex show backends` for backend configuration.
+
+## List All Pipes
+
+```bash
+pipelex show pipes
+```
+
+Lists all pipes discovered in your project and imported packages, showing their pipe codes and basic information.
+
+**Examples:**
+
+```bash
+# List all available pipes
+pipelex show pipes
+```
+
+This includes:
+
+- Internal Pipelex pipes (like the pipe builder)
+- Pipes from your project's `.plx` files
+- Pipes that are part of imported packages
+
+## Show Pipe Definition
+
+```bash
+pipelex show pipe PIPE_CODE
+```
+
+Displays the complete definition of a specific pipe including inputs, outputs, prompts, model settings, and all configuration.
+
+**Arguments:**
+
+- `PIPE_CODE` - The pipe code to inspect
+
+**Examples:**
+
+```bash
+# Show pipe definition
+pipelex show pipe analyze_cv_matching
+pipelex show pipe write_weekly_report
+```
+
+## List AI Models
+
+```bash
+pipelex show models BACKEND_NAME [OPTIONS]
+```
+
+Lists all available models from a configured backend provider by querying the provider's API.
+
+**Arguments:**
+
+- `BACKEND_NAME` - The backend to query (e.g., `openai`, `anthropic`, `mistral`)
+
+**Options:**
+
+- `--flat`, `-f` - Output in flat CSV format for easy copying into other configuration files
+
+**Examples:**
+
+```bash
+# List models
+pipelex show models openai
+pipelex show models mistral
+
+# List models in flat format
+pipelex show models anthropic --flat
+```
+
+**Use case:** When configuring new models in your deck, use this command to see what models are available from each provider.
+
+## Show Backends
+
+```bash
+pipelex show backends [OPTIONS]
+```
+
+Displays all configured inference backends and the active routing profile with its routing rules.
+
+**Options:**
+
+- `--all`, `-a` - Show all backends including disabled ones (by default, only enabled backends are shown)
+
+**Examples:**
+
+```bash
+# Show enabled backends and routing profile
+pipelex show backends
+
+# Show all backends including disabled ones
+pipelex show backends --all
+```
+
+**What it displays:**
+
+- Table of configured backends with status, endpoint, and model count
+- Active routing profile name and description
+- Default backend for the profile
+- Routing rules mapping model patterns to backends
+
+## Related Configuration
+
+- [Inference Backend Configuration](../../7-configuration/config-technical/inference-backend-config.md)
+- Backend configuration files: `.pipelex/inference/backends.toml`
+- Routing configuration: `.pipelex/inference/routing_profiles.toml`
+
diff --git a/docs/home/9-tools/cli/validate.md b/docs/home/9-tools/cli/validate.md
new file mode 100644
index 000000000..ebeaf7d07
--- /dev/null
+++ b/docs/home/9-tools/cli/validate.md
@@ -0,0 +1,123 @@
+# Validate Commands
+
+Validate your pipeline definitions and configuration for correctness.
+
+## Validate All Pipes
+
+```bash
+pipelex validate all
+```
+
+Performs comprehensive validation:
+
+1. Validates all library configurations
+2. Runs static validation on all discovered pipes
+3. Performs dry-run execution to check pipeline logic
+
+This is the recommended validation to run before committing changes or deploying pipelines.
+
+**Examples:**
+
+```bash
+# Validate everything
+pipelex validate all
+```
+
+## Validate Single Pipe
+
+```bash
+pipelex validate PIPE_CODE
+pipelex validate --pipe PIPE_CODE
+```
+
+Validates and dry-runs a specific pipe from your imported packages, useful for iterative development.
+
+**Arguments:**
+
+- `PIPE_CODE` - The pipe code to validate as a positional argument, or use `--pipe` option
+
+**Options:**
+
+- `--pipe PIPE_CODE` - Explicitly specify the pipe code to validate (alternative to positional argument)
+
+**Examples:**
+
+```bash
+# Validate a specific pipe (positional argument)
+pipelex validate analyze_cv_matching
+pipelex validate write_weekly_report
+
+# Validate a specific pipe (explicit option)
+pipelex validate --pipe analyze_cv_matching
+```
+
+## Validate Bundle
+
+```bash
+pipelex validate BUNDLE_FILE.plx
+pipelex validate --bundle BUNDLE_FILE.plx
+```
+
+Validates all pipes defined in a bundle file. The command automatically detects `.plx` files as bundles.
+
+**Arguments:**
+
+- `BUNDLE_FILE.plx` - Path to the bundle file (auto-detected by `.plx` extension)
+
+**Options:**
+
+- `--bundle BUNDLE_FILE.plx` - Explicitly specify the bundle file path
+
+**Examples:**
+
+```bash
+# Validate a bundle (auto-detected)
+pipelex validate my_pipeline.plx
+pipelex validate pipelines/invoice_processor.plx
+
+# Validate a bundle (explicit option)
+pipelex validate --bundle my_pipeline.plx
+```
+
+!!! note
+ When validating a bundle, ALL pipes in that bundle are validated, not just the main pipe.
+
+## Validate Specific Pipe in Bundle
+
+```bash
+pipelex validate --bundle BUNDLE_FILE.plx --pipe PIPE_CODE
+```
+
+Validates all pipes in a bundle, while ensuring a specific pipe exists in that bundle. The entire bundle is validated, not just the specified pipe.
+
+**Options:**
+
+- `--bundle BUNDLE_FILE.plx` - Path to the bundle file
+- `--pipe PIPE_CODE` - Pipe code that must exist in the bundle
+
+**Examples:**
+
+```bash
+# Validate bundle and ensure specific pipe exists in it
+pipelex validate --bundle my_pipeline.plx --pipe extract_invoice
+pipelex validate --bundle invoice_processor.plx --pipe validate_amounts
+```
+
+!!! important "Bundle Validation Behavior"
+ The specified pipe must be defined in the bundle. This is useful when you want to validate a bundle and confirm a specific pipe is present and valid within it. However, the entire bundle will be validated regardless.
+
+## What Validation Checks
+
+All validation commands check:
+
+- Syntax correctness of `.plx` files
+- Concept and pipe definitions are valid
+- Input/output connections are correct
+- All referenced pipes and concepts exist
+- Dry-run execution succeeds without errors, which implies the logic is correct and the pipe can be run
+
+## Related Configuration
+
+- [Static Validation Configuration](../../7-configuration/config-pipeline-validation/static-validation-config.md)
+- [Dry Run Configuration](../../7-configuration/config-pipeline-validation/dry-run-config.md)
+
diff --git a/docs/home/9-tools/pipe-builder.md b/docs/home/9-tools/pipe-builder.md
index 44ba22c71..a0efc748c 100644
--- a/docs/home/9-tools/pipe-builder.md
+++ b/docs/home/9-tools/pipe-builder.md
@@ -3,195 +3,118 @@
!!! warning "Beta Feature"
The Pipe Builder is currently in beta and progressing fast. Expect frequent improvements and changes.
-The Pipe Builder is an AI-powered tool that generates Pipelex pipelines from natural language descriptions. It helps you quickly prototype pipelines by describing what you want to achieve, and the builder translates your requirements into working `.plx` files.
+The Pipe Builder is an AI-powered tool that generates complete Pipelex pipelines from natural language descriptions. Describe what you want to achieve, and the builder creates a production-ready `.plx` file with concepts, pipes, and all the necessary structure.
-## Overview
+## What It Does
-The Pipe Builder uses AI to:
+The Pipe Builder takes a brief description like:
-- Understand your pipeline requirements from a brief description
-- Generate domain concepts, pipe specifications, and complete pipeline structure
-- Validate the generated pipeline for common errors
-- Automatically fix certain deterministic issues
+> "Take a CV and Job offer in PDF, analyze if they match and generate 5 questions for the interview"
-## Usage
+And generates:
-Generate a pipeline with one validation/fix loop that automatically corrects deterministic issues:
-
-```bash
-pipelex build pipe "Brief description of what the pipeline should do" -o path/to/output.plx
-```
-
-**Example:**
-
-```bash
-pipelex build pipe "Given an expense report, apply company rules" -o results/expense_pipeline.plx
-```
-
-This command:
-
-1. Generates a complete pipeline from your brief
-2. Validates the pipeline structure
-3. Attempts to automatically fix common errors
-4. Saves the final pipeline to the specified path
-
-## Options
-
-The build command supports the following options:
-
-- `--output`, `-o`: Path to save the generated file
-- `--no-output`: Skip saving the file (useful for testing)
+- **Domain concepts** - Data structures for your workflow (e.g., `CVAnalysis`, `InterviewQuestion`)
+- **Pipe operators** - LLM calls, extractions, image generation steps
+- **Pipe controllers** - Sequences, batches, parallels, conditions to orchestrate the flow
+- **A complete bundle** - Ready to validate and run
## How It Works
-The Pipe Builder follows this process:
-
-1. **Analysis**: Analyzes your brief to understand the domain and requirements
-2. **Concept Generation**: Creates appropriate domain concepts for your workflow
-3. **Pipe Generation**: Generates pipe operators and controllers to implement the logic
-4. **Validation**: Validates the generated pipeline structure
-5. **Automatic Fixes**: Fixes common errors like missing inputs or incorrect pipe connections
-
-## Generate Runner Code
+!!! tip "Built with Pipelex"
+ The Pipe Builder is itself a Pipelex pipeline! This showcases the power of Pipelex: a tool that builds pipelines... using a pipeline.
-After creating a pipeline, you can generate Python code to run it with the `build runner` command. This creates a ready-to-use Python script with all necessary imports and example input values.
+The builder follows a multi-step process:
-### Usage
-
-```bash
-pipelex build runner [TARGET] [OPTIONS]
-```
+### 1. Draft the Plan
-**Arguments:**
+First, the AI analyzes your brief and creates a narrative plan describing:
-- `TARGET` - Either a pipe code or a bundle file path, auto-detected according to presence of the .plx file extension
+- The sequence of steps needed
+- Inputs and outputs for each step
+- Where structured data or lists are needed
+- Which orchestration patterns to use (sequence, batch, parallel, condition)
-**Options:**
-
-- `--pipe` - Pipe code to use (alternative to positional argument)
-- `--bundle` - Bundle file path (alternative to positional argument)
-- `--output`, `-o` - Path to save the generated Python file (defaults to `results/run_{pipe_code}.py`)
-
-### Examples
-
-**Generate runner for a pipe:**
-
-```bash
-pipelex build runner my_pipe
-```
+### 2. Draft the Concepts
-**Generate runner from a bundle file:**
+Based on the plan, it identifies and defines the concepts needed:
-```bash
-pipelex build runner my_bundle.plx
-```
+- What data structures are required
+- Which fields each concept needs
+- How concepts relate to each other
+- Which native concepts to reuse (Text, Image, PDF, etc.)
-**Specify a pipe from a bundle:**
+### 3. Structure the Concepts
-```bash
-pipelex build runner --bundle my_bundle.plx --pipe my_pipe
-```
+The concept drafts are formalized into proper Pipelex concept specifications with:
-**Custom output path:**
+- Field names and types
+- Descriptions
+- Required/optional flags
+- Default values
-```bash
-pipelex build runner my_pipe --output custom_runner.py
-```
+### 4. Design the Flow
-### What Gets Generated
+The plan is translated into a structured flow with:
-The generated Python file includes:
+- Pipe operators (PipeLLM, PipeExtract, PipeImgGen)
+- Pipe controllers (PipeSequence, PipeBatch, PipeParallel, PipeCondition)
+- Input/output contracts for each pipe
+- Memory flow between steps
-1. **All necessary imports** - Imports for Pipelex execution, content types, and any custom structures
-2. **Input memory setup** - Example input values based on the pipe's input types
-3. **Pipeline execution** - Async function that executes the pipeline
-4. **Output handling** - Code to extract and display the results
-5. **Main execution block** - Pipelex initialization and asyncio setup
+### 5. Review and Refine
-### Input Type Handling
+The flow is reviewed for consistency:
-- **Native concepts** (Text, Image, PDF, etc.) - Automatically generates appropriate content objects
-- **Custom concepts** - Recursively generates the structure with example values
-- **Structured content** - Creates example data matching the concept's fields
+- Are inputs and outputs properly connected?
+- Is the main pipe correctly identified?
+- Are batches and lists handled properly?
-### Next Steps
+### 6. Generate Pipe Signatures
-After generating the runner file:
+Each pipe gets a formal signature with:
-1. Open the generated Python file
-2. Review and customize the example input values
-3. Run the script: `python results/run_{pipe_code}.py`
-4. Iterate and adjust as needed
+- Unique pipe code
+- Description
+- Input/output types with multiplicity
+- Dependencies on other pipes
-For the complete CLI reference, see [CLI Commands](cli.md).
+### 7. Assemble the Bundle
-## Example Use Cases
+Finally, everything is assembled into a complete Pipelex bundle:
-**Document Processing:**
+- Domain header with name and description
+- All concept definitions
+- All pipe definitions
+- Main pipe identification
-```bash
-pipelex build pipe "Take a CV in a PDF file and a Job offer text, and analyze if they match"
-```
+## The Builder Pipeline
-**Data Transformation:**
+The Pipe Builder is defined in [`pipelex/builder/builder.plx`](https://github.com/Pipelex/pipelex/blob/main/pipelex/builder/builder.plx). The main orchestrator is a `PipeSequence` called `pipe_builder` that chains together:
-```bash
-pipelex build pipe "Extract structured data from invoice images"
```
-
-**Multi-step Workflows:**
-
-```bash
-pipelex build pipe "Given an RFP PDF, build a compliance matrix"
+draft_the_plan → draft_the_concepts → structure_concepts → draft_flow → review_flow → design_pipe_signatures → write_bundle_header → detail_pipe_spec (batched) → assemble_pipelex_bundle_spec
```
-## Current Limitations
-
-The Pipe Builder is in active development and currently:
-
-- Can automatically fix input/output connection errors
-- May require manual adjustments for complex conditional logic or custom functions
-- Validation focuses on structural correctness, not business logic
-
-## Tips for Best Results
-
-- You can be specific in your brief about inputs, outputs, data formats, or structures if you know what you need
-- If you're uncertain about the details, let the AI figure it out and see what it generates
-- Include any domain-specific requirements you're aware of upfront
+Each step uses `PipeLLM` with carefully crafted prompts to guide the AI through the generation process.
-## Iterating on Generated Pipelines
+## Using the Pipe Builder
-After generating a pipeline, you can continue refining it using any Software Engineering (SWE) agent. The generated `.plx` file can be iteratively improved through natural language instructions.
-
-Pipelex provides specialized agent rules (`write_pipelex.md` and `run_pipelex.md`) that guide AI assistants in working with pipelines. You can install these rules for your preferred AI coding assistant using:
+To use the Pipe Builder, use the CLI command:
```bash
-pipelex kit rules
+pipelex build pipe "Your description here"
```
-This command installs the rules for:
-
-- **Cursor**
-- **Claude Code**
-- **OpenAI Codex**
-- **GitHub Copilot**
-- **Windsurf**
-- **Blackbox AI**
-
-These rules help AI assistants understand Pipelex syntax, best practices, and common patterns, making it easier to iterate and refine your generated pipelines.
-
-## Next Steps
-
-After generating a pipeline:
+See the [Build Pipe CLI documentation](cli/build/pipe.md) for:
-1. Review the generated `.plx` file
-2. Test it with sample inputs: `pipelex run <pipe_code> --input-memory-from-json input.json`
-3. Continue iterating using your preferred SWE agent with the Pipelex agent rules
-4. Adjust concepts or pipe configurations as needed
+- Command options
+- Generated file structure
+- Example use cases
+- Tips for best results
-For more information on pipeline structure and customization, see:
+## Related Documentation
-- [Design and Run Pipelines](../../home/6-build-reliable-ai-workflows/pipes/index.md)
-- [Pipe Operators](../../home/6-build-reliable-ai-workflows/pipes/pipe-operators/index.md)
-- [Pipe Controllers](../../home/6-build-reliable-ai-workflows/pipes/pipe-controllers/index.md)
+- [Build Pipe CLI](cli/build/pipe.md) - Command reference and options
+- [Design and Run Pipelines](../6-build-reliable-ai-workflows/pipes/index.md) - Pipeline concepts
+- [Concepts](../6-build-reliable-ai-workflows/concepts/define_your_concepts.md) - Understanding concept definitions
diff --git a/mkdocs.yml b/mkdocs.yml
index 3cbcb9358..5feb4b8ac 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -170,8 +170,18 @@ nav:
- Analytics:
- Observer Data Extraction: home/8-analytics/data-extraction.md
- Tools:
- - CLI: home/9-tools/cli.md
- - Kit Commands: home/9-tools/kit.md
+ - CLI:
+ - Overview: home/9-tools/cli/index.md
+ - Init: home/9-tools/cli/init.md
+ - Validate: home/9-tools/cli/validate.md
+ - Run: home/9-tools/cli/run.md
+ - Show: home/9-tools/cli/show.md
+ - Kit: home/9-tools/cli/kit.md
+ - Build:
+ - Overview: home/9-tools/cli/build/index.md
+ - Pipe: home/9-tools/cli/build/pipe.md
+ - Runner: home/9-tools/cli/build/runner.md
+ - Structures: home/9-tools/cli/build/structures.md
- Pipe Builder: home/9-tools/pipe-builder.md
- Logging: home/9-tools/logging.md
- Advanced Customizations:
diff --git a/pipelex/cli/commands/build/runner_cmd.py b/pipelex/cli/commands/build/runner_cmd.py
index b68c6b00b..3f06d954c 100644
--- a/pipelex/cli/commands/build/runner_cmd.py
+++ b/pipelex/cli/commands/build/runner_cmd.py
@@ -1,6 +1,9 @@
import asyncio
from pathlib import Path
-from typing import Annotated
+from typing import TYPE_CHECKING, Annotated
+
+if TYPE_CHECKING:
+ from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
import click
import typer
@@ -17,10 +20,12 @@ from pipelex.cli.error_handlers import (
from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
from pipelex.core.pipes.inputs.exceptions import PipeInputError
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
+from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.hub import get_required_pipe, get_telemetry_manager
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
from pipelex.pipelex import PACKAGE_VERSION
-from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle
+from pipelex.pipeline.validate_bundle import ValidateBundleError, validate_bundle, validate_bundles_from_directory
+from pipelex.system.registries.class_registry_utils import ClassRegistryUtils
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventProperty
from pipelex.tools.misc.file_utils import (
@@ -37,21 +42,15 @@ SUB_COMMAND_RUNNER = "runner"
def prepare_runner_cmd(
target: Annotated[
str | None,
- typer.Argument(help="Pipe code or bundle file path (auto-detected)"),
+ typer.Argument(help="Bundle file path (.plx) or library directory"),
] = None,
pipe: Annotated[
str | None,
- typer.Option("--pipe", help="Pipe code to use, can be omitted if you specify a bundle (.plx) that declares a main pipe"),
- ] = None,
- bundle: Annotated[
- str | None,
- typer.Option("--bundle", help="Bundle file path (.plx) - uses its main_pipe unless you specify a pipe code"),
+ typer.Option("--pipe", help="Pipe code to use (mandatory for library directory, optional for .plx if it declares a main_pipe)"),
] = None,
output_path: Annotated[
str | None,
- typer.Option(
- "--output", "-o", help="Path to save the generated Python file (defaults to bundle's directory if bundle provided, otherwise 'results/')"
- ),
+ typer.Option("--output", "-o", help="Path to save the generated Python file (defaults to target's directory)"),
] = None,
) -> None:
"""Prepare a Python runner file for a pipe.
@@ -64,87 +63,92 @@ def prepare_runner_cmd(
Custom concept types will have their structure recursively generated.
Examples:
- pipelex build runner my_pipe
- pipelex build runner --bundle my_bundle.plx
- pipelex build runner --bundle my_bundle.plx --pipe my_pipe
pipelex build runner my_bundle.plx
- pipelex build runner my_pipe --output runner.py
+ pipelex build runner my_bundle.plx --pipe my_pipe
+ pipelex build runner ./my_library/ --pipe my_pipe
+ pipelex build runner my_bundle.plx --output runner.py
"""
# Import here to avoid circular imports
from pipelex.cli.commands.build.structures_cmd import generate_structures_from_blueprints # noqa: PLC0415
- # Validate mutual exclusivity
- provided_options = sum([target is not None, pipe is not None, bundle is not None])
- if provided_options == 0:
+ # Show help if no target provided
+ if target is None:
ctx: click.Context = click.get_current_context()
typer.echo(ctx.get_help())
raise typer.Exit(0)
- # Let's analyze the options and determine what pipe code to use and if we need to load a bundle
- pipe_code: str | None = None
- bundle_path: str | None = None
-
- # Determine source:
- if target:
- # Check if target is a directory (not allowed for runner command)
- target_path = Path(target)
- if target_path.is_dir():
- typer.secho(
- f"Failed to run: '{target}' is a directory. The runner command requires a .plx file or a pipe code, not a directory.",
- fg=typer.colors.RED,
- err=True,
- )
- raise typer.Exit(1)
+ # Analyze target type
+ target_path = Path(target)
+ is_directory = target_path.is_dir()
+ is_plx_file = target.endswith(".plx")
- if target.endswith(".plx"):
- bundle_path = target
- if bundle:
- typer.secho(
- "Failed to run: cannot use option --bundle if you're already passing a bundle file (.plx) as positional argument",
- fg=typer.colors.RED,
- err=True,
- )
- raise typer.Exit(1)
- else:
- pipe_code = target
- if pipe:
- typer.secho(
- "Failed to run: cannot use option --pipe if you're already passing a pipe code as positional argument",
- fg=typer.colors.RED,
- err=True,
- )
- raise typer.Exit(1)
-
- if bundle:
- assert not bundle_path, "bundle_path should be None at this stage if --bundle is provided"
- bundle_path = bundle
-
- if pipe:
- assert not pipe_code, "pipe_code should be None at this stage if --pipe is provided"
- pipe_code = pipe
-
- if not pipe_code and not bundle_path:
- typer.secho("Failed to run: no pipe code or bundle file specified", fg=typer.colors.RED, err=True)
+ # Validate: directory requires --pipe
+ if is_directory and not pipe:
+ typer.secho(
+ f"Failed to run: '{target}' is a directory. The --pipe option is required when using a library directory.",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+
+ # Validate: target must be a .plx file or directory
+ if not is_directory and not is_plx_file:
+ typer.secho(
+ f"Failed to run: '{target}' is not a .plx file or directory.",
+ fg=typer.colors.RED,
+ err=True,
+ )
raise typer.Exit(1)
- async def prepare_runner(pipe_code: str | None = None, bundle_path: str | None = None):
- bundle_blueprint = None
- if bundle_path:
+ pipe_code = pipe
+ bundle_path = target if is_plx_file else None
+ library_dir = target if is_directory else None
+
+ async def prepare_runner(
+ pipe_code: str | None = None,
+ bundle_path: str | None = None,
+ library_dir: str | None = None,
+ ):
+ all_blueprints: list[PipelexBundleBlueprint] = []
+
+ if library_dir:
+ # Load all bundles from library directory
+ try:
+ typer.echo(f"🔍 Loading bundles from: {library_dir}")
+ validate_result = await validate_bundles_from_directory(directory=Path(library_dir))
+ all_blueprints = validate_result.blueprints
+ typer.echo(f"✅ Loaded {len(all_blueprints)} blueprint(s)")
+
+ # pipe_code is mandatory for library directory (already validated above)
+ assert pipe_code is not None
+ typer.echo(f"Using pipe '{pipe_code}' from library '{library_dir}'")
+
+ except FileNotFoundError as exc:
+ typer.secho(f"Failed to load library '{library_dir}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ except ValidateBundleError as exc:
+ typer.secho(f"Failed to load library '{library_dir}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ except PipeInputError as exc:
+ typer.secho(f"Failed to load library '{library_dir}': {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+
+ elif bundle_path:
try:
validate_bundle_result = await validate_bundle(plx_file_path=bundle_path)
- bundle_blueprint = validate_bundle_result.blueprints[0]
+ all_blueprints = validate_bundle_result.blueprints
+ first_blueprint = all_blueprints[0]
if not pipe_code:
- main_pipe_code = bundle_blueprint.main_pipe
+ main_pipe_code = first_blueprint.main_pipe
if not main_pipe_code:
- # Fall back to first pipe if no main_pipe declared
- if bundle_blueprint.pipe:
- main_pipe_code = next(iter(bundle_blueprint.pipe.keys()))
- typer.echo(f"No main_pipe declared, using first pipe '{main_pipe_code}' from bundle '{bundle_path}'")
- else:
- typer.secho(f"Bundle '{bundle_path}' has no pipes defined", fg=typer.colors.RED, err=True)
- raise typer.Exit(1)
- else:
- typer.echo(f"Using main pipe '{main_pipe_code}' from bundle '{bundle_path}'")
+ # No main_pipe declared - require --pipe option
+ typer.secho(
+ f"Bundle '{bundle_path}' has no main_pipe declared. Use --pipe to specify which pipe to use.",
+ fg=typer.colors.RED,
+ err=True,
+ )
+ raise typer.Exit(1)
+ typer.echo(f"Using main pipe '{main_pipe_code}' from bundle '{bundle_path}'")
pipe_code = main_pipe_code
else:
typer.echo(f"Using pipe '{pipe_code}' from bundle '{bundle_path}'")
@@ -157,8 +161,8 @@ def prepare_runner_cmd(
except PipeInputError as exc:
typer.secho(f"Failed to load bundle '{bundle_path}': {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(1) from exc
- elif not pipe_code:
- typer.secho("Failed to run: no pipe code specified", fg=typer.colors.RED, err=True)
+ else:
+ typer.secho("Failed to run: no bundle file or library directory specified", fg=typer.colors.RED, err=True)
raise typer.Exit(1)
# Get the pipe
@@ -168,16 +172,12 @@ def prepare_runner_cmd(
typer.secho(f"❌ Error: Could not find pipe '{pipe_code}': {exc}", fg=typer.colors.RED)
raise typer.Exit(1) from exc
- # Determine if output is a list from the bundle blueprint
- output_is_list = False
- if bundle_blueprint and bundle_blueprint.pipe and pipe_code in bundle_blueprint.pipe:
- pipe_blueprint = bundle_blueprint.pipe[pipe_code]
- output_parse = parse_concept_with_multiplicity(pipe_blueprint.output)
- output_is_list = output_parse.multiplicity is not None
-
- # Determine output path - use bundle's directory if bundle provided, otherwise results/
+ # Determine output path - use target's directory
if output_path:
final_output_path = output_path
+ elif library_dir:
+ # Place runner in the library directory
+ final_output_path = str(Path(library_dir) / f"run_{pipe_code}.py")
elif bundle_path:
# Place runner in the same directory as the PLX file
bundle_dir = Path(bundle_path).parent
@@ -191,16 +191,41 @@ def prepare_runner_cmd(
output_dir = Path(final_output_path).parent
# Generate structures folder FIRST (before runner, since runner imports from structures)
- if bundle_blueprint:
- structures_output_dir = output_dir / "structures"
+ structures_output_dir = output_dir / "structures"
+ if all_blueprints:
generated_structures = generate_structures_from_blueprints(
- blueprints=[bundle_blueprint],
+ blueprints=all_blueprints,
output_directory=structures_output_dir,
- target_path=output_dir, # Check for existing structures in the bundle's directory
+ target_path=output_dir, # Check for existing structures in the target directory
)
if generated_structures:
typer.secho(f"✅ Generated {len(generated_structures)} structure(s) in: {structures_output_dir}", fg=typer.colors.GREEN)
+ # Register all structure classes from the output directory so generate_runner_code can find them
+ # This includes both newly generated structures and any manually-created ones
+ if structures_output_dir.exists():
+ ClassRegistryUtils.register_classes_in_folder(
+ folder_path=str(structures_output_dir),
+ base_class=StuffContent,
+ is_recursive=True,
+ )
+ # Also register any manually-created classes in the target directory (outside structures/)
+ ClassRegistryUtils.register_classes_in_folder(
+ folder_path=str(output_dir),
+ base_class=StuffContent,
+ is_recursive=True,
+ force_exclude_dirs=[str(structures_output_dir.resolve())] if structures_output_dir.exists() else None,
+ )
+
+ # Determine if output is a list from any of the blueprints
+ output_is_list = False
+ for blueprint in all_blueprints:
+ if blueprint.pipe and pipe_code in blueprint.pipe:
+ pipe_blueprint = blueprint.pipe[pipe_code]
+ output_parse = parse_concept_with_multiplicity(pipe_blueprint.output)
+ output_is_list = output_parse.multiplicity is not None
+ break
+
# Generate the runner code
try:
runner_code = generate_runner_code(the_pipe, output_multiplicity=output_is_list)
@@ -225,7 +250,7 @@ def prepare_runner_cmd(
tag(name=EventProperty.PIPELEX_VERSION, value=PACKAGE_VERSION)
tag(name=EventProperty.CLI_COMMAND, value=f"{COMMAND} {SUB_COMMAND_RUNNER}")
- asyncio.run(prepare_runner(pipe_code=pipe_code, bundle_path=bundle_path))
+ asyncio.run(prepare_runner(pipe_code=pipe_code, bundle_path=bundle_path, library_dir=library_dir))
except PipeOperatorModelChoiceError as exc:
handle_model_choice_error(exc, context=ErrorContext.BUILD)