Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Added docs

Louis Choquelb13a3fd74585
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
new file mode 100644
index 000000000..4fd40f9cc
--- /dev/null
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
@@ -0,0 +1,289 @@
+# Pipeline Creation
+
+Pipelex provides powerful tools to automatically generate pipeline blueprints from natural language requirements. This feature leverages AI to translate your ideas into working pipeline code, dramatically speeding up development.
+
+## Overview
+
+The pipeline creation system works in several stages:
+
+1. **Draft Generation**: Convert natural language requirements into structured markdown drafts
+2. **Blueprint Creation**: Transform drafts into formal pipeline blueprints (TOML format)
+3. **Validation**: Automatically validate and fix common issues
+4. **Implementation**: Generate executable pipeline code
+
+## Core Commands
+
+### Build Blueprint
+
+Generate a complete pipeline blueprint from requirements:
+
+```bash
+pipelex build blueprint PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]
+```
+
+**Example:**
+```bash
+pipelex build blueprint gen_photopposite \
+ --requirements "Take a photo as input, analyze its content and its most important feature, imagine the opposite to that feature, render it as a photo" \
+ -c pipelex/libraries \
+ -o pipelex/libraries/pipelines/temp/result_4/photopposite \
+ --validate
+```
+
+**Options:**
+- `--domain, -d`: Domain for the pipeline (default: "wip_domain")
+- `--requirements, -r`: Requirements text directly
+- `--file, -f`: Path to file containing requirements
+- `--output, -o`: Output path for generated files
+- `--validate`: Validate the generated blueprint
+- `--config-folder-path, -c`: Path to libraries folder
+
+### Build Draft
+
+Generate just a pipeline draft (intermediate step):
+
+```bash
+pipelex build draft PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]
+```
+
+**Options:**
+- `--raw`: Generate raw text draft instead of structured draft
+- All other options same as blueprint command
+
+### Validate Blueprint
+
+Validate an existing blueprint file:
+
+```bash
+pipelex validate blueprint BLUEPRINT_PATH [OPTIONS]
+```
+
+**Example:**
+```bash
+pipelex validate blueprint pipelex/libraries/pipelines/temp/result_3/photopposite.toml \
+ -c pipelex/libraries
+```
+
+**Options:**
+- `--config-folder-path, -c`: Path to libraries folder
+- `--fix/--no-fix, -f/-F`: Enable/disable automatic error fixing (default: enabled)
+
+## Complete Workflow
+
+### 1. Requirements Analysis
+
+Start with clear, specific requirements:
+
+```text
+Take a photo as input, analyze its content and its most important feature,
+imagine the opposite to that feature, render it as a photo
+```
+
+### 2. Draft Generation
+
+The system first creates a structured markdown draft:
+
+```markdown
+# Pipeline: gen_photopposite
+Domain: image_processing
+Purpose: Generate opposite version of photo's main feature
+
+## Inputs
+- `photo` : Input photo to analyze and transform
+
+## Flow
+
+WorkingMemory starts with: {photo}
+
+### Step 1: Analyze photo content
+LLM[analyze]: (photo) → analysis
+
+WorkingMemory now contains: {photo, analysis}
+
+### Step 2: Generate opposite concept
+LLM[conceptualize]: (analysis) → opposite_concept
+
+WorkingMemory now contains: {photo, analysis, opposite_concept}
+
+### Step 3: Render opposite photo
+ImgGen: (opposite_concept) → opposite_photo
+
+WorkingMemory now contains: {photo, analysis, opposite_concept, opposite_photo}
+
+## Output
+Returns: `opposite_photo` - Photo showing opposite of main feature
+```
+
+### 3. Blueprint Creation
+
+The draft is converted into a formal TOML blueprint:
+
+```toml
+domain = "image_processing"
+definition = "Generate opposite version of photo's main feature"
+
+[concept]
+PhotoAnalysis = "Analysis of photo content and main features"
+OppositePhotoPrompt = "Prompt for generating opposite version of photo"
+
+[pipe.gen_photopposite]
+PipeSequence = "Generate opposite version of photo's main feature"
+output = "Image"
+steps = [
+ { pipe = "analyze_photo", result = "analysis" },
+ { pipe = "create_opposite_concept", result = "opposite_concept" },
+ { pipe = "render_opposite", result = "opposite_photo" }
+]
+
+[pipe.analyze_photo]
+PipeLLM = "Analyze photo content and identify main feature"
+inputs = { photo = "Image" }
+output = "PhotoAnalysis"
+prompt_template = """
+Analyze this photo and identify its most important visual feature.
+
+@photo
+
+Focus on the dominant element, color, mood, or characteristic that defines this image.
+"""
+
+[pipe.create_opposite_concept]
+PipeLLM = "Create concept for opposite version"
+inputs = { analysis = "PhotoAnalysis" }
+output = "OppositePhotoPrompt"
+prompt_template = """
+Based on this photo analysis, create a detailed prompt for generating the opposite version.
+
+@analysis
+
+Describe what the opposite would look like, focusing on reversing the main feature while maintaining photo realism.
+"""
+
+[pipe.render_opposite]
+PipeImgGen = "Generate the opposite photo"
+inputs = { opposite_concept = "OppositePhotoPrompt" }
+output = "Image"
+```
+
+### 4. Validation and Fixing
+
+The system automatically:
+
+- **Validates syntax**: Ensures TOML structure is correct
+- **Checks concepts**: Verifies all concepts are properly defined
+- **Validates inputs**: Ensures all pipe inputs are available
+- **Fixes errors**: Automatically adds missing inputs or corrects common issues
+- **Dry runs**: Tests pipeline execution without making API calls
+
+## Generated Files
+
+When you run the build command, several files are created:
+
+```
+output_path_base_draft.md # Original markdown draft
+output_path_base_rough.toml # Initial blueprint
+output_path_base_rough.json # Blueprint in JSON format
+output_path_base_fixed.toml # Validated and fixed blueprint (if --validate used)
+output_path_base_fixed.json # Fixed blueprint in JSON format
+```
+
+## Best Practices
+
+### Writing Requirements
+
+**Good requirements:**
+```text
+Analyze a PDF document, extract key financial data, and generate a summary report with charts
+```
+
+**Better requirements:**
+```text
+Take a PDF financial statement as input, use OCR to extract text and tables,
+identify revenue, expenses, and profit figures, then generate a structured
+summary report with visual charts showing trends
+```
+
+### Domain Selection
+
+Choose descriptive domain names:
+- `financial_analysis` instead of `finance`
+- `document_processing` instead of `docs`
+- `image_generation` instead of `images`
+
+### Output Path Organization
+
+Organize generated pipelines:
+```
+pipelex/libraries/pipelines/
+ temp/ # Temporary/experimental pipelines
+ result_1/
+ result_2/
+ domain_name/ # Production pipelines
+ pipeline_name.toml
+ structures.py
+```
+
+## Advanced Features
+
+### Custom Templates
+
+The system uses built-in templates from:
+- `.pipelex/create/draft_pipelines.md` - Drafting guidelines
+- `.pipelex/create/build_pipelines.md` - Blueprint creation rules
+
+### Error Fixing
+
+The validation system can automatically fix:
+- Missing input variables
+- Incorrect concept references
+- Malformed pipe definitions
+- Input/output mismatches
+
+### Integration with Existing Libraries
+
+Generated blueprints automatically integrate with:
+- Native concepts (`Text`, `Image`, `PDF`, etc.)
+- Library concepts (`documents.TextAndImages`, etc.)
+- Custom domain concepts
+
+## Troubleshooting
+
+### Common Issues
+
+**"Missing input variable" error:**
+- Usually fixed automatically with `--validate`
+- Check that all pipe inputs are available in WorkingMemory
+
+**"Invalid concept" error:**
+- Ensure concept names use PascalCase
+- Don't redefine native concepts
+- Avoid plurals in concept names
+
+**"Pipe not found" error:**
+- Check that all referenced pipes exist in the blueprint
+- Verify pipe names match exactly
+
+### Getting Help
+
+```bash
+# Validate your entire setup
+pipelex validate all -c pipelex/libraries
+
+# Test a specific pipe
+pipelex validate pipe your_pipe_name -c pipelex/libraries
+
+# Check blueprint structure
+pipelex validate blueprint your_blueprint.toml -c pipelex/libraries --no-fix
+```
+
+## Next Steps
+
+After creating your blueprint:
+
+1. **Review the generated code** - Check that it matches your requirements
+2. **Create structure classes** - Add Python classes for custom concepts if needed
+3. **Test the pipeline** - Run validation and dry runs
+4. **Execute the pipeline** - Use `execute_pipeline()` to run it with real data
+5. **Iterate and improve** - Refine based on results
+
+The pipeline creation system makes it easy to go from idea to working code in minutes, while maintaining the full power and flexibility of Pipelex pipelines.