Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Refactor/epic library blueprint 2 generate blueprint (#218)

Louis Choquelda3ed609b97b
diff --git a/.pipelex/create/build_pipelines.md b/.pipelex/create/build_pipelines.md
new file mode 100644
index 000000000..1dc666313
--- /dev/null
+++ b/.pipelex/create/build_pipelines.md
@@ -0,0 +1,316 @@
+# Building Pipelex Pipelines from Drafts
+
+This guide explains how to translate a markdown pipeline draft into a formal `PipelineBlueprint` structure that can be validated and executed by Pipelex.
+
+## Overview
+
+1. Convert the draft's narrative flow into a formal domain/concepts/pipes structure
+2. Map draft notation to specific pipe types and configurations
+3. Ensure all working memory references are properly captured in operator pipe inputs (wrapping up input requirements for controller pipes will be done later)
+
+## Translation Workflow
+
+### Step 1: Extract Domain Information
+
+Create the blueprint root:
+```json
+{
+ "domain": "domain_name",
+ "definition": "What this pipeline does"
+}
+```
+
+### Step 2: Identify and Define Concepts
+
+Review the draft for all data types mentioned:
+
+- **Input variables** → Concepts needed
+- **Intermediate results** → Concepts needed
+- **Final output** → Concept needed
+
+#### Concept Rules
+
+1. **Use native concepts when possible**:
+ - `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`
+ - Don't redefine these
+
+2. **Use library concepts when applicable**:
+ - `documents.TextAndImagesContent`
+ - `images.VisualDescription`
+ - `images.ImgGenPrompt`
+ - `images.Photo`
+
+3. **Create domain concepts for specialized data**:
+ - Use PascalCase: `Analysis`, `TechnicalReport`, `UserQuery`
+ - Never use plurals: `Document` not `Documents`
+ - Never use circumstantial names: `Photo` not `InputPhoto`
+ - Avoid adjectives: `Text` not `LongText`
+
+#### Concept Definition Formats
+
+Quick form (for simple text-based concepts):
+```json
+"concept": {
+ "UserQuery": "Natural language question from a user",
+ "Summary": "Condensed version of content"
+}
+```
+
+Full form (when structure or refinement needed):
+```json
+"concept": {
+ "TechnicalAnalysis": {
+ "definition": "Technical analysis of a system",
+ "refines": "Text"
+ }
+}
+```
+
+### Step 3: Translate Flow Steps to Pipes
+
+Map each step from your draft to the appropriate pipe type and configuration.
+
+#### Operator Mappings
+
+| Draft Notation | Pipe Type | Key Configuration |
+|---------------|-----------|-------------------|
+| `LLM[purpose]: (inputs) → output` | `PipeLLM` | Set `prompt_template` or `prompt` |
+| `OCR: (document) → pages` | `PipeOcr` | Input must be named `ocr_input` |
+| `ImgGen: (prompt) → image` | `PipeImgGen` | Configure image generation settings |
+| `Func[name]: (data) → result` | `PipeFunc` | Reference Python function |
+
+#### Controller Mappings
+
+| Draft Pattern | Pipe Type | Structure |
+|--------------|-----------|-----------|
+| Sequential steps (Step 1, Step 2...) | `PipeSequence` | Use `steps` array |
+| Process each item in list | `PipeSequence` step with batch | Add `batch_over` and `batch_as` to step |
+| Parallel branches (Branch A, Branch B...) | `PipeParallel` | Use `parallels` array |
+| Conditional routing (Case "x"...) | `PipeCondition` | Use `expression` and `pipe_map` |
+
+### Step 4: Configure Each Pipe
+
+#### PipeLLM Configuration
+
+From draft step:
+```markdown
+### Step: Analyze document for key insights
+LLM[analyze]: (document, criteria) → analysis
+```
+
+To blueprint:
+```json
+"analyze_document": {
+ "type": "PipeLLM",
+ "definition": "Analyze document for key insights",
+ "inputs": {
+ "document": "Text",
+ "criteria": "Text"
+ },
+ "output": "Analysis",
+ "prompt_template": "Analyze the following document based on the provided criteria.\n\n@document\n\n@criteria"
+}
+```
+
+**Prompt Template Rules**:
+
+- Use `@variable` for block content (multi-line)
+- Use `$variable` for inline content (within sentences)
+
+#### PipeOcr Configuration
+
+From draft step:
+```markdown
+### Step: Extract content from PDF
+OCR: (document) → pages
+```
+
+To blueprint:
+```json
+"extract_pages": {
+ "type": "PipeOcr",
+ "definition": "Extract content from PDF",
+ "inputs": {
+ "ocr_input": "PDF" // Must be named ocr_input
+ },
+ "output": "Page", // Output is always Page concept
+ "page_images": true,
+ "page_views": true
+}
+```
+
+#### PipeSequence Configuration
+
+From draft sequence:
+```markdown
+Step 1: Extract information
+extract_info(document) → key_info
+
+Step 2: Analyze information
+analyze(key_info) → analysis
+```
+
+To blueprint:
+```json
+"process_document": {
+ "type": "PipeSequence",
+ "definition": "Extract and analyze document",
+ "inputs": {
+ "document": "Text"
+ },
+ "output": "Analysis",
+ "steps": [
+ {
+ "pipe": "extract_info",
+ "result": "key_info"
+ },
+ {
+ "pipe": "analyze_info",
+ "result": "analysis"
+ }
+ ]
+}
+```
+
+#### Batch Processing Configuration
+
+From draft batch:
+```markdown
+Process each item in documents:
+analyze_document(document) → analysis
+```
+
+To blueprint step within PipeSequence:
+```json
+{
+ "pipe": "analyze_single",
+ "batch_over": "documents",
+ "batch_as": "document",
+ "result": "analyses"
+}
+```
+Using the above, the required input used is `documents`, and by looping on the items of that list, each one will be named `document`, and that's how it will be received in the `analyze_single` pipe.
+
+#### PipeCondition Configuration
+
+From draft conditional:
+```markdown
+Route based on category:
+Case "technical": analyze_technical(document) → analysis
+Case "legal": analyze_legal(document) → analysis
+```
+
+To blueprint:
+```json
+"route_analysis": {
+ "type": "PipeCondition",
+ "definition": "Route document to appropriate analyzer",
+ "inputs": {
+ "document": "Text",
+ "category": "Text"
+ },
+ "output": "Analysis",
+ "expression": "category",
+ "pipe_map": {
+ "technical": "analyze_technical",
+ "legal": "analyze_legal"
+ }
+}
+```
+
+## Complete Example: Draft to Blueprint
+
+### Draft (Markdown)
+
+```markdown
+# Pipeline: document_insights
+Domain: analysis
+Purpose: Extract insights from documents based on criteria
+
+## Inputs
+- `document` : Document to analyze
+- `criteria` : Analysis criteria
+
+## Flow
+
+### Step 1: Extract key information
+LLM[extract]: (document) → key_points
+
+### Step 2: Analyze against criteria
+LLM[analyze]: (key_points, criteria) → insights
+
+## Output
+Returns: `insights`
+```
+
+### Blueprint (JSON)
+
+```json
+{
+ "domain": "analysis",
+ "definition": "Extract insights from documents based on criteria",
+ "concept": {
+ "KeyPoints": "Important information extracted from document",
+ "Insights": "Analysis results based on specific criteria"
+ },
+ "pipe": {
+ "document_insights": {
+ "type": "PipeSequence",
+ "definition": "Extract and analyze document insights",
+ "output": "Insights",
+ "steps": [
+ {
+ "pipe": "extract_key_points",
+ "result": "key_points"
+ },
+ {
+ "pipe": "analyze_points",
+ "result": "insights"
+ }
+ ]
+ },
+ "extract_key_points": {
+ "type": "PipeLLM",
+ "definition": "Extract key information",
+ "inputs": {
+ "document": "Text"
+ },
+ "output": "KeyPoints",
+ "prompt_template": "Extract the key points from this document.\n\n@document"
+ },
+ "analyze_points": {
+ "type": "PipeLLM",
+ "definition": "Analyze against criteria",
+ "inputs": {
+ "key_points": "KeyPoints",
+ "criteria": "Text"
+ },
+ "output": "Insights",
+ "prompt_template": "Analyze these key points against the provided criteria.\n\n@key_points\n\n@criteria"
+ }
+ }
+}
+```
+
+## Validation Checklist
+
+Before finalizing your blueprint:
+
+- [ ] Domain is defined with a single word or snake_case code
+- [ ] All concepts follow naming conventions (PascalCase, no plurals, no adjectives, no circumstantial names)
+- [ ] Native concepts are not redefined
+- [ ] Each pipe has `type`, `definition`, `inputs`, and `output`
+- [ ] Controller pipes don't need to declare inputs, they will be inferred from the blueprint
+- [ ] OCR pipes use `ocr_input` as the input variable name
+- [ ] Batch operations specify both `batch_over` and `batch_as`
+- [ ] Variable names are semantic (describe what they are, not their role)
+- [ ] All referenced sub-pipes exist in the blueprint
+- [ ] Prompt templates use `@` for blocks and `$` for inline content
+
+## Common Pitfalls to Avoid
+
+2. **Wrong OCR input name**: Must be `ocr_input`, not `document` or `pdf`
+3. **Redefining native concepts**: Use `Text` directly, don't create `MyText`
+4. **Circumstantial naming**: Use `Photo` not `InputPhoto`, `Analysis` not `FinalAnalysis`
+5. **Missing result names**: Each step in a sequence should specify its `result` name
+6. **Inconsistent variable names**: The same data should have the same name throughout
diff --git a/.pipelex/create/design_pipelines.md b/.pipelex/create/design_pipelines.md
new file mode 100644
index 000000000..2a3fcfb80
--- /dev/null
+++ b/.pipelex/create/design_pipelines.md
@@ -0,0 +1,270 @@
+## Designing Pipelex pipelines
+
+This section explains how to design pipelines for Pipelex.
+
+### Workflow for a designer agent
+
+- Plan the pipeline in natural language (goal, inputs, steps, outputs).
+
+- Translate the plan into a `PipelineBlueprint` object graph: domain, concepts, pipes.
+
+- Define controllers first (`PipeSequence`, `PipeParallel`, `PipeCondition`), then operators (`PipeLLM`, `PipeOcr`, etc.).
+
+- For each new pipe, detail its `type` and `definition`.
+
+- Ensure each pipe’s `inputs` shows all required variables, including those used by nested steps or conditional branches. `inputs` is a dict mapping variable names to concept codes.
+
+- Choose appropriate native or structured output concepts.
+
+### Root blueprint object: PipelineBlueprint
+
+`PipelineBlueprint` fields:
+
+- `domain: str` — required. Single word or snake_case code for the domain.
+
+- `definition: Optional[str]` — optional textual description of the domain.
+
+- `system_prompt: Optional[str]` — default system prompt applied by LLM pipes in this domain if they don’t provide one.
+
+- `system_prompt_to_structure: Optional[str]` — domain-level instructions to structure outputs.
+
+- `prompt_template_to_structure: Optional[str]` — domain-level instructions to structure prompts.
+
+- `concept: Dict[str, Union[str, ConceptBlueprint]]` — concept section. Keys are concept names (PascalCase). Values are either:
+
+ - a string (quick form): the concept definition; or
+
+ - a `ConceptBlueprint` object (full form).
+
+- `pipe: Dict[str, Dict[str, Any]]` — pipes section. Keys are pipe codes (snake_case). Values are per-pipe details dicts that will be validated into concrete `PipeBlueprint` subclasses based on their `type`.
+
+### Concepts
+
+- Conceiving and naming concepts:
+ - Use PascalCase for concept names.
+
+ - DO NOT use plurals in concept names (use `Item` not `Items`).
+
+ - DO NOT use adjectives in concept names (use `Text`, not `LargeText`).
+
+ - When a concept or concept reference is unqualified (no `domain.` prefix), it will be prefixed with the current blueprint `domain` during validation/expansion, unless its name corresponds to one of the native concepts.
+
+ - Do not redefine native concepts (use them directly): `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`.
+
+ - Textual concepts refining `native.Text` are acceptable too, and you don't need to specify their structure: by default it'll be text.
+
+ - Reuse existing concepts whenever possible. For instance if you take a `Poem` and make it dark, the result is a `Poem` too: you MUST NOT create a new `DarkPoem` concept in this kind of situation.
+
+ - Concepts from the base library are also provided and must be used whenever possible:
+
+ - `documents.TextAndImagesContent` = "A content that comprises text and images where the text can include local links to the images"
+
+ - `images.VisualDescription` = "Visual description of something"
+
+ - `images.ImgGenPrompt`:
+ - definition = "Prompt to generate an image"
+ - refines = "Text"
+
+ - `images.Photo`:
+ - definition = "Photo"
+ - structure = "ImageContent"
+ - refines = "Image"
+
+ - DO NOT create a concept qualified with circumstances like `InputPhoto`: it's just a `Photo`, it so happens that we're using it as input, OK, but that will be convened by the rest of the structure.
+
+- Quick form (string):
+
+ - `concept["ConceptName"] = "Definition of the concept"`
+
+ - If no `structure` is specified and the name isn’t a known structure class, it implicitly refines `native.Text`.
+
+- Full form (`ConceptBlueprint` from `pipelex/core/concept_factory.py`):
+
+ - `definition: str` — required.
+
+ - `structure: Optional[str]` — name of a registered `StructuredContent` subclass (typically defined in `pipelex_libraries/pipelines/<domain>.py`). If provided, it must exist in the class registry.
+
+ - `refines: Union[str, List[str]]` — concept codes or names it refines. May be a single string or a list. Unqualified names are prefixed with the current `domain` automatically.
+
+ - `domain: Optional[str]` — not required when provided via the library loader.
+
+### Pipes: overall rules
+
+Each entry under `pipe` is a dict that will be converted into a typed `PipeBlueprint` (see `pipelex/core/pipe_blueprint.py` for the base and factories in `pipe_controllers/` and `pipe_operators/`). Use the new format:
+
+- `type: str` — the pipe class name, e.g., `PipeLLM`, `PipeOcr`, `PipeSequence`, `PipeParallel`, `PipeCondition`, `PipeFunc`, `PipeImgGen`.
+
+- `definition: str` — human-readable description of the pipe’s responsibility.
+
+- `inputs: Optional[Dict[str, str]]` — map of variable name -> concept code or name. Unqualified names are prefixed with `domain`.
+
+- `output: str` — target concept code or name. Unqualified names are prefixed with `domain`.
+
+Important:
+
+- For controller pipes (`PipeSequence`, `PipeParallel`, `PipeCondition`), the parent pipe’s `inputs` must include all variables required by its sub-pipes or conditional branches that come from the initial working memory (i.e., not produced by previous steps).
+
+- Multiplicity: some pipes and sub-steps can produce multiple outputs (see below). Plan result names accordingly.
+
+### Pipe-specific blueprint schemas
+
+Below are the additional fields expected per type, on top of the base fields above (`type`, `definition`, `inputs`, `output`). All fields are validated by their blueprint classes.
+
+#### PipeLLM (operators/pipe_llm_factory.py)
+
+- Prompt sources (choose at most one per group):
+
+ - System prompt: one of `system_prompt`, `system_prompt_name`, `system_prompt_template`, `system_prompt_template_name`.
+
+ - User prompt: one of `prompt`, `prompt_name`, `prompt_template`, `template_name`.
+
+- LLM selection:
+
+ - `llm: Optional[LLMSettingOrPresetId]` — model for text.
+
+ - `llm_to_structure: Optional[LLMSettingOrPresetId]` — model for object structuring.
+
+- Structuring:
+
+ - `structuring_method: Optional[StructuringMethod]`
+
+ - `prompt_template_to_structure: Optional[str]`
+
+ - `system_prompt_to_structure: Optional[str]`
+
+- Multiplicity:
+
+ - `nb_output: Optional[int]` — exact number of outputs; mutually exclusive with `multiple_output`.
+
+ - `multiple_output: Optional[bool]` — allow variable number of outputs.
+
+Behavioral notes:
+
+- If no system prompt is provided, the domain’s `system_prompt` (from the root blueprint) is used if available.
+
+- Inputs that are `Image` concepts are automatically exposed as user images to the LLM; other inputs become text blocks.
+
+- Prompt template insertion style for the agent:
+
+ - Use block insertion (equivalent to `@name`) for long, multi-line stuff.
+
+ - Use inline insertion (equivalent to `$name`) for short tokens embedded in sentences.
+
+ - Do not synthesize literal markers; provide structured prompt fields and let the runtime render appropriately.
+
+#### PipeOcr (operators/pipe_ocr_factory.py)
+
+- Required input variable name: `ocr_input`.
+
+- The `ocr_input` concept must be `Image` or `PDF`.
+
+- Additional options:
+
+ - `ocr_platform: Optional[OcrPlatform]` — defaults to Mistral.
+
+ - `page_images: bool` — include detected images.
+
+ - `page_image_captions: bool` — caption images.
+
+ - `page_views: bool` — include rendered page views.
+
+ - `page_views_dpi: Optional[int]` — DPI for page views; defaults to config.
+
+- Output concept should be `Page`. The operational result is a list of `Page` items.
+
+#### PipeSequence (controllers/pipe_sequence_factory.py)
+
+- `steps: List[SubPipeBlueprint]`
+
+- `SubPipeBlueprint` fields (controllers/sub_pipe_factory.py):
+
+ - `pipe: str` — referenced pipe code to execute.
+
+ - `result: Optional[str]` — name to assign to this step’s output in working memory.
+
+ - Multiplicity (choose at most one): `nb_output: Optional[int]`, `multiple_output: Optional[bool]`.
+
+ - Batch (both or none): `batch_over: Union[bool, str]` and `batch_as: Optional[str]`.
+
+Rules:
+
+- If `batch_over` is provided, `batch_as` is required; if `batch_as` is provided, `batch_over` is required.
+
+- When batching, the step runs in parallel over each element of the specified list, and the `result` becomes a list.
+
+#### PipeParallel (controllers/pipe_parallel_factory.py)
+
+- `parallels: List[SubPipeBlueprint]` — each must specify `result`.
+
+- `add_each_output: bool = True` — whether to add each parallel result separately to memory.
+
+- `combined_output: Optional[str]` — optional combined output concept code; if provided without a domain prefix, it’s prefixed automatically.
+
+Rules:
+
+- At least one of `add_each_output` or `combined_output` must be set appropriately (cannot disable both behaviors).
+
+#### PipeCondition (controllers/pipe_condition_factory.py)
+
+- Choose exactly one of:
+
+ - `expression: Optional[str]` — direct expression.
+
+ - `expression_template: Optional[str]` — Jinja2 template expression.
+
+- `pipe_map: Dict[str, str]` — map of evaluated expression values to pipe codes to execute.
+
+- `default_pipe_code: Optional[str]` — optional fallback pipe code if no key matches.
+
+- `add_alias_from_expression_to: Optional[str]` — if set, store the evaluated expression under this variable name in working memory.
+
+### Input and output concepts resolution
+
+- Unqualified concept names in `inputs` and `output` are automatically prefixed with the current `domain` during blueprint validation (see `PipeBlueprint.add_domain_prefix`).
+
+- For `PipeParallel.combined_output`, the same domain-prefixing rule applies.
+
+### Checklist for the agent
+
+- Define `domain` and optional domain-level prompts (`system_prompt`, `system_prompt_to_structure`, `prompt_template_to_structure`).
+
+- Create `concept` entries using either quick string form or full `ConceptBlueprint` with `definition`, optional `structure`, and `refines`.
+
+- For each pipe under `pipe`:
+
+ - Provide `type`, `definition`, `inputs`, `output`.
+
+ - Add type-specific fields (see sections above).
+
+ - Ensure controller pipes include all upstream inputs needed by nested steps/branches.
+
+ - Set multiplicity (`nb_output`/`multiple_output`) and batch settings where needed.
+
+ - For OCR, use `ocr_input` with `Image` or `PDF`, and set `output` to `Page`.
+
+- Validate mentally that every referenced sub-pipe exists and every input it requires is either provided by the parent `inputs` or produced by a previous step.
+
+### Minimal skeleton (object form)
+
+The agent should generate a structure similar to the following (illustrative, not exhaustive):
+
+```json
+{
+ "domain": "my_domain",
+ "definition": "What this domain covers",
+ "system_prompt": "Global system prompt for LLM pipes (optional)",
+ "concept": {
+ "UserQuery": "Natural language question from a user",
+ "Answer": { "definition": "Answer text" }
+ },
+ "pipe": {
+ "answer_question": {
+ "type": "PipeLLM",
+ "definition": "Produce an answer from a question and context",
+ "inputs": { "question": "UserQuery", "context": "Text" },
+ "output": "Answer",
+ "prompt_template": "Analyze the question and context, then answer concisely.\n\n@question\n\n@context"
+ }
+ }
+}
+```
diff --git a/.pipelex/create/draft_pipelines.md b/.pipelex/create/draft_pipelines.md
new file mode 100644
index 000000000..3217d4255
--- /dev/null
+++ b/.pipelex/create/draft_pipelines.md
@@ -0,0 +1,259 @@
+# Drafting Pipelex Pipelines
+
+This guide explains how to draft pipelines in natural language using markdown notation before translating them into formal Pipelex definitions.
+
+## Core Concept: WorkingMemory
+
+Every pipeline execution maintains a `WorkingMemory` that acts as a shared data space. When a pipeline starts, the initial inputs are placed in this memory. As each pipe executes, its output is added to the memory with a specific name (the `result` name). Subsequent pipes can access any data in the memory by referencing these names.
+
+**Key Rules**:
+
+- When a pipe produces an output with name `X`, any later pipe can use `X` as an input. The names must match exactly.
+- Controller pipes (PipeSequence, PipeParallel, PipeCondition) automatically inherit the input requirements of their nested operators - you don't need to specify their inputs explicitly.
+
+## Drafting Workflow
+
+1. **Start with the goal**: What does this pipeline accomplish?
+2. **Define inputs**: What data does it need to start?
+3. **Narrate the flow**: Describe each step and how data flows through WorkingMemory
+4. **Specify outputs**: What final result emerges?
+
+## Pipeline Draft Structure
+
+### Header
+```markdown
+# Pipeline: [pipeline_name]
+Domain: [domain_name]
+Purpose: [What this pipeline does]
+```
+
+### Inputs Section
+```markdown
+## Inputs
+- `document` : PDF document to analyze
+- `criteria` : Analysis criteria to apply
+```
+
+### Flow Section
+
+Describe the pipeline flow using concise step notation:
+
+```markdown
+## Flow
+
+WorkingMemory starts with: {document, criteria}
+
+### Step 1: Process document content
+LLM[process]: (document) → processed_content
+
+WorkingMemory now contains: {document, criteria, processed_content}
+
+### Step 2: Apply analysis criteria to processed content
+LLM[synthesize]: (processed_content, criteria) → analysis_report
+
+WorkingMemory now contains: {document, criteria, processed_content, analysis_report}
+```
+
+### Output Section
+```markdown
+## Output
+Returns: `analysis_report` - Comprehensive analysis based on criteria
+```
+
+## Pipe Types and Notation
+
+## Operator Notations
+
+**Important**: During drafting, DO NOT write actual prompts or prompt templates. Focus on describing what each operator does and how data flows. Prompts will be written during implementation.
+
+### LLM Operations (including vision, to generate text from text and images)
+```markdown
+LLM[purpose]: (inputs) → output
+
+Examples:
+LLM[summarize]: (long_text) → summary
+LLM[analyze]: (document, criteria) → analysis
+```
+
+### OCR Operations (to extract text and images from PDF pages, including full page views)
+```markdown
+OCR: (pdf_document) → pages_list
+```
+
+### Image Generation (to generate images from text prompts)
+```markdown
+ImgGen: (prompt_text) → generated_image
+```
+
+### Function Operations (to call external python functions)
+```markdown
+Func[function_name]: (data) → result
+```
+
+### Sequential Processing (to run steps in sequence, use it when steps need results from previous steps)
+```markdown
+Step 1: Extract key information from document
+extract_info(document) → key_info
+
+Step 2: Analyze extracted information
+analyze(key_info) → analysis
+
+Step 3: Generate report with original document context
+generate_report(analysis, document) → report
+```
+
+### Batch Processing (to process each item in a list independently and in parallel)
+```markdown
+Process each item in documents:
+analyze_document(document) → analysis
+
+Collect all results → document_analyses
+```
+
+### Parallel Processing (to run different pipes in parallel from the same memory, use it when steps can be run independently, each with a copy of the working memory)
+```markdown
+Parallel execution:
+
+Branch A: Extract technical details
+extract_technical(document) → technical_info
+
+Branch B: Extract business insights
+extract_business(document) → business_info
+
+Combine: Merge analysis results
+merge_insights(technical_info, business_info) → comprehensive_analysis
+```
+
+### Conditional Processing (to run different pipes based on a condition)
+```markdown
+Route based on expression(document_type):
+
+Case "technical": Handle technical documents
+analyze_technical(document) → analysis
+
+Case "legal": Handle legal documents
+analyze_legal(document) → analysis
+
+Default: Handle general documents
+analyze_general(document) → analysis
+```
+
+## Memory Flow Examples
+
+### Example 1: Document Analysis Pipeline
+```markdown
+# Pipeline: analyze_document
+Domain: document_analysis
+Purpose: Extract and analyze key information from documents
+
+## Inputs
+- `document` : PDF document to analyze
+
+## Flow
+
+WorkingMemory: {document}
+
+### Step 1: Extract content from PDF
+OCR: (document) → pages
+
+WorkingMemory: {document, pages}
+
+### Step 2: Summarize each page in parallel
+Process each page in pages:
+LLM[summarize]: (page) → summary
+
+Collect all results → summaries
+
+WorkingMemory: {document, pages, summaries}
+
+### Step 3: Create comprehensive analysis from summaries
+LLM[synthesize]: (summaries) → analysis
+
+WorkingMemory: {document, pages, summaries, analysis}
+
+## Output
+Returns: `analysis`
+```
+
+### Example 2: Conditional Processing
+```markdown
+# Pipeline: smart_responder
+Domain: customer_service
+Purpose: Route and respond to customer queries
+
+## Inputs
+- `query` : Customer question
+- `context` : Available context information
+
+## Flow
+
+WorkingMemory: {query, context}
+
+### Step 1: Classify query into support category
+LLM[classify]: (query) → category
+
+WorkingMemory: {query, context, category}
+
+### Step 2: Route to appropriate support handler
+Route based on category:
+
+Case "technical": Technical support handler
+LLM[technical_support]: (query, context) → response
+
+Case "billing": Billing support handler
+LLM[billing_support]: (query, context) → response
+
+Default: General support handler
+LLM[general_support]: (query) → response
+
+WorkingMemory: {query, context, category, response}
+
+## Output
+Returns: `response`
+```
+
+## Best Practices
+
+1. **Use semantic variable names**: Name variables based on what they represent, not how they're used. Use `document` instead of `input_document`, `text` instead of `input_text`, `analysis` instead of `result_1`.
+
+2. **Name consistently**: Once you name a result (e.g., `analysis`), use that exact name when referencing it later.
+
+3. **Show memory state**: After complex steps, show what's in WorkingMemory to track data flow.
+
+4. **Be explicit about inputs**: For each operator step, clearly indicate which items from WorkingMemory are being used.
+
+5. **Use descriptive names**: Choose names that clearly describe the content: `summaries`, `technical_info`, `user_preferences`.
+
+6. **Concise step descriptions**: Each step should have a clear title that describes what it accomplishes. Avoid repeating the same information in multiple places.
+
+7. **No prompts during drafting**: Focus on describing WHAT each step does, not HOW (the actual prompts). Prompts are implementation details added later.
+
+8. **Block vs Inline references**:
+ - Use `@variable_name` notation for block insertions (multi-line content)
+ - Use `$variable_name` notation for inline insertions (single values in sentences)
+
+9. **Batch operations**: When processing lists, clearly indicate:
+ - What list is being iterated (`batch_over`)
+ - What each item is called during processing (`batch_as`)
+ - What the collected results are named
+
+## Validation Checklist
+
+Before finalizing your draft:
+
+- [ ] Every input used by an operator pipe exists in WorkingMemory (either from initial inputs or previous steps)
+- [ ] Every result name is unique (no overwriting unless intentional)
+- [ ] The final output exists in WorkingMemory
+- [ ] Names are consistent throughout (no typos or variations)
+- [ ] Variable names are semantic (describe what they are, not how they're used)
+- [ ] The flow clearly shows data transformation from inputs to outputs
+- [ ] Each step has a concise, non-repetitive description
+
+## From Draft to Implementation
+
+Once your markdown draft is complete, it can be translated into:
+1. A formal TOML pipeline definition with explicit pipe declarations
+2. Python structure classes for complex data types (when needed)
+3. Validated, executable Pipelex pipeline code
+
+The narrative draft serves as both documentation and a blueprint for implementation, making the pipeline's logic transparent and maintainable.
diff --git a/AGENTS.md b/AGENTS.md
index 50a542992..71e3063d5 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -30,7 +30,7 @@ This python >=3.10 code is in the `pipelex` directory.
## Testing
- Always test with `make codex-tests` -> it runs pytest on our `tests/` directory using proper settings
-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading
+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all` under the hood)
## PR Instructions
diff --git a/CHANGELOG.md b/CHANGELOG.md
index abad2c5dc..8930716e2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -163,7 +163,7 @@ Simplified input memory:
### Documentation & Development Experience
- **Cursor Rules Enhancement**: Comprehensive pipe controller documentation covering `PipeSequence`, `PipeCondition`, `PipeBatch`, and `PipeParallel`, improved PipeOperator documentation for `PipeLLM`, `PipeOCR`
-- **Pipeline Validation CLI**: Enhanced `pipelex validate` command with better error reporting and validation coverage
+- **Pipeline Validation CLI**: Enhanced `pipelex validate all` command with better error reporting and validation coverage
- **Improved Error Messages**: Better formatting and context for pipeline configuration errors
### Changed
@@ -480,7 +480,7 @@ is_reporting_enabled = true
- Use a minimum version for `uv` instead of a fixed version
- Implement `AGENTS.md` for Codex
- Add tests for some of the `tools.misc`
-- pipelex cli: Rename `pipelex run-setup` to `pipelex validate`
+- pipelex cli: Rename `pipelex run-setup` to `pipelex validate all`
## [v0.2.8] - 2025-05-28
diff --git a/CLAUDE.md b/CLAUDE.md
index ed571e800..2a57cb258 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -82,7 +82,7 @@ class TestFooBar:
- Always test with `make t` -> it runs pytest using proper settings
- If some pytest tests fail, run pytest on the failed ones with the required verbosity to diagnose the issue
-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading
+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all` under the hood)
## PR Instructions
diff --git a/Makefile b/Makefile
index 9add6d431..938d01179 100644
--- a/Makefile
+++ b/Makefile
@@ -153,7 +153,7 @@ update: env
validate: env
$(call PRINT_TITLE,"Running setup sequence")
- $(VENV_PIPELEX) validate -c pipelex/libraries
+ $(VENV_PIPELEX) validate all -c pipelex/libraries
build: env
$(call PRINT_TITLE,"Building the wheels")
@@ -419,10 +419,6 @@ docs-deploy: env
### SHORTHANDS
##########################################################################################
-check-unused-imports: env
- $(call PRINT_TITLE,"Checking for unused imports without fixing")
- $(VENV_RUFF) check --select=F401 --no-fix .
-
c: format lint pyright mypy
@echo "> done: c = check"
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
new file mode 100644
index 000000000..4fd40f9cc
--- /dev/null
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
@@ -0,0 +1,289 @@
+# Pipeline Creation
+
+Pipelex provides powerful tools to automatically generate pipeline blueprints from natural language requirements. This feature leverages AI to translate your ideas into working pipeline code, dramatically speeding up development.
+
+## Overview
+
+The pipeline creation system works in several stages:
+
+1. **Draft Generation**: Convert natural language requirements into structured markdown drafts
+2. **Blueprint Creation**: Transform drafts into formal pipeline blueprints (TOML format)
+3. **Validation**: Automatically validate and fix common issues
+4. **Implementation**: Generate executable pipeline code
+
+## Core Commands
+
+### Build Blueprint
+
+Generate a complete pipeline blueprint from requirements:
+
+```bash
+pipelex build blueprint PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]
+```
+
+**Example:**
+```bash
+pipelex build blueprint gen_photopposite \
+ --requirements "Take a photo as input, analyze its content and its most important feature, imagine the opposite to that feature, render it as a photo" \
+ -c pipelex/libraries \
+ -o pipelex/libraries/pipelines/temp/result_4/photopposite \
+ --validate
+```
+
+**Options:**
+- `--domain, -d`: Domain for the pipeline (default: "wip_domain")
+- `--requirements, -r`: Requirements text directly
+- `--file, -f`: Path to file containing requirements
+- `--output, -o`: Output path for generated files
+- `--validate`: Validate the generated blueprint
+- `--config-folder-path, -c`: Path to libraries folder
+
+### Build Draft
+
+Generate just a pipeline draft (intermediate step):
+
+```bash
+pipelex build draft PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]
+```
+
+**Options:**
+- `--raw`: Generate raw text draft instead of structured draft
+- All other options same as blueprint command
+
+### Validate Blueprint
+
+Validate an existing blueprint file:
+
+```bash
+pipelex validate blueprint BLUEPRINT_PATH [OPTIONS]
+```
+
+**Example:**
+```bash
+pipelex validate blueprint pipelex/libraries/pipelines/temp/result_3/photopposite.toml \
+ -c pipelex/libraries
+```
+
+**Options:**
+- `--config-folder-path, -c`: Path to libraries folder
+- `--fix/--no-fix, -f/-F`: Enable/disable automatic error fixing (default: enabled)
+
+## Complete Workflow
+
+### 1. Requirements Analysis
+
+Start with clear, specific requirements:
+
+```text
+Take a photo as input, analyze its content and its most important feature,
+imagine the opposite to that feature, render it as a photo
+```
+
+### 2. Draft Generation
+
+The system first creates a structured markdown draft:
+
+```markdown
+# Pipeline: gen_photopposite
+Domain: image_processing
+Purpose: Generate opposite version of photo's main feature
+
+## Inputs
+- `photo` : Input photo to analyze and transform
+
+## Flow
+
+WorkingMemory starts with: {photo}
+
+### Step 1: Analyze photo content
+LLM[analyze]: (photo) → analysis
+
+WorkingMemory now contains: {photo, analysis}
+
+### Step 2: Generate opposite concept
+LLM[conceptualize]: (analysis) → opposite_concept
+
+WorkingMemory now contains: {photo, analysis, opposite_concept}
+
+### Step 3: Render opposite photo
+ImgGen: (opposite_concept) → opposite_photo
+
+WorkingMemory now contains: {photo, analysis, opposite_concept, opposite_photo}
+
+## Output
+Returns: `opposite_photo` - Photo showing opposite of main feature
+```
+
+### 3. Blueprint Creation
+
+The draft is converted into a formal TOML blueprint:
+
+```toml
+domain = "image_processing"
+definition = "Generate opposite version of photo's main feature"
+
+[concept]
+PhotoAnalysis = "Analysis of photo content and main features"
+OppositePhotoPrompt = "Prompt for generating opposite version of photo"
+
+[pipe.gen_photopposite]
+PipeSequence = "Generate opposite version of photo's main feature"
+output = "Image"
+steps = [
+ { pipe = "analyze_photo", result = "analysis" },
+ { pipe = "create_opposite_concept", result = "opposite_concept" },
+ { pipe = "render_opposite", result = "opposite_photo" }
+]
+
+[pipe.analyze_photo]
+PipeLLM = "Analyze photo content and identify main feature"
+inputs = { photo = "Image" }
+output = "PhotoAnalysis"
+prompt_template = """
+Analyze this photo and identify its most important visual feature.
+
+@photo
+
+Focus on the dominant element, color, mood, or characteristic that defines this image.
+"""
+
+[pipe.create_opposite_concept]
+PipeLLM = "Create concept for opposite version"
+inputs = { analysis = "PhotoAnalysis" }
+output = "OppositePhotoPrompt"
+prompt_template = """
+Based on this photo analysis, create a detailed prompt for generating the opposite version.
+
+@analysis
+
+Describe what the opposite would look like, focusing on reversing the main feature while maintaining photo realism.
+"""
+
+[pipe.render_opposite]
+PipeImgGen = "Generate the opposite photo"
+inputs = { opposite_concept = "OppositePhotoPrompt" }
+output = "Image"
+```
+
+### 4. Validation and Fixing
+
+The system automatically:
+
+- **Validates syntax**: Ensures TOML structure is correct
+- **Checks concepts**: Verifies all concepts are properly defined
+- **Validates inputs**: Ensures all pipe inputs are available
+- **Fixes errors**: Automatically adds missing inputs or corrects common issues
+- **Dry runs**: Tests pipeline execution without making API calls
+
+## Generated Files
+
+When you run the build command, several files are created:
+
+```
+output_path_base_draft.md # Original markdown draft
+output_path_base_rough.toml # Initial blueprint
+output_path_base_rough.json # Blueprint in JSON format
+output_path_base_fixed.toml # Validated and fixed blueprint (if --validate used)
+output_path_base_fixed.json # Fixed blueprint in JSON format
+```
+
+## Best Practices
+
+### Writing Requirements
+
+**Good requirements:**
+```text
+Analyze a PDF document, extract key financial data, and generate a summary report with charts
+```
+
+**Better requirements:**
+```text
+Take a PDF financial statement as input, use OCR to extract text and tables,
+identify revenue, expenses, and profit figures, then generate a structured
+summary report with visual charts showing trends
+```
+
+### Domain Selection
+
+Choose descriptive domain names:
+- `financial_analysis` instead of `finance`
+- `document_processing` instead of `docs`
+- `image_generation` instead of `images`
+
+### Output Path Organization
+
+Organize generated pipelines:
+```
+pipelex/libraries/pipelines/
+ temp/ # Temporary/experimental pipelines
+ result_1/
+ result_2/
+ domain_name/ # Production pipelines
+ pipeline_name.toml
+ structures.py
+```
+
+## Advanced Features
+
+### Custom Templates
+
+The system uses built-in templates from:
+- `.pipelex/create/draft_pipelines.md` - Drafting guidelines
+- `.pipelex/create/build_pipelines.md` - Blueprint creation rules
+
+### Error Fixing
+
+The validation system can automatically fix:
+- Missing input variables
+- Incorrect concept references
+- Malformed pipe definitions
+- Input/output mismatches
+
+### Integration with Existing Libraries
+
+Generated blueprints automatically integrate with:
+- Native concepts (`Text`, `Image`, `PDF`, etc.)
+- Library concepts (`documents.TextAndImages`, etc.)
+- Custom domain concepts
+
+## Troubleshooting
+
+### Common Issues
+
+**"Missing input variable" error:**
+- Usually fixed automatically with `--validate`
+- Check that all pipe inputs are available in WorkingMemory
+
+**"Invalid concept" error:**
+- Ensure concept names use PascalCase
+- Don't redefine native concepts
+- Avoid plurals in concept names
+
+**"Pipe not found" error:**
+- Check that all referenced pipes exist in the blueprint
+- Verify pipe names match exactly
+
+### Getting Help
+
+```bash
+# Validate your entire setup
+pipelex validate all -c pipelex/libraries
+
+# Test a specific pipe
+pipelex validate pipe your_pipe_name -c pipelex/libraries
+
+# Check blueprint structure
+pipelex validate blueprint your_blueprint.toml -c pipelex/libraries --no-fix
+```
+
+## Next Steps
+
+After creating your blueprint:
+
+1. **Review the generated code** - Check that it matches your requirements
+2. **Create structure classes** - Add Python classes for custom concepts if needed
+3. **Test the pipeline** - Run validation and dry runs
+4. **Execute the pipeline** - Use `execute_pipeline()` to run it with real data
+5. **Iterate and improve** - Refine based on results
+
+The pipeline creation system makes it easy to go from idea to working code in minutes, while maintaining the full power and flexibility of Pipelex pipelines.
diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.md
index e621067ec..633ea7850 100644
--- a/docs/pages/configuration/config-technical/library-config.md
+++ b/docs/pages/configuration/config-technical/library-config.md
@@ -61,7 +61,7 @@ exported_templates_path = "pipelex_libraries/templates"
Use the CLI command to initialize libraries:
```bash
-pipelex init-libraries
+pipelex init libraries
```
This will:
@@ -122,7 +122,7 @@ The library system includes specific error types:
2. **Validation**:
- - Run `pipelex validate` after making changes
+ - Run `pipelex validate all` after making changes
- Check for domain consistency
- Verify concept relationships
diff --git a/docs/pages/configuration/index.md b/docs/pages/configuration/index.md
index 714539c88..9e8cab9df 100644
--- a/docs/pages/configuration/index.md
+++ b/docs/pages/configuration/index.md
@@ -5,13 +5,13 @@
Pipelex uses a TOML-based configuration system. The main configuration file `pipelex.toml` must be located at the root of your project. You can create this file by running:
```bash
-pipelex init-config
+pipelex init config
```
💡 **Important Notes**:
-1. `pipelex init-config` creates a **template** configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
-2. Using `pipelex init-config --reset` will **overwrite** your existing `pipelex.toml` file without warning. Make sure to backup your configuration before using this flag.
+1. `pipelex init config` creates a **template** configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
+2. Using `pipelex init config --reset` will **overwrite** your existing `pipelex.toml` file without warning. Make sure to backup your configuration before using this flag.
For a complete list of all possible configuration options, refer to the configuration group documentation below.
@@ -69,5 +69,5 @@ NB: The run_mode unit_test is used for testing purposes.
1. **Version Control**: Include your base `pipelex.toml` in version control
2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings
3. **Documentation**: Comment any custom settings for team reference
-4. **Validation**: Run `pipelex validate` after making configuration changes
+4. **Validation**: Run `pipelex validate all` after making configuration changes
5. **Gitignore**: Add local and sensitive override files to `.gitignore`
diff --git a/docs/pages/installation/index.md b/docs/pages/installation/index.md
index b2ee24c26..183ed10ef 100644
--- a/docs/pages/installation/index.md
+++ b/docs/pages/installation/index.md
@@ -40,7 +40,7 @@ All the secret keys used by `pipelex` are specified in the `.env.example` file.
In order to set the pipelex configuration files, you need to run 2 commands using the CLI (we recommend to run it at the root of your project):
-- `pipelex init-libraries`: This will create a `pipelex_libraries` folder, with the base llm configuration and the base pipelines.
+- `pipelex init libraries`: This will create a `pipelex_libraries` folder, with the base llm configuration and the base pipelines.
This is the directory where you should add your pipelines.
The structure is like this:
@@ -58,7 +58,7 @@ The structure is like this:
Learn more about pipelex_libraries in our [Libraries documentation](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md)
-- `pipelex init-config`: This cli command will create a `pipelex.toml` file at the root of the project, with basic configuration. This configuration file gathers all configuration for feature flags, logging, cost reporting, and so on... Learn more in our [Configuration documentation](../configuration/index.md)
+- `pipelex init config`: This cli command will create a `pipelex.toml` file at the root of the project, with basic configuration. This configuration file gathers all configuration for feature flags, logging, cost reporting, and so on... Learn more in our [Configuration documentation](../configuration/index.md)
💡 _Any troubles? Have a look at our [Cookbook](https://github.com/Pipelex/pipelex-cookbook)! and come ask for help on our [Discord](https://go.pipelex.com/discord)_
diff --git a/docs/pages/tools/migration.md b/docs/pages/migration/index.md
similarity index 86%
rename from docs/pages/tools/migration.md
rename to docs/pages/migration/index.md
index 21927e078..53f686a34 100644
--- a/docs/pages/tools/migration.md
+++ b/docs/pages/migration/index.md
@@ -2,22 +2,22 @@
This document provides guidance for migrating between different versions of Pipelex that introduce breaking changes.
-## Version 0.6.11: Concept Definition Syntax Change
+## Concepts Migration
### Breaking Change
-In version 0.6.11, we changed the syntax for defining concepts in TOML library files. The key name for concept definitions has changed from `Concept` to `definition`.
+We changed the syntax for defining concepts in TOML library files. The key name for concept definitions has changed from `Concept` to `definition`.
### What Changed
-**Old syntax (before v0.6.11):**
+**Old syntax:**
```toml
[concept.MyConceptName]
Concept = "Description of the concept"
refines = "BaseConceptName"
```
-**New syntax (v0.6.11+):**
+**New syntax:**
```toml
[concept.MyConceptName]
definition = "Description of the concept"
@@ -27,6 +27,7 @@ refines = "BaseConceptName"
### Why This Change
This change improves consistency in our TOML schema by:
+
- Using more descriptive field names (`definition` instead of `Concept`)
- Aligning with the internal `ConceptBlueprint` data structure
- Making the TOML files more self-documenting and intuitive
@@ -46,6 +47,7 @@ pipelex migrate
```
The migration command will:
+
- Find all `.toml` files in your configured pipelines directory
- Replace `Concept =` with `definition =` in concept definitions
- Preserve all other formatting and comments
@@ -57,7 +59,7 @@ If you prefer to migrate manually or need to handle special cases:
1. **Locate your TOML files**: Find all pipeline library files (typically in your configured pipelines directory)
2. **Update concept definitions**: Change `Concept =` to `definition =` in all `[concept.ConceptName]` sections
-3. **Validate syntax**: Run `pipelex validate` to ensure your files are correctly formatted
+3. **Validate syntax**: Run `pipelex validate all` to ensure your files are correctly formatted
### Examples
@@ -101,7 +103,7 @@ After migration, validate your pipeline files:
```bash
# Validate all pipeline files
-pipelex validate
+pipelex validate all
# Run a specific pipeline to test
pipelex run your-pipeline-name
@@ -110,11 +112,13 @@ pipelex run your-pipeline-name
### Troubleshooting
**Error: "ConceptBlueprint validation failed"**
+
- Check that all `Concept =` entries have been changed to `definition =`
- Ensure no typos were introduced during migration
- Verify TOML syntax is still valid
**Error: "Failed to load TOML file"**
+
- Check file permissions
- Ensure TOML syntax is valid (no missing quotes, brackets, etc.)
- Review any custom modifications that might conflict
@@ -122,25 +126,26 @@ pipelex run your-pipeline-name
### Rollback
If you need to rollback the migration:
+
1. Stop using the new version of Pipelex
2. Restore from the `.backup` files created during migration
3. Or manually change `definition =` back to `Concept =`
-## Version 0.3.0: Pipe Definition Syntax Change
+## Pipes Migration
### Breaking Change
-Starting in version 0.3.0, we changed the syntax for defining pipes in TOML library files. The pipe definition format has been restructured to use explicit `type` and `definition` fields instead of the implicit `PipeClassName = "description"` format.
+We changed the syntax for defining pipes in TOML library files. The pipe definition format has been restructured to use explicit `type` and `definition` fields instead of the implicit `PipeClassName = "description"` format.
### What Changed
-**Old syntax (before v0.3.0):**
+**Old syntax:**
```toml
[pipe.my_pipe_name]
SomePipeClass = "Description of what this pipe does"
```
-**New syntax (v0.3.0+):**
+**New syntax:**
```toml
[pipe.my_pipe_name]
type = "SomePipeClass"
@@ -150,6 +155,7 @@ definition = "Description of what this pipe does"
### Why This Change
This change improves consistency and clarity by:
+
- Making the pipe class type explicit and separate from its description
- Aligning pipe definitions with concept definitions (both use `definition`)
- Improving readability and maintainability of TOML files
@@ -170,6 +176,7 @@ pipelex migrate
```
The migration command will:
+
- Find all `.toml` files in your configured pipelines directory
- Convert `PipeClassName = "description"` to `type = "PipeClassName"` and `definition = "description"`
- Preserve all other formatting and comments
@@ -185,7 +192,7 @@ If you prefer to migrate manually:
- Replace `PipeClassName = "description"` with:
- `type = "PipeClassName"`
- `definition = "description"`
-3. **Validate syntax**: Run `pipelex validate` to ensure your files are correctly formatted
+3. **Validate syntax**: Run `pipelex validate all` to ensure your files are correctly formatted
### Examples
@@ -247,7 +254,7 @@ After migration, validate your pipeline files:
```bash
# Validate all pipeline files
-pipelex validate
+pipelex validate all
# Run a specific pipeline to test
pipelex run your-pipeline-name
diff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.md
index a447a4020..4da126632 100644
--- a/docs/pages/quick-start/index.md
+++ b/docs/pages/quick-start/index.md
@@ -14,7 +14,7 @@ For illustration purposes, let's build **a character generator**. Each example r
### Write your first pipeline
First, create a `.toml` library file in the `pipelex_libraries/pipelines` directory to store your pipe definition.
-Run `pipelex init-libraries` to create this directory if it doesn't exist. For now, keep all your pipeline definitions inside that folder only.
+Run `pipelex init libraries` to create this directory if it doesn't exist. For now, keep all your pipeline definitions inside that folder only.
`character.toml`
```toml
diff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.md
index 15fb31849..2fe5e781b 100644
--- a/docs/pages/tools/cli.md
+++ b/docs/pages/tools/cli.md
@@ -4,76 +4,47 @@ The Pipelex CLI provides a command-line interface for managing and interacting w
## Available Commands
-### `pipelex init-libraries`
+### Init group
-Initialize Pipelex libraries in the current directory.
+Initialize project assets.
```bash
-pipelex init-libraries [--overwrite/-o]
+pipelex init libraries [DIRECTORY] [--overwrite/-o]
+pipelex init config [--reset/-r]
```
-**Options:**
-- `--overwrite`, `-o`: If set, existing library files will be overwritten. Otherwise, only missing files will be created.
+### Validate group
-### `pipelex init-config`
-
-Initialize Pipelex configuration in the current directory.
-
-```bash
-pipelex init-config [--reset/-r]
-```
-
-**Options:**
-- `--reset`, `-r`: If set, existing configuration file (pipelex.toml) will be overwritten. Otherwise, the command will warn if the file already exists.
-
-### `pipelex validate`
-
-Run the setup sequence to validate your Pipelex configuration and pipelines.
+Validate configuration and pipelines.
```bash
-pipelex validate
+pipelex validate all [-c/--config-folder-path PATH]
+pipelex validate pipe PIPE_CODE [-c/--config-folder-path PATH]
```
-This command:
+### Show group
-1. Exports libraries
-2. Validates the configuration
-3. Ensures all pipelines are properly set up
-
-### `pipelex show-config`
-
-Display the current Pipelex configuration.
+Inspect configuration and pipes.
```bash
-pipelex show-config
+pipelex show config
+pipelex show pipes [-c/--config-folder-path PATH]
+pipelex show pipe PIPE_CODE [-c/--config-folder-path PATH]
```
-Shows the complete configuration for your project, including all settings and parameters.
-See more in our [Configuration documentation](../configuration/index.md)
+### Migrate group
-### `pipelex list-pipes`
-
-List all available pipes in your project.
+Migrate TOML files to the new syntax.
```bash
-pipelex list-pipes
+pipelex migrate run [-c/--config-folder-path PATH] [--dry-run] [--backups/--no-backups]
```
-Displays a table of pipes organized by domain, showing:
-
-- Code: The unique identifier for each pipe
-- Definition: Description of the pipe's purpose
-- Input: Required input parameters and their concept codes
-- Output: The output concept code
-
-The output is formatted as tables grouped by domain, with concept codes simplified when they belong to the current domain.
-
## Usage Tips
-1. Always run `pipelex validate` after making changes to your configuration or pipelines
-2. Use `pipelex show-config` to debug configuration issues
+1. Always run `pipelex validate all` after making changes to your configuration or pipelines
+2. Use `pipelex show config` to debug configuration issues
3. When initializing a new project:
-
- - Start with `pipelex init-config`
- - Then run `pipelex init-libraries`
- - Finally, validate your setup with `pipelex validate`
+ - Start with `pipelex init config`
+ - Then run `pipelex init libraries`
+ - Finally, validate your setup with `pipelex validate all`
diff --git a/mkdocs.yml b/mkdocs.yml
index 178c3ecb8..357eb6e0e 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -109,6 +109,7 @@ nav:
- Kick off a Pipeline Project: pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md
- Define Your Concepts: pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md
- Design and Run Pipelines: pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md
+ - Pipeline Creation: pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md
- Pipe Operators:
- Overview: pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/index.md
- PipeLLM: pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md
@@ -146,7 +147,7 @@ nav:
- Tools:
- CLI: pages/tools/cli.md
- Logging: pages/tools/logging.md
- - Migration: pages/tools/migration.md
+ - Migration: pages/migration/index.md
- Advanced Customizations:
- Overview: pages/advanced-customization/index.md
- Secrets Provider: pages/advanced-customization/secrets-provider-injection.md
diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.py
index d7461c13f..8ca16df2e 100644
--- a/pipelex/cli/_cli.py
+++ b/pipelex/cli/_cli.py
@@ -1,57 +1,12 @@
-import asyncio
-import os
-import shutil
-from pathlib import Path
-from typing import Annotated, Optional
+from typing import Optional
import typer
from click import Command, Context
from typer.core import TyperGroup
from typing_extensions import override
-from pipelex import log, pretty_print
-from pipelex.exceptions import PipelexCLIError, PipelexConfigError
-from pipelex.hub import get_pipe_provider, get_pipeline_tracker, get_required_pipe
-from pipelex.libraries.library_config import LibraryConfig
-from pipelex.pipe_works.pipe_dry import dry_run_all_pipes, dry_run_single_pipe
-from pipelex.pipelex import Pipelex
-from pipelex.tools.config.manager import config_manager
-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import TomlMigrator, migrate_concept_syntax
-
-
-def is_pipelex_libraries_folder(folder_path: str) -> bool:
- """Check if the given folder path contains a valid pipelex libraries structure.
-
- A valid pipelex libraries folder should contain the following subdirectories:
- - pipelines
- - llm_deck
- - llm_integrations
- - plugins
- - templates
-
- Args:
- folder_path: Path to the folder to check
-
- Returns:
- True if the folder contains all required subdirectories, False otherwise
- """
- if not os.path.exists(folder_path) or not os.path.isdir(folder_path):
- return False
-
- required_subdirs = [
- "pipelines",
- "llm_deck",
- "llm_integrations",
- "plugins",
- "templates",
- ]
-
- for subdir in required_subdirs:
- subdir_path = os.path.join(folder_path, subdir)
- if not os.path.exists(subdir_path) or not os.path.isdir(subdir_path):
- return False
-
- return True
+from pipelex.cli.commands import init_app, migrate_app, show_app, validate_app
+from pipelex.cli.commands.build_cmd import build_app
class PipelexCLI(TyperGroup):
@@ -65,6 +20,11 @@ class PipelexCLI(TyperGroup):
return cmd
+def main() -> None:
+ """Entry point for the pipelex CLI."""
+ app()
+
+
app = typer.Typer(
add_completion=False,
no_args_is_help=True,
@@ -73,253 +33,8 @@ app = typer.Typer(
)
[email protected]("init-libraries")
-def init_libraries(
- directory: Annotated[
- str,
- typer.Argument(help="Directory where to create the pipelex_libraries folder"),
- ] = ".",
- overwrite: Annotated[
- bool,
- typer.Option(
- "--overwrite",
- "-o",
- help="Warning: If set, existing files will be overwritten.",
- ),
- ] = False,
-) -> None:
- """Initialize pipelex libraries in a pipelex_libraries folder in the specified directory.
-
- If overwrite is False, only create files that don't exist yet.
- If overwrite is True, all files will be overwritten even if they exist.
- """
- try:
- # Always create a pipelex_libraries folder in the specified directory
- target_path = os.path.join(directory, "pipelex_libraries")
-
- # Create the target directory if it doesn't exist
- os.makedirs(directory, exist_ok=True)
-
- # Create a LibraryConfig instance with the target path
- library_config = LibraryConfig(config_folder_path=target_path)
- library_config.export_libraries(overwrite=overwrite)
-
- if overwrite:
- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (all files overwritten)")
- else:
- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (only created non-existing files)")
- except Exception as exc:
- raise PipelexCLIError(f"Failed to initialize libraries at '{directory}': {exc}")
-
-
[email protected]("init-config")
-def init_config(
- reset: Annotated[
- bool,
- typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten."),
- ] = False,
-) -> None:
- """Initialize pipelex configuration in the current directory."""
- pipelex_template_path = os.path.join(config_manager.pipelex_root_dir, "pipelex_template.toml")
- target_config_path = os.path.join(config_manager.local_root_dir, "pipelex.toml")
-
- if os.path.exists(target_config_path) and not reset:
- typer.echo("Warning: pipelex.toml already exists. Use --reset to force creation.")
- return
-
- try:
- shutil.copy2(pipelex_template_path, target_config_path)
- typer.echo(f"Created pipelex.toml at {target_config_path}")
- except Exception as exc:
- raise PipelexCLIError(f"Failed to create pipelex.toml: {exc}")
-
-
-def validate(
- relative_config_folder_path: Annotated[
- str,
- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),
- ] = "./pipelex_libraries",
-) -> None:
- """Run the setup sequence."""
- # Check if pipelex libraries folder exists
- if not is_pipelex_libraries_folder(relative_config_folder_path):
- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")
- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")
- raise typer.Exit(1)
-
- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
- pipelex_instance.validate_libraries()
- asyncio.run(dry_run_all_pipes())
- log.info("Setup sequence passed OK, config and pipelines are validated.")
-
-
-def dry_run_pipe(
- pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],
- relative_config_folder_path: Annotated[
- str,
- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),
- ] = "./pipelex_libraries",
-) -> None:
- """Dry run a single pipe."""
- # Check if pipelex libraries folder exists
- if not is_pipelex_libraries_folder(relative_config_folder_path):
- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")
- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")
- raise typer.Exit(1)
-
- try:
- # Initialize Pipelex
- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
- pipelex_instance.validate_libraries()
-
- # Run the single pipe dry run
- asyncio.run(dry_run_single_pipe(pipe_code))
- get_pipeline_tracker().output_flowchart()
-
- except Exception as exc:
- typer.echo(f"❌ Error running dry run for pipe '{pipe_code}': {exc}")
- raise typer.Exit(1)
-
-
-def show_config() -> None:
- """Show the pipelex configuration."""
- try:
- final_config = config_manager.load_config()
- pretty_print(
- final_config,
- title=f"Pipelex configuration for project: {config_manager.get_project_name()}",
- )
- except Exception as exc:
- raise PipelexConfigError(f"Error loading configuration: {exc}")
-
-
-def list_pipes(
- relative_config_folder_path: Annotated[
- str,
- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),
- ] = "pipelex_libraries",
-) -> None:
- """List all available pipes."""
- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
-
- try:
- get_pipe_provider().pretty_list_pipes()
-
- except Exception as exc:
- raise PipelexCLIError(f"Failed to list pipes: {exc}")
-
-
[email protected]("show-pipe")
-def show_pipe(
- pipe_code: Annotated[
- str,
- typer.Argument(help="Pipeline code to show definition for"),
- ],
- relative_config_folder_path: Annotated[
- str,
- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),
- ] = "./pipelex_libraries",
-) -> None:
- """Show pipe from the pipe library."""
- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
- pipe = get_required_pipe(pipe_code=pipe_code)
- pretty_print(pipe, title=f"Pipe '{pipe_code}'")
-
-
-def migrate(
- relative_config_folder_path: Annotated[
- str,
- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),
- ] = "./pipelex_libraries",
- dry_run: Annotated[
- bool,
- typer.Option("--dry-run", help="Preview changes without applying them"),
- ] = False,
- backups: Annotated[
- bool,
- typer.Option("--backups/--no-backups", help="Create backup files before migration"),
- ] = True,
-) -> None:
- """Migrate TOML files to new syntax (Concept = -> definition = and PipeClassName = -> type/definition)."""
- config_path = Path(relative_config_folder_path)
-
- # Check if it's a pipelex libraries folder structure or just a directory with TOML files
- if is_pipelex_libraries_folder(relative_config_folder_path):
- pipelines_dir = config_path / "pipelines"
- else:
- # Assume it's a directory containing TOML files directly
- pipelines_dir = config_path
-
- if not pipelines_dir.exists():
- typer.echo(f"❌ Directory not found at '{pipelines_dir}'")
- raise typer.Exit(1)
-
- try:
- # Use the migration module (now handles both concepts and pipes)
- result = migrate_concept_syntax(
- directory=pipelines_dir,
- create_backups=backups and not dry_run, # Create backups if enabled and not dry-run
- dry_run=dry_run,
- )
-
- # Handle any errors that occurred
- for error in result.errors:
- typer.echo(f"❌ {error}")
-
- if result.errors and not result.files_modified:
- typer.echo("❌ Migration failed due to errors")
- raise typer.Exit(1)
-
- typer.echo(f"Found {result.files_processed} TOML file(s) to check")
-
- if result.files_modified == 0:
- typer.echo("✅ All TOML files are already using the new syntax")
- return
-
- if dry_run:
- # Show detailed preview for dry run
- migrator = TomlMigrator()
- for file_path in result.modified_files:
- try:
- with open(file_path, "r", encoding="utf-8") as f:
- content = f.read()
-
- changes = migrator.get_migration_preview(content)
- typer.echo(f"\n📄 {file_path.relative_to(pipelines_dir)}:")
- for change in changes:
- typer.echo(f" Line {change['line_number']}: {change['old_line']} → {change['new_line']}")
- except Exception as e:
- typer.echo(f"❌ Error reading {file_path}: {e}")
-
- typer.echo(f"\n📋 Summary: Found {result.total_changes} change(s) in {result.files_modified} file(s)")
- typer.echo(" Run without --dry-run to apply these changes")
- else:
- # Show migration results
- create_backups = backups and not dry_run
- for file_path in result.modified_files:
- typer.echo(f"✅ Migrated {file_path.relative_to(pipelines_dir)}")
- if create_backups:
- backup_path = file_path.with_suffix(".toml.backup")
- typer.echo(f" Backup saved to {backup_path.name}")
-
- typer.echo(f"\n✅ Migration completed: {result.total_changes} change(s) applied to {result.files_modified} file(s)")
- if create_backups:
- typer.echo(" Backup files created with .backup extension")
- typer.echo(" Run 'pipelex validate' to verify the migration")
-
- except FileNotFoundError as e:
- typer.echo(f"❌ {e}")
- raise typer.Exit(1)
- except Exception as e:
- typer.echo(f"❌ Migration failed: {e}")
- raise typer.Exit(1)
-
-
-def main() -> None:
- """Entry point for the pipelex CLI."""
- app()
+app.add_typer(init_app, name="init", help="Initialization commands")
+app.add_typer(validate_app, name="validate", help="Validation and dry-run commands")
+app.add_typer(show_app, name="show", help="Show and list commands")
+app.add_typer(migrate_app, name="migrate", help="Migration commands")
+app.add_typer(build_app, name="build", help="Build artifacts like pipeline blueprints")
diff --git a/pipelex/cli/commands/__init__.py b/pipelex/cli/commands/__init__.py
new file mode 100644
index 000000000..19b9cd5a6
--- /dev/null
+++ b/pipelex/cli/commands/__init__.py
@@ -0,0 +1,11 @@
+"""Command groups for Pipelex CLI.
+
+This package organizes CLI commands into logical modules.
+"""
+
+from .init_cmd import init_app
+from .migrate_cmd import migrate_app
+from .show_cmd import show_app
+from .validate_cmd import validate_app
+
+__all__ = ["init_app", "migrate_app", "show_app", "validate_app"]
diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.py
new file mode 100644
index 000000000..266704a8b
--- /dev/null
+++ b/pipelex/cli/commands/build_cmd.py
@@ -0,0 +1,145 @@
+from __future__ import annotations
+
+import asyncio
+from typing import Annotated, Optional
+
+import typer
+
+from pipelex.create.build_blueprint import do_build_blueprint
+from pipelex.create.draft_pipeline import do_draft_pipeline, do_draft_pipeline_text
+from pipelex.exceptions import PipelexCLIError
+from pipelex.pipelex import Pipelex
+from pipelex.tools.misc.file_utils import load_text_from_path
+
+# Typer group for build commands
+build_app = typer.Typer(help="Build artifacts like pipeline blueprints")
+
+
+@build_app.command("draft")
+def build_draft_pipeline_cmd(
+ pipeline_name: Annotated[
+ str,
+ typer.Argument(help="Name/code of the pipeline to generate"),
+ ],
+ domain: Annotated[
+ str,
+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),
+ ] = "wip_domain",
+ requirements: Annotated[
+ Optional[str],
+ typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),
+ ] = None,
+ requirements_file: Annotated[
+ Optional[str],
+ typer.Option("--file", "-f", help="Path to a file containing the requirements text"),
+ ] = None,
+ output_path: Annotated[
+ Optional[str],
+ typer.Option("--output", "-o", help="Path to save the generated TOML blueprint (optional)"),
+ ] = None,
+ raw: Annotated[
+ bool,
+ typer.Option("--raw", help="Raw text draft"),
+ ] = False,
+ relative_config_folder_path: Annotated[
+ str,
+ typer.Option(
+ "--config-folder-path",
+ "-c",
+ help="Relative path to the config folder path (libraries)",
+ ),
+ ] = "./pipelex_libraries",
+) -> None:
+ # Initialize Pipelex (loads libraries and pipes)
+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+
+ # Get requirements text
+ try:
+ requirements_text: str
+ if requirements_file:
+ requirements_text = load_text_from_path(requirements_file)
+ else:
+ if not requirements:
+ raise PipelexCLIError("You must provide requirements text via --requirements or a file via --file")
+ requirements_text = requirements
+ except Exception as exc:
+ raise PipelexCLIError(f"Failed to load requirements: {exc}") from exc
+
+ if raw:
+ asyncio.run(
+ do_draft_pipeline_text(
+ domain=domain,
+ pipeline_name=pipeline_name,
+ requirements=requirements_text,
+ output_path=output_path,
+ )
+ )
+ else:
+ asyncio.run(
+ do_draft_pipeline(
+ pipeline_name=pipeline_name,
+ requirements=requirements_text,
+ output_path=output_path,
+ )
+ )
+
+
+@build_app.command("blueprint")
+def build_blueprint_cmd(
+ pipeline_name: Annotated[
+ str,
+ typer.Argument(help="Name/code of the pipeline to generate"),
+ ],
+ domain: Annotated[
+ str,
+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),
+ ] = "wip_domain",
+ requirements: Annotated[
+ Optional[str],
+ typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),
+ ] = None,
+ requirements_file: Annotated[
+ Optional[str],
+ typer.Option("--file", "-f", help="Path to a file containing the requirements text"),
+ ] = None,
+ output_path: Annotated[
+ Optional[str],
+ typer.Option("--output", "-o", help="Path to save the generated TOML blueprint (optional)"),
+ ] = None,
+ validate: Annotated[
+ bool,
+ typer.Option("--validate", help="Dry-run the first generated pipe from the blueprint"),
+ ] = False,
+ relative_config_folder_path: Annotated[
+ str,
+ typer.Option(
+ "--config-folder-path",
+ "-c",
+ help="Relative path to the config folder path (libraries)",
+ ),
+ ] = "./pipelex_libraries",
+) -> None:
+ # Initialize Pipelex (loads libraries and pipes)
+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+
+ # Get requirements text
+ try:
+ requirements_text: str
+ if requirements_file:
+ requirements_text = load_text_from_path(requirements_file)
+ else:
+ if not requirements:
+ raise PipelexCLIError("You must provide requirements text via --requirements or a file via --file")
+ requirements_text = requirements
+ except Exception as exc:
+ raise PipelexCLIError(f"Failed to load requirements: {exc}") from exc
+
+ asyncio.run(
+ do_build_blueprint(
+ domain=domain,
+ pipeline_name=pipeline_name,
+ requirements=requirements_text,
+ output_path=output_path,
+ validate=validate,
+ )
+ )
diff --git a/pipelex/cli/commands/common.py b/pipelex/cli/commands/common.py
new file mode 100644
index 000000000..f4c948780
--- /dev/null
+++ b/pipelex/cli/commands/common.py
@@ -0,0 +1,25 @@
+from __future__ import annotations
+
+import os
+from typing import Final
+
+REQUIRED_PIPELEX_SUBDIRS: Final[list[str]] = [
+ "pipelines",
+ "llm_deck",
+ "llm_integrations",
+ "plugins",
+ "templates",
+]
+
+
+def is_pipelex_libraries_folder(folder_path: str) -> bool:
+ """Check if the given folder path contains a valid pipelex libraries structure."""
+ if not os.path.exists(folder_path) or not os.path.isdir(folder_path):
+ return False
+
+ for subdir in REQUIRED_PIPELEX_SUBDIRS:
+ subdir_path = os.path.join(folder_path, subdir)
+ if not os.path.exists(subdir_path) or not os.path.isdir(subdir_path):
+ return False
+
+ return True
diff --git a/pipelex/cli/commands/init_cmd.py b/pipelex/cli/commands/init_cmd.py
new file mode 100644
index 000000000..22a942796
--- /dev/null
+++ b/pipelex/cli/commands/init_cmd.py
@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+import os
+import shutil
+from typing import Annotated
+
+import typer
+
+from pipelex.exceptions import PipelexCLIError
+from pipelex.libraries.library_config import LibraryConfig
+from pipelex.tools.config.manager import config_manager
+
+
+def do_init_libraries(directory: str = ".", overwrite: bool = False) -> None:
+ """Initialize pipelex libraries in a pipelex_libraries folder in the specified directory."""
+ try:
+ target_path = os.path.join(directory, "pipelex_libraries")
+ os.makedirs(directory, exist_ok=True)
+
+ library_config = LibraryConfig(config_folder_path=target_path)
+ library_config.export_libraries(overwrite=overwrite)
+
+ if overwrite:
+ typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (all files overwritten)")
+ else:
+ typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (only created non-existing files)")
+ except Exception as exc:
+ raise PipelexCLIError(f"Failed to initialize libraries at '{directory}': {exc}") from exc
+
+
+def do_init_config(reset: bool = False) -> None:
+ """Initialize pipelex configuration in the current directory."""
+ pipelex_template_path = os.path.join(config_manager.pipelex_root_dir, "pipelex_template.toml")
+ target_config_path = os.path.join(config_manager.local_root_dir, "pipelex.toml")
+
+ if os.path.exists(target_config_path) and not reset:
+ typer.echo("Warning: pipelex.toml already exists. Use --reset to force creation.")
+ return
+
+ try:
+ shutil.copy2(pipelex_template_path, target_config_path)
+ typer.echo(f"Created pipelex.toml at {target_config_path}")
+ except Exception as exc:
+ raise PipelexCLIError(f"Failed to create pipelex.toml: {exc}") from exc
+
+
+# Typer group for init commands
+init_app = typer.Typer(help="Initialization commands")
+
+
+@init_app.command("libraries")
+def init_libraries_cmd(
+ directory: Annotated[str, typer.Argument(help="Directory where to create the pipelex_libraries folder")] = ".",
+ overwrite: Annotated[bool, typer.Option("--overwrite", "-o", help="Warning: If set, existing files will be overwritten.")] = False,
+) -> None:
+ do_init_libraries(directory=directory, overwrite=overwrite)
+
+
+@init_app.command("config")
+def init_config_cmd(
+ reset: Annotated[bool, typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten.")] = False,
+) -> None:
+ do_init_config(reset=reset)
diff --git a/pipelex/cli/commands/migrate_cmd.py b/pipelex/cli/commands/migrate_cmd.py
new file mode 100644
index 000000000..91e02a8d3
--- /dev/null
+++ b/pipelex/cli/commands/migrate_cmd.py
@@ -0,0 +1,120 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Annotated
+
+import typer
+
+from pipelex.cli.commands.common import is_pipelex_libraries_folder
+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import TomlMigrator, migrate_concept_syntax
+from pipelex.migration.migration_result import MigrationResult
+
+
+def do_migrate(
+ target_path: str = "./pipelex_libraries",
+ dry_run: bool = False,
+ backups: bool = True,
+) -> None:
+ """Migrate TOML files to new syntax (Concept = -> definition = and PipeClassName = -> type/definition).
+
+ The path can be:
+ - A Pipelex libraries folder (we'll use its `pipelines` subfolder)
+ - A directory containing TOML files
+ - A single TOML file
+ """
+ config_path = Path(target_path)
+
+ if is_pipelex_libraries_folder(target_path):
+ pipelines_dir_or_file = config_path / "pipelines"
+ else:
+ pipelines_dir_or_file = config_path
+
+ if not pipelines_dir_or_file.exists():
+ typer.echo(f"❌ Path not found at '{pipelines_dir_or_file}'")
+ raise typer.Exit(1)
+
+ try:
+ result: MigrationResult = migrate_concept_syntax(
+ directory=pipelines_dir_or_file,
+ create_backups=backups and not dry_run,
+ dry_run=dry_run,
+ )
+
+ for error in result.errors:
+ typer.echo(f"❌ {error}")
+
+ if result.errors and not result.files_modified:
+ typer.echo("❌ Migration failed due to errors")
+ raise typer.Exit(1)
+
+ typer.echo(f"Found {result.files_processed} TOML file(s) to check")
+
+ if result.files_modified == 0:
+ typer.echo("✅ All TOML files are already using the new syntax")
+ return
+
+ if dry_run:
+ migrator = TomlMigrator()
+ # For display, if a single file path was provided, show relative to its parent
+ base_for_print = pipelines_dir_or_file if pipelines_dir_or_file.is_dir() else pipelines_dir_or_file.parent
+ for file_path in result.modified_files:
+ try:
+ with open(file_path, "r", encoding="utf-8") as f:
+ content = f.read()
+
+ changes = migrator.get_migration_preview(content)
+ try:
+ rel = file_path.relative_to(base_for_print)
+ except Exception:
+ rel = Path(file_path.name)
+ typer.echo(f"\n📄 {rel}:")
+ for change in changes:
+ typer.echo(f" Line {change['line_number']}: {change['old_line']} → {change['new_line']}")
+ except Exception as exc:
+ typer.echo(f"❌ Error reading {file_path}: {exc}")
+
+ typer.echo(f"\n📋 Summary: Found {result.total_changes} change(s) in {result.files_modified} file(s)")
+ typer.echo(" Run without --dry-run to apply these changes")
+ else:
+ create_backups = backups and not dry_run
+ base_for_print = pipelines_dir_or_file if pipelines_dir_or_file.is_dir() else pipelines_dir_or_file.parent
+ for file_path in result.modified_files:
+ try:
+ rel = file_path.relative_to(base_for_print)
+ except Exception:
+ rel = Path(file_path.name)
+ typer.echo(f"✅ Migrated {rel}")
+ if create_backups:
+ backup_path = file_path.with_suffix(".toml.backup")
+ typer.echo(f" Backup saved to {backup_path.name}")
+
+ typer.echo(f"\n✅ Migration completed: {result.total_changes} change(s) applied to {result.files_modified} file(s)")
+ if create_backups:
+ typer.echo(" Backup files created with .backup extension")
+ typer.echo(" Run 'pipelex validate all' to verify the migration")
+ except FileNotFoundError as exc:
+ typer.echo(f"❌ {exc}")
+ raise typer.Exit(1)
+ except Exception as exc:
+ typer.echo(f"❌ Migration failed: {exc}")
+ raise typer.Exit(1)
+
+
+# Typer group for migration commands
+migrate_app = typer.Typer(help="Migration commands")
+
+
+@migrate_app.command("run")
+def migrate_cmd(
+ target_path: Annotated[
+ str,
+ typer.Option(
+ "--path",
+ "-p",
+ help=("Path to Pipelex libraries folder (uses its pipelines), a directory of TOML files, or a single TOML file"),
+ ),
+ ] = "./pipelex_libraries",
+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Preview changes without applying them")] = False,
+ backups: Annotated[bool, typer.Option("--backups/--no-backups", help="Create backup files before migration")] = False,
+) -> None:
+ do_migrate(target_path=target_path, dry_run=dry_run, backups=backups)
diff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.py
new file mode 100644
index 000000000..e4f0ffc15
--- /dev/null
+++ b/pipelex/cli/commands/show_cmd.py
@@ -0,0 +1,68 @@
+from __future__ import annotations
+
+from typing import Annotated
+
+import typer
+
+from pipelex import pretty_print
+from pipelex.exceptions import PipelexCLIError, PipelexConfigError
+from pipelex.hub import get_pipe_provider, get_required_pipe
+from pipelex.pipelex import Pipelex
+from pipelex.tools.config.manager import config_manager
+
+
+def do_show_config() -> None:
+ """Show the pipelex configuration."""
+ try:
+ final_config = config_manager.load_config()
+ pretty_print(
+ final_config,
+ title=f"Pipelex configuration for project: {config_manager.get_project_name()}",
+ )
+ except Exception as exc:
+ raise PipelexConfigError(f"Error loading configuration: {exc}") from exc
+
+
+def do_list_pipes(relative_config_folder_path: str = "pipelex_libraries") -> None:
+ """List all available pipes."""
+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+
+ try:
+ get_pipe_provider().pretty_list_pipes()
+ except Exception as exc:
+ raise PipelexCLIError(f"Failed to list pipes: {exc}") from exc
+
+
+def do_show_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:
+ """Show a single pipe definition from the library."""
+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ pretty_print(pipe, title=f"Pipe '{pipe_code}'")
+
+
+# Typer group for show commands
+show_app = typer.Typer(help="Show and list commands")
+
+
+@show_app.command("config")
+def show_config_cmd() -> None:
+ do_show_config()
+
+
+@show_app.command("pipes")
+def list_pipes_cmd(
+ relative_config_folder_path: Annotated[
+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")
+ ] = "pipelex_libraries",
+) -> None:
+ do_list_pipes(relative_config_folder_path=relative_config_folder_path)
+
+
+@show_app.command("pipe")
+def show_pipe_cmd(
+ pipe_code: Annotated[str, typer.Argument(help="Pipeline code to show definition for")],
+ relative_config_folder_path: Annotated[
+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")
+ ] = "./pipelex_libraries",
+) -> None:
+ do_show_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)
diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.py
new file mode 100644
index 000000000..ca7754153
--- /dev/null
+++ b/pipelex/cli/commands/validate_cmd.py
@@ -0,0 +1,65 @@
+from __future__ import annotations
+
+import asyncio
+from typing import Annotated
+
+import typer
+
+from pipelex import log
+from pipelex.cli.commands.common import is_pipelex_libraries_folder
+from pipelex.hub import get_pipeline_tracker
+from pipelex.pipe_works.pipe_dry import dry_run_all_pipes, dry_run_single_pipe
+from pipelex.pipelex import Pipelex
+
+
+def do_validate(relative_config_folder_path: str = "./pipelex_libraries") -> None:
+ """Validate libraries and dry-run all pipes."""
+ if not is_pipelex_libraries_folder(relative_config_folder_path):
+ typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")
+ typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")
+ raise typer.Exit(1)
+
+ pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+ pipelex_instance.validate_libraries()
+ asyncio.run(dry_run_all_pipes())
+ log.info("Setup sequence passed OK, config and pipelines are validated.")
+
+
+def do_dry_run_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:
+ """Dry run a single pipe."""
+ if not is_pipelex_libraries_folder(relative_config_folder_path):
+ typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")
+ typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")
+ raise typer.Exit(1)
+
+ try:
+ pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)
+ pipelex_instance.validate_libraries()
+ asyncio.run(dry_run_single_pipe(pipe_code))
+ get_pipeline_tracker().output_flowchart()
+ except Exception as exc:
+ typer.echo(f"❌ Error running dry run for pipe '{pipe_code}': {exc}")
+ raise typer.Exit(1)
+
+
+# Typer group for validation commands
+validate_app = typer.Typer(help="Validation and dry-run commands")
+
+
+@validate_app.command("all")
+def validate_all_cmd(
+ relative_config_folder_path: Annotated[
+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")
+ ] = "./pipelex_libraries",
+) -> None:
+ do_validate(relative_config_folder_path=relative_config_folder_path)
+
+
+@validate_app.command("pipe")
+def dry_run_pipe_cmd(
+ pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],
+ relative_config_folder_path: Annotated[
+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")
+ ] = "./pipelex_libraries",
+) -> None:
+ do_dry_run_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)
diff --git a/pipelex/core/pipe_run_params.py b/pipelex/core/pipe_run_params.py
index da680fe4e..7c93ee3c2 100644
--- a/pipelex/core/pipe_run_params.py
+++ b/pipelex/core/pipe_run_params.py
@@ -111,6 +111,7 @@ class BatchParams(BaseModel):
class PipeRunParams(BaseModel):
run_mode: PipeRunMode = PipeRunMode.LIVE
final_stuff_code: Optional[str] = None
+ is_with_preliminary_text: Optional[bool] = None
output_multiplicity: Optional[PipeOutputMultiplicity] = None
dynamic_output_concept_code: Optional[str] = None
batch_params: Optional[BatchParams] = None
@@ -139,6 +140,7 @@ class PipeRunParams(BaseModel):
cls,
pipe_run_params: Self,
applied_output_multiplicity: Optional[PipeOutputMultiplicity],
+ is_with_preliminary_text: Optional[bool] = None,
) -> Self:
"""
Copy the run params the nb_output into the params, and remove the attribute.
@@ -152,6 +154,8 @@ class PipeRunParams(BaseModel):
elif isinstance(applied_output_multiplicity, int):
new_run_params.output_multiplicity = False
new_run_params.params[PipeRunParamKey.NB_OUTPUT] = applied_output_multiplicity
+ if is_with_preliminary_text is not None:
+ new_run_params.is_with_preliminary_text = is_with_preliminary_text
return new_run_params
@property
diff --git a/pipelex/core/registry_models.py b/pipelex/core/registry_models.py
index 3f400ac52..c9b2bfd95 100644
--- a/pipelex/core/registry_models.py
+++ b/pipelex/core/registry_models.py
@@ -17,6 +17,8 @@ from pipelex.core.stuff_content import (
TextAndImagesContent,
TextContent,
)
+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint
+from pipelex.libraries.pipelines.meta.pipeline_draft import PipelineDraft
from pipelex.pipe_controllers.pipe_batch import PipeBatch
from pipelex.pipe_controllers.pipe_batch_factory import PipeBatchFactory
from pipelex.pipe_controllers.pipe_condition import PipeCondition
@@ -86,6 +88,8 @@ class PipelexRegistryModels(RegistryModels):
PDFContent,
TextAndImagesContent,
PageContent,
+ PipelineBlueprint,
+ PipelineDraft,
]
EXPERIMENTAL: ClassVar[List[ModelType]] = [
diff --git a/pipelex/create/build_blueprint.py b/pipelex/create/build_blueprint.py
new file mode 100644
index 000000000..affce8844
--- /dev/null
+++ b/pipelex/create/build_blueprint.py
@@ -0,0 +1,97 @@
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from typing import Any, Dict, Optional, cast
+
+import typer
+from pydantic import ValidationError
+
+from pipelex import pretty_print
+from pipelex.create.helpers import get_support_file
+from pipelex.create.pipeline_toml import save_pipeline_blueprint_toml_to_path
+from pipelex.exceptions import PipeDefinitionError, PipelexCLIError
+from pipelex.hub import get_library_manager
+from pipelex.libraries.library_manager import LibraryManager
+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint
+from pipelex.pipe_works.pipe_dry import dry_run_pipe_codes
+from pipelex.pipeline.execute import execute_pipeline
+from pipelex.tools.misc.file_utils import save_text_to_path
+from pipelex.tools.misc.json_utils import save_as_json_to_path
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+
+
+async def do_build_blueprint(
+ domain: str,
+ pipeline_name: str,
+ requirements: str,
+ output_path: Optional[str],
+ validate: bool,
+) -> None:
+ pipe_output = await execute_pipeline(
+ pipe_code="build_blueprint",
+ input_memory={
+ "domain": domain,
+ "pipeline_name": pipeline_name,
+ "requirements": requirements,
+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),
+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),
+ },
+ )
+ blueprint = pipe_output.main_stuff_as(content_type=PipelineBlueprint)
+ pretty_print(blueprint, title="Pipeline Blueprint")
+ pipeline_draft = pipe_output.working_memory.get_stuff_as_str(name="pipeline_draft")
+ pretty_print(pipeline_draft, title="Pipeline Draft")
+
+ # Save or display result
+ output_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"
+ # blueprint_dict = blueprint.smart_dump()
+ # save_toml_to_path(data=blueprint_dict, path=output_path)
+ save_pipeline_blueprint_toml_to_path(blueprint=blueprint, path=output_path)
+ json_path = output_path.replace(Path(output_path).suffix, ".json")
+ save_as_json_to_path(object_to_save=blueprint, path=json_path)
+ draft_path = output_path.replace(Path(output_path).suffix, "_draft.md")
+ save_text_to_path(text=pipeline_draft, path=draft_path)
+ typer.echo(f"✅ Blueprint saved to '{output_path}'")
+
+ if validate:
+ try:
+ _load_pipes_from_generated_blueprint(blueprint=blueprint)
+ except PipeDefinitionError as exc:
+ raise PipelexCLIError(f"Failed to load pipes from generated blueprint at '{output_path}': {exc}") from exc
+
+ generated_pipe_codes = list(blueprint.pipe.keys())
+ if not generated_pipe_codes:
+ raise PipelexCLIError("No pipe found in generated blueprint to validate")
+ asyncio.run(dry_run_pipe_codes(pipe_codes=generated_pipe_codes))
+
+
+def _load_pipes_from_generated_blueprint(blueprint: PipelineBlueprint) -> None:
+ """Instantiate and register all pipes from a generated PipelineBlueprint.
+
+ This constructs the appropriate typed PipeBlueprint for each pipe entry and
+ uses LibraryManager.load_pipe_from_blueprint to create the Pipe, then adds it
+ to the in-memory PipeLibrary so that validation can run against them.
+ """
+ library_manager = cast(LibraryManager, get_library_manager())
+ domain_code = blueprint.domain
+
+ for pipe_code, details in blueprint.pipe.items():
+ details_dict: Dict[str, Any] = details.copy()
+
+ # Build the concrete PipeBlueprint via LibraryManager helper
+ try:
+ pipe_blueprint = LibraryManager.make_pipe_blueprint_from_details(
+ domain_code=domain_code,
+ details_dict=details_dict,
+ )
+ except ValidationError as exc:
+ error_msg = format_pydantic_validation_error(exc=exc)
+ raise PipeDefinitionError(f"Failed to build pipe blueprint for pipe '{pipe_code}': {error_msg}") from exc
+
+ # Create pipe and register in library
+ pipe = LibraryManager.load_pipe_from_blueprint(
+ pipe_code=pipe_code,
+ pipe_blueprint=pipe_blueprint,
+ )
+ library_manager.pipe_library.add_new_pipe(pipe=pipe)
diff --git a/pipelex/create/draft_pipeline.py b/pipelex/create/draft_pipeline.py
new file mode 100644
index 000000000..f87dd736e
--- /dev/null
+++ b/pipelex/create/draft_pipeline.py
@@ -0,0 +1,60 @@
+from __future__ import annotations
+
+from typing import Optional
+
+import typer
+
+from pipelex import pretty_print
+from pipelex.create.helpers import get_support_file
+from pipelex.create.pipeline_toml import dict_to_toml, save_toml_to_path
+from pipelex.libraries.pipelines.meta.pipeline_draft import PipelineDraft
+from pipelex.pipeline.execute import execute_pipeline
+from pipelex.tools.misc.file_utils import save_text_to_path
+
+
+async def do_draft_pipeline_text(
+ domain: str,
+ pipeline_name: str,
+ requirements: str,
+ output_path: Optional[str],
+) -> None:
+ pipe_output = await execute_pipeline(
+ pipe_code="draft_pipeline_text",
+ input_memory={
+ "domain": domain,
+ "pipeline_name": pipeline_name,
+ "requirements": requirements,
+ "rules": get_support_file(subpath="create/draft_pipelines.md"),
+ },
+ )
+ draft_text = pipe_output.main_stuff_as_str
+ pretty_print(draft_text, title="Pipeline Draft Text")
+
+ # Save or display result
+ output_path = output_path or "pipelex/libraries/pipelines/temp/pipeline_draft.md"
+ save_text_to_path(text=draft_text, path=output_path)
+ typer.echo(f"✅ Pipeline draft text saved to '{output_path}'")
+
+
+async def do_draft_pipeline(
+ pipeline_name: str,
+ requirements: str,
+ output_path: Optional[str],
+) -> None:
+ pipe_output = await execute_pipeline(
+ pipe_code="draft_pipeline",
+ input_memory={
+ "pipeline_name": pipeline_name,
+ "requirements": requirements,
+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),
+ },
+ )
+ draft = pipe_output.main_stuff_as(content_type=PipelineDraft)
+ pretty_print(draft, title="Pipeline Draft")
+ draft_dict = draft.smart_dump()
+ pretty_print(dict_to_toml(data=draft_dict), title="Pipeline Draft (TOML)")
+
+ # Save or display result
+ output_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"
+ save_toml_to_path(data=draft_dict, path=output_path)
+ typer.echo(f"✅ Blueprint saved to '{output_path}'")
diff --git a/pipelex/create/helpers.py b/pipelex/create/helpers.py
new file mode 100644
index 000000000..664f4df53
--- /dev/null
+++ b/pipelex/create/helpers.py
@@ -0,0 +1,55 @@
+from pathlib import Path
+from typing import Dict, List, Optional
+
+from pipelex.tools.misc.file_utils import load_text_from_path
+
+
+class PipelexSupportFiles:
+ """Dictionary-like interface to access all files in .pipelex by their subpath."""
+
+ def __init__(self, base_path: str = ".pipelex"):
+ self.base_path = Path(base_path)
+ self._file_cache: Dict[str, str] = {}
+
+ def __getitem__(self, subpath: str) -> str:
+ """Get file content by subpath (e.g., 'design_pipelines.md')."""
+ if subpath not in self._file_cache:
+ file_path = self.base_path / subpath
+ if not file_path.exists():
+ raise KeyError(f"File '{subpath}' not found in {self.base_path}")
+ self._file_cache[subpath] = load_text_from_path(str(file_path))
+ return self._file_cache[subpath]
+
+ def __contains__(self, subpath: str) -> bool:
+ """Check if a file exists by subpath."""
+ file_path = self.base_path / subpath
+ return file_path.exists() and file_path.is_file()
+
+ def get(self, subpath: str, default: Optional[str] = None) -> Optional[str]:
+ """Get file content by subpath with optional default."""
+ try:
+ return self[subpath]
+ except KeyError:
+ return default
+
+ def keys(self) -> List[str]:
+ """Get all available file subpaths."""
+ if not self.base_path.exists():
+ return []
+
+ files: List[str] = []
+ for file_path in self.base_path.rglob("*"):
+ if file_path.is_file():
+ subpath = str(file_path.relative_to(self.base_path))
+ files.append(subpath)
+ return sorted(files)
+
+ def clear_cache(self) -> None:
+ """Clear the internal file cache."""
+ self._file_cache.clear()
+
+
+def get_support_file(subpath: str) -> str:
+ """Get pipeline creation rules from design_pipelines.md."""
+ support_files = PipelexSupportFiles()
+ return support_files[subpath]
diff --git a/pipelex/create/pipeline_toml.py b/pipelex/create/pipeline_toml.py
new file mode 100644
index 000000000..9e6bd8a5c
--- /dev/null
+++ b/pipelex/create/pipeline_toml.py
@@ -0,0 +1,138 @@
+from __future__ import annotations
+
+from typing import Any, Dict, List, Mapping, cast
+
+import tomlkit
+from tomlkit import array, document, inline_table, table
+from tomlkit import string as tk_string
+
+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint
+from pipelex.tools.misc.file_utils import save_text_to_path
+from pipelex.tools.misc.json_utils import remove_none_values_from_dict
+from pipelex.tools.misc.toml_utils import clean_trailing_whitespace
+
+
+def make_toml_string(
+ text: str,
+ prefer_literal: bool = False,
+ force_multiline: bool = False,
+ ensure_trailing_newline: bool = True,
+ ensure_leading_blank_line_in_value: bool = False,
+):
+ """
+ Build a tomlkit string node.
+ - If `force_multiline` or the text contains '\\n', we emit a triple-quoted multiline string.
+ - When multiline, `ensure_trailing_newline` puts the closing quotes on their own line.
+ - When multiline, `ensure_leading_blank_line_in_value` inserts a real blank line at the start of the value.
+ """
+ needs_multiline = force_multiline or ("\n" in text)
+ normalized = text
+
+ if needs_multiline:
+ if ensure_leading_blank_line_in_value and not normalized.startswith("\n"):
+ normalized = "\n" + normalized
+ if ensure_trailing_newline and not normalized.endswith("\n"):
+ normalized = normalized + "\n"
+
+ use_literal = prefer_literal and ("'''" not in normalized)
+ return tk_string(normalized, multiline=needs_multiline, literal=use_literal)
+
+
+def _convert_to_inline(value: Any):
+ """Recursively convert Python values; dicts -> inline tables; lists kept as arrays."""
+ if isinstance(value, Mapping):
+ value = cast(Mapping[str, Any], value)
+ inline_table_obj = inline_table()
+ for key, value_item in value.items():
+ inline_table_obj[key] = _convert_to_inline(value_item)
+ return inline_table_obj
+
+ if isinstance(value, list):
+ value = cast(List[Any], value)
+ array_obj = array()
+ array_obj.multiline(True) # set to False for single-line arrays
+ for element in value:
+ if isinstance(element, Mapping):
+ element = cast(Mapping[str, Any], element)
+ inline_element = inline_table()
+ for inner_key, inner_value in element.items():
+ inline_element[inner_key] = _convert_to_inline(inner_value)
+ array_obj.append(inline_element) # pyright: ignore[reportUnknownMemberType]
+ else:
+ array_obj.append(_convert_to_inline(element)) # pyright: ignore[reportUnknownMemberType]
+ return array_obj
+
+ if isinstance(value, str):
+ # Triple quotes if needed (or forced); closing quotes on their own line.
+ return make_toml_string(
+ value,
+ prefer_literal=False, # flip to True for '''...'''
+ force_multiline=False, # flip to True to force """...""" even without \n
+ ensure_trailing_newline=True, # keep closing """ on its own line
+ ensure_leading_blank_line_in_value=False, # flip to True to keep a blank first line
+ )
+ return value
+
+
+def _convert_mapping_to_table(mapping: Mapping[str, Any]) -> Any:
+ """Convert a mapping into a TOML Table where any nested mappings (third level+)
+ are converted to inline tables.
+
+ This creates a second-level standard table, and only uses inline tables for
+ third level and deeper structures.
+ """
+ tbl = table()
+ for field_key, field_value in mapping.items():
+ if isinstance(field_value, Mapping):
+ # Third-level mapping -> inline table
+ tbl.add(field_key, _convert_to_inline(field_value))
+ else:
+ tbl.add(field_key, _convert_to_inline(field_value))
+ return tbl
+
+
+def dict_to_toml(data: Mapping[str, Any]) -> str:
+ """Top-level keys become tables; second-level mappings become tables; inline tables start at third level."""
+ data = remove_none_values_from_dict(data=data)
+ document_root = document()
+ for section_key, section_value in data.items():
+ if isinstance(section_value, Mapping):
+ section_value = cast(Mapping[str, Any], section_value)
+ table_obj = table()
+ for field_key, field_value in section_value.items():
+ if isinstance(field_value, Mapping):
+ # Second level mapping -> standard table [section.field]
+ sub_table = _convert_mapping_to_table(cast(Mapping[str, Any], field_value))
+ table_obj.add(field_key, sub_table)
+ else:
+ table_obj.add(field_key, _convert_to_inline(field_value))
+ document_root.add(section_key, table_obj) # `[section_key]` section
+ else:
+ document_root.add(section_key, _convert_to_inline(section_value))
+ dumped_content = tomlkit.dumps(document_root) # pyright: ignore[reportUnknownMemberType]
+ return dumped_content
+
+
+def pipeline_blueprint_to_toml(blueprint: PipelineBlueprint) -> str:
+ blueprint_dict = blueprint.smart_dump()
+ return dict_to_toml(data=blueprint_dict)
+
+
+def save_toml_to_path(data: Dict[str, Any], path: str) -> None:
+ """Save dictionary as TOML to file path.
+
+ Args:
+ data: Dictionary to save as TOML
+ path: Path where to save the TOML file
+ """
+ # data_cleaned = remove_none_values_from_dict(data=data)
+ data_cleaned = data
+ with open(path, "w", encoding="utf-8") as file:
+ toml_content: str = dict_to_toml(data=data_cleaned)
+ cleaned_content = clean_trailing_whitespace(toml_content)
+ file.write(cleaned_content)
+
+
+def save_pipeline_blueprint_toml_to_path(blueprint: PipelineBlueprint, path: str) -> None:
+ pipeline_blueprint_toml = pipeline_blueprint_to_toml(blueprint=blueprint)
+ save_text_to_path(text=pipeline_blueprint_toml, path=path)
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 90757d578..90c8cad2e 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -1,3 +1,4 @@
+import importlib
import os
from pathlib import Path
from typing import Any, ClassVar, Dict, List, Optional, Type
@@ -15,7 +16,7 @@ from pipelex.core.concept_library import ConceptLibrary
from pipelex.core.domain import Domain
from pipelex.core.domain_library import DomainLibrary
from pipelex.core.pipe_abstract import PipeAbstract
-from pipelex.core.pipe_blueprint import PipeSpecificFactoryProtocol
+from pipelex.core.pipe_blueprint import PipeBlueprint, PipeSpecificFactoryProtocol
from pipelex.core.pipe_library import PipeLibrary
from pipelex.exceptions import (
ConceptLibraryError,
@@ -29,8 +30,8 @@ from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
from pipelex.libraries.pipeline_blueprint import (
ConceptBlueprintError,
PipeBlueprintError,
+ PipelineBlueprint,
PipelineBlueprintValidationError,
- PipelineLibraryBlueprint,
)
from pipelex.tools.class_registry_utils import ClassRegistryUtils
from pipelex.tools.misc.file_utils import find_files_in_dir
@@ -202,11 +203,11 @@ class LibraryManager(LibraryManagerAbstract):
nb_pipes_loaded = len(self.pipe_library.root) - nb_pipes_before
log.verbose(f"Loaded {nb_pipes_loaded} pipes from '{toml_path.name}'")
- def _load_blueprint_from_file(self, toml_path: Path) -> PipelineLibraryBlueprint:
+ def _load_blueprint_from_file(self, toml_path: Path) -> PipelineBlueprint:
"""Load and validate a pipeline blueprint from a TOML file."""
try:
toml_data = load_toml_from_path(path=str(toml_path))
- blueprint = PipelineLibraryBlueprint.model_validate(toml_data)
+ blueprint = PipelineBlueprint.model_validate(toml_data)
return blueprint
except ValidationError as exc:
error_msg = format_pydantic_validation_error(exc)
@@ -217,7 +218,7 @@ class LibraryManager(LibraryManagerAbstract):
except Exception as exc:
raise LibraryError(f"Failed to load TOML file '{toml_path}': {exc}") from exc
- def _load_concepts_from_blueprint(self, blueprint: PipelineLibraryBlueprint, file_path: str):
+ def _load_concepts_from_blueprint(self, blueprint: PipelineBlueprint, file_path: str):
"""Load concepts from a validated blueprint."""
for concept_name, concept_data in blueprint.concept.items():
try:
@@ -245,7 +246,7 @@ class LibraryManager(LibraryManagerAbstract):
error_msg=error_msg,
) from exc
- def _load_pipes_from_blueprint(self, blueprint: PipelineLibraryBlueprint, file_path: str):
+ def _load_pipes_from_blueprint(self, blueprint: PipelineBlueprint, file_path: str):
"""Load pipes from a validated blueprint."""
for pipe_name, pipe_data in blueprint.pipe.items():
try:
@@ -378,3 +379,114 @@ Old syntax will be removed in v0.3.0.
details_dict=details_dict,
)
return pipe_from_blueprint
+
+ @classmethod
+ def make_pipe_from_details_dict(
+ cls,
+ domain_code: str,
+ pipe_code: str,
+ details_dict: Dict[str, Any],
+ ) -> PipeAbstract:
+ # Delegate to the new make_pipe_from_blueprint method
+ return cls.make_pipe_from_blueprint(
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ details_dict=details_dict,
+ )
+
+ @classmethod
+ def load_pipe_from_blueprint(
+ cls,
+ pipe_code: str,
+ pipe_blueprint: PipeBlueprint,
+ ) -> PipeAbstract:
+ """Create a Pipe from a concrete PipeBlueprint instance.
+
+ This resolves the corresponding factory by converting the blueprint class name
+ (e.g. "PipeLLMBlueprint") to its factory name ("PipeLLMFactory").
+ """
+ # The blueprint must be a concrete subclass like PipeLLMBlueprint, PipeOcrBlueprint, etc.
+ blueprint_class_name = type(pipe_blueprint).__name__
+ if blueprint_class_name == "PipeBlueprint":
+ raise PipeFactoryError("Cannot load pipe from base PipeBlueprint. Please provide a specific blueprint subclass (e.g. PipeLLMBlueprint).")
+
+ # Derive factory name: PipeLLMBlueprint -> PipeLLMFactory
+ factory_class_name = blueprint_class_name.replace("Blueprint", "Factory")
+
+ try:
+ pipe_factory: Type[PipeSpecificFactoryProtocol[Any, Any]] = KajsonManager.get_class_registry().get_required_subclass(
+ name=factory_class_name,
+ base_class=PipeSpecificFactoryProtocol,
+ )
+ except ClassRegistryNotFoundError as factory_not_found_error:
+ raise PipeFactoryError(
+ f"Pipe '{pipe_code}' couldn't be created: factory '{factory_class_name}' not found: {factory_not_found_error}"
+ ) from factory_not_found_error
+ except ClassRegistryInheritanceError as factory_inheritance_error:
+ raise PipeFactoryError(
+ f"Pipe '{pipe_code}' couldn't be created: factory '{factory_class_name}' is not a subclass of {type(PipeSpecificFactoryProtocol)}."
+ ) from factory_inheritance_error
+
+ # Domain is part of the blueprint
+ domain_code = pipe_blueprint.domain
+
+ # Let the specific factory build the concrete Pipe instance
+ pipe_from_blueprint: PipeAbstract = pipe_factory.make_pipe_from_blueprint(
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ pipe_blueprint=pipe_blueprint, # type: ignore[arg-type]
+ )
+ return pipe_from_blueprint
+
+ @classmethod
+ def make_pipe_blueprint_from_details(
+ cls,
+ domain_code: str,
+ details_dict: Dict[str, Any],
+ ) -> PipeBlueprint:
+ """Create a concrete PipeBlueprint subclass instance from details.
+
+ This resolves the factory based on the provided details (supports both
+ new and legacy formats), locates the corresponding <PipeClassName>Blueprint
+ in the factory module, and validates the blueprint.
+ """
+ # Determine pipe class name from details
+ if "type" in details_dict and "definition" in details_dict:
+ pipe_class_name = details_dict["type"]
+ normalized_details = details_dict.copy()
+ else:
+ try:
+ pipe_class_name, legacy_definition = next(iter(details_dict.items()))
+ normalized_details = {"definition": legacy_definition}
+ except StopIteration as empty_err:
+ raise PipeFactoryError("Pipe details are empty; cannot determine pipe type") from empty_err
+
+ factory_class_name = f"{pipe_class_name}Factory"
+
+ # Resolve factory to locate its module (where the Blueprint class is defined)
+ try:
+ pipe_factory: Type[PipeSpecificFactoryProtocol[Any, Any]] = KajsonManager.get_class_registry().get_required_subclass(
+ name=factory_class_name,
+ base_class=PipeSpecificFactoryProtocol,
+ )
+ except ClassRegistryNotFoundError as factory_not_found_error:
+ raise PipeFactoryError(
+ f"Factory '{factory_class_name}' not found for pipe type '{pipe_class_name}': {factory_not_found_error}"
+ ) from factory_not_found_error
+ except ClassRegistryInheritanceError as factory_inheritance_error:
+ raise PipeFactoryError(
+ f"Factory '{factory_class_name}' is not a subclass of {type(PipeSpecificFactoryProtocol)}."
+ ) from factory_inheritance_error
+
+ factory_module = importlib.import_module(pipe_factory.__module__)
+ blueprint_class_name = f"{pipe_class_name}Blueprint"
+
+ try:
+ blueprint_cls: Type[PipeBlueprint] = getattr(factory_module, blueprint_class_name)
+ except AttributeError as exc:
+ raise PipeFactoryError(f"Cannot find blueprint class '{blueprint_class_name}' in module '{pipe_factory.__module__}'") from exc
+
+ details_with_domain = normalized_details.copy()
+ details_with_domain["domain"] = domain_code
+
+ return blueprint_cls.model_validate(details_with_domain)
diff --git a/pipelex/libraries/llm_deck/base_llm_deck.toml b/pipelex/libraries/llm_deck/base_llm_deck.toml
index 1c9271aeb..620509679 100644
--- a/pipelex/libraries/llm_deck/base_llm_deck.toml
+++ b/pipelex/libraries/llm_deck/base_llm_deck.toml
@@ -55,7 +55,9 @@ llm_to_enrich = { llm_handle = "best-claude", temperature = 0.1 }
llm_for_question_and_excerpt_reformulation = { llm_handle = "gpt-4o", temperature = 0.9 }
# Engineering skills
-llm_to_engineer = { llm_handle = "best-claude", temperature = 0.5 }
+llm_to_engineer = { llm_handle = "best-claude", temperature = 0.2 }
+llm_to_pipelex = { llm_handle = "best-claude", temperature = 0.2 }
+# llm_to_pipelex = { llm_handle = "gpt-5-mini", temperature = 0.2 }
# Image skills
llm_to_write_imgg_prompt = { llm_handle = "best-claude", temperature = 0.2 }
diff --git a/pipelex/libraries/pipeline_blueprint.py b/pipelex/libraries/pipeline_blueprint.py
index 5b60fb821..54ebb0331 100644
--- a/pipelex/libraries/pipeline_blueprint.py
+++ b/pipelex/libraries/pipeline_blueprint.py
@@ -1,11 +1,10 @@
-from pathlib import Path
from typing import Any, Dict, Optional, Union
-from pydantic import BaseModel, Field
+from pydantic import Field
from pipelex.core.concept_factory import ConceptBlueprint
+from pipelex.core.stuff_content import StructuredContent
from pipelex.exceptions import LibraryError
-from pipelex.tools.misc.toml_utils import dict_to_toml_string, save_toml_to_path
class PipelineBlueprintValidationError(LibraryError):
@@ -35,15 +34,8 @@ class PipeBlueprintError(LibraryError):
super().__init__(f"Error loading pipe '{pipe_name}' from '{file_path}': {error_msg}")
-class PipeStepBlueprint(BaseModel):
- """Blueprint of a step in a pipe sequence."""
-
- pipe: str
- result: str
-
-
-class PipelineLibraryBlueprint(BaseModel):
- """Complete blueprint of a pipeline library TOML file."""
+class PipelineBlueprint(StructuredContent):
+ """Complete blueprint of a pipeline TOML definition."""
# Domain information (required)
domain: str
@@ -57,37 +49,3 @@ class PipelineLibraryBlueprint(BaseModel):
# Pipes section - pipe_name -> blueprint dict
pipe: Dict[str, Dict[str, Any]] = Field(default_factory=dict)
-
- def to_toml_dict(self) -> Dict[str, Any]:
- """Convert blueprint back to TOML-compatible dictionary structure."""
- result: Dict[str, Any] = {"domain": self.domain}
-
- # Add optional domain-level fields
- if self.definition:
- result["definition"] = self.definition
- if self.system_prompt:
- result["system_prompt"] = self.system_prompt
- if self.system_prompt_to_structure:
- result["system_prompt_to_structure"] = self.system_prompt_to_structure
- if self.prompt_template_to_structure:
- result["prompt_template_to_structure"] = self.prompt_template_to_structure
-
- # Add concepts section if not empty
- if self.concept:
- result["concept"] = self.concept
-
- # Add pipes section if not empty
- if self.pipe:
- result["pipe"] = self.pipe
-
- return result
-
- def to_toml_string(self) -> str:
- """Convert blueprint to TOML string format."""
- return dict_to_toml_string(self.to_toml_dict())
-
- def save_to_file(self, file_path: Union[str, Path]) -> None:
- """Save blueprint to a TOML file."""
- path = Path(file_path)
- path.parent.mkdir(parents=True, exist_ok=True)
- save_toml_to_path(data=self.to_toml_dict(), path=str(path))
diff --git a/pipelex/libraries/pipelines/documents.toml b/pipelex/libraries/pipelines/documents.toml
index 77fdfd536..c3826d8ad 100644
--- a/pipelex/libraries/pipelines/documents.toml
+++ b/pipelex/libraries/pipelines/documents.toml
@@ -11,14 +11,16 @@ TextAndImagesContent = "A content that comprises text and images where the text
# It can be named however you want
# but it must be either an image or a pdf or a concept which refines one of them
[pipe.ocr_page_contents_from_pdf]
-PipeOcr = "Extract page contents from a PDF document"
+type = "PipeOcr"
+definition = "Extract page contents from a PDF document"
inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
page_views = false
[pipe.ocr_page_contents_and_views_from_pdf]
-PipeOcr = "Extract page contents from a PDF document as well as full page views"
+type = "PipeOcr"
+definition = "Extract page contents from a PDF document as well as full page views"
inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
diff --git a/pipelex/libraries/pipelines/images.toml b/pipelex/libraries/pipelines/images.toml
index f708d0e89..9d4197279 100644
--- a/pipelex/libraries/pipelines/images.toml
+++ b/pipelex/libraries/pipelines/images.toml
@@ -20,7 +20,8 @@ refines = "Image"
#################################################################
[pipe.describe_image]
-PipeLLM = "Describe an image"
+type = "PipeLLM"
+definition = "Describe an image"
inputs = { image = "Image" }
output = "VisualDescription"
system_prompt = "You are a very good observer."
@@ -30,7 +31,8 @@ Describe the provided image in great detail.
"""
[pipe.describe_photo]
-PipeLLM = "Describe a photo"
+type = "PipeLLM"
+definition = "Describe a photo"
inputs = { photo = "Photo" }
output = "VisualDescription"
system_prompt = "You are a very good observer."
@@ -48,14 +50,16 @@ Describe the provided photo and how it was shot: scene, lighting, camera, etc.
# It can be named however you want,
# but it must be either an ImgGenPrompt or a concept which refines ImgGenPrompt
[pipe.generate_image]
-PipeImgGen = "Generate an image"
+type = "PipeImgGen"
+definition = "Generate an image"
inputs = { prompt = "ImgGenPrompt" }
output = "Image"
nb_steps = 2
[pipe.generate_photo]
-PipeImgGen = "Generate a photo"
+type = "PipeImgGen"
+definition = "Generate a photo"
inputs = { prompt = "ImgGenPrompt" }
output = "images.Photo"
nb_steps = 8
diff --git a/pipelex/libraries/pipelines/meta/meta_pipeline.toml b/pipelex/libraries/pipelines/meta/meta_pipeline.toml
new file mode 100644
index 000000000..21d6d99d0
--- /dev/null
+++ b/pipelex/libraries/pipelines/meta/meta_pipeline.toml
@@ -0,0 +1,64 @@
+domain = "meta"
+definition = "Meta domain is the domain of pipelines about pipelines"
+
+[concept]
+PipelineDraft = "Rough solution to create a pipeline"
+PipelineDraftText = "Rough solution to create a pipeline in text format"
+PipeDraft = "Rough solution to create a pipe"
+PipelineBlueprint = "Details enabling to create a pipeline"
+ConceptBlueprint = "Details enabling to create a concept"
+PipeBlueprint = "Details enabling to create a pipe"
+
+[pipe]
+
+[pipe.build_blueprint]
+type = "PipeSequence"
+definition = "Build a pipeline blueprint from natural language requirements"
+inputs = { draft_pipeline_rules = "Text", build_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }
+output = "PipelineBlueprint"
+steps = [
+ { pipe = "draft_pipeline_text", result = "pipeline_draft" },
+ { pipe = "structure_blueprint", result = "pipeline_blueprint" },
+]
+
+[pipe.draft_pipeline_text]
+type = "PipeLLM"
+definition = "Generate a rough pipeline draft from natural language requirements"
+inputs = { draft_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }
+output = "PipelineDraftText"
+llm = "llm_to_pipelex"
+prompt_template = """
+You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.
+
+@draft_pipeline_rules
+
+---
+Name of the pipeline: $pipeline_name.
+
+Domain: $domain.
+
+Requirements: $requirements
+"""
+
+[pipe.structure_blueprint]
+type = "PipeLLM"
+definition = "Generate a pipeline blueprint from natural language requirements"
+inputs = { build_pipeline_rules = "Text", pipeline_draft = "Text", pipeline_name = "Text", domain = "Text", requirements = "Text" }
+output = "PipelineBlueprint"
+llm = "llm_to_pipelex"
+# structuring_method = "preliminary_text"
+prompt_template = """
+You are a Pipelex pipeline architect. Build a structured pipeline blueprint from the provided pipeline draft text.
+
+@build_pipeline_rules
+
+@pipeline_draft
+
+---
+Name of the pipeline: $pipeline_name.
+
+Domain: $domain.
+
+Requirements: $requirements.
+"""
+
diff --git a/pipelex/libraries/pipelines/meta/pipeline_draft.py b/pipelex/libraries/pipelines/meta/pipeline_draft.py
new file mode 100644
index 000000000..26e5ab970
--- /dev/null
+++ b/pipelex/libraries/pipelines/meta/pipeline_draft.py
@@ -0,0 +1,27 @@
+from typing import Dict
+
+from pydantic import Field
+
+from pipelex.core.stuff_content import StructuredContent
+
+
+class PipeDraft(StructuredContent):
+ code: str
+ type: str
+ definition: str
+ inputs: Dict[str, str]
+ output: str
+
+
+class PipelineDraft(StructuredContent):
+ """Complete blueprint of a pipeline library TOML file."""
+
+ # Domain information (required)
+ domain: str
+ definition: str
+
+ # Concepts section - concept_name -> definition (string) or blueprint (dict)
+ concept: Dict[str, str] = Field(default_factory=dict)
+
+ # Pipes section - pipe_name -> blueprint dict
+ pipe: Dict[str, PipeDraft] = Field(default_factory=dict)
diff --git a/pipelex/tools/migrate/__init__.py b/pipelex/migration/__init__.py
similarity index 100%
rename from pipelex/tools/migrate/__init__.py
rename to pipelex/migration/__init__.py
diff --git a/pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.py b/pipelex/migration/migrate_v0_1_0_to_v0_2_0.py
similarity index 80%
rename from pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.py
rename to pipelex/migration/migrate_v0_1_0_to_v0_2_0.py
index adf889cf8..b11e84fd7 100644
--- a/pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.py
+++ b/pipelex/migration/migrate_v0_1_0_to_v0_2_0.py
@@ -5,7 +5,7 @@ import shutil
from pathlib import Path
from typing import Any, ClassVar, Dict, List, Match
-from pipelex.tools.migrate.migration_result import MigrationResult
+from pipelex.migration.migration_result import MigrationResult
class TomlMigrator:
@@ -247,13 +247,68 @@ def migrate_concept_syntax(directory: Path, create_backups: bool = True, dry_run
"""
Convenience function to migrate TOML files from Concept = to definition = syntax.
+ This function accepts either a directory (the historical behavior) or a path to a single TOML file.
+
Args:
- directory: Directory containing TOML files to migrate
- create_backups: Whether to create backup files before migration
+ directory: Path to a directory containing TOML files to migrate, or a single TOML file
+ create_backups: Whether to create backup files before migration (ignored in dry-run)
dry_run: If True, only preview changes without applying them
Returns:
MigrationResult with migration statistics
"""
migrator = TomlMigrator()
- return migrator.migrate_directory(directory, create_backups, dry_run)
+
+ # Support passing a single TOML file path in addition to a directory
+ input_path = directory
+ if input_path.is_file():
+ result = MigrationResult()
+ result.files_processed = 1
+
+ if input_path.suffix.lower() != ".toml":
+ result.errors.append(f"Provided file is not a .toml file: {input_path}")
+ return result
+
+ try:
+ with open(input_path, "r", encoding="utf-8") as f:
+ content = f.read()
+ except Exception as e:
+ result.errors.append(f"Failed to read file {input_path}: {e}")
+ return result
+
+ if not migrator.needs_migration(content):
+ return result
+
+ # Count only concept changes for consistency with directory migration statistics
+ def _is_inside_multiline_string(text: str, position: int) -> bool:
+ """Local helper to check if a position is inside a TOML multiline string."""
+ text_before = text[:position]
+ triple_double_quotes = text_before.count('"""')
+ triple_single_quotes = text_before.count("'''")
+ return (triple_double_quotes % 2 == 1) or (triple_single_quotes % 2 == 1)
+
+ matches = list(migrator.concept_pattern.finditer(content))
+ changes_count = 0
+ for match in matches:
+ if not _is_inside_multiline_string(content, match.start()):
+ changes_count += 1
+
+ if dry_run:
+ result.files_modified = 1
+ result.total_changes = changes_count
+ result.modified_files.append(input_path)
+ return result
+
+ # Perform actual migration
+ try:
+ actual_changes = migrator.migrate_file(input_path, create_backup=create_backups)
+ result.files_modified = 1
+ result.total_changes = actual_changes
+ result.modified_files.append(input_path)
+ except Exception as e:
+ result.errors.append(f"Error processing {input_path}: {e}")
+
+ return result
+
+ # Default: treat as directory (existing behavior)
+ return migrator.migrate_directory(input_path, create_backups, dry_run)
diff --git a/pipelex/tools/migrate/migration_result.py b/pipelex/migration/migration_result.py
similarity index 100%
rename from pipelex/tools/migrate/migration_result.py
rename to pipelex/migration/migration_result.py
diff --git a/pipelex/pipe_operators/pipe_llm.py b/pipelex/pipe_operators/pipe_llm.py
index f92f8ac05..62c2294f8 100644
--- a/pipelex/pipe_operators/pipe_llm.py
+++ b/pipelex/pipe_operators/pipe_llm.py
@@ -11,6 +11,7 @@ from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingChoice
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.config import StaticValidationReaction, get_config
+from pipelex.core.concept import Concept
from pipelex.core.concept_code_factory import ConceptCodeFactory
from pipelex.core.concept_native import NativeConcept, NativeConceptClass
from pipelex.core.domain import Domain, SpecialDomain
@@ -79,7 +80,11 @@ class PipeLLM(PipeOperator):
if self.structuring_method is not None:
output_concept = get_required_concept(concept_code=self.output_concept_code)
if output_concept.structure_class_name == NativeConceptClass.TEXT:
- raise PipeDefinitionError(f"Output concept '{self.output_concept_code}' is a Text concept, so it cannot be structured")
+ concept_name = Concept.extract_concept_name_from_str(concept_str=self.output_concept_code)
+ raise PipeDefinitionError(
+ f"Output concept '{self.output_concept_code}' is considered a Text concept, "
+ f"so it cannot be structured. Maybe you forgot to add '{concept_name}' to the class registry?"
+ )
return self
@override
@@ -305,9 +310,13 @@ class PipeLLM(PipeOperator):
)
)
+ is_with_preliminary_text = (
+ self.structuring_method == StructuringMethod.PRELIMINARY_TEXT
+ ) or get_config().pipelex.structure_config.is_default_text_then_structure
llm_prompt_run_params = PipeRunParams.copy_by_injecting_multiplicity(
pipe_run_params=pipe_run_params,
applied_output_multiplicity=applied_output_multiplicity,
+ is_with_preliminary_text=is_with_preliminary_text,
)
# llm_prompt_1: LLMPrompt = (
# await self.pipe_llm_prompt.run_pipe(
@@ -340,7 +349,8 @@ class PipeLLM(PipeOperator):
log.debug(f"PipeLLM generating {fixed_nb_output} output(s)" if fixed_nb_output else "PipeLLM generating a list of output(s)")
llm_prompt_2_factory: Optional[LLMPromptFactoryAbstract]
- if structuring_method := self.structuring_method:
+ if self.structuring_method:
+ structuring_method = cast(StructuringMethod, self.structuring_method)
log.debug(f"PipeLLM pipe_code is '{self.code}' and structuring_method is '{structuring_method}'")
match structuring_method:
case StructuringMethod.DIRECT:
diff --git a/pipelex/pipe_operators/pipe_llm_factory.py b/pipelex/pipe_operators/pipe_llm_factory.py
index d5c3ce043..69d6d420c 100644
--- a/pipelex/pipe_operators/pipe_llm_factory.py
+++ b/pipelex/pipe_operators/pipe_llm_factory.py
@@ -14,6 +14,7 @@ from pipelex.pipe_operators.pipe_jinja2_factory import PipeJinja2Factory
from pipelex.pipe_operators.pipe_llm import PipeLLM, StructuringMethod
from pipelex.pipe_operators.pipe_llm_prompt import PipeLLMPrompt
from pipelex.tools.templating.jinja2_errors import Jinja2TemplateError
+from pipelex.tools.templating.template_provider_abstract import TemplateNotFoundError
from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_lists
@@ -100,12 +101,16 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):
error_msg += "The prompt template is not provided."
raise PipeDefinitionError(error_msg) from exc
elif pipe_blueprint.prompt is None and pipe_blueprint.prompt_name is None:
- # no jinja2 provided, no verbatim name, no fixed text, let's use the pipe code as jinja2 name
- user_pipe_jinja2 = PipeJinja2(
- code="adhoc_for_user_prompt",
- domain=domain_code,
- jinja2_name=pipe_code,
- )
+ # no jinja2 provided, no verbatim name, no fixed text, let's try and use the pipe code as jinja2 name
+ try:
+ user_pipe_jinja2 = PipeJinja2(
+ code="adhoc_for_user_prompt",
+ domain=domain_code,
+ jinja2_name=pipe_code,
+ )
+ except TemplateNotFoundError as exc:
+ error_msg = f"Jinja2 template not found for pipe '{pipe_code}' in domain '{domain_code}': {exc}."
+ raise PipeDefinitionError(error_msg) from exc
user_images: List[str] = []
if pipe_blueprint.inputs:
diff --git a/pipelex/pipe_operators/pipe_llm_prompt.py b/pipelex/pipe_operators/pipe_llm_prompt.py
index 218bc40c2..7f65b58e4 100644
--- a/pipelex/pipe_operators/pipe_llm_prompt.py
+++ b/pipelex/pipe_operators/pipe_llm_prompt.py
@@ -171,9 +171,15 @@ class PipeLLMPrompt(PipeOperator):
# Append output structure prompt if needed
if pipe_run_params.dynamic_output_concept_code:
- user_text += PipeLLMPrompt.get_output_structure_prompt(output_concept=pipe_run_params.dynamic_output_concept_code)
+ user_text += PipeLLMPrompt.get_output_structure_prompt(
+ output_concept=pipe_run_params.dynamic_output_concept_code,
+ is_with_preliminary_text=pipe_run_params.is_with_preliminary_text or False,
+ )
else:
- user_text += PipeLLMPrompt.get_output_structure_prompt(output_concept=self.output_concept_code)
+ user_text += PipeLLMPrompt.get_output_structure_prompt(
+ output_concept=self.output_concept_code,
+ is_with_preliminary_text=pipe_run_params.is_with_preliminary_text or False,
+ )
log.verbose(f"User text with {self.output_concept_code=}:\n {user_text}")
@@ -231,7 +237,7 @@ class PipeLLMPrompt(PipeOperator):
)
@staticmethod
- def get_output_structure_prompt(output_concept: str) -> str:
+ def get_output_structure_prompt(output_concept: str, is_with_preliminary_text: bool) -> str:
class_name = Concept.extract_concept_name_from_str(concept_str=output_concept)
output_class = get_class_registry().get_class(class_name)
if not output_class:
@@ -242,14 +248,26 @@ class PipeLLMPrompt(PipeOperator):
if not class_structure:
return ""
- output_structure_prompt = (
- f"\n\n---\nRequested output format: The output should be the following class: {class_name}\n"
- f"{chr(10).join(class_structure)}\n"
- "You do NOT need to output a formatted JSON object, another LLM will take care of that. "
- "If you cannot find a value that is Optional, output None for that field."
- "However, you MUST clearly output the values for each of these fields in your response.\n---\n"
- "DO NOT create information. If the information is not present, output None."
- )
+ class_structure_str = "\n".join(class_structure)
+
+ # TODO: use proper prompt templating for this
+ if is_with_preliminary_text:
+ output_structure_prompt = (
+ f"\n\n---\nRequested output format: The requested output will be used to define the following class: {class_name}\n"
+ f"{class_structure_str}\n"
+ "You do NOT need to output a formatted JSON object, another LLM will take care of that. "
+ "If you cannot find a value that is Optional, output None for that field. "
+ "However, you MUST clearly output the values for each of these fields in your response.\n---\n"
+ "DO NOT create information. If the information is not present, output None."
+ )
+ else:
+ output_structure_prompt = (
+ f"\n\n---\nRequested output format: The output must conform to the following BaseModel: {class_name}\n"
+ f"{class_structure_str}\n"
+ "If you cannot find a value that is Optional, output None for that field. "
+ "However, you MUST clearly output the values for each of these fields in your response.\n---\n"
+ "DO NOT create information. If the information is not present, output None."
+ )
return output_structure_prompt
async def _unravel_text(
diff --git a/pipelex/pipe_works/pipe_dry.py b/pipelex/pipe_works/pipe_dry.py
index 65a5537aa..01be2580a 100644
--- a/pipelex/pipe_works/pipe_dry.py
+++ b/pipelex/pipe_works/pipe_dry.py
@@ -44,6 +44,15 @@ async def dry_run_single_pipe(pipe_code: str) -> str:
return f"FAILED: {str(exc)}"
+async def dry_run_pipe_codes(pipe_codes: List[str]) -> Dict[str, str]:
+ """
+ Dry run a list of pipe codes.
+ """
+ pipe_provider = get_pipe_provider()
+ pipes = [pipe_provider.get_required_pipe(pipe_code=pipe_code) for pipe_code in pipe_codes]
+ return await dry_run_pipes(pipes=pipes)
+
+
async def dry_run_pipes(pipes: List[PipeAbstract]) -> Dict[str, str]:
"""
Dry run all pipes in the library using ThreadPoolExecutor for true parallelism.
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index f97129248..068171963 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -239,7 +239,10 @@ class Pipelex(metaclass=MetaSingleton):
# TODO: add kwargs to make() so that subclasses can employ specific parameters
@classmethod
def make(
- cls, relative_config_folder_path: Optional[str] = None, absolute_config_folder_path: Optional[str] = None, from_file: Optional[bool] = True
+ cls,
+ relative_config_folder_path: Optional[str] = None,
+ absolute_config_folder_path: Optional[str] = None,
+ from_file: Optional[bool] = True,
) -> Self:
"""Create and initialize a Pipelex instance.
diff --git a/pipelex/tools/misc/json_utils.py b/pipelex/tools/misc/json_utils.py
index 630937d93..a1c528128 100644
--- a/pipelex/tools/misc/json_utils.py
+++ b/pipelex/tools/misc/json_utils.py
@@ -1,5 +1,5 @@
import json
-from typing import Any, Dict, List, Optional, Tuple, Union, cast
+from typing import Any, Dict, List, Mapping, Optional, Tuple, Union, cast
from kajson import kajson
from pydantic import BaseModel
@@ -8,7 +8,7 @@ from pipelex.tools.exceptions import ToolException
from pipelex.tools.misc.file_utils import save_text_to_path
from pipelex.tools.typing.pydantic_utils import CustomBaseModel
-JsonContent = Union[Dict[Any, Any], List[Any]]
+JsonContent = Union[Dict[str, Any], List[Any]]
class ArgumentTypeError(ToolException):
@@ -182,7 +182,7 @@ def deep_update(target_dict: Dict[str, Any], updates: Dict[str, Any]):
target_dict[key] = value
-def remove_none_values(json_content: JsonContent) -> JsonContent:
+def remove_none_values(json_content: JsonContent | Any) -> JsonContent | Any:
"""
Recursively removes all None values from a JSON-compatible data structure.
@@ -214,10 +214,25 @@ def remove_none_values(json_content: JsonContent) -> JsonContent:
}
"""
if isinstance(json_content, dict):
- json_content = cast(Dict[str, Any], json_content)
- return {key: remove_none_values(json_content=value) for key, value in json_content.items() if value is not None}
- else:
+ json_content = cast(Dict[str, Any], json_content) # pyright: ignore[reportUnnecessaryCast]
+ cleaned_dict: Dict[str, Any] = {}
+ for key, value in json_content.items():
+ if value is not None:
+ cleaned_dict[key] = remove_none_values(json_content=value)
+ return cleaned_dict
+ elif isinstance(json_content, list):
+ json_content = cast(List[Any], json_content) # pyright: ignore[reportUnnecessaryCast]
return [remove_none_values(item) for item in json_content]
+ else:
+ return json_content
+
+
+def remove_none_values_from_dict(data: Mapping[str, Any]) -> Dict[str, Any]:
+ processed = remove_none_values(json_content=data)
+ if not isinstance(processed, dict):
+ raise JsonTypeError("Removing None values from a dict, we expected a dict in return")
+ processed = cast(Dict[str, Any], processed) # pyright: ignore[reportUnnecessaryCast]
+ return processed
def purify_json(
diff --git a/pipelex/tools/misc/toml_utils.py b/pipelex/tools/misc/toml_utils.py
index 2316100cf..4dd7d9a79 100644
--- a/pipelex/tools/misc/toml_utils.py
+++ b/pipelex/tools/misc/toml_utils.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
from typing import Any, Dict, List, Optional
import toml
@@ -53,7 +55,7 @@ def validate_toml_file(path: str) -> None:
_validate_toml_content(content, path)
-def _clean_trailing_whitespace(content: str) -> str:
+def clean_trailing_whitespace(content: str) -> str:
"""Clean trailing whitespace from TOML content.
This function:
@@ -98,7 +100,7 @@ def load_toml_from_path(path: str) -> Dict[str, Any]:
with open(path, "r", encoding="utf-8") as file:
content = file.read()
- cleaned_content = _clean_trailing_whitespace(content)
+ cleaned_content = clean_trailing_whitespace(content)
# If content changed, write it back
if content != cleaned_content:
@@ -120,29 +122,3 @@ def failable_load_toml_from_path(path: str) -> Optional[Dict[str, Any]]:
except toml.TomlDecodeError as exc:
print(f"Failed to parse TOML file '{path}': {exc}")
return None
-
-
-def save_toml_to_path(data: Dict[str, Any], path: str) -> None:
- """Save dictionary as TOML to file path.
-
- Args:
- data: Dictionary to save as TOML
- path: Path where to save the TOML file
- """
- with open(path, "w", encoding="utf-8") as file:
- toml_content = toml.dumps(data)
- cleaned_content = _clean_trailing_whitespace(toml_content)
- file.write(cleaned_content)
-
-
-def dict_to_toml_string(data: Dict[str, Any]) -> str:
- """Convert dictionary to TOML string format.
-
- Args:
- data: Dictionary to convert to TOML
-
- Returns:
- TOML formatted string
- """
- toml_content = toml.dumps(data)
- return _clean_trailing_whitespace(toml_content)
diff --git a/pyproject.toml b/pyproject.toml
index 2c7636cd0..8d2103775 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,6 +39,7 @@ dependencies = [
"rich>=13.8.1",
"shortuuid>=1.0.13",
"toml>=0.10.2",
+ "tomlkit>=0.13.2",
"typer>=0.16.0",
"typing-extensions>=4.13.2",
"yattag>=1.15.2",
diff --git a/tests/e2e/pipelex/tools/migrate/__init__.py b/tests/e2e/pipelex/migration/__init__.py
similarity index 100%
rename from tests/e2e/pipelex/tools/migrate/__init__.py
rename to tests/e2e/pipelex/migration/__init__.py
diff --git a/tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.py b/tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.py
similarity index 99%
rename from tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.py
rename to tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.py
index a896b429a..635c55de3 100644
--- a/tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.py
+++ b/tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.py
@@ -5,7 +5,7 @@ from typing import Any, List, Tuple
import pytest
-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import migrate_concept_syntax
+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import migrate_concept_syntax
class TestMigrationE2E:
@@ -336,7 +336,7 @@ Concept = "Test concept"
import shutil
# Copy the test data file to our temp directory
- test_data_path = Path(__file__).parent.parent.parent.parent.parent / "data" / "test_migrate_v0_1_0_to_v0_2_0.toml"
+ test_data_path = Path(__file__).parent.parent.parent.parent / "data" / "test_migrate_v0_1_0_to_v0_2_0.toml"
test_file = tmp_path / "test_migrate.toml"
shutil.copy2(test_data_path, test_file)
diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py
new file mode 100644
index 000000000..1e1805bba
--- /dev/null
+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py
@@ -0,0 +1,46 @@
+import pytest
+
+from pipelex import pretty_print
+from pipelex.core.pipe_run_params import PipeRunMode
+from pipelex.create.helpers import get_support_file
+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint
+from pipelex.pipeline.execute import execute_pipeline
+
+
[email protected](loop_scope="class")
+ "requirements",
+ [
+ (
+ "Create a domain 'sample' to describe simple demo pipelines. "
+ "Define concept ThingDescription = 'A short natural language description of a thing'. "
+ "Add a pipe write_thing that takes input { text = Text } and outputs ThingDescription using PipeLLM."
+ ),
+ (
+ "Domain: gadgets. Concept: GadgetSpec = 'Technical description of a gadget'. "
+ "Add pipe summarize_gadget: inputs { text = Text } -> output GadgetSpec via PipeLLM."
+ ),
+ ],
+)
+async def test_generate_pipeline_blueprint(pipe_run_mode: PipeRunMode, requirements: str):
+ pipe_output = await execute_pipeline(
+ pipe_code="build_blueprint",
+ input_memory={
+ "domain": "test_domain",
+ "pipeline_name": "test_pipeline",
+ "requirements": requirements,
+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),
+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),
+ },
+ pipe_run_mode=pipe_run_mode,
+ )
+
+ blueprint = pipe_output.main_stuff_as(content_type=PipelineBlueprint)
+ assert isinstance(blueprint, PipelineBlueprint)
+ # Basic sanity checks
+ assert isinstance(blueprint.domain, str) and blueprint.domain != ""
+
+ pretty_print(blueprint, title="PipelineBlueprint")
diff --git a/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py b/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py
index be5872265..7aa68df24 100644
--- a/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py
+++ b/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py
@@ -49,7 +49,9 @@ class TestPipeLLMPrompt:
assert pipe_output.main_stuff is not None
assert pipe_output.llm_prompt is not None
assert pipe_output.llm_prompt.user_text is not None
- assert pipe_output.llm_prompt.user_text.endswith(PipeLLMPrompt.get_output_structure_prompt(NativeConcept.TEXT.code))
+ assert pipe_output.llm_prompt.user_text.endswith(
+ PipeLLMPrompt.get_output_structure_prompt(NativeConcept.TEXT.code, is_with_preliminary_text=False)
+ )
async def test_prompt_with_images(self, pipe_run_mode: PipeRunMode):
"""Test prompt with image inputs."""
@@ -109,7 +111,7 @@ class TestPipeLLMPrompt:
assert pipe_output.main_stuff is not None
# Verify output structure is appended
- output_structure = pipe_llm_prompt.get_output_structure_prompt(concept_code)
+ output_structure = pipe_llm_prompt.get_output_structure_prompt(concept_code, is_with_preliminary_text=False)
assert pipe_output.llm_prompt is not None
assert pipe_output.llm_prompt.user_text is not None
diff --git a/tests/test_pipelines/discord_newsletter.toml b/tests/test_pipelines/discord_newsletter.toml
index 600f6c780..eb84f8682 100644
--- a/tests/test_pipelines/discord_newsletter.toml
+++ b/tests/test_pipelines/discord_newsletter.toml
@@ -7,7 +7,8 @@ ChannelSummary = "A summarized Discord channel for newsletter inclusion"
Newsletter = "The final newsletter content with organized channel summaries"
[pipe.write_discord_newsletter]
-PipeSequence = "Create a newsletter from Discord articles by summarizing channels and organizing content"
+type = "PipeSequence"
+definition = "Create a newsletter from Discord articles by summarizing channels and organizing content"
inputs = { discord_channel_updates = "DiscordChannelUpdate" }
output = "Newsletter"
steps = [
@@ -15,7 +16,8 @@ steps = [
]
[pipe.summarize_discord_channel_update]
-PipeLLM = "Summarize a Discord channel's messages into newsletter-friendly content"
+type = "PipeLLM"
+definition = "Summarize a Discord channel's messages into newsletter-friendly content"
inputs = { discord_channel_update = "DiscordChannelUpdate" }
output = "ChannelSummary"
system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."
diff --git a/tests/test_pipelines/failing_pipelines.toml b/tests/test_pipelines/failing_pipelines.toml
index ac775a9ff..64323fac3 100644
--- a/tests/test_pipelines/failing_pipelines.toml
+++ b/tests/test_pipelines/failing_pipelines.toml
@@ -4,7 +4,8 @@ domain = "failing_pipelines"
[pipe]
[pipe.dummy]
-PipeLLM = "This pipe is a dummy pipe"
+type = "PipeLLM"
+definition = "This pipe is a dummy pipe"
output = "Text"
llm = { llm_handle = "gpt-4o-mini", temperature = 1, max_tokens = 50 }
prompt_template = """
@@ -12,7 +13,8 @@ This is a dummy prompt, do whatever you want. Something funny but not a joke, ok
"""
[pipe.infinite_loop_1]
-PipeSequence = "This pipe will cause an infinite loop"
+type = "PipeSequence"
+definition = "This pipe will cause an infinite loop"
output = "Text"
steps = [
{ pipe = "dummy", result = "dummy_result" },
diff --git a/tests/test_pipelines/misc_tests/flows.toml b/tests/test_pipelines/misc_tests/flows.toml
index dfb49b264..86674a112 100644
--- a/tests/test_pipelines/misc_tests/flows.toml
+++ b/tests/test_pipelines/misc_tests/flows.toml
@@ -8,7 +8,8 @@ Color = "A color"
[pipe]
[pipe.choose_colors]
-PipeLLM = "Choose Colors"
+type = "PipeLLM"
+definition = "Choose Colors"
output = "Color"
prompt_template = """
Choose {{ _nb_output }} colors.
@@ -16,7 +17,8 @@ Choose {{ _nb_output }} colors.
llm = "llm_for_creative_writing"
[pipe.sequence_for_batch_test]
-PipeSequence = "Sequence for parallel test"
+type = "PipeSequence"
+definition = "Sequence for parallel test"
inputs = { color = "Color" }
output = "Color"
steps = [
@@ -25,7 +27,8 @@ steps = [
]
[pipe.capitalize_color]
-PipeLLM = "Capitalize Colors"
+type = "PipeLLM"
+definition = "Capitalize Colors"
inputs = { color = "Color" }
output = "Color"
prompt_template = """
@@ -38,7 +41,8 @@ Output only the word, nothing else.
"""
[pipe.capitalize_last_letter]
-PipeLLM = "Capitalize Last Letter"
+type = "PipeLLM"
+definition = "Capitalize Last Letter"
inputs = { capitalized_color = "Color" }
output = "Color"
prompt_template = """
diff --git a/tests/test_pipelines/misc_tests/multiplicity.toml b/tests/test_pipelines/misc_tests/multiplicity.toml
index 91c182b2b..1f8957e2a 100644
--- a/tests/test_pipelines/misc_tests/multiplicity.toml
+++ b/tests/test_pipelines/misc_tests/multiplicity.toml
@@ -9,7 +9,8 @@ ProductOfNature = "Something produced by Nature"
FantasyScene = "A fantasy scene"
[pipe.original_power_ranger_colors]
-PipeLLM = "Recall the colors of the original power rangers"
+type = "PipeLLM"
+definition = "Recall the colors of the original power rangers"
output = "Color"
prompt_template = """
Remind me of the colors of the {{ _nb_output }} original power rangers.
@@ -18,7 +19,8 @@ llm = "llm_for_creative_writing"
nb_output = 5
[pipe.alltime_power_ranger_colors]
-PipeLLM = "Recall the colors of all the power rangers"
+type = "PipeLLM"
+definition = "Recall the colors of all the power rangers"
output = "Color"
prompt_template = """
Remind me of the colors of all the power rangers over all series.
@@ -27,7 +29,8 @@ Don't list the names of the characters or the name of the series: just list the
llm = "llm_for_creative_writing"
[pipe.imagine_nature_product]
-PipeLLM = "Imagine a product of nature"
+type = "PipeLLM"
+definition = "Imagine a product of nature"
inputs = { color = "Color" }
output = "ProductOfNature"
prompt_template = """
@@ -36,7 +39,8 @@ Just state what it is in a single sentence.
"""
[pipe.imagine_fantasy_scene_including_products_of_nature]
-PipeLLM = "Imagine a fantasy scene including products of nature"
+type = "PipeLLM"
+definition = "Imagine a fantasy scene including products of nature"
inputs = { product_of_nature = "ProductOfNature" }
output = "FantasyScene"
prompt_template = """
@@ -48,7 +52,8 @@ Keep it short and concise, just one sentence.
"""
[pipe.imagine_nature_scene_of_original_power_rangers_colors]
-PipeSequence = "Imagine nature scenes of Power Rangers colors"
+type = "PipeSequence"
+definition = "Imagine nature scenes of Power Rangers colors"
output = "Image"
steps = [
{ pipe = "original_power_ranger_colors", result = "color" },
@@ -58,7 +63,8 @@ steps = [
[pipe.imagine_nature_scene_of_alltime_power_rangers_colors]
-PipeSequence = "Imagine nature scenes of Power Rangers colors"
+type = "PipeSequence"
+definition = "Imagine nature scenes of Power Rangers colors"
output = "Image"
steps = [
{ pipe = "alltime_power_ranger_colors", result = "color", multiple_output = true },
diff --git a/tests/test_pipelines/misc_tests/subfolder_1/cars.toml b/tests/test_pipelines/misc_tests/subfolder_1/cars.toml
index e07230d48..47c961a64 100644
--- a/tests/test_pipelines/misc_tests/subfolder_1/cars.toml
+++ b/tests/test_pipelines/misc_tests/subfolder_1/cars.toml
@@ -10,7 +10,8 @@ CarDescription = "A detailed description of a car model"
[pipe]
[pipe.generate_car_description]
-PipeLLM = "Generate a description of a car"
+type = "PipeLLM"
+definition = "Generate a description of a car"
inputs = { car = "Car" }
output = "CarDescription"
prompt_template = """
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/animals.toml b/tests/test_pipelines/misc_tests/subfolder_2/animals.toml
index 02dc21291..f35a6c846 100644
--- a/tests/test_pipelines/misc_tests/subfolder_2/animals.toml
+++ b/tests/test_pipelines/misc_tests/subfolder_2/animals.toml
@@ -10,7 +10,8 @@ AnimalDescription = "A detailed description of an animal"
[pipe]
[pipe.generate_animal_description]
-PipeLLM = "Generate a description of an animal"
+type = "PipeLLM"
+definition = "Generate a description of an animal"
inputs = { animal = "Animal" }
output = "AnimalDescription"
prompt_template = """
diff --git a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml b/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml
index dfe99d7ea..2a879f1d9 100644
--- a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml
+++ b/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml
@@ -8,7 +8,8 @@ FlowerDescription = "A detailed description of a flower"
[pipe]
[pipe.generate_flower_description]
-PipeLLM = "Generate a description of a flower"
+type = "PipeLLM"
+definition = "Generate a description of a flower"
inputs = { flower = "Flower" }
output = "FlowerDescription"
prompt_template = """
diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.toml b/tests/test_pipelines/misc_tests/test_image_inputs.toml
index 0acd25588..329418215 100644
--- a/tests/test_pipelines/misc_tests/test_image_inputs.toml
+++ b/tests/test_pipelines/misc_tests/test_image_inputs.toml
@@ -6,7 +6,8 @@ Article = "Article"
[pipe]
[pipe.extract_article_from_image]
-PipeLLM = "Describe a page content"
+type = "PipeLLM"
+definition = "Describe a page content"
inputs = { image = "Image" }
output = "Article"
system_prompt = """
@@ -17,7 +18,8 @@ Extract the text and the date of the article in the image.
"""
[pipe.describe_page]
-PipeLLM = "Describe a page"
+type = "PipeLLM"
+definition = "Describe a page"
inputs = { "page.page_view" = "Image", page = "Page" }
output = "Article"
system_prompt = """
diff --git a/tests/test_pipelines/misc_tests/test_jinja2.toml b/tests/test_pipelines/misc_tests/test_jinja2.toml
index 664922f79..9a0d44398 100644
--- a/tests/test_pipelines/misc_tests/test_jinja2.toml
+++ b/tests/test_pipelines/misc_tests/test_jinja2.toml
@@ -2,7 +2,8 @@ domain = "test_jinja2"
definition = "This library is intended for testing Jinja2"
[pipe.jinja2_test_1]
-PipeJinja2 = "Jinja2 test 1"
+type = "PipeJinja2"
+definition = "Jinja2 test 1"
inputs = { text = "Text" }
output = "Text"
jinja2 = """
diff --git a/tests/test_pipelines/misc_tests/tests.toml b/tests/test_pipelines/misc_tests/tests.toml
index 378121f5a..d5e91c743 100644
--- a/tests/test_pipelines/misc_tests/tests.toml
+++ b/tests/test_pipelines/misc_tests/tests.toml
@@ -9,7 +9,8 @@ ArticleAndCritic = "An article and a critique of it"
Complex = "A complex object"
[pipe.simple_llm_test_from_image]
-PipeLLM = "Simple LLM test from image"
+type = "PipeLLM"
+definition = "Simple LLM test from image"
inputs = { image = "Image" }
output = "Text"
prompt_template = """
@@ -17,7 +18,8 @@ Describe the using rap lyrics, including puns and references to the image.
"""
[pipe.test_no_input]
-PipeLLM = "No Input"
+type = "PipeLLM"
+definition = "No Input"
output = "Text"
llm = "llm_for_testing_gen_text"
prompt_template = """
@@ -25,7 +27,8 @@ Explain that this is a test prompt which took no input from the user.
"""
[pipe.test_no_input_that_could_be_long]
-PipeLLM = "No Input but generates a text that could be long"
+type = "PipeLLM"
+definition = "No Input but generates a text that could be long"
output = "Text"
llm = { llm_handle = "gpt-4o-mini", temperature = 0.5, max_tokens = 100 }
prompt_template = """
diff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml
index 166ed5844..b13d3ad15 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml
@@ -7,7 +7,8 @@ TextList = "List of Text"
[pipe]
[pipe.test_pipe_batch]
-PipeBatch = "Test pipe batch"
+type = "PipeBatch"
+definition = "Test pipe batch"
inputs = { text_item = "Text", text_list = "TextList" }
output = "UppercaseText"
branch_pipe_code = "uppercase_transformer"
@@ -15,7 +16,8 @@ input_list_name = "text_list"
input_item_name = "text_item"
[pipe.uppercase_transformer]
-PipeLLM = "Transform text to uppercase"
+type = "PipeLLM"
+definition = "Transform text to uppercase"
inputs = { text_item = "Text" }
output = "UppercaseText"
prompt_template = """
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml
index bef33ccdc..531fafd66 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml
@@ -6,7 +6,8 @@ CategoryInput = "Input with a category field"
[pipe]
[pipe.basic_condition_by_category]
-PipeCondition = "Route based on category field"
+type = "PipeCondition"
+definition = "Route based on category field"
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression_template = "{{ input_data.category }}"
@@ -17,21 +18,24 @@ medium = "process_medium"
large = "process_large"
[pipe.process_small]
-PipeLLM = "Generate random text for small items"
+type = "PipeLLM"
+definition = "Generate random text for small items"
output = "native.Text"
prompt_template = """
Output this only: "small"
"""
[pipe.process_medium]
-PipeLLM = "Generate random text for medium items"
+type = "PipeLLM"
+definition = "Generate random text for medium items"
output = "native.Text"
prompt_template = """
Output this only: "medium"
"""
[pipe.process_large]
-PipeLLM = "Generate random text for large items"
+type = "PipeLLM"
+definition = "Generate random text for large items"
output = "native.Text"
prompt_template = """
Output this only: "large"
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml
index 0b4050843..a9c21cd11 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml
@@ -6,7 +6,8 @@ CategoryInput = "Input with a category field"
[pipe]
[pipe.basic_condition_by_category_2]
-PipeCondition = "Route based on category field using expression"
+type = "PipeCondition"
+definition = "Route based on category field using expression"
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression = "input_data.category"
@@ -17,21 +18,24 @@ medium = "process_medium_2"
large = "process_large_2"
[pipe.process_small_2]
-PipeLLM = "Generate random text for small items"
+type = "PipeLLM"
+definition = "Generate random text for small items"
output = "native.Text"
prompt_template = """
Output this only: "small"
"""
[pipe.process_medium_2]
-PipeLLM = "Generate random text for medium items"
+type = "PipeLLM"
+definition = "Generate random text for medium items"
output = "native.Text"
prompt_template = """
Output this only: "medium"
"""
[pipe.process_large_2]
-PipeLLM = "Generate random text for large items"
+type = "PipeLLM"
+definition = "Generate random text for large items"
output = "native.Text"
prompt_template = """
Output this only: "large"
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml
index 9ea535763..54c1ea33d 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml
@@ -9,7 +9,8 @@ ProcessingContext = "Combined processing context"
[pipe]
# Main entry point - routes by document type first
[pipe.complex_document_processor]
-PipeCondition = "Primary routing by document type"
+type = "PipeCondition"
+definition = "Primary routing by document type"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
expression_template = "{{ doc_request.document_type }}"
@@ -21,7 +22,8 @@ legal = "legal_document_router"
# Technical document processing branch
[pipe.technical_document_router]
-PipeCondition = "Route technical documents by priority and user level"
+type = "PipeCondition"
+definition = "Route technical documents by priority and user level"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
expression_template = "{% if doc_request.priority == 'urgent' %}urgent_tech{% elif user_profile.user_level == 'expert' and doc_request.complexity == 'high' %}expert_tech{% else %}standard_tech{% endif %}"
@@ -33,7 +35,8 @@ standard_tech = "standard_technical_processor"
# Business document processing branch
[pipe.business_document_router]
-PipeCondition = "Route business documents by department and priority"
+type = "PipeCondition"
+definition = "Route business documents by department and priority"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
expression_template = "{% if doc_request.priority == 'urgent' %}urgent_business{% elif user_profile.department == 'finance' %}finance_business{% elif user_profile.department == 'marketing' %}marketing_business{% else %}general_business{% endif %}"
@@ -46,7 +49,8 @@ general_business = "general_business_processor"
# Legal document processing branch
[pipe.legal_document_router]
-PipeCondition = "Route legal documents by complexity and user level"
+type = "PipeCondition"
+definition = "Route legal documents by complexity and user level"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
expression_template = "{% if doc_request.complexity == 'high' and user_profile.user_level != 'beginner' %}complex_legal{% elif doc_request.language != 'english' %}international_legal{% else %}standard_legal{% endif %}"
@@ -58,7 +62,8 @@ standard_legal = "standard_legal_processor"
# Leaf processors - Technical
[pipe.urgent_technical_processor]
-PipeLLM = "Process urgent technical documents with high priority"
+type = "PipeLLM"
+definition = "Process urgent technical documents with high priority"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -73,7 +78,8 @@ Output: "URGENT_TECHNICAL_PROCESSED"
"""
[pipe.expert_technical_processor]
-PipeLLM = "Process complex technical documents for expert users"
+type = "PipeLLM"
+definition = "Process complex technical documents for expert users"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -86,7 +92,8 @@ Output: "EXPERT_TECHNICAL_PROCESSED"
"""
[pipe.standard_technical_processor]
-PipeLLM = "Process standard technical documents"
+type = "PipeLLM"
+definition = "Process standard technical documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -99,7 +106,8 @@ Output: "STANDARD_TECHNICAL_PROCESSED"
# Leaf processors - Business
[pipe.urgent_business_processor]
-PipeLLM = "Process urgent business documents"
+type = "PipeLLM"
+definition = "Process urgent business documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -111,7 +119,8 @@ Output: "URGENT_BUSINESS_PROCESSED"
"""
[pipe.finance_business_processor]
-PipeLLM = "Process finance business documents"
+type = "PipeLLM"
+definition = "Process finance business documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -123,7 +132,8 @@ Output: "FINANCE_BUSINESS_PROCESSED"
"""
[pipe.marketing_business_processor]
-PipeLLM = "Process marketing business documents"
+type = "PipeLLM"
+definition = "Process marketing business documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -135,7 +145,8 @@ Output: "MARKETING_BUSINESS_PROCESSED"
"""
[pipe.general_business_processor]
-PipeLLM = "Process general business documents"
+type = "PipeLLM"
+definition = "Process general business documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -148,7 +159,8 @@ Output: "GENERAL_BUSINESS_PROCESSED"
# Leaf processors - Legal
[pipe.complex_legal_processor]
-PipeLLM = "Process complex legal documents"
+type = "PipeLLM"
+definition = "Process complex legal documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
@@ -161,7 +173,8 @@ Output: "COMPLEX_LEGAL_PROCESSED"
"""
[pipe.international_legal_processor]
-PipeLLM = "Process international legal documents"
+type = "PipeLLM"
+definition = "Process international legal documents"
inputs = { doc_request = "DocumentRequest" }
output = "native.Text"
prompt_template = """
@@ -173,7 +186,8 @@ Output: "INTERNATIONAL_LEGAL_PROCESSED"
"""
[pipe.standard_legal_processor]
-PipeLLM = "Process standard legal documents"
+type = "PipeLLM"
+definition = "Process standard legal documents"
inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }
output = "native.Text"
prompt_template = """
diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml
index 46a1c41ab..6410f0f55 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml
@@ -4,7 +4,8 @@ definition = "Test pipes for PipeCondition based on text length"
[pipe]
[pipe.capitalize_long_text]
-PipeLLM = "Capitalize text for long inputs"
+type = "PipeLLM"
+definition = "Capitalize text for long inputs"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Transform the following LONG text to uppercase and add "LONG: " prefix:
@@ -14,7 +15,8 @@ prompt_template = """Transform the following LONG text to uppercase and add "LON
Return only the prefixed uppercase text, nothing else."""
[pipe.add_prefix_short_text]
-PipeLLM = "Add prefix for short text inputs"
+type = "PipeLLM"
+definition = "Add prefix for short text inputs"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Add the prefix "SHORT: " to the beginning of the following text:
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml b/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml
index a898a83c3..6c359aab5 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml
@@ -4,7 +4,8 @@ definition = "Test pipes for PipeParallel text analysis"
[pipe]
[pipe.analyze_sentiment]
-PipeLLM = "Analyze sentiment of text"
+type = "PipeLLM"
+definition = "Analyze sentiment of text"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Analyze the sentiment of the following text and return only one word: positive, negative, or neutral.
@@ -14,7 +15,8 @@ prompt_template = """Analyze the sentiment of the following text and return only
Sentiment:"""
[pipe.count_words]
-PipeLLM = "Count words in text"
+type = "PipeLLM"
+definition = "Count words in text"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Count the number of words in the following text and return only the number:
@@ -24,7 +26,8 @@ prompt_template = """Count the number of words in the following text and return
Word count:"""
[pipe.extract_keywords]
-PipeLLM = "Extract keywords from text"
+type = "PipeLLM"
+definition = "Extract keywords from text"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Extract the top 3 keywords from the following text. Return them as a comma-separated list:
diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml b/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml
index 7fb132521..78c8d8619 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml
@@ -9,7 +9,8 @@ CombinedAnalysis = "Combined analysis results from parallel processing"
[pipe]
[pipe.parallel_document_analysis]
-PipeParallel = "Analyze document length and content in parallel"
+type = "PipeParallel"
+definition = "Analyze document length and content in parallel"
inputs = { document = "DocumentInput" }
output = "CombinedAnalysis"
add_each_output = true
@@ -20,7 +21,8 @@ parallels = [
]
[pipe.analyze_length]
-PipeLLM = "Analyze document length and structure"
+type = "PipeLLM"
+definition = "Analyze document length and structure"
inputs = { document = "DocumentInput" }
output = "LengthAnalysis"
llm = "llm_for_testing_gen_text"
@@ -36,7 +38,8 @@ Provide a brief analysis focusing on:
"""
[pipe.analyze_content]
-PipeLLM = "Analyze document content and themes"
+type = "PipeLLM"
+definition = "Analyze document content and themes"
inputs = { document = "DocumentInput" }
output = "ContentAnalysis"
llm = "llm_for_testing_gen_text"
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml
index c4d279e63..87694b705 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml
@@ -3,7 +3,8 @@ definition = "Test pipe for capitalizing text to uppercase"
[pipe]
[pipe.capitalize_text]
-PipeLLM = "Transform text to uppercase"
+type = "PipeLLM"
+definition = "Transform text to uppercase"
inputs = { input_text = "Text" }
output = "Text"
prompt_template = """Transform the following text to uppercase:
@@ -13,7 +14,8 @@ prompt_template = """Transform the following text to uppercase:
Return only the uppercase text, nothing else."""
[pipe.add_prefix]
-PipeLLM = "Add PROCESSED prefix to text"
+type = "PipeLLM"
+definition = "Add PROCESSED prefix to text"
inputs = { capitalized_text = "Text" }
output = "Text"
prompt_template = """Add the prefix "PROCESSED: " to the beginning of the following text:
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml
index 383388cf5..5a8fdb1d3 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml
@@ -9,7 +9,8 @@ SummaryText = "A summary of the processed text"
[pipe]
[pipe.simple_text_sequence]
-PipeSequence = "Process text through cleaning and summarization"
+type = "PipeSequence"
+definition = "Process text through cleaning and summarization"
inputs = { raw_text = "RawText" }
output = "SummaryText"
steps = [
@@ -18,7 +19,8 @@ steps = [
]
[pipe.clean_text]
-PipeLLM = "Clean and normalize the input text"
+type = "PipeLLM"
+definition = "Clean and normalize the input text"
inputs = { raw_text = "RawText" }
output = "CleanedText"
llm = "llm_for_testing_gen_text"
@@ -32,7 +34,8 @@ Clean and normalize the following text by:
"""
[pipe.generate_summary]
-PipeLLM = "Generate a summary of the cleaned text"
+type = "PipeLLM"
+definition = "Generate a summary of the cleaned text"
inputs = { cleaned_text = "CleanedText" }
output = "SummaryText"
llm = "llm_for_testing_gen_text"
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml
index b8613da31..689564787 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml
@@ -10,7 +10,8 @@ Document = "A document containing multiple customer reviews"
[pipe]
[pipe.analyze_reviews_sequence]
-PipeSequence = "Process customer reviews with sentiment analysis"
+type = "PipeSequence"
+definition = "Process customer reviews with sentiment analysis"
inputs = { document = "Document" }
output = "ProductRating"
steps = [
@@ -20,7 +21,8 @@ steps = [
]
[pipe.extract_individual_reviews]
-PipeLLM = "Extract individual reviews from document"
+type = "PipeLLM"
+definition = "Extract individual reviews from document"
inputs = { document = "Document" }
output = "CustomerReview"
nb_output = 1
@@ -32,7 +34,8 @@ Extract each individual customer review from this document as separate items:
"""
[pipe.analyze_review_sentiment]
-PipeLLM = "Analyze sentiment of a single review"
+type = "PipeLLM"
+definition = "Analyze sentiment of a single review"
inputs = { single_review = "CustomerReview" }
output = "SentimentAnalysis"
llm = "llm_for_testing_gen_text"
@@ -45,7 +48,8 @@ Provide sentiment (positive/negative/neutral) and confidence score.
"""
[pipe.aggregate_review_results]
-PipeLLM = "Aggregate all review analyses into final rating"
+type = "PipeLLM"
+definition = "Aggregate all review analyses into final rating"
inputs = { sentiment_analyses = "SentimentAnalysis" }
output = "ProductRating"
llm = "llm_for_testing_gen_text"
diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml
index bd9f2220e..ee60d0f72 100644
--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml
+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml
@@ -11,7 +11,8 @@ BestIdea = "The top-ranked creative idea with justification"
[pipe]
[pipe.creative_ideation_sequence]
-PipeSequence = "Generate multiple ideas, analyze each individually, then select the best"
+type = "PipeSequence"
+definition = "Generate multiple ideas, analyze each individually, then select the best"
inputs = { topic = "CreativeTopic" }
output = "BestIdea"
steps = [
@@ -22,7 +23,8 @@ steps = [
]
[pipe.generate_multiple_ideas]
-PipeLLM = "Generate exactly 5 creative ideas for a topic"
+type = "PipeLLM"
+definition = "Generate exactly 5 creative ideas for a topic"
inputs = { topic = "CreativeTopic" }
output = "CreativeIdea"
nb_output = 5
@@ -41,7 +43,8 @@ Ideas:
"""
[pipe.brainstorm_concepts]
-PipeLLM = "Brainstorm variable number of concepts"
+type = "PipeLLM"
+definition = "Brainstorm variable number of concepts"
inputs = { topic = "CreativeTopic" }
output = "CreativeIdea"
multiple_output = true
@@ -55,7 +58,8 @@ Each concept should be briefly described.
"""
[pipe.analyze_single_idea]
-PipeLLM = "Analyze a single idea in detail"
+type = "PipeLLM"
+definition = "Analyze a single idea in detail"
inputs = { single_idea = "CreativeIdea" }
output = "IdeaAnalysis"
llm = "llm_for_testing_gen_text"
@@ -75,7 +79,8 @@ Provide a comprehensive assessment.
"""
[pipe.evaluate_all_ideas]
-PipeLLM = "Evaluate and rank all analyzed ideas"
+type = "PipeLLM"
+definition = "Evaluate and rank all analyzed ideas"
inputs = { detailed_analyses = "IdeaAnalysis" }
output = "IdeaEvaluation"
llm = "llm_for_testing_gen_text"
@@ -91,7 +96,8 @@ Provide:
"""
[pipe.select_best_idea]
-PipeLLM = "Select the best idea based on evaluation"
+type = "PipeLLM"
+definition = "Select the best idea based on evaluation"
inputs = { evaluation = "IdeaEvaluation" }
output = "BestIdea"
llm = "llm_for_testing_gen_text"
diff --git a/tests/test_pipelines/tricky_questions.toml b/tests/test_pipelines/tricky_questions.toml
index f15dcd7e3..04e0648f1 100644
--- a/tests/test_pipelines/tricky_questions.toml
+++ b/tests/test_pipelines/tricky_questions.toml
@@ -14,7 +14,8 @@ refines = "AnswerToAQuestion"
[pipe]
[pipe.analyse_question_tricky]
-PipeLLM = "Analyze a question to determine whether it's straightforward or tricky"
+type = "PipeLLM"
+definition = "Analyze a question to determine whether it's straightforward or tricky"
inputs = { question = "Question" }
output = "QuestionAnalysis"
llm = "llm_to_reason"
@@ -29,7 +30,8 @@ If there's an obvious trap, state it without getting into details.
"""
[pipe.answer_after_analysis]
-PipeLLM = "Answer knowingly after analyzing a question"
+type = "PipeLLM"
+definition = "Answer knowingly after analyzing a question"
inputs = { question = "Question", question_analysis = "QuestionAnalysis" }
output = "ThoughtfulAnswer"
llm = "llm_to_reason"
@@ -47,14 +49,16 @@ Answer in 4 parts:
"""
[pipe.conclude_thoughtful_answer]
-PipeJinja2 = "Conclude a thoughtful answer"
+type = "PipeJinja2"
+definition = "Conclude a thoughtful answer"
inputs = { thoughtful_answer = "ThoughtfulAnswer" }
output = "ThoughtfulAnswerConclusion"
jinja2 = "After analyzing the question, here is my answer: $thoughtful_answer.the_answer"
[pipe.conclude_tricky_question_by_steps]
-PipeSequence = "Answer a tricky question by first analyzing its trickiness and then concluding"
+type = "PipeSequence"
+definition = "Answer a tricky question by first analyzing its trickiness and then concluding"
inputs = { question = "Question" }
output = "ThoughtfulAnswerConclusion"
steps = [
diff --git a/tests/unit/pipelex/tools/migrate/__init__.py b/tests/unit/pipelex/migration/__init__.py
similarity index 100%
rename from tests/unit/pipelex/tools/migrate/__init__.py
rename to tests/unit/pipelex/migration/__init__.py
diff --git a/tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.py b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
similarity index 99%
rename from tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.py
rename to tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
index ae61ecfa0..747ecbaa7 100644
--- a/tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.py
+++ b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py
@@ -2,7 +2,7 @@ from pathlib import Path
import pytest
-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import MigrationResult, TomlMigrator, migrate_concept_syntax
+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import MigrationResult, TomlMigrator, migrate_concept_syntax
class TestTomlMigrator:
diff --git a/uv.lock b/uv.lock
index 573525370..ae4ea3189 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1773,6 +1773,7 @@ dependencies = [
{ name = "rich" },
{ name = "shortuuid" },
{ name = "toml" },
+ { name = "tomlkit" },
{ name = "typer" },
{ name = "typing-extensions" },
{ name = "yattag" },
@@ -1874,6 +1875,7 @@ requires-dist = [
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6.8" },
{ name = "shortuuid", specifier = ">=1.0.13" },
{ name = "toml", specifier = ">=0.10.2" },
+ { name = "tomlkit", specifier = ">=0.13.2" },
{ name = "typer", specifier = ">=0.16.0" },
{ name = "types-aioboto3", extras = ["bedrock", "bedrock-runtime"], marker = "extra == 'dev'", specifier = ">=13.4.0" },
{ name = "types-aiofiles", marker = "extra == 'dev'", specifier = ">=24.1.0.20240626" },
@@ -2596,6 +2598,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/6e/c2/61d3e0f47e2b74ef40a68b9e6ad5984f6241a942f7cd3bbfbdbd03861ea9/tomli-2.2.1-py3-none-any.whl", hash = "sha256:cb55c73c5f4408779d0cf3eef9f762b9c9f147a77de7b258bef0a5628adc85cc", size = 14257, upload-time = "2024-11-27T22:38:35.385Z" },
]
+[[package]]
+name = "tomlkit"
+version = "0.13.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cc/18/0bbf3884e9eaa38819ebe46a7bd25dcd56b67434402b66a58c4b8e552575/tomlkit-0.13.3.tar.gz", hash = "sha256:430cf247ee57df2b94ee3fbe588e71d362a941ebb545dec29b53961d61add2a1", size = 185207, upload-time = "2025-06-05T07:13:44.947Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bd/75/8539d011f6be8e29f339c42e633aae3cb73bffa95dd0f9adec09b9c58e85/tomlkit-0.13.3-py3-none-any.whl", hash = "sha256:c89c649d79ee40629a9fda55f8ace8c6a1b42deb912b2a8fd8d942ddadb606b0", size = 38901, upload-time = "2025-06-05T07:13:43.546Z" },
+]
+
[[package]]
name = "tqdm"
version = "4.67.1"