Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Build the blueprint based on the draft

Louis Choquel1df6d07279ae
diff --git a/.pipelex/create/build_pipelines.md b/.pipelex/create/build_pipelines.md
new file mode 100644
index 000000000..deab04c86
--- /dev/null
+++ b/.pipelex/create/build_pipelines.md
@@ -0,0 +1,316 @@
+# Building Pipelex Pipelines from Drafts
+
+This guide explains how to translate a markdown pipeline draft into a formal `PipelineLibraryBlueprint` structure that can be validated and executed by Pipelex.
+
+## Overview
+
+1. Convert the draft's narrative flow into a formal domain/concepts/pipes structure
+2. Map draft notation to specific pipe types and configurations
+3. Ensure all working memory references are properly captured in operator pipe inputs (wrapping up input requirements for controller pipes will be done later)
+
+## Translation Workflow
+
+### Step 1: Extract Domain Information
+
+Create the blueprint root:
+```json
+{
+ "domain": "domain_name",
+ "definition": "What this pipeline does"
+}
+```
+
+### Step 2: Identify and Define Concepts
+
+Review the draft for all data types mentioned:
+
+- **Input variables** → Concepts needed
+- **Intermediate results** → Concepts needed
+- **Final output** → Concept needed
+
+#### Concept Rules
+
+1. **Use native concepts when possible**:
+ - `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`
+ - Don't redefine these
+
+2. **Use library concepts when applicable**:
+ - `documents.TextAndImagesContent`
+ - `images.VisualDescription`
+ - `images.ImgGenPrompt`
+ - `images.Photo`
+
+3. **Create domain concepts for specialized data**:
+ - Use PascalCase: `Analysis`, `TechnicalReport`, `UserQuery`
+ - Never use plurals: `Document` not `Documents`
+ - Never use circumstantial names: `Photo` not `InputPhoto`
+ - Avoid adjectives: `Text` not `LongText`
+
+#### Concept Definition Formats
+
+Quick form (for simple text-based concepts):
+```json
+"concept": {
+ "UserQuery": "Natural language question from a user",
+ "Summary": "Condensed version of content"
+}
+```
+
+Full form (when structure or refinement needed):
+```json
+"concept": {
+ "TechnicalAnalysis": {
+ "definition": "Technical analysis of a system",
+ "refines": "Text"
+ }
+}
+```
+
+### Step 3: Translate Flow Steps to Pipes
+
+Map each step from your draft to the appropriate pipe type and configuration.
+
+#### Operator Mappings
+
+| Draft Notation | Pipe Type | Key Configuration |
+|---------------|-----------|-------------------|
+| `LLM[purpose]: (inputs) → output` | `PipeLLM` | Set `prompt_template` or `prompt` |
+| `OCR: (document) → pages` | `PipeOcr` | Input must be named `ocr_input` |
+| `ImgGen: (prompt) → image` | `PipeImgGen` | Configure image generation settings |
+| `Func[name]: (data) → result` | `PipeFunc` | Reference Python function |
+
+#### Controller Mappings
+
+| Draft Pattern | Pipe Type | Structure |
+|--------------|-----------|-----------|
+| Sequential steps (Step 1, Step 2...) | `PipeSequence` | Use `steps` array |
+| Process each item in list | `PipeSequence` step with batch | Add `batch_over` and `batch_as` to step |
+| Parallel branches (Branch A, Branch B...) | `PipeParallel` | Use `parallels` array |
+| Conditional routing (Case "x"...) | `PipeCondition` | Use `expression` and `pipe_map` |
+
+### Step 4: Configure Each Pipe
+
+#### PipeLLM Configuration
+
+From draft step:
+```markdown
+### Step: Analyze document for key insights
+LLM[analyze]: (document, criteria) → analysis
+```
+
+To blueprint:
+```json
+"analyze_document": {
+ "type": "PipeLLM",
+ "definition": "Analyze document for key insights",
+ "inputs": {
+ "document": "Text",
+ "criteria": "Text"
+ },
+ "output": "Analysis",
+ "prompt_template": "Analyze the following document based on the provided criteria.\n\n@document\n\n@criteria"
+}
+```
+
+**Prompt Template Rules**:
+
+- Use `@variable` for block content (multi-line)
+- Use `$variable` for inline content (within sentences)
+
+#### PipeOcr Configuration
+
+From draft step:
+```markdown
+### Step: Extract content from PDF
+OCR: (document) → pages
+```
+
+To blueprint:
+```json
+"extract_pages": {
+ "type": "PipeOcr",
+ "definition": "Extract content from PDF",
+ "inputs": {
+ "ocr_input": "PDF" // Must be named ocr_input
+ },
+ "output": "Page", // Output is always Page concept
+ "page_images": true,
+ "page_views": true
+}
+```
+
+#### PipeSequence Configuration
+
+From draft sequence:
+```markdown
+Step 1: Extract information
+extract_info(document) → key_info
+
+Step 2: Analyze information
+analyze(key_info) → analysis
+```
+
+To blueprint:
+```json
+"process_document": {
+ "type": "PipeSequence",
+ "definition": "Extract and analyze document",
+ "inputs": {
+ "document": "Text"
+ },
+ "output": "Analysis",
+ "steps": [
+ {
+ "pipe": "extract_info",
+ "result": "key_info"
+ },
+ {
+ "pipe": "analyze_info",
+ "result": "analysis"
+ }
+ ]
+}
+```
+
+#### Batch Processing Configuration
+
+From draft batch:
+```markdown
+Process each item in documents:
+analyze_document(document) → analysis
+```
+
+To blueprint step within PipeSequence:
+```json
+{
+ "pipe": "analyze_single",
+ "batch_over": "documents",
+ "batch_as": "document",
+ "result": "analyses"
+}
+```
+Using the above, the required input used is `documents`, and by looping on the items of that list, each one will be named `document`, and that's how it will be received in the `analyze_single` pipe.
+
+#### PipeCondition Configuration
+
+From draft conditional:
+```markdown
+Route based on category:
+Case "technical": analyze_technical(document) → analysis
+Case "legal": analyze_legal(document) → analysis
+```
+
+To blueprint:
+```json
+"route_analysis": {
+ "type": "PipeCondition",
+ "definition": "Route document to appropriate analyzer",
+ "inputs": {
+ "document": "Text",
+ "category": "Text"
+ },
+ "output": "Analysis",
+ "expression": "category",
+ "pipe_map": {
+ "technical": "analyze_technical",
+ "legal": "analyze_legal"
+ }
+}
+```
+
+## Complete Example: Draft to Blueprint
+
+### Draft (Markdown)
+
+```markdown
+# Pipeline: document_insights
+Domain: analysis
+Purpose: Extract insights from documents based on criteria
+
+## Inputs
+- `document` : Document to analyze
+- `criteria` : Analysis criteria
+
+## Flow
+
+### Step 1: Extract key information
+LLM[extract]: (document) → key_points
+
+### Step 2: Analyze against criteria
+LLM[analyze]: (key_points, criteria) → insights
+
+## Output
+Returns: `insights`
+```
+
+### Blueprint (JSON)
+
+```json
+{
+ "domain": "analysis",
+ "definition": "Extract insights from documents based on criteria",
+ "concept": {
+ "KeyPoints": "Important information extracted from document",
+ "Insights": "Analysis results based on specific criteria"
+ },
+ "pipe": {
+ "document_insights": {
+ "type": "PipeSequence",
+ "definition": "Extract and analyze document insights",
+ "output": "Insights",
+ "steps": [
+ {
+ "pipe": "extract_key_points",
+ "result": "key_points"
+ },
+ {
+ "pipe": "analyze_points",
+ "result": "insights"
+ }
+ ]
+ },
+ "extract_key_points": {
+ "type": "PipeLLM",
+ "definition": "Extract key information",
+ "inputs": {
+ "document": "Text"
+ },
+ "output": "KeyPoints",
+ "prompt_template": "Extract the key points from this document.\n\n@document"
+ },
+ "analyze_points": {
+ "type": "PipeLLM",
+ "definition": "Analyze against criteria",
+ "inputs": {
+ "key_points": "KeyPoints",
+ "criteria": "Text"
+ },
+ "output": "Insights",
+ "prompt_template": "Analyze these key points against the provided criteria.\n\n@key_points\n\n@criteria"
+ }
+ }
+}
+```
+
+## Validation Checklist
+
+Before finalizing your blueprint:
+
+- [ ] Domain is defined with a single word or snake_case code
+- [ ] All concepts follow naming conventions (PascalCase, no plurals, no adjectives, no circumstantial names)
+- [ ] Native concepts are not redefined
+- [ ] Each pipe has `type`, `definition`, `inputs`, and `output`
+- [ ] Controller pipes don't need to declare inputs, they will be inferred from the blueprint
+- [ ] OCR pipes use `ocr_input` as the input variable name
+- [ ] Batch operations specify both `batch_over` and `batch_as`
+- [ ] Variable names are semantic (describe what they are, not their role)
+- [ ] All referenced sub-pipes exist in the blueprint
+- [ ] Prompt templates use `@` for blocks and `$` for inline content
+
+## Common Pitfalls to Avoid
+
+2. **Wrong OCR input name**: Must be `ocr_input`, not `document` or `pdf`
+3. **Redefining native concepts**: Use `Text` directly, don't create `MyText`
+4. **Circumstantial naming**: Use `Photo` not `InputPhoto`, `Analysis` not `FinalAnalysis`
+5. **Missing result names**: Each step in a sequence should specify its `result` name
+6. **Inconsistent variable names**: The same data should have the same name throughout
diff --git a/.pipelex/design_pipelines.md b/.pipelex/create/design_pipelines.md
similarity index 100%
rename from .pipelex/design_pipelines.md
rename to .pipelex/create/design_pipelines.md
diff --git a/.pipelex/draft_pipelines.md b/.pipelex/create/draft_pipelines.md
similarity index 100%
rename from .pipelex/draft_pipelines.md
rename to .pipelex/create/draft_pipelines.md
diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.py
index bf0601ee0..266704a8b 100644
--- a/pipelex/cli/commands/build_cmd.py
+++ b/pipelex/cli/commands/build_cmd.py
@@ -88,8 +88,12 @@ def build_draft_pipeline_cmd(
def build_blueprint_cmd(
pipeline_name: Annotated[
str,
- typer.Argument(help="Name/code of the pipeline to generate (used inside the draft)"),
+ typer.Argument(help="Name/code of the pipeline to generate"),
],
+ domain: Annotated[
+ str,
+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),
+ ] = "wip_domain",
requirements: Annotated[
Optional[str],
typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),
@@ -132,6 +136,7 @@ def build_blueprint_cmd(
asyncio.run(
do_build_blueprint(
+ domain=domain,
pipeline_name=pipeline_name,
requirements=requirements_text,
output_path=output_path,
diff --git a/pipelex/create/build_blueprint.py b/pipelex/create/build_blueprint.py
index e5b08c68c..8f594d160 100644
--- a/pipelex/create/build_blueprint.py
+++ b/pipelex/create/build_blueprint.py
@@ -16,11 +16,13 @@ from pipelex.libraries.library_manager import LibraryManager
from pipelex.libraries.pipeline_blueprint import PipelineLibraryBlueprint
from pipelex.pipe_works.pipe_dry import dry_run_pipe_codes
from pipelex.pipeline.execute import execute_pipeline
+from pipelex.tools.misc.file_utils import save_text_to_path
from pipelex.tools.misc.json_utils import save_as_json_to_path
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
async def do_build_blueprint(
+ domain: str,
pipeline_name: str,
requirements: str,
output_path: Optional[str],
@@ -29,13 +31,17 @@ async def do_build_blueprint(
pipe_output = await execute_pipeline(
pipe_code="build_blueprint",
input_memory={
+ "domain": domain,
"pipeline_name": pipeline_name,
"requirements": requirements,
- "rules": get_support_file(subpath="design_pipelines.md"),
+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),
+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),
},
)
blueprint = pipe_output.main_stuff_as(content_type=PipelineLibraryBlueprint)
pretty_print(blueprint, title="Pipeline Blueprint")
+ pipeline_draft = pipe_output.working_memory.get_stuff_as_str(name="pipeline_draft")
+ pretty_print(pipeline_draft, title="Pipeline Draft")
# Save or display result
output_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"
@@ -44,6 +50,8 @@ async def do_build_blueprint(
save_pipeline_blueprint_toml_to_path(blueprint=blueprint, path=output_path)
json_path = output_path.replace(Path(output_path).suffix, ".json")
save_as_json_to_path(object_to_save=blueprint, path=json_path)
+ draft_path = output_path.replace(Path(output_path).suffix, "_draft.md")
+ save_text_to_path(text=pipeline_draft, path=draft_path)
typer.echo(f"✅ Blueprint saved to '{output_path}'")
if validate:
diff --git a/pipelex/create/draft_pipeline.py b/pipelex/create/draft_pipeline.py
index 3b739cfd2..f87dd736e 100644
--- a/pipelex/create/draft_pipeline.py
+++ b/pipelex/create/draft_pipeline.py
@@ -24,7 +24,7 @@ async def do_draft_pipeline_text(
"domain": domain,
"pipeline_name": pipeline_name,
"requirements": requirements,
- "rules": get_support_file(subpath="draft_pipelines.md"),
+ "rules": get_support_file(subpath="create/draft_pipelines.md"),
},
)
draft_text = pipe_output.main_stuff_as_str
@@ -46,7 +46,7 @@ async def do_draft_pipeline(
input_memory={
"pipeline_name": pipeline_name,
"requirements": requirements,
- "rules": get_support_file(subpath="draft_pipelines.md"),
+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),
},
)
draft = pipe_output.main_stuff_as(content_type=PipelineDraft)
diff --git a/pipelex/libraries/pipelines/meta/meta_pipeline.toml b/pipelex/libraries/pipelines/meta/meta_pipeline.toml
index bf9565f6c..7b29af98b 100644
--- a/pipelex/libraries/pipelines/meta/meta_pipeline.toml
+++ b/pipelex/libraries/pipelines/meta/meta_pipeline.toml
@@ -11,16 +11,26 @@ PipeBlueprint = "Details enabling to create a pipe"
[pipe]
+[pipe.build_blueprint]
+type = "PipeSequence"
+definition = "Build a pipeline blueprint from natural language requirements"
+inputs = { draft_pipeline_rules = "Text", build_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }
+output = "PipelineLibraryBlueprint"
+steps = [
+ { pipe = "draft_pipeline_text", result = "pipeline_draft" },
+ { pipe = "structure_blueprint", result = "pipeline_blueprint" },
+]
+
[pipe.draft_pipeline_text]
type = "PipeLLM"
definition = "Generate a rough pipeline draft from natural language requirements"
-inputs = { rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }
+inputs = { draft_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }
output = "PipelineDraftText"
llm = "llm_to_pipelex"
prompt_template = """
You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.
-@rules
+@draft_pipeline_rules
@requirements
@@ -28,40 +38,21 @@ Domain: $domain.
Name of the pipeline: $pipeline_name.
"""
-
-[pipe.draft_pipeline]
-type = "PipeLLM"
-definition = "Generate a rough pipeline draft from natural language requirements"
-inputs = { requirements = "Text" }
-output = "PipelineDraft"
-structuring_method = "preliminary_text"
-prompt_template = """
-You are a Pipelex pipeline architect. Build a pipeline draft showing how we solve the problem.
-
-Think on the pipeline requirements: what are the pipeline's inputs and what is the expected output?
-
-- Keep domain and pipe codes short, lowercase, and meaningful.
-- In concept, map ConceptName (PascalCase, singular) to its natural language definition string. Do not redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page).
-- In pipe, define at least one pipe that accomplishes the main goal implied by the requirements. Use operators/controllers supported by Pipelex (e.g., PipeLLM, PipeOcr, PipeSequence...).
-- Ensure all input variables referenced in prompts are listed in the pipe `inputs`.
-
-@requirements
-"""
-
-[pipe.build_blueprint]
+[pipe.structure_blueprint]
type = "PipeLLM"
definition = "Generate a pipeline blueprint from natural language requirements"
-inputs = { rules = "Text", requirements = "Text", pipeline_name = "Text" }
+inputs = { build_pipeline_rules = "Text", pipeline_draft = "Text", pipeline_name = "Text", domain = "Text" }
output = "PipelineLibraryBlueprint"
llm = "llm_to_pipelex"
# structuring_method = "preliminary_text"
prompt_template = """
-You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.
+You are a Pipelex pipeline architect. Build a structured pipeline blueprint from the provided pipeline draft text.
-@rules
+@build_pipeline_rules
-@requirements
+@pipeline_draft
-Name of the pipeline, i.e. name of the main pipe: $pipeline_name.
+Domain: $domain.
+Name of the pipeline: $pipeline_name.
"""
diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py
index 043ca1b6a..51e28d102 100644
--- a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py
+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py
@@ -29,9 +29,10 @@ async def test_generate_pipeline_blueprint(pipe_run_mode: PipeRunMode, requireme
pipe_output = await execute_pipeline(
pipe_code="build_blueprint",
input_memory={
+ "domain": "test_domain",
"pipeline_name": "test_pipeline",
"requirements": requirements,
- "rules": get_support_file(),
+ "rules": get_support_file("create/design_pipelines.md"),
},
pipe_run_mode=pipe_run_mode,
)