Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Build the blueprint based on the draft
Louis Choquel1df6d07279ae
diff --git a/.pipelex/create/build_pipelines.md b/.pipelex/create/build_pipelines.mdnew file mode 100644index 000000000..deab04c86--- /dev/null+++ b/.pipelex/create/build_pipelines.md@@ -0,0 +1,316 @@+# Building Pipelex Pipelines from Drafts++This guide explains how to translate a markdown pipeline draft into a formal `PipelineLibraryBlueprint` structure that can be validated and executed by Pipelex.++## Overview++1. Convert the draft's narrative flow into a formal domain/concepts/pipes structure+2. Map draft notation to specific pipe types and configurations+3. Ensure all working memory references are properly captured in operator pipe inputs (wrapping up input requirements for controller pipes will be done later)++## Translation Workflow++### Step 1: Extract Domain Information++Create the blueprint root:+```json+{+ "domain": "domain_name",+ "definition": "What this pipeline does"+}+```++### Step 2: Identify and Define Concepts++Review the draft for all data types mentioned:++- **Input variables** → Concepts needed+- **Intermediate results** → Concepts needed+- **Final output** → Concept needed++#### Concept Rules++1. **Use native concepts when possible**:+ - `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`+ - Don't redefine these++2. **Use library concepts when applicable**:+ - `documents.TextAndImagesContent`+ - `images.VisualDescription`+ - `images.ImgGenPrompt`+ - `images.Photo`++3. **Create domain concepts for specialized data**:+ - Use PascalCase: `Analysis`, `TechnicalReport`, `UserQuery`+ - Never use plurals: `Document` not `Documents`+ - Never use circumstantial names: `Photo` not `InputPhoto`+ - Avoid adjectives: `Text` not `LongText`++#### Concept Definition Formats++Quick form (for simple text-based concepts):+```json+"concept": {+ "UserQuery": "Natural language question from a user",+ "Summary": "Condensed version of content"+}+```++Full form (when structure or refinement needed):+```json+"concept": {+ "TechnicalAnalysis": {+ "definition": "Technical analysis of a system",+ "refines": "Text"+ }+}+```++### Step 3: Translate Flow Steps to Pipes++Map each step from your draft to the appropriate pipe type and configuration.++#### Operator Mappings++| Draft Notation | Pipe Type | Key Configuration |+|---------------|-----------|-------------------|+| `LLM[purpose]: (inputs) → output` | `PipeLLM` | Set `prompt_template` or `prompt` |+| `OCR: (document) → pages` | `PipeOcr` | Input must be named `ocr_input` |+| `ImgGen: (prompt) → image` | `PipeImgGen` | Configure image generation settings |+| `Func[name]: (data) → result` | `PipeFunc` | Reference Python function |++#### Controller Mappings++| Draft Pattern | Pipe Type | Structure |+|--------------|-----------|-----------|+| Sequential steps (Step 1, Step 2...) | `PipeSequence` | Use `steps` array |+| Process each item in list | `PipeSequence` step with batch | Add `batch_over` and `batch_as` to step |+| Parallel branches (Branch A, Branch B...) | `PipeParallel` | Use `parallels` array |+| Conditional routing (Case "x"...) | `PipeCondition` | Use `expression` and `pipe_map` |++### Step 4: Configure Each Pipe++#### PipeLLM Configuration++From draft step:+```markdown+### Step: Analyze document for key insights+LLM[analyze]: (document, criteria) → analysis+```++To blueprint:+```json+"analyze_document": {+ "type": "PipeLLM",+ "definition": "Analyze document for key insights",+ "inputs": {+ "document": "Text",+ "criteria": "Text"+ },+ "output": "Analysis",+ "prompt_template": "Analyze the following document based on the provided criteria.\n\n@document\n\n@criteria"+}+```++**Prompt Template Rules**:++- Use `@variable` for block content (multi-line)+- Use `$variable` for inline content (within sentences)++#### PipeOcr Configuration++From draft step:+```markdown+### Step: Extract content from PDF+OCR: (document) → pages+```++To blueprint:+```json+"extract_pages": {+ "type": "PipeOcr",+ "definition": "Extract content from PDF",+ "inputs": {+ "ocr_input": "PDF" // Must be named ocr_input+ },+ "output": "Page", // Output is always Page concept+ "page_images": true,+ "page_views": true+}+```++#### PipeSequence Configuration++From draft sequence:+```markdown+Step 1: Extract information+extract_info(document) → key_info++Step 2: Analyze information+analyze(key_info) → analysis+```++To blueprint:+```json+"process_document": {+ "type": "PipeSequence",+ "definition": "Extract and analyze document",+ "inputs": {+ "document": "Text"+ },+ "output": "Analysis",+ "steps": [+ {+ "pipe": "extract_info",+ "result": "key_info"+ },+ {+ "pipe": "analyze_info",+ "result": "analysis"+ }+ ]+}+```++#### Batch Processing Configuration++From draft batch:+```markdown+Process each item in documents:+analyze_document(document) → analysis+```++To blueprint step within PipeSequence:+```json+{+ "pipe": "analyze_single",+ "batch_over": "documents",+ "batch_as": "document",+ "result": "analyses"+}+```+Using the above, the required input used is `documents`, and by looping on the items of that list, each one will be named `document`, and that's how it will be received in the `analyze_single` pipe.++#### PipeCondition Configuration++From draft conditional:+```markdown+Route based on category:+Case "technical": analyze_technical(document) → analysis+Case "legal": analyze_legal(document) → analysis+```++To blueprint:+```json+"route_analysis": {+ "type": "PipeCondition",+ "definition": "Route document to appropriate analyzer",+ "inputs": {+ "document": "Text",+ "category": "Text"+ },+ "output": "Analysis",+ "expression": "category",+ "pipe_map": {+ "technical": "analyze_technical",+ "legal": "analyze_legal"+ }+}+```++## Complete Example: Draft to Blueprint++### Draft (Markdown)++```markdown+# Pipeline: document_insights+Domain: analysis+Purpose: Extract insights from documents based on criteria++## Inputs+- `document` : Document to analyze+- `criteria` : Analysis criteria++## Flow++### Step 1: Extract key information+LLM[extract]: (document) → key_points++### Step 2: Analyze against criteria+LLM[analyze]: (key_points, criteria) → insights++## Output+Returns: `insights`+```++### Blueprint (JSON)++```json+{+ "domain": "analysis",+ "definition": "Extract insights from documents based on criteria",+ "concept": {+ "KeyPoints": "Important information extracted from document",+ "Insights": "Analysis results based on specific criteria"+ },+ "pipe": {+ "document_insights": {+ "type": "PipeSequence",+ "definition": "Extract and analyze document insights",+ "output": "Insights",+ "steps": [+ {+ "pipe": "extract_key_points",+ "result": "key_points"+ },+ {+ "pipe": "analyze_points",+ "result": "insights"+ }+ ]+ },+ "extract_key_points": {+ "type": "PipeLLM",+ "definition": "Extract key information",+ "inputs": {+ "document": "Text"+ },+ "output": "KeyPoints",+ "prompt_template": "Extract the key points from this document.\n\n@document"+ },+ "analyze_points": {+ "type": "PipeLLM",+ "definition": "Analyze against criteria",+ "inputs": {+ "key_points": "KeyPoints",+ "criteria": "Text"+ },+ "output": "Insights",+ "prompt_template": "Analyze these key points against the provided criteria.\n\n@key_points\n\n@criteria"+ }+ }+}+```++## Validation Checklist++Before finalizing your blueprint:++- [ ] Domain is defined with a single word or snake_case code+- [ ] All concepts follow naming conventions (PascalCase, no plurals, no adjectives, no circumstantial names)+- [ ] Native concepts are not redefined+- [ ] Each pipe has `type`, `definition`, `inputs`, and `output`+- [ ] Controller pipes don't need to declare inputs, they will be inferred from the blueprint+- [ ] OCR pipes use `ocr_input` as the input variable name+- [ ] Batch operations specify both `batch_over` and `batch_as`+- [ ] Variable names are semantic (describe what they are, not their role)+- [ ] All referenced sub-pipes exist in the blueprint+- [ ] Prompt templates use `@` for blocks and `$` for inline content++## Common Pitfalls to Avoid++2. **Wrong OCR input name**: Must be `ocr_input`, not `document` or `pdf`+3. **Redefining native concepts**: Use `Text` directly, don't create `MyText`+4. **Circumstantial naming**: Use `Photo` not `InputPhoto`, `Analysis` not `FinalAnalysis`+5. **Missing result names**: Each step in a sequence should specify its `result` name+6. **Inconsistent variable names**: The same data should have the same name throughoutdiff --git a/.pipelex/design_pipelines.md b/.pipelex/create/design_pipelines.mdsimilarity index 100%rename from .pipelex/design_pipelines.mdrename to .pipelex/create/design_pipelines.mddiff --git a/.pipelex/draft_pipelines.md b/.pipelex/create/draft_pipelines.mdsimilarity index 100%rename from .pipelex/draft_pipelines.mdrename to .pipelex/create/draft_pipelines.mddiff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.pyindex bf0601ee0..266704a8b 100644--- a/pipelex/cli/commands/build_cmd.py+++ b/pipelex/cli/commands/build_cmd.py@@ -88,8 +88,12 @@ def build_draft_pipeline_cmd(def build_blueprint_cmd(pipeline_name: Annotated[str,- typer.Argument(help="Name/code of the pipeline to generate (used inside the draft)"),+ typer.Argument(help="Name/code of the pipeline to generate"),],+ domain: Annotated[+ str,+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),+ ] = "wip_domain",requirements: Annotated[Optional[str],typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),@@ -132,6 +136,7 @@ def build_blueprint_cmd(asyncio.run(do_build_blueprint(+ domain=domain,pipeline_name=pipeline_name,requirements=requirements_text,output_path=output_path,diff --git a/pipelex/create/build_blueprint.py b/pipelex/create/build_blueprint.pyindex e5b08c68c..8f594d160 100644--- a/pipelex/create/build_blueprint.py+++ b/pipelex/create/build_blueprint.py@@ -16,11 +16,13 @@ from pipelex.libraries.library_manager import LibraryManagerfrom pipelex.libraries.pipeline_blueprint import PipelineLibraryBlueprintfrom pipelex.pipe_works.pipe_dry import dry_run_pipe_codesfrom pipelex.pipeline.execute import execute_pipeline+from pipelex.tools.misc.file_utils import save_text_to_pathfrom pipelex.tools.misc.json_utils import save_as_json_to_pathfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_errorasync def do_build_blueprint(+ domain: str,pipeline_name: str,requirements: str,output_path: Optional[str],@@ -29,13 +31,17 @@ async def do_build_blueprint(pipe_output = await execute_pipeline(pipe_code="build_blueprint",input_memory={+ "domain": domain,"pipeline_name": pipeline_name,"requirements": requirements,- "rules": get_support_file(subpath="design_pipelines.md"),+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),},)blueprint = pipe_output.main_stuff_as(content_type=PipelineLibraryBlueprint)pretty_print(blueprint, title="Pipeline Blueprint")+ pipeline_draft = pipe_output.working_memory.get_stuff_as_str(name="pipeline_draft")+ pretty_print(pipeline_draft, title="Pipeline Draft")# Save or display resultoutput_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"@@ -44,6 +50,8 @@ async def do_build_blueprint(save_pipeline_blueprint_toml_to_path(blueprint=blueprint, path=output_path)json_path = output_path.replace(Path(output_path).suffix, ".json")save_as_json_to_path(object_to_save=blueprint, path=json_path)+ draft_path = output_path.replace(Path(output_path).suffix, "_draft.md")+ save_text_to_path(text=pipeline_draft, path=draft_path)typer.echo(f"✅ Blueprint saved to '{output_path}'")if validate:diff --git a/pipelex/create/draft_pipeline.py b/pipelex/create/draft_pipeline.pyindex 3b739cfd2..f87dd736e 100644--- a/pipelex/create/draft_pipeline.py+++ b/pipelex/create/draft_pipeline.py@@ -24,7 +24,7 @@ async def do_draft_pipeline_text("domain": domain,"pipeline_name": pipeline_name,"requirements": requirements,- "rules": get_support_file(subpath="draft_pipelines.md"),+ "rules": get_support_file(subpath="create/draft_pipelines.md"),},)draft_text = pipe_output.main_stuff_as_str@@ -46,7 +46,7 @@ async def do_draft_pipeline(input_memory={"pipeline_name": pipeline_name,"requirements": requirements,- "rules": get_support_file(subpath="draft_pipelines.md"),+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),},)draft = pipe_output.main_stuff_as(content_type=PipelineDraft)diff --git a/pipelex/libraries/pipelines/meta/meta_pipeline.toml b/pipelex/libraries/pipelines/meta/meta_pipeline.tomlindex bf9565f6c..7b29af98b 100644--- a/pipelex/libraries/pipelines/meta/meta_pipeline.toml+++ b/pipelex/libraries/pipelines/meta/meta_pipeline.toml@@ -11,16 +11,26 @@ PipeBlueprint = "Details enabling to create a pipe"[pipe]+[pipe.build_blueprint]+type = "PipeSequence"+definition = "Build a pipeline blueprint from natural language requirements"+inputs = { draft_pipeline_rules = "Text", build_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }+output = "PipelineLibraryBlueprint"+steps = [+ { pipe = "draft_pipeline_text", result = "pipeline_draft" },+ { pipe = "structure_blueprint", result = "pipeline_blueprint" },+]+[pipe.draft_pipeline_text]type = "PipeLLM"definition = "Generate a rough pipeline draft from natural language requirements"-inputs = { rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }+inputs = { draft_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }output = "PipelineDraftText"llm = "llm_to_pipelex"prompt_template = """You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.-@rules+@draft_pipeline_rules@requirements@@ -28,40 +38,21 @@ Domain: $domain.Name of the pipeline: $pipeline_name."""--[pipe.draft_pipeline]-type = "PipeLLM"-definition = "Generate a rough pipeline draft from natural language requirements"-inputs = { requirements = "Text" }-output = "PipelineDraft"-structuring_method = "preliminary_text"-prompt_template = """-You are a Pipelex pipeline architect. Build a pipeline draft showing how we solve the problem.--Think on the pipeline requirements: what are the pipeline's inputs and what is the expected output?--- Keep domain and pipe codes short, lowercase, and meaningful.-- In concept, map ConceptName (PascalCase, singular) to its natural language definition string. Do not redefine native concepts (Text, Image, PDF, TextAndImages, Number, Page).-- In pipe, define at least one pipe that accomplishes the main goal implied by the requirements. Use operators/controllers supported by Pipelex (e.g., PipeLLM, PipeOcr, PipeSequence...).-- Ensure all input variables referenced in prompts are listed in the pipe `inputs`.--@requirements-"""--[pipe.build_blueprint]+[pipe.structure_blueprint]type = "PipeLLM"definition = "Generate a pipeline blueprint from natural language requirements"-inputs = { rules = "Text", requirements = "Text", pipeline_name = "Text" }+inputs = { build_pipeline_rules = "Text", pipeline_draft = "Text", pipeline_name = "Text", domain = "Text" }output = "PipelineLibraryBlueprint"llm = "llm_to_pipelex"# structuring_method = "preliminary_text"prompt_template = """-You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.+You are a Pipelex pipeline architect. Build a structured pipeline blueprint from the provided pipeline draft text.-@rules+@build_pipeline_rules-@requirements+@pipeline_draft-Name of the pipeline, i.e. name of the main pipe: $pipeline_name.+Domain: $domain.+Name of the pipeline: $pipeline_name."""diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.pyindex 043ca1b6a..51e28d102 100644--- a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py@@ -29,9 +29,10 @@ async def test_generate_pipeline_blueprint(pipe_run_mode: PipeRunMode, requiremepipe_output = await execute_pipeline(pipe_code="build_blueprint",input_memory={+ "domain": "test_domain","pipeline_name": "test_pipeline","requirements": requirements,- "rules": get_support_file(),+ "rules": get_support_file("create/design_pipelines.md"),},pipe_run_mode=pipe_run_mode,)