Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Stable pipeline drafting
Louis Choquel74888098e69a
diff --git a/.pipelex/draft_pipelines.md b/.pipelex/draft_pipelines.mdindex c20b50dc9..df059f3af 100644--- a/.pipelex/draft_pipelines.md+++ b/.pipelex/draft_pipelines.md@@ -7,6 +7,7 @@ This guide explains how to draft pipelines in natural language using markdown noEvery pipeline execution maintains a `WorkingMemory` that acts as a shared data space. When a pipeline starts, the initial inputs are placed in this memory. As each pipe executes, its output is added to the memory with a specific name (the `result` name). Subsequent pipes can access any data in the memory by referencing these names.**Key Rules**:+- When a pipe produces an output with name `X`, any later pipe can use `X` as an input. The names must match exactly.- Controller pipes (PipeSequence, PipeParallel, PipeCondition) automatically inherit the input requirements of their nested operators - you don't need to specify their inputs explicitly.@@ -35,24 +36,20 @@ Purpose: [What this pipeline does]### Flow Section-Describe the pipeline flow narratively, using a pseudo-code style that shows how data moves through WorkingMemory:+Describe the pipeline flow using concise step notation:```markdown## FlowWorkingMemory starts with: {input_name, another_input}-### Step 1: [Description]-Use `input_name` to generate `intermediate_result`-→ LLM[process]: (input_name) → intermediate_result- Purpose: Processes the input data+### Step 1: Process input data+LLM[process]: (input_name) → intermediate_resultWorkingMemory now contains: {input_name, another_input, intermediate_result}-### Step 2: [Description]-Combine `intermediate_result` and `another_input` to create `final_output`-→ LLM[synthesize]: (intermediate_result, another_input) → final_output- Purpose: Synthesizes the intermediate result with additional context+### Step 2: Synthesize results with additional context+LLM[synthesize]: (intermediate_result, another_input) → final_outputWorkingMemory now contains: {input_name, another_input, intermediate_result, final_output}```@@ -65,66 +62,80 @@ Returns: `final_output` - Description of the final result## Pipe Types and Notation-### Sequential Processing (PipeSequence)+## Operator Notations++**Important**: During drafting, DO NOT write actual prompts or prompt templates. Focus on describing what each operator does and how data flows. Prompts will be written during implementation.++### LLM Operations (including vision, to generate text from text and images)```markdown-Step 1: process_input → result_1-Step 2: analyze(result_1) → result_2-Step 3: finalize(result_2, original_input) → final_output+LLM[purpose]: (inputs) → output++Examples:+LLM[summarize]: (long_text) → summary+LLM[analyze]: (document, criteria) → analysis```-### Parallel Processing (PipeParallel)+### OCR Operations (to extract text and images from PDF pages, including full page views)```markdown-Parallel:- Branch A: process_method_a(input) → result_a- Branch B: process_method_b(input) → result_b-Combine: merge(result_a, result_b) → combined_result+OCR: (pdf_document) → pages_list```-### Conditional Processing (PipeCondition)+### Image Generation (to generate images from text prompts)```markdown-If expression(input_data):- Case "option_1": pipe_a(input_data) → result- Case "option_2": pipe_b(input_data) → result- Default: pipe_default(input_data) → result+ImgGen: (prompt_text) → generated_image```-### Batch Processing+### Function Operations (to call external python functions)```markdown-For each item in `list_of_items`:- process_single(item) → processed_item-Collect all → processed_items_list+Func[function_name]: (data) → result```-## Operator Notations+### Sequential Processing (to run steps in sequence, use it when steps need results from previous steps)+```markdown+Step 1: Process input data+process_input → result_1-**Important**: During drafting, DO NOT write actual prompts or prompt templates. Focus on describing what each operator does and how data flows. Prompts will be written during implementation.+Step 2: Analyze processed results+analyze(result_1) → result_2-### LLM Operations-```markdown-LLM[purpose]: (inputs) → output-Example: LLM[summarize]: (long_text) → summary+Step 3: Finalize with original context+finalize(result_2, original_input) → final_output```-Describe the purpose without the actual prompt:+### Batch Processing (to process each item in a list independently and in parallel)```markdown-LLM[analyze]: (document, criteria) → analysis- Purpose: Analyzes the document based on provided criteria-```+Process each item in list_of_items:+process_single(item) → processed_item-### OCR Operations-```markdown-OCR: (pdf_document) → pages_list-Each page contains text and images+Collect all results → processed_items_list```-### Image Generation+### Parallel Processing (to run different pipes in parallel from the same memory, use it when steps can be run independently, each with a copy of the working memory)```markdown-ImgGen: (prompt_text) → generated_image+Parallel execution:++Branch A: Process applying pipe A+pipe_a(input) → result_a++Branch B: Process applying pipe B+pipe_b(input) → result_b++Combine: Merge parallel results+merge(result_a, result_b) → combined_result```-### Function Operations+### Conditional Processing (to run different pipes based on a condition)```markdown-Func[calculate_metrics]: (data) → metrics+Route based on expression(input_data):++Case "option_1": Handle option 1+pipe_a(input_data) → result++Case "option_2": Handle option 2+pipe_b(input_data) → result++Default: Handle other cases+pipe_default(input_data) → result```## Memory Flow Examples@@ -142,19 +153,22 @@ Purpose: Extract and analyze key information from documentsWorkingMemory: {raw_document}-### Step 1: Extract content+### Step 1: Extract content from PDFOCR: (raw_document) → pages+WorkingMemory: {raw_document, pages}-### Step 2: Summarize each page-For each page in pages:- LLM[summarize]: (page) → page_summary-Collect all → page_summaries+### Step 2: Summarize each page in parallel+Process each page in pages:+LLM[summarize]: (page) → page_summary++Collect all results → page_summaries+WorkingMemory: {raw_document, pages, page_summaries}-### Step 3: Create final analysis+### Step 3: Create comprehensive analysis from summariesLLM[synthesize]: (page_summaries) → final_analysis- Purpose: Creates comprehensive analysis from all page summaries+WorkingMemory: {raw_document, pages, page_summaries, final_analysis}## Output@@ -175,22 +189,22 @@ Purpose: Route and respond to customer queriesWorkingMemory: {query, context}-### Step 1: Classify query type+### Step 1: Classify query into support categoryLLM[classify]: (query) → query_type- Purpose: Classifies the query into a category+WorkingMemory: {query, context, query_type}-### Step 2: Route based on type-If query_type:- Case "technical":- LLM[technical_support]: (query, context) → response- Purpose: Provides technical support based on query and context- Case "billing":- LLM[billing_support]: (query, context) → response- Purpose: Handles billing-related queries with available context- Default:- LLM[general_support]: (query) → response- Purpose: Provides general assistance for unclassified queries+### Step 2: Route to appropriate support handler+Route based on query_type:++Case "technical": Technical support handler+LLM[technical_support]: (query, context) → response++Case "billing": Billing support handler+LLM[billing_support]: (query, context) → response++Default: General support handler+LLM[general_support]: (query) → responseWorkingMemory: {query, context, query_type, response}@@ -204,17 +218,19 @@ Returns: `response`2. **Show memory state**: After complex steps, show what's in WorkingMemory to track data flow.-3. **Be explicit about inputs**: For each operator step, clearly indicate which items from WorkingMemory are being used. Controller pipes will automatically inherit these requirements.+3. **Be explicit about inputs**: For each operator step, clearly indicate which items from WorkingMemory are being used.4. **Use descriptive names**: Instead of `result_1`, use `extracted_entities` or `summarized_content`.-5. **No prompts during drafting**: Focus on describing WHAT each step does (its purpose), not HOW (the actual prompts). Prompts are implementation details added later.+5. **Concise step descriptions**: Each step should have a clear title that describes what it accomplishes. Avoid repeating the same information in multiple places.++6. **No prompts during drafting**: Focus on describing WHAT each step does, not HOW (the actual prompts). Prompts are implementation details added later.-6. **Block vs Inline references**:+7. **Block vs Inline references**:- Use `@variable_name` notation for block insertions (multi-line content)- Use `$variable_name` notation for inline insertions (single values in sentences)-7. **Batch operations**: When processing lists, clearly indicate:+8. **Batch operations**: When processing lists, clearly indicate:- What list is being iterated (`batch_over`)- What each item is called during processing (`batch_as`)- What the collected results are named@@ -228,7 +244,7 @@ Before finalizing your draft:- [ ] The final output exists in WorkingMemory- [ ] Names are consistent throughout (no typos or variations)- [ ] The flow clearly shows data transformation from inputs to outputs-- [ ] Each LLM step has a clear purpose (but no actual prompts yet)+- [ ] Each step has a concise, non-repetitive description## From Draft to Implementationdiff --git a/pipelex/libraries/llm_deck/base_llm_deck.toml b/pipelex/libraries/llm_deck/base_llm_deck.tomlindex 0ca5bf63a..620509679 100644--- a/pipelex/libraries/llm_deck/base_llm_deck.toml+++ b/pipelex/libraries/llm_deck/base_llm_deck.toml@@ -56,7 +56,8 @@ llm_for_question_and_excerpt_reformulation = { llm_handle = "gpt-4o", temperatur# Engineering skillsllm_to_engineer = { llm_handle = "best-claude", temperature = 0.2 }-llm_to_pipelex = { llm_handle = "gpt-5-mini", temperature = 0.2 }+llm_to_pipelex = { llm_handle = "best-claude", temperature = 0.2 }+# llm_to_pipelex = { llm_handle = "gpt-5-mini", temperature = 0.2 }# Image skillsllm_to_write_imgg_prompt = { llm_handle = "best-claude", temperature = 0.2 }