Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Refactor/epic library blueprint 2 generate blueprint (#218)
Louis Choquelda3ed609b97b
diff --git a/.pipelex/create/build_pipelines.md b/.pipelex/create/build_pipelines.mdnew file mode 100644index 000000000..1dc666313--- /dev/null+++ b/.pipelex/create/build_pipelines.md@@ -0,0 +1,316 @@+# Building Pipelex Pipelines from Drafts++This guide explains how to translate a markdown pipeline draft into a formal `PipelineBlueprint` structure that can be validated and executed by Pipelex.++## Overview++1. Convert the draft's narrative flow into a formal domain/concepts/pipes structure+2. Map draft notation to specific pipe types and configurations+3. Ensure all working memory references are properly captured in operator pipe inputs (wrapping up input requirements for controller pipes will be done later)++## Translation Workflow++### Step 1: Extract Domain Information++Create the blueprint root:+```json+{+ "domain": "domain_name",+ "definition": "What this pipeline does"+}+```++### Step 2: Identify and Define Concepts++Review the draft for all data types mentioned:++- **Input variables** → Concepts needed+- **Intermediate results** → Concepts needed+- **Final output** → Concept needed++#### Concept Rules++1. **Use native concepts when possible**:+ - `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`+ - Don't redefine these++2. **Use library concepts when applicable**:+ - `documents.TextAndImagesContent`+ - `images.VisualDescription`+ - `images.ImgGenPrompt`+ - `images.Photo`++3. **Create domain concepts for specialized data**:+ - Use PascalCase: `Analysis`, `TechnicalReport`, `UserQuery`+ - Never use plurals: `Document` not `Documents`+ - Never use circumstantial names: `Photo` not `InputPhoto`+ - Avoid adjectives: `Text` not `LongText`++#### Concept Definition Formats++Quick form (for simple text-based concepts):+```json+"concept": {+ "UserQuery": "Natural language question from a user",+ "Summary": "Condensed version of content"+}+```++Full form (when structure or refinement needed):+```json+"concept": {+ "TechnicalAnalysis": {+ "definition": "Technical analysis of a system",+ "refines": "Text"+ }+}+```++### Step 3: Translate Flow Steps to Pipes++Map each step from your draft to the appropriate pipe type and configuration.++#### Operator Mappings++| Draft Notation | Pipe Type | Key Configuration |+|---------------|-----------|-------------------|+| `LLM[purpose]: (inputs) → output` | `PipeLLM` | Set `prompt_template` or `prompt` |+| `OCR: (document) → pages` | `PipeOcr` | Input must be named `ocr_input` |+| `ImgGen: (prompt) → image` | `PipeImgGen` | Configure image generation settings |+| `Func[name]: (data) → result` | `PipeFunc` | Reference Python function |++#### Controller Mappings++| Draft Pattern | Pipe Type | Structure |+|--------------|-----------|-----------|+| Sequential steps (Step 1, Step 2...) | `PipeSequence` | Use `steps` array |+| Process each item in list | `PipeSequence` step with batch | Add `batch_over` and `batch_as` to step |+| Parallel branches (Branch A, Branch B...) | `PipeParallel` | Use `parallels` array |+| Conditional routing (Case "x"...) | `PipeCondition` | Use `expression` and `pipe_map` |++### Step 4: Configure Each Pipe++#### PipeLLM Configuration++From draft step:+```markdown+### Step: Analyze document for key insights+LLM[analyze]: (document, criteria) → analysis+```++To blueprint:+```json+"analyze_document": {+ "type": "PipeLLM",+ "definition": "Analyze document for key insights",+ "inputs": {+ "document": "Text",+ "criteria": "Text"+ },+ "output": "Analysis",+ "prompt_template": "Analyze the following document based on the provided criteria.\n\n@document\n\n@criteria"+}+```++**Prompt Template Rules**:++- Use `@variable` for block content (multi-line)+- Use `$variable` for inline content (within sentences)++#### PipeOcr Configuration++From draft step:+```markdown+### Step: Extract content from PDF+OCR: (document) → pages+```++To blueprint:+```json+"extract_pages": {+ "type": "PipeOcr",+ "definition": "Extract content from PDF",+ "inputs": {+ "ocr_input": "PDF" // Must be named ocr_input+ },+ "output": "Page", // Output is always Page concept+ "page_images": true,+ "page_views": true+}+```++#### PipeSequence Configuration++From draft sequence:+```markdown+Step 1: Extract information+extract_info(document) → key_info++Step 2: Analyze information+analyze(key_info) → analysis+```++To blueprint:+```json+"process_document": {+ "type": "PipeSequence",+ "definition": "Extract and analyze document",+ "inputs": {+ "document": "Text"+ },+ "output": "Analysis",+ "steps": [+ {+ "pipe": "extract_info",+ "result": "key_info"+ },+ {+ "pipe": "analyze_info",+ "result": "analysis"+ }+ ]+}+```++#### Batch Processing Configuration++From draft batch:+```markdown+Process each item in documents:+analyze_document(document) → analysis+```++To blueprint step within PipeSequence:+```json+{+ "pipe": "analyze_single",+ "batch_over": "documents",+ "batch_as": "document",+ "result": "analyses"+}+```+Using the above, the required input used is `documents`, and by looping on the items of that list, each one will be named `document`, and that's how it will be received in the `analyze_single` pipe.++#### PipeCondition Configuration++From draft conditional:+```markdown+Route based on category:+Case "technical": analyze_technical(document) → analysis+Case "legal": analyze_legal(document) → analysis+```++To blueprint:+```json+"route_analysis": {+ "type": "PipeCondition",+ "definition": "Route document to appropriate analyzer",+ "inputs": {+ "document": "Text",+ "category": "Text"+ },+ "output": "Analysis",+ "expression": "category",+ "pipe_map": {+ "technical": "analyze_technical",+ "legal": "analyze_legal"+ }+}+```++## Complete Example: Draft to Blueprint++### Draft (Markdown)++```markdown+# Pipeline: document_insights+Domain: analysis+Purpose: Extract insights from documents based on criteria++## Inputs+- `document` : Document to analyze+- `criteria` : Analysis criteria++## Flow++### Step 1: Extract key information+LLM[extract]: (document) → key_points++### Step 2: Analyze against criteria+LLM[analyze]: (key_points, criteria) → insights++## Output+Returns: `insights`+```++### Blueprint (JSON)++```json+{+ "domain": "analysis",+ "definition": "Extract insights from documents based on criteria",+ "concept": {+ "KeyPoints": "Important information extracted from document",+ "Insights": "Analysis results based on specific criteria"+ },+ "pipe": {+ "document_insights": {+ "type": "PipeSequence",+ "definition": "Extract and analyze document insights",+ "output": "Insights",+ "steps": [+ {+ "pipe": "extract_key_points",+ "result": "key_points"+ },+ {+ "pipe": "analyze_points",+ "result": "insights"+ }+ ]+ },+ "extract_key_points": {+ "type": "PipeLLM",+ "definition": "Extract key information",+ "inputs": {+ "document": "Text"+ },+ "output": "KeyPoints",+ "prompt_template": "Extract the key points from this document.\n\n@document"+ },+ "analyze_points": {+ "type": "PipeLLM",+ "definition": "Analyze against criteria",+ "inputs": {+ "key_points": "KeyPoints",+ "criteria": "Text"+ },+ "output": "Insights",+ "prompt_template": "Analyze these key points against the provided criteria.\n\n@key_points\n\n@criteria"+ }+ }+}+```++## Validation Checklist++Before finalizing your blueprint:++- [ ] Domain is defined with a single word or snake_case code+- [ ] All concepts follow naming conventions (PascalCase, no plurals, no adjectives, no circumstantial names)+- [ ] Native concepts are not redefined+- [ ] Each pipe has `type`, `definition`, `inputs`, and `output`+- [ ] Controller pipes don't need to declare inputs, they will be inferred from the blueprint+- [ ] OCR pipes use `ocr_input` as the input variable name+- [ ] Batch operations specify both `batch_over` and `batch_as`+- [ ] Variable names are semantic (describe what they are, not their role)+- [ ] All referenced sub-pipes exist in the blueprint+- [ ] Prompt templates use `@` for blocks and `$` for inline content++## Common Pitfalls to Avoid++2. **Wrong OCR input name**: Must be `ocr_input`, not `document` or `pdf`+3. **Redefining native concepts**: Use `Text` directly, don't create `MyText`+4. **Circumstantial naming**: Use `Photo` not `InputPhoto`, `Analysis` not `FinalAnalysis`+5. **Missing result names**: Each step in a sequence should specify its `result` name+6. **Inconsistent variable names**: The same data should have the same name throughoutdiff --git a/.pipelex/create/design_pipelines.md b/.pipelex/create/design_pipelines.mdnew file mode 100644index 000000000..2a3fcfb80--- /dev/null+++ b/.pipelex/create/design_pipelines.md@@ -0,0 +1,270 @@+## Designing Pipelex pipelines++This section explains how to design pipelines for Pipelex.++### Workflow for a designer agent++- Plan the pipeline in natural language (goal, inputs, steps, outputs).++- Translate the plan into a `PipelineBlueprint` object graph: domain, concepts, pipes.++- Define controllers first (`PipeSequence`, `PipeParallel`, `PipeCondition`), then operators (`PipeLLM`, `PipeOcr`, etc.).++- For each new pipe, detail its `type` and `definition`.++- Ensure each pipe’s `inputs` shows all required variables, including those used by nested steps or conditional branches. `inputs` is a dict mapping variable names to concept codes.++- Choose appropriate native or structured output concepts.++### Root blueprint object: PipelineBlueprint++`PipelineBlueprint` fields:++- `domain: str` — required. Single word or snake_case code for the domain.++- `definition: Optional[str]` — optional textual description of the domain.++- `system_prompt: Optional[str]` — default system prompt applied by LLM pipes in this domain if they don’t provide one.++- `system_prompt_to_structure: Optional[str]` — domain-level instructions to structure outputs.++- `prompt_template_to_structure: Optional[str]` — domain-level instructions to structure prompts.++- `concept: Dict[str, Union[str, ConceptBlueprint]]` — concept section. Keys are concept names (PascalCase). Values are either:++ - a string (quick form): the concept definition; or++ - a `ConceptBlueprint` object (full form).++- `pipe: Dict[str, Dict[str, Any]]` — pipes section. Keys are pipe codes (snake_case). Values are per-pipe details dicts that will be validated into concrete `PipeBlueprint` subclasses based on their `type`.++### Concepts++- Conceiving and naming concepts:+ - Use PascalCase for concept names.++ - DO NOT use plurals in concept names (use `Item` not `Items`).++ - DO NOT use adjectives in concept names (use `Text`, not `LargeText`).++ - When a concept or concept reference is unqualified (no `domain.` prefix), it will be prefixed with the current blueprint `domain` during validation/expansion, unless its name corresponds to one of the native concepts.++ - Do not redefine native concepts (use them directly): `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`.++ - Textual concepts refining `native.Text` are acceptable too, and you don't need to specify their structure: by default it'll be text.++ - Reuse existing concepts whenever possible. For instance if you take a `Poem` and make it dark, the result is a `Poem` too: you MUST NOT create a new `DarkPoem` concept in this kind of situation.++ - Concepts from the base library are also provided and must be used whenever possible:++ - `documents.TextAndImagesContent` = "A content that comprises text and images where the text can include local links to the images"++ - `images.VisualDescription` = "Visual description of something"++ - `images.ImgGenPrompt`:+ - definition = "Prompt to generate an image"+ - refines = "Text"++ - `images.Photo`:+ - definition = "Photo"+ - structure = "ImageContent"+ - refines = "Image"++ - DO NOT create a concept qualified with circumstances like `InputPhoto`: it's just a `Photo`, it so happens that we're using it as input, OK, but that will be convened by the rest of the structure.++- Quick form (string):++ - `concept["ConceptName"] = "Definition of the concept"`++ - If no `structure` is specified and the name isn’t a known structure class, it implicitly refines `native.Text`.++- Full form (`ConceptBlueprint` from `pipelex/core/concept_factory.py`):++ - `definition: str` — required.++ - `structure: Optional[str]` — name of a registered `StructuredContent` subclass (typically defined in `pipelex_libraries/pipelines/<domain>.py`). If provided, it must exist in the class registry.++ - `refines: Union[str, List[str]]` — concept codes or names it refines. May be a single string or a list. Unqualified names are prefixed with the current `domain` automatically.++ - `domain: Optional[str]` — not required when provided via the library loader.++### Pipes: overall rules++Each entry under `pipe` is a dict that will be converted into a typed `PipeBlueprint` (see `pipelex/core/pipe_blueprint.py` for the base and factories in `pipe_controllers/` and `pipe_operators/`). Use the new format:++- `type: str` — the pipe class name, e.g., `PipeLLM`, `PipeOcr`, `PipeSequence`, `PipeParallel`, `PipeCondition`, `PipeFunc`, `PipeImgGen`.++- `definition: str` — human-readable description of the pipe’s responsibility.++- `inputs: Optional[Dict[str, str]]` — map of variable name -> concept code or name. Unqualified names are prefixed with `domain`.++- `output: str` — target concept code or name. Unqualified names are prefixed with `domain`.++Important:++- For controller pipes (`PipeSequence`, `PipeParallel`, `PipeCondition`), the parent pipe’s `inputs` must include all variables required by its sub-pipes or conditional branches that come from the initial working memory (i.e., not produced by previous steps).++- Multiplicity: some pipes and sub-steps can produce multiple outputs (see below). Plan result names accordingly.++### Pipe-specific blueprint schemas++Below are the additional fields expected per type, on top of the base fields above (`type`, `definition`, `inputs`, `output`). All fields are validated by their blueprint classes.++#### PipeLLM (operators/pipe_llm_factory.py)++- Prompt sources (choose at most one per group):++ - System prompt: one of `system_prompt`, `system_prompt_name`, `system_prompt_template`, `system_prompt_template_name`.++ - User prompt: one of `prompt`, `prompt_name`, `prompt_template`, `template_name`.++- LLM selection:++ - `llm: Optional[LLMSettingOrPresetId]` — model for text.++ - `llm_to_structure: Optional[LLMSettingOrPresetId]` — model for object structuring.++- Structuring:++ - `structuring_method: Optional[StructuringMethod]`++ - `prompt_template_to_structure: Optional[str]`++ - `system_prompt_to_structure: Optional[str]`++- Multiplicity:++ - `nb_output: Optional[int]` — exact number of outputs; mutually exclusive with `multiple_output`.++ - `multiple_output: Optional[bool]` — allow variable number of outputs.++Behavioral notes:++- If no system prompt is provided, the domain’s `system_prompt` (from the root blueprint) is used if available.++- Inputs that are `Image` concepts are automatically exposed as user images to the LLM; other inputs become text blocks.++- Prompt template insertion style for the agent:++ - Use block insertion (equivalent to `@name`) for long, multi-line stuff.++ - Use inline insertion (equivalent to `$name`) for short tokens embedded in sentences.++ - Do not synthesize literal markers; provide structured prompt fields and let the runtime render appropriately.++#### PipeOcr (operators/pipe_ocr_factory.py)++- Required input variable name: `ocr_input`.++- The `ocr_input` concept must be `Image` or `PDF`.++- Additional options:++ - `ocr_platform: Optional[OcrPlatform]` — defaults to Mistral.++ - `page_images: bool` — include detected images.++ - `page_image_captions: bool` — caption images.++ - `page_views: bool` — include rendered page views.++ - `page_views_dpi: Optional[int]` — DPI for page views; defaults to config.++- Output concept should be `Page`. The operational result is a list of `Page` items.++#### PipeSequence (controllers/pipe_sequence_factory.py)++- `steps: List[SubPipeBlueprint]`++- `SubPipeBlueprint` fields (controllers/sub_pipe_factory.py):++ - `pipe: str` — referenced pipe code to execute.++ - `result: Optional[str]` — name to assign to this step’s output in working memory.++ - Multiplicity (choose at most one): `nb_output: Optional[int]`, `multiple_output: Optional[bool]`.++ - Batch (both or none): `batch_over: Union[bool, str]` and `batch_as: Optional[str]`.++Rules:++- If `batch_over` is provided, `batch_as` is required; if `batch_as` is provided, `batch_over` is required.++- When batching, the step runs in parallel over each element of the specified list, and the `result` becomes a list.++#### PipeParallel (controllers/pipe_parallel_factory.py)++- `parallels: List[SubPipeBlueprint]` — each must specify `result`.++- `add_each_output: bool = True` — whether to add each parallel result separately to memory.++- `combined_output: Optional[str]` — optional combined output concept code; if provided without a domain prefix, it’s prefixed automatically.++Rules:++- At least one of `add_each_output` or `combined_output` must be set appropriately (cannot disable both behaviors).++#### PipeCondition (controllers/pipe_condition_factory.py)++- Choose exactly one of:++ - `expression: Optional[str]` — direct expression.++ - `expression_template: Optional[str]` — Jinja2 template expression.++- `pipe_map: Dict[str, str]` — map of evaluated expression values to pipe codes to execute.++- `default_pipe_code: Optional[str]` — optional fallback pipe code if no key matches.++- `add_alias_from_expression_to: Optional[str]` — if set, store the evaluated expression under this variable name in working memory.++### Input and output concepts resolution++- Unqualified concept names in `inputs` and `output` are automatically prefixed with the current `domain` during blueprint validation (see `PipeBlueprint.add_domain_prefix`).++- For `PipeParallel.combined_output`, the same domain-prefixing rule applies.++### Checklist for the agent++- Define `domain` and optional domain-level prompts (`system_prompt`, `system_prompt_to_structure`, `prompt_template_to_structure`).++- Create `concept` entries using either quick string form or full `ConceptBlueprint` with `definition`, optional `structure`, and `refines`.++- For each pipe under `pipe`:++ - Provide `type`, `definition`, `inputs`, `output`.++ - Add type-specific fields (see sections above).++ - Ensure controller pipes include all upstream inputs needed by nested steps/branches.++ - Set multiplicity (`nb_output`/`multiple_output`) and batch settings where needed.++ - For OCR, use `ocr_input` with `Image` or `PDF`, and set `output` to `Page`.++- Validate mentally that every referenced sub-pipe exists and every input it requires is either provided by the parent `inputs` or produced by a previous step.++### Minimal skeleton (object form)++The agent should generate a structure similar to the following (illustrative, not exhaustive):++```json+{+ "domain": "my_domain",+ "definition": "What this domain covers",+ "system_prompt": "Global system prompt for LLM pipes (optional)",+ "concept": {+ "UserQuery": "Natural language question from a user",+ "Answer": { "definition": "Answer text" }+ },+ "pipe": {+ "answer_question": {+ "type": "PipeLLM",+ "definition": "Produce an answer from a question and context",+ "inputs": { "question": "UserQuery", "context": "Text" },+ "output": "Answer",+ "prompt_template": "Analyze the question and context, then answer concisely.\n\n@question\n\n@context"+ }+ }+}+```diff --git a/.pipelex/create/draft_pipelines.md b/.pipelex/create/draft_pipelines.mdnew file mode 100644index 000000000..3217d4255--- /dev/null+++ b/.pipelex/create/draft_pipelines.md@@ -0,0 +1,259 @@+# Drafting Pipelex Pipelines++This guide explains how to draft pipelines in natural language using markdown notation before translating them into formal Pipelex definitions.++## Core Concept: WorkingMemory++Every pipeline execution maintains a `WorkingMemory` that acts as a shared data space. When a pipeline starts, the initial inputs are placed in this memory. As each pipe executes, its output is added to the memory with a specific name (the `result` name). Subsequent pipes can access any data in the memory by referencing these names.++**Key Rules**:++- When a pipe produces an output with name `X`, any later pipe can use `X` as an input. The names must match exactly.+- Controller pipes (PipeSequence, PipeParallel, PipeCondition) automatically inherit the input requirements of their nested operators - you don't need to specify their inputs explicitly.++## Drafting Workflow++1. **Start with the goal**: What does this pipeline accomplish?+2. **Define inputs**: What data does it need to start?+3. **Narrate the flow**: Describe each step and how data flows through WorkingMemory+4. **Specify outputs**: What final result emerges?++## Pipeline Draft Structure++### Header+```markdown+# Pipeline: [pipeline_name]+Domain: [domain_name]+Purpose: [What this pipeline does]+```++### Inputs Section+```markdown+## Inputs+- `document` : PDF document to analyze+- `criteria` : Analysis criteria to apply+```++### Flow Section++Describe the pipeline flow using concise step notation:++```markdown+## Flow++WorkingMemory starts with: {document, criteria}++### Step 1: Process document content+LLM[process]: (document) → processed_content++WorkingMemory now contains: {document, criteria, processed_content}++### Step 2: Apply analysis criteria to processed content+LLM[synthesize]: (processed_content, criteria) → analysis_report++WorkingMemory now contains: {document, criteria, processed_content, analysis_report}+```++### Output Section+```markdown+## Output+Returns: `analysis_report` - Comprehensive analysis based on criteria+```++## Pipe Types and Notation++## Operator Notations++**Important**: During drafting, DO NOT write actual prompts or prompt templates. Focus on describing what each operator does and how data flows. Prompts will be written during implementation.++### LLM Operations (including vision, to generate text from text and images)+```markdown+LLM[purpose]: (inputs) → output++Examples:+LLM[summarize]: (long_text) → summary+LLM[analyze]: (document, criteria) → analysis+```++### OCR Operations (to extract text and images from PDF pages, including full page views)+```markdown+OCR: (pdf_document) → pages_list+```++### Image Generation (to generate images from text prompts)+```markdown+ImgGen: (prompt_text) → generated_image+```++### Function Operations (to call external python functions)+```markdown+Func[function_name]: (data) → result+```++### Sequential Processing (to run steps in sequence, use it when steps need results from previous steps)+```markdown+Step 1: Extract key information from document+extract_info(document) → key_info++Step 2: Analyze extracted information+analyze(key_info) → analysis++Step 3: Generate report with original document context+generate_report(analysis, document) → report+```++### Batch Processing (to process each item in a list independently and in parallel)+```markdown+Process each item in documents:+analyze_document(document) → analysis++Collect all results → document_analyses+```++### Parallel Processing (to run different pipes in parallel from the same memory, use it when steps can be run independently, each with a copy of the working memory)+```markdown+Parallel execution:++Branch A: Extract technical details+extract_technical(document) → technical_info++Branch B: Extract business insights+extract_business(document) → business_info++Combine: Merge analysis results+merge_insights(technical_info, business_info) → comprehensive_analysis+```++### Conditional Processing (to run different pipes based on a condition)+```markdown+Route based on expression(document_type):++Case "technical": Handle technical documents+analyze_technical(document) → analysis++Case "legal": Handle legal documents+analyze_legal(document) → analysis++Default: Handle general documents+analyze_general(document) → analysis+```++## Memory Flow Examples++### Example 1: Document Analysis Pipeline+```markdown+# Pipeline: analyze_document+Domain: document_analysis+Purpose: Extract and analyze key information from documents++## Inputs+- `document` : PDF document to analyze++## Flow++WorkingMemory: {document}++### Step 1: Extract content from PDF+OCR: (document) → pages++WorkingMemory: {document, pages}++### Step 2: Summarize each page in parallel+Process each page in pages:+LLM[summarize]: (page) → summary++Collect all results → summaries++WorkingMemory: {document, pages, summaries}++### Step 3: Create comprehensive analysis from summaries+LLM[synthesize]: (summaries) → analysis++WorkingMemory: {document, pages, summaries, analysis}++## Output+Returns: `analysis`+```++### Example 2: Conditional Processing+```markdown+# Pipeline: smart_responder+Domain: customer_service+Purpose: Route and respond to customer queries++## Inputs+- `query` : Customer question+- `context` : Available context information++## Flow++WorkingMemory: {query, context}++### Step 1: Classify query into support category+LLM[classify]: (query) → category++WorkingMemory: {query, context, category}++### Step 2: Route to appropriate support handler+Route based on category:++Case "technical": Technical support handler+LLM[technical_support]: (query, context) → response++Case "billing": Billing support handler+LLM[billing_support]: (query, context) → response++Default: General support handler+LLM[general_support]: (query) → response++WorkingMemory: {query, context, category, response}++## Output+Returns: `response`+```++## Best Practices++1. **Use semantic variable names**: Name variables based on what they represent, not how they're used. Use `document` instead of `input_document`, `text` instead of `input_text`, `analysis` instead of `result_1`.++2. **Name consistently**: Once you name a result (e.g., `analysis`), use that exact name when referencing it later.++3. **Show memory state**: After complex steps, show what's in WorkingMemory to track data flow.++4. **Be explicit about inputs**: For each operator step, clearly indicate which items from WorkingMemory are being used.++5. **Use descriptive names**: Choose names that clearly describe the content: `summaries`, `technical_info`, `user_preferences`.++6. **Concise step descriptions**: Each step should have a clear title that describes what it accomplishes. Avoid repeating the same information in multiple places.++7. **No prompts during drafting**: Focus on describing WHAT each step does, not HOW (the actual prompts). Prompts are implementation details added later.++8. **Block vs Inline references**:+ - Use `@variable_name` notation for block insertions (multi-line content)+ - Use `$variable_name` notation for inline insertions (single values in sentences)++9. **Batch operations**: When processing lists, clearly indicate:+ - What list is being iterated (`batch_over`)+ - What each item is called during processing (`batch_as`)+ - What the collected results are named++## Validation Checklist++Before finalizing your draft:++- [ ] Every input used by an operator pipe exists in WorkingMemory (either from initial inputs or previous steps)+- [ ] Every result name is unique (no overwriting unless intentional)+- [ ] The final output exists in WorkingMemory+- [ ] Names are consistent throughout (no typos or variations)+- [ ] Variable names are semantic (describe what they are, not how they're used)+- [ ] The flow clearly shows data transformation from inputs to outputs+- [ ] Each step has a concise, non-repetitive description++## From Draft to Implementation++Once your markdown draft is complete, it can be translated into:+1. A formal TOML pipeline definition with explicit pipe declarations+2. Python structure classes for complex data types (when needed)+3. Validated, executable Pipelex pipeline code++The narrative draft serves as both documentation and a blueprint for implementation, making the pipeline's logic transparent and maintainable.diff --git a/AGENTS.md b/AGENTS.mdindex 50a542992..71e3063d5 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -30,7 +30,7 @@ This python >=3.10 code is in the `pipelex` directory.## Testing- Always test with `make codex-tests` -> it runs pytest on our `tests/` directory using proper settings-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all` under the hood)## PR Instructionsdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex abad2c5dc..8930716e2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -163,7 +163,7 @@ Simplified input memory:### Documentation & Development Experience- **Cursor Rules Enhancement**: Comprehensive pipe controller documentation covering `PipeSequence`, `PipeCondition`, `PipeBatch`, and `PipeParallel`, improved PipeOperator documentation for `PipeLLM`, `PipeOCR`-- **Pipeline Validation CLI**: Enhanced `pipelex validate` command with better error reporting and validation coverage+- **Pipeline Validation CLI**: Enhanced `pipelex validate all` command with better error reporting and validation coverage- **Improved Error Messages**: Better formatting and context for pipeline configuration errors### Changed@@ -480,7 +480,7 @@ is_reporting_enabled = true- Use a minimum version for `uv` instead of a fixed version- Implement `AGENTS.md` for Codex- Add tests for some of the `tools.misc`-- pipelex cli: Rename `pipelex run-setup` to `pipelex validate`+- pipelex cli: Rename `pipelex run-setup` to `pipelex validate all`## [v0.2.8] - 2025-05-28diff --git a/CLAUDE.md b/CLAUDE.mdindex ed571e800..2a57cb258 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -82,7 +82,7 @@ class TestFooBar:- Always test with `make t` -> it runs pytest using proper settings- If some pytest tests fail, run pytest on the failed ones with the required verbosity to diagnose the issue-- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading+- If all unit tests pass, run `make validate` -> it runs a minimal version of our app with just the inits and data loading (`pipelex validate all` under the hood)## PR Instructionsdiff --git a/Makefile b/Makefileindex 9add6d431..938d01179 100644--- a/Makefile+++ b/Makefile@@ -153,7 +153,7 @@ update: envvalidate: env$(call PRINT_TITLE,"Running setup sequence")- $(VENV_PIPELEX) validate -c pipelex/libraries+ $(VENV_PIPELEX) validate all -c pipelex/librariesbuild: env$(call PRINT_TITLE,"Building the wheels")@@ -419,10 +419,6 @@ docs-deploy: env### SHORTHANDS##########################################################################################-check-unused-imports: env- $(call PRINT_TITLE,"Checking for unused imports without fixing")- $(VENV_RUFF) check --select=F401 --no-fix .-c: format lint pyright mypy@echo "> done: c = check"diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.mdnew file mode 100644index 000000000..4fd40f9cc--- /dev/null+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md@@ -0,0 +1,289 @@+# Pipeline Creation++Pipelex provides powerful tools to automatically generate pipeline blueprints from natural language requirements. This feature leverages AI to translate your ideas into working pipeline code, dramatically speeding up development.++## Overview++The pipeline creation system works in several stages:++1. **Draft Generation**: Convert natural language requirements into structured markdown drafts+2. **Blueprint Creation**: Transform drafts into formal pipeline blueprints (TOML format)+3. **Validation**: Automatically validate and fix common issues+4. **Implementation**: Generate executable pipeline code++## Core Commands++### Build Blueprint++Generate a complete pipeline blueprint from requirements:++```bash+pipelex build blueprint PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]+```++**Example:**+```bash+pipelex build blueprint gen_photopposite \+ --requirements "Take a photo as input, analyze its content and its most important feature, imagine the opposite to that feature, render it as a photo" \+ -c pipelex/libraries \+ -o pipelex/libraries/pipelines/temp/result_4/photopposite \+ --validate+```++**Options:**+- `--domain, -d`: Domain for the pipeline (default: "wip_domain")+- `--requirements, -r`: Requirements text directly+- `--file, -f`: Path to file containing requirements+- `--output, -o`: Output path for generated files+- `--validate`: Validate the generated blueprint+- `--config-folder-path, -c`: Path to libraries folder++### Build Draft++Generate just a pipeline draft (intermediate step):++```bash+pipelex build draft PIPELINE_NAME --requirements "REQUIREMENTS_TEXT" [OPTIONS]+```++**Options:**+- `--raw`: Generate raw text draft instead of structured draft+- All other options same as blueprint command++### Validate Blueprint++Validate an existing blueprint file:++```bash+pipelex validate blueprint BLUEPRINT_PATH [OPTIONS]+```++**Example:**+```bash+pipelex validate blueprint pipelex/libraries/pipelines/temp/result_3/photopposite.toml \+ -c pipelex/libraries+```++**Options:**+- `--config-folder-path, -c`: Path to libraries folder+- `--fix/--no-fix, -f/-F`: Enable/disable automatic error fixing (default: enabled)++## Complete Workflow++### 1. Requirements Analysis++Start with clear, specific requirements:++```text+Take a photo as input, analyze its content and its most important feature,+imagine the opposite to that feature, render it as a photo+```++### 2. Draft Generation++The system first creates a structured markdown draft:++```markdown+# Pipeline: gen_photopposite+Domain: image_processing+Purpose: Generate opposite version of photo's main feature++## Inputs+- `photo` : Input photo to analyze and transform++## Flow++WorkingMemory starts with: {photo}++### Step 1: Analyze photo content+LLM[analyze]: (photo) → analysis++WorkingMemory now contains: {photo, analysis}++### Step 2: Generate opposite concept+LLM[conceptualize]: (analysis) → opposite_concept++WorkingMemory now contains: {photo, analysis, opposite_concept}++### Step 3: Render opposite photo+ImgGen: (opposite_concept) → opposite_photo++WorkingMemory now contains: {photo, analysis, opposite_concept, opposite_photo}++## Output+Returns: `opposite_photo` - Photo showing opposite of main feature+```++### 3. Blueprint Creation++The draft is converted into a formal TOML blueprint:++```toml+domain = "image_processing"+definition = "Generate opposite version of photo's main feature"++[concept]+PhotoAnalysis = "Analysis of photo content and main features"+OppositePhotoPrompt = "Prompt for generating opposite version of photo"++[pipe.gen_photopposite]+PipeSequence = "Generate opposite version of photo's main feature"+output = "Image"+steps = [+ { pipe = "analyze_photo", result = "analysis" },+ { pipe = "create_opposite_concept", result = "opposite_concept" },+ { pipe = "render_opposite", result = "opposite_photo" }+]++[pipe.analyze_photo]+PipeLLM = "Analyze photo content and identify main feature"+inputs = { photo = "Image" }+output = "PhotoAnalysis"+prompt_template = """+Analyze this photo and identify its most important visual feature.++@photo++Focus on the dominant element, color, mood, or characteristic that defines this image.+"""++[pipe.create_opposite_concept]+PipeLLM = "Create concept for opposite version"+inputs = { analysis = "PhotoAnalysis" }+output = "OppositePhotoPrompt"+prompt_template = """+Based on this photo analysis, create a detailed prompt for generating the opposite version.++@analysis++Describe what the opposite would look like, focusing on reversing the main feature while maintaining photo realism.+"""++[pipe.render_opposite]+PipeImgGen = "Generate the opposite photo"+inputs = { opposite_concept = "OppositePhotoPrompt" }+output = "Image"+```++### 4. Validation and Fixing++The system automatically:++- **Validates syntax**: Ensures TOML structure is correct+- **Checks concepts**: Verifies all concepts are properly defined+- **Validates inputs**: Ensures all pipe inputs are available+- **Fixes errors**: Automatically adds missing inputs or corrects common issues+- **Dry runs**: Tests pipeline execution without making API calls++## Generated Files++When you run the build command, several files are created:++```+output_path_base_draft.md # Original markdown draft+output_path_base_rough.toml # Initial blueprint+output_path_base_rough.json # Blueprint in JSON format+output_path_base_fixed.toml # Validated and fixed blueprint (if --validate used)+output_path_base_fixed.json # Fixed blueprint in JSON format+```++## Best Practices++### Writing Requirements++**Good requirements:**+```text+Analyze a PDF document, extract key financial data, and generate a summary report with charts+```++**Better requirements:**+```text+Take a PDF financial statement as input, use OCR to extract text and tables,+identify revenue, expenses, and profit figures, then generate a structured+summary report with visual charts showing trends+```++### Domain Selection++Choose descriptive domain names:+- `financial_analysis` instead of `finance`+- `document_processing` instead of `docs`+- `image_generation` instead of `images`++### Output Path Organization++Organize generated pipelines:+```+pipelex/libraries/pipelines/+ temp/ # Temporary/experimental pipelines+ result_1/+ result_2/+ domain_name/ # Production pipelines+ pipeline_name.toml+ structures.py+```++## Advanced Features++### Custom Templates++The system uses built-in templates from:+- `.pipelex/create/draft_pipelines.md` - Drafting guidelines+- `.pipelex/create/build_pipelines.md` - Blueprint creation rules++### Error Fixing++The validation system can automatically fix:+- Missing input variables+- Incorrect concept references+- Malformed pipe definitions+- Input/output mismatches++### Integration with Existing Libraries++Generated blueprints automatically integrate with:+- Native concepts (`Text`, `Image`, `PDF`, etc.)+- Library concepts (`documents.TextAndImages`, etc.)+- Custom domain concepts++## Troubleshooting++### Common Issues++**"Missing input variable" error:**+- Usually fixed automatically with `--validate`+- Check that all pipe inputs are available in WorkingMemory++**"Invalid concept" error:**+- Ensure concept names use PascalCase+- Don't redefine native concepts+- Avoid plurals in concept names++**"Pipe not found" error:**+- Check that all referenced pipes exist in the blueprint+- Verify pipe names match exactly++### Getting Help++```bash+# Validate your entire setup+pipelex validate all -c pipelex/libraries++# Test a specific pipe+pipelex validate pipe your_pipe_name -c pipelex/libraries++# Check blueprint structure+pipelex validate blueprint your_blueprint.toml -c pipelex/libraries --no-fix+```++## Next Steps++After creating your blueprint:++1. **Review the generated code** - Check that it matches your requirements+2. **Create structure classes** - Add Python classes for custom concepts if needed+3. **Test the pipeline** - Run validation and dry runs+4. **Execute the pipeline** - Use `execute_pipeline()` to run it with real data+5. **Iterate and improve** - Refine based on results++The pipeline creation system makes it easy to go from idea to working code in minutes, while maintaining the full power and flexibility of Pipelex pipelines.diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.mdindex e621067ec..633ea7850 100644--- a/docs/pages/configuration/config-technical/library-config.md+++ b/docs/pages/configuration/config-technical/library-config.md@@ -61,7 +61,7 @@ exported_templates_path = "pipelex_libraries/templates"Use the CLI command to initialize libraries:```bash-pipelex init-libraries+pipelex init libraries```This will:@@ -122,7 +122,7 @@ The library system includes specific error types:2. **Validation**:- - Run `pipelex validate` after making changes+ - Run `pipelex validate all` after making changes- Check for domain consistency- Verify concept relationshipsdiff --git a/docs/pages/configuration/index.md b/docs/pages/configuration/index.mdindex 714539c88..9e8cab9df 100644--- a/docs/pages/configuration/index.md+++ b/docs/pages/configuration/index.md@@ -5,13 +5,13 @@Pipelex uses a TOML-based configuration system. The main configuration file `pipelex.toml` must be located at the root of your project. You can create this file by running:```bash-pipelex init-config+pipelex init config```💡 **Important Notes**:-1. `pipelex init-config` creates a **template** configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.-2. Using `pipelex init-config --reset` will **overwrite** your existing `pipelex.toml` file without warning. Make sure to backup your configuration before using this flag.+1. `pipelex init config` creates a **template** configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.+2. Using `pipelex init config --reset` will **overwrite** your existing `pipelex.toml` file without warning. Make sure to backup your configuration before using this flag.For a complete list of all possible configuration options, refer to the configuration group documentation below.@@ -69,5 +69,5 @@ NB: The run_mode unit_test is used for testing purposes.1. **Version Control**: Include your base `pipelex.toml` in version control2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings3. **Documentation**: Comment any custom settings for team reference-4. **Validation**: Run `pipelex validate` after making configuration changes+4. **Validation**: Run `pipelex validate all` after making configuration changes5. **Gitignore**: Add local and sensitive override files to `.gitignore`diff --git a/docs/pages/installation/index.md b/docs/pages/installation/index.mdindex b2ee24c26..183ed10ef 100644--- a/docs/pages/installation/index.md+++ b/docs/pages/installation/index.md@@ -40,7 +40,7 @@ All the secret keys used by `pipelex` are specified in the `.env.example` file.In order to set the pipelex configuration files, you need to run 2 commands using the CLI (we recommend to run it at the root of your project):-- `pipelex init-libraries`: This will create a `pipelex_libraries` folder, with the base llm configuration and the base pipelines.+- `pipelex init libraries`: This will create a `pipelex_libraries` folder, with the base llm configuration and the base pipelines.This is the directory where you should add your pipelines.The structure is like this:@@ -58,7 +58,7 @@ The structure is like this:Learn more about pipelex_libraries in our [Libraries documentation](../build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md)-- `pipelex init-config`: This cli command will create a `pipelex.toml` file at the root of the project, with basic configuration. This configuration file gathers all configuration for feature flags, logging, cost reporting, and so on... Learn more in our [Configuration documentation](../configuration/index.md)+- `pipelex init config`: This cli command will create a `pipelex.toml` file at the root of the project, with basic configuration. This configuration file gathers all configuration for feature flags, logging, cost reporting, and so on... Learn more in our [Configuration documentation](../configuration/index.md)💡 _Any troubles? Have a look at our [Cookbook](https://github.com/Pipelex/pipelex-cookbook)! and come ask for help on our [Discord](https://go.pipelex.com/discord)_diff --git a/docs/pages/tools/migration.md b/docs/pages/migration/index.mdsimilarity index 86%rename from docs/pages/tools/migration.mdrename to docs/pages/migration/index.mdindex 21927e078..53f686a34 100644--- a/docs/pages/tools/migration.md+++ b/docs/pages/migration/index.md@@ -2,22 +2,22 @@This document provides guidance for migrating between different versions of Pipelex that introduce breaking changes.-## Version 0.6.11: Concept Definition Syntax Change+## Concepts Migration### Breaking Change-In version 0.6.11, we changed the syntax for defining concepts in TOML library files. The key name for concept definitions has changed from `Concept` to `definition`.+We changed the syntax for defining concepts in TOML library files. The key name for concept definitions has changed from `Concept` to `definition`.### What Changed-**Old syntax (before v0.6.11):**+**Old syntax:**```toml[concept.MyConceptName]Concept = "Description of the concept"refines = "BaseConceptName"```-**New syntax (v0.6.11+):**+**New syntax:**```toml[concept.MyConceptName]definition = "Description of the concept"@@ -27,6 +27,7 @@ refines = "BaseConceptName"### Why This ChangeThis change improves consistency in our TOML schema by:+- Using more descriptive field names (`definition` instead of `Concept`)- Aligning with the internal `ConceptBlueprint` data structure- Making the TOML files more self-documenting and intuitive@@ -46,6 +47,7 @@ pipelex migrate```The migration command will:+- Find all `.toml` files in your configured pipelines directory- Replace `Concept =` with `definition =` in concept definitions- Preserve all other formatting and comments@@ -57,7 +59,7 @@ If you prefer to migrate manually or need to handle special cases:1. **Locate your TOML files**: Find all pipeline library files (typically in your configured pipelines directory)2. **Update concept definitions**: Change `Concept =` to `definition =` in all `[concept.ConceptName]` sections-3. **Validate syntax**: Run `pipelex validate` to ensure your files are correctly formatted+3. **Validate syntax**: Run `pipelex validate all` to ensure your files are correctly formatted### Examples@@ -101,7 +103,7 @@ After migration, validate your pipeline files:```bash# Validate all pipeline files-pipelex validate+pipelex validate all# Run a specific pipeline to testpipelex run your-pipeline-name@@ -110,11 +112,13 @@ pipelex run your-pipeline-name### Troubleshooting**Error: "ConceptBlueprint validation failed"**+- Check that all `Concept =` entries have been changed to `definition =`- Ensure no typos were introduced during migration- Verify TOML syntax is still valid**Error: "Failed to load TOML file"**+- Check file permissions- Ensure TOML syntax is valid (no missing quotes, brackets, etc.)- Review any custom modifications that might conflict@@ -122,25 +126,26 @@ pipelex run your-pipeline-name### RollbackIf you need to rollback the migration:+1. Stop using the new version of Pipelex2. Restore from the `.backup` files created during migration3. Or manually change `definition =` back to `Concept =`-## Version 0.3.0: Pipe Definition Syntax Change+## Pipes Migration### Breaking Change-Starting in version 0.3.0, we changed the syntax for defining pipes in TOML library files. The pipe definition format has been restructured to use explicit `type` and `definition` fields instead of the implicit `PipeClassName = "description"` format.+We changed the syntax for defining pipes in TOML library files. The pipe definition format has been restructured to use explicit `type` and `definition` fields instead of the implicit `PipeClassName = "description"` format.### What Changed-**Old syntax (before v0.3.0):**+**Old syntax:**```toml[pipe.my_pipe_name]SomePipeClass = "Description of what this pipe does"```-**New syntax (v0.3.0+):**+**New syntax:**```toml[pipe.my_pipe_name]type = "SomePipeClass"@@ -150,6 +155,7 @@ definition = "Description of what this pipe does"### Why This ChangeThis change improves consistency and clarity by:+- Making the pipe class type explicit and separate from its description- Aligning pipe definitions with concept definitions (both use `definition`)- Improving readability and maintainability of TOML files@@ -170,6 +176,7 @@ pipelex migrate```The migration command will:+- Find all `.toml` files in your configured pipelines directory- Convert `PipeClassName = "description"` to `type = "PipeClassName"` and `definition = "description"`- Preserve all other formatting and comments@@ -185,7 +192,7 @@ If you prefer to migrate manually:- Replace `PipeClassName = "description"` with:- `type = "PipeClassName"`- `definition = "description"`-3. **Validate syntax**: Run `pipelex validate` to ensure your files are correctly formatted+3. **Validate syntax**: Run `pipelex validate all` to ensure your files are correctly formatted### Examples@@ -247,7 +254,7 @@ After migration, validate your pipeline files:```bash# Validate all pipeline files-pipelex validate+pipelex validate all# Run a specific pipeline to testpipelex run your-pipeline-namediff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.mdindex a447a4020..4da126632 100644--- a/docs/pages/quick-start/index.md+++ b/docs/pages/quick-start/index.md@@ -14,7 +14,7 @@ For illustration purposes, let's build **a character generator**. Each example r### Write your first pipelineFirst, create a `.toml` library file in the `pipelex_libraries/pipelines` directory to store your pipe definition.-Run `pipelex init-libraries` to create this directory if it doesn't exist. For now, keep all your pipeline definitions inside that folder only.+Run `pipelex init libraries` to create this directory if it doesn't exist. For now, keep all your pipeline definitions inside that folder only.`character.toml````tomldiff --git a/docs/pages/tools/cli.md b/docs/pages/tools/cli.mdindex 15fb31849..2fe5e781b 100644--- a/docs/pages/tools/cli.md+++ b/docs/pages/tools/cli.md@@ -4,76 +4,47 @@ The Pipelex CLI provides a command-line interface for managing and interacting w## Available Commands-### `pipelex init-libraries`+### Init group-Initialize Pipelex libraries in the current directory.+Initialize project assets.```bash-pipelex init-libraries [--overwrite/-o]+pipelex init libraries [DIRECTORY] [--overwrite/-o]+pipelex init config [--reset/-r]```-**Options:**-- `--overwrite`, `-o`: If set, existing library files will be overwritten. Otherwise, only missing files will be created.+### Validate group-### `pipelex init-config`--Initialize Pipelex configuration in the current directory.--```bash-pipelex init-config [--reset/-r]-```--**Options:**-- `--reset`, `-r`: If set, existing configuration file (pipelex.toml) will be overwritten. Otherwise, the command will warn if the file already exists.--### `pipelex validate`--Run the setup sequence to validate your Pipelex configuration and pipelines.+Validate configuration and pipelines.```bash-pipelex validate+pipelex validate all [-c/--config-folder-path PATH]+pipelex validate pipe PIPE_CODE [-c/--config-folder-path PATH]```-This command:+### Show group-1. Exports libraries-2. Validates the configuration-3. Ensures all pipelines are properly set up--### `pipelex show-config`--Display the current Pipelex configuration.+Inspect configuration and pipes.```bash-pipelex show-config+pipelex show config+pipelex show pipes [-c/--config-folder-path PATH]+pipelex show pipe PIPE_CODE [-c/--config-folder-path PATH]```-Shows the complete configuration for your project, including all settings and parameters.-See more in our [Configuration documentation](../configuration/index.md)+### Migrate group-### `pipelex list-pipes`--List all available pipes in your project.+Migrate TOML files to the new syntax.```bash-pipelex list-pipes+pipelex migrate run [-c/--config-folder-path PATH] [--dry-run] [--backups/--no-backups]```-Displays a table of pipes organized by domain, showing:--- Code: The unique identifier for each pipe-- Definition: Description of the pipe's purpose-- Input: Required input parameters and their concept codes-- Output: The output concept code--The output is formatted as tables grouped by domain, with concept codes simplified when they belong to the current domain.-## Usage Tips-1. Always run `pipelex validate` after making changes to your configuration or pipelines-2. Use `pipelex show-config` to debug configuration issues+1. Always run `pipelex validate all` after making changes to your configuration or pipelines+2. Use `pipelex show config` to debug configuration issues3. When initializing a new project:-- - Start with `pipelex init-config`- - Then run `pipelex init-libraries`- - Finally, validate your setup with `pipelex validate`+ - Start with `pipelex init config`+ - Then run `pipelex init libraries`+ - Finally, validate your setup with `pipelex validate all`diff --git a/mkdocs.yml b/mkdocs.ymlindex 178c3ecb8..357eb6e0e 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -109,6 +109,7 @@ nav:- Kick off a Pipeline Project: pages/build-reliable-ai-workflows-with-pipelex/kick-off-a-knowledge-pipeline-project.md- Define Your Concepts: pages/build-reliable-ai-workflows-with-pipelex/define_your_concepts.md- Design and Run Pipelines: pages/build-reliable-ai-workflows-with-pipelex/design_and_run_pipelines.md+ - Pipeline Creation: pages/build-reliable-ai-workflows-with-pipelex/pipeline-creation.md- Pipe Operators:- Overview: pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/index.md- PipeLLM: pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeLLM.md@@ -146,7 +147,7 @@ nav:- Tools:- CLI: pages/tools/cli.md- Logging: pages/tools/logging.md- - Migration: pages/tools/migration.md+ - Migration: pages/migration/index.md- Advanced Customizations:- Overview: pages/advanced-customization/index.md- Secrets Provider: pages/advanced-customization/secrets-provider-injection.mddiff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex d7461c13f..8ca16df2e 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -1,57 +1,12 @@-import asyncio-import os-import shutil-from pathlib import Path-from typing import Annotated, Optional+from typing import Optionalimport typerfrom click import Command, Contextfrom typer.core import TyperGroupfrom typing_extensions import override-from pipelex import log, pretty_print-from pipelex.exceptions import PipelexCLIError, PipelexConfigError-from pipelex.hub import get_pipe_provider, get_pipeline_tracker, get_required_pipe-from pipelex.libraries.library_config import LibraryConfig-from pipelex.pipe_works.pipe_dry import dry_run_all_pipes, dry_run_single_pipe-from pipelex.pipelex import Pipelex-from pipelex.tools.config.manager import config_manager-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import TomlMigrator, migrate_concept_syntax---def is_pipelex_libraries_folder(folder_path: str) -> bool:- """Check if the given folder path contains a valid pipelex libraries structure.-- A valid pipelex libraries folder should contain the following subdirectories:- - pipelines- - llm_deck- - llm_integrations- - plugins- - templates-- Args:- folder_path: Path to the folder to check-- Returns:- True if the folder contains all required subdirectories, False otherwise- """- if not os.path.exists(folder_path) or not os.path.isdir(folder_path):- return False-- required_subdirs = [- "pipelines",- "llm_deck",- "llm_integrations",- "plugins",- "templates",- ]-- for subdir in required_subdirs:- subdir_path = os.path.join(folder_path, subdir)- if not os.path.exists(subdir_path) or not os.path.isdir(subdir_path):- return False-- return True+from pipelex.cli.commands import init_app, migrate_app, show_app, validate_app+from pipelex.cli.commands.build_cmd import build_appclass PipelexCLI(TyperGroup):@@ -65,6 +20,11 @@ class PipelexCLI(TyperGroup):return cmd+def main() -> None:+ """Entry point for the pipelex CLI."""+ app()++app = typer.Typer(add_completion=False,no_args_is_help=True,@@ -73,253 +33,8 @@ app = typer.Typer()[email protected]("init-libraries")-def init_libraries(- directory: Annotated[- str,- typer.Argument(help="Directory where to create the pipelex_libraries folder"),- ] = ".",- overwrite: Annotated[- bool,- typer.Option(- "--overwrite",- "-o",- help="Warning: If set, existing files will be overwritten.",- ),- ] = False,-) -> None:- """Initialize pipelex libraries in a pipelex_libraries folder in the specified directory.-- If overwrite is False, only create files that don't exist yet.- If overwrite is True, all files will be overwritten even if they exist.- """- try:- # Always create a pipelex_libraries folder in the specified directory- target_path = os.path.join(directory, "pipelex_libraries")-- # Create the target directory if it doesn't exist- os.makedirs(directory, exist_ok=True)-- # Create a LibraryConfig instance with the target path- library_config = LibraryConfig(config_folder_path=target_path)- library_config.export_libraries(overwrite=overwrite)-- if overwrite:- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (all files overwritten)")- else:- typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (only created non-existing files)")- except Exception as exc:- raise PipelexCLIError(f"Failed to initialize libraries at '{directory}': {exc}")--[email protected]("init-config")-def init_config(- reset: Annotated[- bool,- typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten."),- ] = False,-) -> None:- """Initialize pipelex configuration in the current directory."""- pipelex_template_path = os.path.join(config_manager.pipelex_root_dir, "pipelex_template.toml")- target_config_path = os.path.join(config_manager.local_root_dir, "pipelex.toml")-- if os.path.exists(target_config_path) and not reset:- typer.echo("Warning: pipelex.toml already exists. Use --reset to force creation.")- return-- try:- shutil.copy2(pipelex_template_path, target_config_path)- typer.echo(f"Created pipelex.toml at {target_config_path}")- except Exception as exc:- raise PipelexCLIError(f"Failed to create pipelex.toml: {exc}")---def validate(- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",-) -> None:- """Run the setup sequence."""- # Check if pipelex libraries folder exists- if not is_pipelex_libraries_folder(relative_config_folder_path):- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")- raise typer.Exit(1)-- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)- pipelex_instance.validate_libraries()- asyncio.run(dry_run_all_pipes())- log.info("Setup sequence passed OK, config and pipelines are validated.")---def dry_run_pipe(- pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",-) -> None:- """Dry run a single pipe."""- # Check if pipelex libraries folder exists- if not is_pipelex_libraries_folder(relative_config_folder_path):- typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")- typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")- raise typer.Exit(1)-- try:- # Initialize Pipelex- pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)- pipelex_instance.validate_libraries()-- # Run the single pipe dry run- asyncio.run(dry_run_single_pipe(pipe_code))- get_pipeline_tracker().output_flowchart()-- except Exception as exc:- typer.echo(f"❌ Error running dry run for pipe '{pipe_code}': {exc}")- raise typer.Exit(1)---def show_config() -> None:- """Show the pipelex configuration."""- try:- final_config = config_manager.load_config()- pretty_print(- final_config,- title=f"Pipelex configuration for project: {config_manager.get_project_name()}",- )- except Exception as exc:- raise PipelexConfigError(f"Error loading configuration: {exc}")---def list_pipes(- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "pipelex_libraries",-) -> None:- """List all available pipes."""- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)-- try:- get_pipe_provider().pretty_list_pipes()-- except Exception as exc:- raise PipelexCLIError(f"Failed to list pipes: {exc}")--[email protected]("show-pipe")-def show_pipe(- pipe_code: Annotated[- str,- typer.Argument(help="Pipeline code to show definition for"),- ],- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",-) -> None:- """Show pipe from the pipe library."""- Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)- pipe = get_required_pipe(pipe_code=pipe_code)- pretty_print(pipe, title=f"Pipe '{pipe_code}'")--[email protected]("migrate")-def migrate(- relative_config_folder_path: Annotated[- str,- typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path"),- ] = "./pipelex_libraries",- dry_run: Annotated[- bool,- typer.Option("--dry-run", help="Preview changes without applying them"),- ] = False,- backups: Annotated[- bool,- typer.Option("--backups/--no-backups", help="Create backup files before migration"),- ] = True,-) -> None:- """Migrate TOML files to new syntax (Concept = -> definition = and PipeClassName = -> type/definition)."""- config_path = Path(relative_config_folder_path)-- # Check if it's a pipelex libraries folder structure or just a directory with TOML files- if is_pipelex_libraries_folder(relative_config_folder_path):- pipelines_dir = config_path / "pipelines"- else:- # Assume it's a directory containing TOML files directly- pipelines_dir = config_path-- if not pipelines_dir.exists():- typer.echo(f"❌ Directory not found at '{pipelines_dir}'")- raise typer.Exit(1)-- try:- # Use the migration module (now handles both concepts and pipes)- result = migrate_concept_syntax(- directory=pipelines_dir,- create_backups=backups and not dry_run, # Create backups if enabled and not dry-run- dry_run=dry_run,- )-- # Handle any errors that occurred- for error in result.errors:- typer.echo(f"❌ {error}")-- if result.errors and not result.files_modified:- typer.echo("❌ Migration failed due to errors")- raise typer.Exit(1)-- typer.echo(f"Found {result.files_processed} TOML file(s) to check")-- if result.files_modified == 0:- typer.echo("✅ All TOML files are already using the new syntax")- return-- if dry_run:- # Show detailed preview for dry run- migrator = TomlMigrator()- for file_path in result.modified_files:- try:- with open(file_path, "r", encoding="utf-8") as f:- content = f.read()-- changes = migrator.get_migration_preview(content)- typer.echo(f"\n📄 {file_path.relative_to(pipelines_dir)}:")- for change in changes:- typer.echo(f" Line {change['line_number']}: {change['old_line']} → {change['new_line']}")- except Exception as e:- typer.echo(f"❌ Error reading {file_path}: {e}")-- typer.echo(f"\n📋 Summary: Found {result.total_changes} change(s) in {result.files_modified} file(s)")- typer.echo(" Run without --dry-run to apply these changes")- else:- # Show migration results- create_backups = backups and not dry_run- for file_path in result.modified_files:- typer.echo(f"✅ Migrated {file_path.relative_to(pipelines_dir)}")- if create_backups:- backup_path = file_path.with_suffix(".toml.backup")- typer.echo(f" Backup saved to {backup_path.name}")-- typer.echo(f"\n✅ Migration completed: {result.total_changes} change(s) applied to {result.files_modified} file(s)")- if create_backups:- typer.echo(" Backup files created with .backup extension")- typer.echo(" Run 'pipelex validate' to verify the migration")-- except FileNotFoundError as e:- typer.echo(f"❌ {e}")- raise typer.Exit(1)- except Exception as e:- typer.echo(f"❌ Migration failed: {e}")- raise typer.Exit(1)---def main() -> None:- """Entry point for the pipelex CLI."""- app()+app.add_typer(init_app, name="init", help="Initialization commands")+app.add_typer(validate_app, name="validate", help="Validation and dry-run commands")+app.add_typer(show_app, name="show", help="Show and list commands")+app.add_typer(migrate_app, name="migrate", help="Migration commands")+app.add_typer(build_app, name="build", help="Build artifacts like pipeline blueprints")diff --git a/pipelex/cli/commands/__init__.py b/pipelex/cli/commands/__init__.pynew file mode 100644index 000000000..19b9cd5a6--- /dev/null+++ b/pipelex/cli/commands/__init__.py@@ -0,0 +1,11 @@+"""Command groups for Pipelex CLI.++This package organizes CLI commands into logical modules.+"""++from .init_cmd import init_app+from .migrate_cmd import migrate_app+from .show_cmd import show_app+from .validate_cmd import validate_app++__all__ = ["init_app", "migrate_app", "show_app", "validate_app"]diff --git a/pipelex/cli/commands/build_cmd.py b/pipelex/cli/commands/build_cmd.pynew file mode 100644index 000000000..266704a8b--- /dev/null+++ b/pipelex/cli/commands/build_cmd.py@@ -0,0 +1,145 @@+from __future__ import annotations++import asyncio+from typing import Annotated, Optional++import typer++from pipelex.create.build_blueprint import do_build_blueprint+from pipelex.create.draft_pipeline import do_draft_pipeline, do_draft_pipeline_text+from pipelex.exceptions import PipelexCLIError+from pipelex.pipelex import Pipelex+from pipelex.tools.misc.file_utils import load_text_from_path++# Typer group for build commands+build_app = typer.Typer(help="Build artifacts like pipeline blueprints")+++@build_app.command("draft")+def build_draft_pipeline_cmd(+ pipeline_name: Annotated[+ str,+ typer.Argument(help="Name/code of the pipeline to generate"),+ ],+ domain: Annotated[+ str,+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),+ ] = "wip_domain",+ requirements: Annotated[+ Optional[str],+ typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),+ ] = None,+ requirements_file: Annotated[+ Optional[str],+ typer.Option("--file", "-f", help="Path to a file containing the requirements text"),+ ] = None,+ output_path: Annotated[+ Optional[str],+ typer.Option("--output", "-o", help="Path to save the generated TOML blueprint (optional)"),+ ] = None,+ raw: Annotated[+ bool,+ typer.Option("--raw", help="Raw text draft"),+ ] = False,+ relative_config_folder_path: Annotated[+ str,+ typer.Option(+ "--config-folder-path",+ "-c",+ help="Relative path to the config folder path (libraries)",+ ),+ ] = "./pipelex_libraries",+) -> None:+ # Initialize Pipelex (loads libraries and pipes)+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)++ # Get requirements text+ try:+ requirements_text: str+ if requirements_file:+ requirements_text = load_text_from_path(requirements_file)+ else:+ if not requirements:+ raise PipelexCLIError("You must provide requirements text via --requirements or a file via --file")+ requirements_text = requirements+ except Exception as exc:+ raise PipelexCLIError(f"Failed to load requirements: {exc}") from exc++ if raw:+ asyncio.run(+ do_draft_pipeline_text(+ domain=domain,+ pipeline_name=pipeline_name,+ requirements=requirements_text,+ output_path=output_path,+ )+ )+ else:+ asyncio.run(+ do_draft_pipeline(+ pipeline_name=pipeline_name,+ requirements=requirements_text,+ output_path=output_path,+ )+ )+++@build_app.command("blueprint")+def build_blueprint_cmd(+ pipeline_name: Annotated[+ str,+ typer.Argument(help="Name/code of the pipeline to generate"),+ ],+ domain: Annotated[+ str,+ typer.Option("--domain", "-d", help="Domain of the pipeline to generate"),+ ] = "wip_domain",+ requirements: Annotated[+ Optional[str],+ typer.Option("--requirements", "-r", help="Requirements text to generate the pipeline blueprint from"),+ ] = None,+ requirements_file: Annotated[+ Optional[str],+ typer.Option("--file", "-f", help="Path to a file containing the requirements text"),+ ] = None,+ output_path: Annotated[+ Optional[str],+ typer.Option("--output", "-o", help="Path to save the generated TOML blueprint (optional)"),+ ] = None,+ validate: Annotated[+ bool,+ typer.Option("--validate", help="Dry-run the first generated pipe from the blueprint"),+ ] = False,+ relative_config_folder_path: Annotated[+ str,+ typer.Option(+ "--config-folder-path",+ "-c",+ help="Relative path to the config folder path (libraries)",+ ),+ ] = "./pipelex_libraries",+) -> None:+ # Initialize Pipelex (loads libraries and pipes)+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)++ # Get requirements text+ try:+ requirements_text: str+ if requirements_file:+ requirements_text = load_text_from_path(requirements_file)+ else:+ if not requirements:+ raise PipelexCLIError("You must provide requirements text via --requirements or a file via --file")+ requirements_text = requirements+ except Exception as exc:+ raise PipelexCLIError(f"Failed to load requirements: {exc}") from exc++ asyncio.run(+ do_build_blueprint(+ domain=domain,+ pipeline_name=pipeline_name,+ requirements=requirements_text,+ output_path=output_path,+ validate=validate,+ )+ )diff --git a/pipelex/cli/commands/common.py b/pipelex/cli/commands/common.pynew file mode 100644index 000000000..f4c948780--- /dev/null+++ b/pipelex/cli/commands/common.py@@ -0,0 +1,25 @@+from __future__ import annotations++import os+from typing import Final++REQUIRED_PIPELEX_SUBDIRS: Final[list[str]] = [+ "pipelines",+ "llm_deck",+ "llm_integrations",+ "plugins",+ "templates",+]+++def is_pipelex_libraries_folder(folder_path: str) -> bool:+ """Check if the given folder path contains a valid pipelex libraries structure."""+ if not os.path.exists(folder_path) or not os.path.isdir(folder_path):+ return False++ for subdir in REQUIRED_PIPELEX_SUBDIRS:+ subdir_path = os.path.join(folder_path, subdir)+ if not os.path.exists(subdir_path) or not os.path.isdir(subdir_path):+ return False++ return Truediff --git a/pipelex/cli/commands/init_cmd.py b/pipelex/cli/commands/init_cmd.pynew file mode 100644index 000000000..22a942796--- /dev/null+++ b/pipelex/cli/commands/init_cmd.py@@ -0,0 +1,63 @@+from __future__ import annotations++import os+import shutil+from typing import Annotated++import typer++from pipelex.exceptions import PipelexCLIError+from pipelex.libraries.library_config import LibraryConfig+from pipelex.tools.config.manager import config_manager+++def do_init_libraries(directory: str = ".", overwrite: bool = False) -> None:+ """Initialize pipelex libraries in a pipelex_libraries folder in the specified directory."""+ try:+ target_path = os.path.join(directory, "pipelex_libraries")+ os.makedirs(directory, exist_ok=True)++ library_config = LibraryConfig(config_folder_path=target_path)+ library_config.export_libraries(overwrite=overwrite)++ if overwrite:+ typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (all files overwritten)")+ else:+ typer.echo(f"✅ Successfully initialized pipelex libraries at '{target_path}' (only created non-existing files)")+ except Exception as exc:+ raise PipelexCLIError(f"Failed to initialize libraries at '{directory}': {exc}") from exc+++def do_init_config(reset: bool = False) -> None:+ """Initialize pipelex configuration in the current directory."""+ pipelex_template_path = os.path.join(config_manager.pipelex_root_dir, "pipelex_template.toml")+ target_config_path = os.path.join(config_manager.local_root_dir, "pipelex.toml")++ if os.path.exists(target_config_path) and not reset:+ typer.echo("Warning: pipelex.toml already exists. Use --reset to force creation.")+ return++ try:+ shutil.copy2(pipelex_template_path, target_config_path)+ typer.echo(f"Created pipelex.toml at {target_config_path}")+ except Exception as exc:+ raise PipelexCLIError(f"Failed to create pipelex.toml: {exc}") from exc+++# Typer group for init commands+init_app = typer.Typer(help="Initialization commands")+++@init_app.command("libraries")+def init_libraries_cmd(+ directory: Annotated[str, typer.Argument(help="Directory where to create the pipelex_libraries folder")] = ".",+ overwrite: Annotated[bool, typer.Option("--overwrite", "-o", help="Warning: If set, existing files will be overwritten.")] = False,+) -> None:+ do_init_libraries(directory=directory, overwrite=overwrite)+++@init_app.command("config")+def init_config_cmd(+ reset: Annotated[bool, typer.Option("--reset", "-r", help="Warning: If set, existing files will be overwritten.")] = False,+) -> None:+ do_init_config(reset=reset)diff --git a/pipelex/cli/commands/migrate_cmd.py b/pipelex/cli/commands/migrate_cmd.pynew file mode 100644index 000000000..91e02a8d3--- /dev/null+++ b/pipelex/cli/commands/migrate_cmd.py@@ -0,0 +1,120 @@+from __future__ import annotations++from pathlib import Path+from typing import Annotated++import typer++from pipelex.cli.commands.common import is_pipelex_libraries_folder+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import TomlMigrator, migrate_concept_syntax+from pipelex.migration.migration_result import MigrationResult+++def do_migrate(+ target_path: str = "./pipelex_libraries",+ dry_run: bool = False,+ backups: bool = True,+) -> None:+ """Migrate TOML files to new syntax (Concept = -> definition = and PipeClassName = -> type/definition).++ The path can be:+ - A Pipelex libraries folder (we'll use its `pipelines` subfolder)+ - A directory containing TOML files+ - A single TOML file+ """+ config_path = Path(target_path)++ if is_pipelex_libraries_folder(target_path):+ pipelines_dir_or_file = config_path / "pipelines"+ else:+ pipelines_dir_or_file = config_path++ if not pipelines_dir_or_file.exists():+ typer.echo(f"❌ Path not found at '{pipelines_dir_or_file}'")+ raise typer.Exit(1)++ try:+ result: MigrationResult = migrate_concept_syntax(+ directory=pipelines_dir_or_file,+ create_backups=backups and not dry_run,+ dry_run=dry_run,+ )++ for error in result.errors:+ typer.echo(f"❌ {error}")++ if result.errors and not result.files_modified:+ typer.echo("❌ Migration failed due to errors")+ raise typer.Exit(1)++ typer.echo(f"Found {result.files_processed} TOML file(s) to check")++ if result.files_modified == 0:+ typer.echo("✅ All TOML files are already using the new syntax")+ return++ if dry_run:+ migrator = TomlMigrator()+ # For display, if a single file path was provided, show relative to its parent+ base_for_print = pipelines_dir_or_file if pipelines_dir_or_file.is_dir() else pipelines_dir_or_file.parent+ for file_path in result.modified_files:+ try:+ with open(file_path, "r", encoding="utf-8") as f:+ content = f.read()++ changes = migrator.get_migration_preview(content)+ try:+ rel = file_path.relative_to(base_for_print)+ except Exception:+ rel = Path(file_path.name)+ typer.echo(f"\n📄 {rel}:")+ for change in changes:+ typer.echo(f" Line {change['line_number']}: {change['old_line']} → {change['new_line']}")+ except Exception as exc:+ typer.echo(f"❌ Error reading {file_path}: {exc}")++ typer.echo(f"\n📋 Summary: Found {result.total_changes} change(s) in {result.files_modified} file(s)")+ typer.echo(" Run without --dry-run to apply these changes")+ else:+ create_backups = backups and not dry_run+ base_for_print = pipelines_dir_or_file if pipelines_dir_or_file.is_dir() else pipelines_dir_or_file.parent+ for file_path in result.modified_files:+ try:+ rel = file_path.relative_to(base_for_print)+ except Exception:+ rel = Path(file_path.name)+ typer.echo(f"✅ Migrated {rel}")+ if create_backups:+ backup_path = file_path.with_suffix(".toml.backup")+ typer.echo(f" Backup saved to {backup_path.name}")++ typer.echo(f"\n✅ Migration completed: {result.total_changes} change(s) applied to {result.files_modified} file(s)")+ if create_backups:+ typer.echo(" Backup files created with .backup extension")+ typer.echo(" Run 'pipelex validate all' to verify the migration")+ except FileNotFoundError as exc:+ typer.echo(f"❌ {exc}")+ raise typer.Exit(1)+ except Exception as exc:+ typer.echo(f"❌ Migration failed: {exc}")+ raise typer.Exit(1)+++# Typer group for migration commands+migrate_app = typer.Typer(help="Migration commands")+++@migrate_app.command("run")+def migrate_cmd(+ target_path: Annotated[+ str,+ typer.Option(+ "--path",+ "-p",+ help=("Path to Pipelex libraries folder (uses its pipelines), a directory of TOML files, or a single TOML file"),+ ),+ ] = "./pipelex_libraries",+ dry_run: Annotated[bool, typer.Option("--dry-run", help="Preview changes without applying them")] = False,+ backups: Annotated[bool, typer.Option("--backups/--no-backups", help="Create backup files before migration")] = False,+) -> None:+ do_migrate(target_path=target_path, dry_run=dry_run, backups=backups)diff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.pynew file mode 100644index 000000000..e4f0ffc15--- /dev/null+++ b/pipelex/cli/commands/show_cmd.py@@ -0,0 +1,68 @@+from __future__ import annotations++from typing import Annotated++import typer++from pipelex import pretty_print+from pipelex.exceptions import PipelexCLIError, PipelexConfigError+from pipelex.hub import get_pipe_provider, get_required_pipe+from pipelex.pipelex import Pipelex+from pipelex.tools.config.manager import config_manager+++def do_show_config() -> None:+ """Show the pipelex configuration."""+ try:+ final_config = config_manager.load_config()+ pretty_print(+ final_config,+ title=f"Pipelex configuration for project: {config_manager.get_project_name()}",+ )+ except Exception as exc:+ raise PipelexConfigError(f"Error loading configuration: {exc}") from exc+++def do_list_pipes(relative_config_folder_path: str = "pipelex_libraries") -> None:+ """List all available pipes."""+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)++ try:+ get_pipe_provider().pretty_list_pipes()+ except Exception as exc:+ raise PipelexCLIError(f"Failed to list pipes: {exc}") from exc+++def do_show_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:+ """Show a single pipe definition from the library."""+ Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ pipe = get_required_pipe(pipe_code=pipe_code)+ pretty_print(pipe, title=f"Pipe '{pipe_code}'")+++# Typer group for show commands+show_app = typer.Typer(help="Show and list commands")+++@show_app.command("config")+def show_config_cmd() -> None:+ do_show_config()+++@show_app.command("pipes")+def list_pipes_cmd(+ relative_config_folder_path: Annotated[+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")+ ] = "pipelex_libraries",+) -> None:+ do_list_pipes(relative_config_folder_path=relative_config_folder_path)+++@show_app.command("pipe")+def show_pipe_cmd(+ pipe_code: Annotated[str, typer.Argument(help="Pipeline code to show definition for")],+ relative_config_folder_path: Annotated[+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")+ ] = "./pipelex_libraries",+) -> None:+ do_show_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)diff --git a/pipelex/cli/commands/validate_cmd.py b/pipelex/cli/commands/validate_cmd.pynew file mode 100644index 000000000..ca7754153--- /dev/null+++ b/pipelex/cli/commands/validate_cmd.py@@ -0,0 +1,65 @@+from __future__ import annotations++import asyncio+from typing import Annotated++import typer++from pipelex import log+from pipelex.cli.commands.common import is_pipelex_libraries_folder+from pipelex.hub import get_pipeline_tracker+from pipelex.pipe_works.pipe_dry import dry_run_all_pipes, dry_run_single_pipe+from pipelex.pipelex import Pipelex+++def do_validate(relative_config_folder_path: str = "./pipelex_libraries") -> None:+ """Validate libraries and dry-run all pipes."""+ if not is_pipelex_libraries_folder(relative_config_folder_path):+ typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")+ typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")+ raise typer.Exit(1)++ pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ pipelex_instance.validate_libraries()+ asyncio.run(dry_run_all_pipes())+ log.info("Setup sequence passed OK, config and pipelines are validated.")+++def do_dry_run_pipe(pipe_code: str, relative_config_folder_path: str = "./pipelex_libraries") -> None:+ """Dry run a single pipe."""+ if not is_pipelex_libraries_folder(relative_config_folder_path):+ typer.echo(f"❌ No pipelex libraries folder found at '{relative_config_folder_path}'")+ typer.echo("To create a pipelex libraries folder, run: pipelex init-libraries")+ raise typer.Exit(1)++ try:+ pipelex_instance = Pipelex.make(relative_config_folder_path=relative_config_folder_path, from_file=False)+ pipelex_instance.validate_libraries()+ asyncio.run(dry_run_single_pipe(pipe_code))+ get_pipeline_tracker().output_flowchart()+ except Exception as exc:+ typer.echo(f"❌ Error running dry run for pipe '{pipe_code}': {exc}")+ raise typer.Exit(1)+++# Typer group for validation commands+validate_app = typer.Typer(help="Validation and dry-run commands")+++@validate_app.command("all")+def validate_all_cmd(+ relative_config_folder_path: Annotated[+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")+ ] = "./pipelex_libraries",+) -> None:+ do_validate(relative_config_folder_path=relative_config_folder_path)+++@validate_app.command("pipe")+def dry_run_pipe_cmd(+ pipe_code: Annotated[str, typer.Argument(help="The pipe code to dry run")],+ relative_config_folder_path: Annotated[+ str, typer.Option("--config-folder-path", "-c", help="Relative path to the config folder path")+ ] = "./pipelex_libraries",+) -> None:+ do_dry_run_pipe(pipe_code=pipe_code, relative_config_folder_path=relative_config_folder_path)diff --git a/pipelex/core/pipe_run_params.py b/pipelex/core/pipe_run_params.pyindex da680fe4e..7c93ee3c2 100644--- a/pipelex/core/pipe_run_params.py+++ b/pipelex/core/pipe_run_params.py@@ -111,6 +111,7 @@ class BatchParams(BaseModel):class PipeRunParams(BaseModel):run_mode: PipeRunMode = PipeRunMode.LIVEfinal_stuff_code: Optional[str] = None+ is_with_preliminary_text: Optional[bool] = Noneoutput_multiplicity: Optional[PipeOutputMultiplicity] = Nonedynamic_output_concept_code: Optional[str] = Nonebatch_params: Optional[BatchParams] = None@@ -139,6 +140,7 @@ class PipeRunParams(BaseModel):cls,pipe_run_params: Self,applied_output_multiplicity: Optional[PipeOutputMultiplicity],+ is_with_preliminary_text: Optional[bool] = None,) -> Self:"""Copy the run params the nb_output into the params, and remove the attribute.@@ -152,6 +154,8 @@ class PipeRunParams(BaseModel):elif isinstance(applied_output_multiplicity, int):new_run_params.output_multiplicity = Falsenew_run_params.params[PipeRunParamKey.NB_OUTPUT] = applied_output_multiplicity+ if is_with_preliminary_text is not None:+ new_run_params.is_with_preliminary_text = is_with_preliminary_textreturn new_run_params@propertydiff --git a/pipelex/core/registry_models.py b/pipelex/core/registry_models.pyindex 3f400ac52..c9b2bfd95 100644--- a/pipelex/core/registry_models.py+++ b/pipelex/core/registry_models.py@@ -17,6 +17,8 @@ from pipelex.core.stuff_content import (TextAndImagesContent,TextContent,)+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint+from pipelex.libraries.pipelines.meta.pipeline_draft import PipelineDraftfrom pipelex.pipe_controllers.pipe_batch import PipeBatchfrom pipelex.pipe_controllers.pipe_batch_factory import PipeBatchFactoryfrom pipelex.pipe_controllers.pipe_condition import PipeCondition@@ -86,6 +88,8 @@ class PipelexRegistryModels(RegistryModels):PDFContent,TextAndImagesContent,PageContent,+ PipelineBlueprint,+ PipelineDraft,]EXPERIMENTAL: ClassVar[List[ModelType]] = [diff --git a/pipelex/create/build_blueprint.py b/pipelex/create/build_blueprint.pynew file mode 100644index 000000000..affce8844--- /dev/null+++ b/pipelex/create/build_blueprint.py@@ -0,0 +1,97 @@+from __future__ import annotations++import asyncio+from pathlib import Path+from typing import Any, Dict, Optional, cast++import typer+from pydantic import ValidationError++from pipelex import pretty_print+from pipelex.create.helpers import get_support_file+from pipelex.create.pipeline_toml import save_pipeline_blueprint_toml_to_path+from pipelex.exceptions import PipeDefinitionError, PipelexCLIError+from pipelex.hub import get_library_manager+from pipelex.libraries.library_manager import LibraryManager+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint+from pipelex.pipe_works.pipe_dry import dry_run_pipe_codes+from pipelex.pipeline.execute import execute_pipeline+from pipelex.tools.misc.file_utils import save_text_to_path+from pipelex.tools.misc.json_utils import save_as_json_to_path+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+++async def do_build_blueprint(+ domain: str,+ pipeline_name: str,+ requirements: str,+ output_path: Optional[str],+ validate: bool,+) -> None:+ pipe_output = await execute_pipeline(+ pipe_code="build_blueprint",+ input_memory={+ "domain": domain,+ "pipeline_name": pipeline_name,+ "requirements": requirements,+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),+ },+ )+ blueprint = pipe_output.main_stuff_as(content_type=PipelineBlueprint)+ pretty_print(blueprint, title="Pipeline Blueprint")+ pipeline_draft = pipe_output.working_memory.get_stuff_as_str(name="pipeline_draft")+ pretty_print(pipeline_draft, title="Pipeline Draft")++ # Save or display result+ output_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"+ # blueprint_dict = blueprint.smart_dump()+ # save_toml_to_path(data=blueprint_dict, path=output_path)+ save_pipeline_blueprint_toml_to_path(blueprint=blueprint, path=output_path)+ json_path = output_path.replace(Path(output_path).suffix, ".json")+ save_as_json_to_path(object_to_save=blueprint, path=json_path)+ draft_path = output_path.replace(Path(output_path).suffix, "_draft.md")+ save_text_to_path(text=pipeline_draft, path=draft_path)+ typer.echo(f"✅ Blueprint saved to '{output_path}'")++ if validate:+ try:+ _load_pipes_from_generated_blueprint(blueprint=blueprint)+ except PipeDefinitionError as exc:+ raise PipelexCLIError(f"Failed to load pipes from generated blueprint at '{output_path}': {exc}") from exc++ generated_pipe_codes = list(blueprint.pipe.keys())+ if not generated_pipe_codes:+ raise PipelexCLIError("No pipe found in generated blueprint to validate")+ asyncio.run(dry_run_pipe_codes(pipe_codes=generated_pipe_codes))+++def _load_pipes_from_generated_blueprint(blueprint: PipelineBlueprint) -> None:+ """Instantiate and register all pipes from a generated PipelineBlueprint.++ This constructs the appropriate typed PipeBlueprint for each pipe entry and+ uses LibraryManager.load_pipe_from_blueprint to create the Pipe, then adds it+ to the in-memory PipeLibrary so that validation can run against them.+ """+ library_manager = cast(LibraryManager, get_library_manager())+ domain_code = blueprint.domain++ for pipe_code, details in blueprint.pipe.items():+ details_dict: Dict[str, Any] = details.copy()++ # Build the concrete PipeBlueprint via LibraryManager helper+ try:+ pipe_blueprint = LibraryManager.make_pipe_blueprint_from_details(+ domain_code=domain_code,+ details_dict=details_dict,+ )+ except ValidationError as exc:+ error_msg = format_pydantic_validation_error(exc=exc)+ raise PipeDefinitionError(f"Failed to build pipe blueprint for pipe '{pipe_code}': {error_msg}") from exc++ # Create pipe and register in library+ pipe = LibraryManager.load_pipe_from_blueprint(+ pipe_code=pipe_code,+ pipe_blueprint=pipe_blueprint,+ )+ library_manager.pipe_library.add_new_pipe(pipe=pipe)diff --git a/pipelex/create/draft_pipeline.py b/pipelex/create/draft_pipeline.pynew file mode 100644index 000000000..f87dd736e--- /dev/null+++ b/pipelex/create/draft_pipeline.py@@ -0,0 +1,60 @@+from __future__ import annotations++from typing import Optional++import typer++from pipelex import pretty_print+from pipelex.create.helpers import get_support_file+from pipelex.create.pipeline_toml import dict_to_toml, save_toml_to_path+from pipelex.libraries.pipelines.meta.pipeline_draft import PipelineDraft+from pipelex.pipeline.execute import execute_pipeline+from pipelex.tools.misc.file_utils import save_text_to_path+++async def do_draft_pipeline_text(+ domain: str,+ pipeline_name: str,+ requirements: str,+ output_path: Optional[str],+) -> None:+ pipe_output = await execute_pipeline(+ pipe_code="draft_pipeline_text",+ input_memory={+ "domain": domain,+ "pipeline_name": pipeline_name,+ "requirements": requirements,+ "rules": get_support_file(subpath="create/draft_pipelines.md"),+ },+ )+ draft_text = pipe_output.main_stuff_as_str+ pretty_print(draft_text, title="Pipeline Draft Text")++ # Save or display result+ output_path = output_path or "pipelex/libraries/pipelines/temp/pipeline_draft.md"+ save_text_to_path(text=draft_text, path=output_path)+ typer.echo(f"✅ Pipeline draft text saved to '{output_path}'")+++async def do_draft_pipeline(+ pipeline_name: str,+ requirements: str,+ output_path: Optional[str],+) -> None:+ pipe_output = await execute_pipeline(+ pipe_code="draft_pipeline",+ input_memory={+ "pipeline_name": pipeline_name,+ "requirements": requirements,+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),+ },+ )+ draft = pipe_output.main_stuff_as(content_type=PipelineDraft)+ pretty_print(draft, title="Pipeline Draft")+ draft_dict = draft.smart_dump()+ pretty_print(dict_to_toml(data=draft_dict), title="Pipeline Draft (TOML)")++ # Save or display result+ output_path = output_path or "pipelex/libraries/pipelines/temp/generated_blueprint.toml"+ save_toml_to_path(data=draft_dict, path=output_path)+ typer.echo(f"✅ Blueprint saved to '{output_path}'")diff --git a/pipelex/create/helpers.py b/pipelex/create/helpers.pynew file mode 100644index 000000000..664f4df53--- /dev/null+++ b/pipelex/create/helpers.py@@ -0,0 +1,55 @@+from pathlib import Path+from typing import Dict, List, Optional++from pipelex.tools.misc.file_utils import load_text_from_path+++class PipelexSupportFiles:+ """Dictionary-like interface to access all files in .pipelex by their subpath."""++ def __init__(self, base_path: str = ".pipelex"):+ self.base_path = Path(base_path)+ self._file_cache: Dict[str, str] = {}++ def __getitem__(self, subpath: str) -> str:+ """Get file content by subpath (e.g., 'design_pipelines.md')."""+ if subpath not in self._file_cache:+ file_path = self.base_path / subpath+ if not file_path.exists():+ raise KeyError(f"File '{subpath}' not found in {self.base_path}")+ self._file_cache[subpath] = load_text_from_path(str(file_path))+ return self._file_cache[subpath]++ def __contains__(self, subpath: str) -> bool:+ """Check if a file exists by subpath."""+ file_path = self.base_path / subpath+ return file_path.exists() and file_path.is_file()++ def get(self, subpath: str, default: Optional[str] = None) -> Optional[str]:+ """Get file content by subpath with optional default."""+ try:+ return self[subpath]+ except KeyError:+ return default++ def keys(self) -> List[str]:+ """Get all available file subpaths."""+ if not self.base_path.exists():+ return []++ files: List[str] = []+ for file_path in self.base_path.rglob("*"):+ if file_path.is_file():+ subpath = str(file_path.relative_to(self.base_path))+ files.append(subpath)+ return sorted(files)++ def clear_cache(self) -> None:+ """Clear the internal file cache."""+ self._file_cache.clear()+++def get_support_file(subpath: str) -> str:+ """Get pipeline creation rules from design_pipelines.md."""+ support_files = PipelexSupportFiles()+ return support_files[subpath]diff --git a/pipelex/create/pipeline_toml.py b/pipelex/create/pipeline_toml.pynew file mode 100644index 000000000..9e6bd8a5c--- /dev/null+++ b/pipelex/create/pipeline_toml.py@@ -0,0 +1,138 @@+from __future__ import annotations++from typing import Any, Dict, List, Mapping, cast++import tomlkit+from tomlkit import array, document, inline_table, table+from tomlkit import string as tk_string++from pipelex.libraries.pipeline_blueprint import PipelineBlueprint+from pipelex.tools.misc.file_utils import save_text_to_path+from pipelex.tools.misc.json_utils import remove_none_values_from_dict+from pipelex.tools.misc.toml_utils import clean_trailing_whitespace+++def make_toml_string(+ text: str,+ prefer_literal: bool = False,+ force_multiline: bool = False,+ ensure_trailing_newline: bool = True,+ ensure_leading_blank_line_in_value: bool = False,+):+ """+ Build a tomlkit string node.+ - If `force_multiline` or the text contains '\\n', we emit a triple-quoted multiline string.+ - When multiline, `ensure_trailing_newline` puts the closing quotes on their own line.+ - When multiline, `ensure_leading_blank_line_in_value` inserts a real blank line at the start of the value.+ """+ needs_multiline = force_multiline or ("\n" in text)+ normalized = text++ if needs_multiline:+ if ensure_leading_blank_line_in_value and not normalized.startswith("\n"):+ normalized = "\n" + normalized+ if ensure_trailing_newline and not normalized.endswith("\n"):+ normalized = normalized + "\n"++ use_literal = prefer_literal and ("'''" not in normalized)+ return tk_string(normalized, multiline=needs_multiline, literal=use_literal)+++def _convert_to_inline(value: Any):+ """Recursively convert Python values; dicts -> inline tables; lists kept as arrays."""+ if isinstance(value, Mapping):+ value = cast(Mapping[str, Any], value)+ inline_table_obj = inline_table()+ for key, value_item in value.items():+ inline_table_obj[key] = _convert_to_inline(value_item)+ return inline_table_obj++ if isinstance(value, list):+ value = cast(List[Any], value)+ array_obj = array()+ array_obj.multiline(True) # set to False for single-line arrays+ for element in value:+ if isinstance(element, Mapping):+ element = cast(Mapping[str, Any], element)+ inline_element = inline_table()+ for inner_key, inner_value in element.items():+ inline_element[inner_key] = _convert_to_inline(inner_value)+ array_obj.append(inline_element) # pyright: ignore[reportUnknownMemberType]+ else:+ array_obj.append(_convert_to_inline(element)) # pyright: ignore[reportUnknownMemberType]+ return array_obj++ if isinstance(value, str):+ # Triple quotes if needed (or forced); closing quotes on their own line.+ return make_toml_string(+ value,+ prefer_literal=False, # flip to True for '''...'''+ force_multiline=False, # flip to True to force """...""" even without \n+ ensure_trailing_newline=True, # keep closing """ on its own line+ ensure_leading_blank_line_in_value=False, # flip to True to keep a blank first line+ )+ return value+++def _convert_mapping_to_table(mapping: Mapping[str, Any]) -> Any:+ """Convert a mapping into a TOML Table where any nested mappings (third level+)+ are converted to inline tables.++ This creates a second-level standard table, and only uses inline tables for+ third level and deeper structures.+ """+ tbl = table()+ for field_key, field_value in mapping.items():+ if isinstance(field_value, Mapping):+ # Third-level mapping -> inline table+ tbl.add(field_key, _convert_to_inline(field_value))+ else:+ tbl.add(field_key, _convert_to_inline(field_value))+ return tbl+++def dict_to_toml(data: Mapping[str, Any]) -> str:+ """Top-level keys become tables; second-level mappings become tables; inline tables start at third level."""+ data = remove_none_values_from_dict(data=data)+ document_root = document()+ for section_key, section_value in data.items():+ if isinstance(section_value, Mapping):+ section_value = cast(Mapping[str, Any], section_value)+ table_obj = table()+ for field_key, field_value in section_value.items():+ if isinstance(field_value, Mapping):+ # Second level mapping -> standard table [section.field]+ sub_table = _convert_mapping_to_table(cast(Mapping[str, Any], field_value))+ table_obj.add(field_key, sub_table)+ else:+ table_obj.add(field_key, _convert_to_inline(field_value))+ document_root.add(section_key, table_obj) # `[section_key]` section+ else:+ document_root.add(section_key, _convert_to_inline(section_value))+ dumped_content = tomlkit.dumps(document_root) # pyright: ignore[reportUnknownMemberType]+ return dumped_content+++def pipeline_blueprint_to_toml(blueprint: PipelineBlueprint) -> str:+ blueprint_dict = blueprint.smart_dump()+ return dict_to_toml(data=blueprint_dict)+++def save_toml_to_path(data: Dict[str, Any], path: str) -> None:+ """Save dictionary as TOML to file path.++ Args:+ data: Dictionary to save as TOML+ path: Path where to save the TOML file+ """+ # data_cleaned = remove_none_values_from_dict(data=data)+ data_cleaned = data+ with open(path, "w", encoding="utf-8") as file:+ toml_content: str = dict_to_toml(data=data_cleaned)+ cleaned_content = clean_trailing_whitespace(toml_content)+ file.write(cleaned_content)+++def save_pipeline_blueprint_toml_to_path(blueprint: PipelineBlueprint, path: str) -> None:+ pipeline_blueprint_toml = pipeline_blueprint_to_toml(blueprint=blueprint)+ save_text_to_path(text=pipeline_blueprint_toml, path=path)diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 90757d578..90c8cad2e 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -1,3 +1,4 @@+import importlibimport osfrom pathlib import Pathfrom typing import Any, ClassVar, Dict, List, Optional, Type@@ -15,7 +16,7 @@ from pipelex.core.concept_library import ConceptLibraryfrom pipelex.core.domain import Domainfrom pipelex.core.domain_library import DomainLibraryfrom pipelex.core.pipe_abstract import PipeAbstract-from pipelex.core.pipe_blueprint import PipeSpecificFactoryProtocol+from pipelex.core.pipe_blueprint import PipeBlueprint, PipeSpecificFactoryProtocolfrom pipelex.core.pipe_library import PipeLibraryfrom pipelex.exceptions import (ConceptLibraryError,@@ -29,8 +30,8 @@ from pipelex.libraries.library_manager_abstract import LibraryManagerAbstractfrom pipelex.libraries.pipeline_blueprint import (ConceptBlueprintError,PipeBlueprintError,+ PipelineBlueprint,PipelineBlueprintValidationError,- PipelineLibraryBlueprint,)from pipelex.tools.class_registry_utils import ClassRegistryUtilsfrom pipelex.tools.misc.file_utils import find_files_in_dir@@ -202,11 +203,11 @@ class LibraryManager(LibraryManagerAbstract):nb_pipes_loaded = len(self.pipe_library.root) - nb_pipes_beforelog.verbose(f"Loaded {nb_pipes_loaded} pipes from '{toml_path.name}'")- def _load_blueprint_from_file(self, toml_path: Path) -> PipelineLibraryBlueprint:+ def _load_blueprint_from_file(self, toml_path: Path) -> PipelineBlueprint:"""Load and validate a pipeline blueprint from a TOML file."""try:toml_data = load_toml_from_path(path=str(toml_path))- blueprint = PipelineLibraryBlueprint.model_validate(toml_data)+ blueprint = PipelineBlueprint.model_validate(toml_data)return blueprintexcept ValidationError as exc:error_msg = format_pydantic_validation_error(exc)@@ -217,7 +218,7 @@ class LibraryManager(LibraryManagerAbstract):except Exception as exc:raise LibraryError(f"Failed to load TOML file '{toml_path}': {exc}") from exc- def _load_concepts_from_blueprint(self, blueprint: PipelineLibraryBlueprint, file_path: str):+ def _load_concepts_from_blueprint(self, blueprint: PipelineBlueprint, file_path: str):"""Load concepts from a validated blueprint."""for concept_name, concept_data in blueprint.concept.items():try:@@ -245,7 +246,7 @@ class LibraryManager(LibraryManagerAbstract):error_msg=error_msg,) from exc- def _load_pipes_from_blueprint(self, blueprint: PipelineLibraryBlueprint, file_path: str):+ def _load_pipes_from_blueprint(self, blueprint: PipelineBlueprint, file_path: str):"""Load pipes from a validated blueprint."""for pipe_name, pipe_data in blueprint.pipe.items():try:@@ -378,3 +379,114 @@ Old syntax will be removed in v0.3.0.details_dict=details_dict,)return pipe_from_blueprint++ @classmethod+ def make_pipe_from_details_dict(+ cls,+ domain_code: str,+ pipe_code: str,+ details_dict: Dict[str, Any],+ ) -> PipeAbstract:+ # Delegate to the new make_pipe_from_blueprint method+ return cls.make_pipe_from_blueprint(+ domain_code=domain_code,+ pipe_code=pipe_code,+ details_dict=details_dict,+ )++ @classmethod+ def load_pipe_from_blueprint(+ cls,+ pipe_code: str,+ pipe_blueprint: PipeBlueprint,+ ) -> PipeAbstract:+ """Create a Pipe from a concrete PipeBlueprint instance.++ This resolves the corresponding factory by converting the blueprint class name+ (e.g. "PipeLLMBlueprint") to its factory name ("PipeLLMFactory").+ """+ # The blueprint must be a concrete subclass like PipeLLMBlueprint, PipeOcrBlueprint, etc.+ blueprint_class_name = type(pipe_blueprint).__name__+ if blueprint_class_name == "PipeBlueprint":+ raise PipeFactoryError("Cannot load pipe from base PipeBlueprint. Please provide a specific blueprint subclass (e.g. PipeLLMBlueprint).")++ # Derive factory name: PipeLLMBlueprint -> PipeLLMFactory+ factory_class_name = blueprint_class_name.replace("Blueprint", "Factory")++ try:+ pipe_factory: Type[PipeSpecificFactoryProtocol[Any, Any]] = KajsonManager.get_class_registry().get_required_subclass(+ name=factory_class_name,+ base_class=PipeSpecificFactoryProtocol,+ )+ except ClassRegistryNotFoundError as factory_not_found_error:+ raise PipeFactoryError(+ f"Pipe '{pipe_code}' couldn't be created: factory '{factory_class_name}' not found: {factory_not_found_error}"+ ) from factory_not_found_error+ except ClassRegistryInheritanceError as factory_inheritance_error:+ raise PipeFactoryError(+ f"Pipe '{pipe_code}' couldn't be created: factory '{factory_class_name}' is not a subclass of {type(PipeSpecificFactoryProtocol)}."+ ) from factory_inheritance_error++ # Domain is part of the blueprint+ domain_code = pipe_blueprint.domain++ # Let the specific factory build the concrete Pipe instance+ pipe_from_blueprint: PipeAbstract = pipe_factory.make_pipe_from_blueprint(+ domain_code=domain_code,+ pipe_code=pipe_code,+ pipe_blueprint=pipe_blueprint, # type: ignore[arg-type]+ )+ return pipe_from_blueprint++ @classmethod+ def make_pipe_blueprint_from_details(+ cls,+ domain_code: str,+ details_dict: Dict[str, Any],+ ) -> PipeBlueprint:+ """Create a concrete PipeBlueprint subclass instance from details.++ This resolves the factory based on the provided details (supports both+ new and legacy formats), locates the corresponding <PipeClassName>Blueprint+ in the factory module, and validates the blueprint.+ """+ # Determine pipe class name from details+ if "type" in details_dict and "definition" in details_dict:+ pipe_class_name = details_dict["type"]+ normalized_details = details_dict.copy()+ else:+ try:+ pipe_class_name, legacy_definition = next(iter(details_dict.items()))+ normalized_details = {"definition": legacy_definition}+ except StopIteration as empty_err:+ raise PipeFactoryError("Pipe details are empty; cannot determine pipe type") from empty_err++ factory_class_name = f"{pipe_class_name}Factory"++ # Resolve factory to locate its module (where the Blueprint class is defined)+ try:+ pipe_factory: Type[PipeSpecificFactoryProtocol[Any, Any]] = KajsonManager.get_class_registry().get_required_subclass(+ name=factory_class_name,+ base_class=PipeSpecificFactoryProtocol,+ )+ except ClassRegistryNotFoundError as factory_not_found_error:+ raise PipeFactoryError(+ f"Factory '{factory_class_name}' not found for pipe type '{pipe_class_name}': {factory_not_found_error}"+ ) from factory_not_found_error+ except ClassRegistryInheritanceError as factory_inheritance_error:+ raise PipeFactoryError(+ f"Factory '{factory_class_name}' is not a subclass of {type(PipeSpecificFactoryProtocol)}."+ ) from factory_inheritance_error++ factory_module = importlib.import_module(pipe_factory.__module__)+ blueprint_class_name = f"{pipe_class_name}Blueprint"++ try:+ blueprint_cls: Type[PipeBlueprint] = getattr(factory_module, blueprint_class_name)+ except AttributeError as exc:+ raise PipeFactoryError(f"Cannot find blueprint class '{blueprint_class_name}' in module '{pipe_factory.__module__}'") from exc++ details_with_domain = normalized_details.copy()+ details_with_domain["domain"] = domain_code++ return blueprint_cls.model_validate(details_with_domain)diff --git a/pipelex/libraries/llm_deck/base_llm_deck.toml b/pipelex/libraries/llm_deck/base_llm_deck.tomlindex 1c9271aeb..620509679 100644--- a/pipelex/libraries/llm_deck/base_llm_deck.toml+++ b/pipelex/libraries/llm_deck/base_llm_deck.toml@@ -55,7 +55,9 @@ llm_to_enrich = { llm_handle = "best-claude", temperature = 0.1 }llm_for_question_and_excerpt_reformulation = { llm_handle = "gpt-4o", temperature = 0.9 }# Engineering skills-llm_to_engineer = { llm_handle = "best-claude", temperature = 0.5 }+llm_to_engineer = { llm_handle = "best-claude", temperature = 0.2 }+llm_to_pipelex = { llm_handle = "best-claude", temperature = 0.2 }+# llm_to_pipelex = { llm_handle = "gpt-5-mini", temperature = 0.2 }# Image skillsllm_to_write_imgg_prompt = { llm_handle = "best-claude", temperature = 0.2 }diff --git a/pipelex/libraries/pipeline_blueprint.py b/pipelex/libraries/pipeline_blueprint.pyindex 5b60fb821..54ebb0331 100644--- a/pipelex/libraries/pipeline_blueprint.py+++ b/pipelex/libraries/pipeline_blueprint.py@@ -1,11 +1,10 @@-from pathlib import Pathfrom typing import Any, Dict, Optional, Union-from pydantic import BaseModel, Field+from pydantic import Fieldfrom pipelex.core.concept_factory import ConceptBlueprint+from pipelex.core.stuff_content import StructuredContentfrom pipelex.exceptions import LibraryError-from pipelex.tools.misc.toml_utils import dict_to_toml_string, save_toml_to_pathclass PipelineBlueprintValidationError(LibraryError):@@ -35,15 +34,8 @@ class PipeBlueprintError(LibraryError):super().__init__(f"Error loading pipe '{pipe_name}' from '{file_path}': {error_msg}")-class PipeStepBlueprint(BaseModel):- """Blueprint of a step in a pipe sequence."""-- pipe: str- result: str---class PipelineLibraryBlueprint(BaseModel):- """Complete blueprint of a pipeline library TOML file."""+class PipelineBlueprint(StructuredContent):+ """Complete blueprint of a pipeline TOML definition."""# Domain information (required)domain: str@@ -57,37 +49,3 @@ class PipelineLibraryBlueprint(BaseModel):# Pipes section - pipe_name -> blueprint dictpipe: Dict[str, Dict[str, Any]] = Field(default_factory=dict)-- def to_toml_dict(self) -> Dict[str, Any]:- """Convert blueprint back to TOML-compatible dictionary structure."""- result: Dict[str, Any] = {"domain": self.domain}-- # Add optional domain-level fields- if self.definition:- result["definition"] = self.definition- if self.system_prompt:- result["system_prompt"] = self.system_prompt- if self.system_prompt_to_structure:- result["system_prompt_to_structure"] = self.system_prompt_to_structure- if self.prompt_template_to_structure:- result["prompt_template_to_structure"] = self.prompt_template_to_structure-- # Add concepts section if not empty- if self.concept:- result["concept"] = self.concept-- # Add pipes section if not empty- if self.pipe:- result["pipe"] = self.pipe-- return result-- def to_toml_string(self) -> str:- """Convert blueprint to TOML string format."""- return dict_to_toml_string(self.to_toml_dict())-- def save_to_file(self, file_path: Union[str, Path]) -> None:- """Save blueprint to a TOML file."""- path = Path(file_path)- path.parent.mkdir(parents=True, exist_ok=True)- save_toml_to_path(data=self.to_toml_dict(), path=str(path))diff --git a/pipelex/libraries/pipelines/documents.toml b/pipelex/libraries/pipelines/documents.tomlindex 77fdfd536..c3826d8ad 100644--- a/pipelex/libraries/pipelines/documents.toml+++ b/pipelex/libraries/pipelines/documents.toml@@ -11,14 +11,16 @@ TextAndImagesContent = "A content that comprises text and images where the text# It can be named however you want# but it must be either an image or a pdf or a concept which refines one of them[pipe.ocr_page_contents_from_pdf]-PipeOcr = "Extract page contents from a PDF document"+type = "PipeOcr"+definition = "Extract page contents from a PDF document"inputs = { ocr_input = "PDF" }output = "Page"page_images = truepage_views = false[pipe.ocr_page_contents_and_views_from_pdf]-PipeOcr = "Extract page contents from a PDF document as well as full page views"+type = "PipeOcr"+definition = "Extract page contents from a PDF document as well as full page views"inputs = { ocr_input = "PDF" }output = "Page"page_images = truediff --git a/pipelex/libraries/pipelines/images.toml b/pipelex/libraries/pipelines/images.tomlindex f708d0e89..9d4197279 100644--- a/pipelex/libraries/pipelines/images.toml+++ b/pipelex/libraries/pipelines/images.toml@@ -20,7 +20,8 @@ refines = "Image"#################################################################[pipe.describe_image]-PipeLLM = "Describe an image"+type = "PipeLLM"+definition = "Describe an image"inputs = { image = "Image" }output = "VisualDescription"system_prompt = "You are a very good observer."@@ -30,7 +31,8 @@ Describe the provided image in great detail."""[pipe.describe_photo]-PipeLLM = "Describe a photo"+type = "PipeLLM"+definition = "Describe a photo"inputs = { photo = "Photo" }output = "VisualDescription"system_prompt = "You are a very good observer."@@ -48,14 +50,16 @@ Describe the provided photo and how it was shot: scene, lighting, camera, etc.# It can be named however you want,# but it must be either an ImgGenPrompt or a concept which refines ImgGenPrompt[pipe.generate_image]-PipeImgGen = "Generate an image"+type = "PipeImgGen"+definition = "Generate an image"inputs = { prompt = "ImgGenPrompt" }output = "Image"nb_steps = 2[pipe.generate_photo]-PipeImgGen = "Generate a photo"+type = "PipeImgGen"+definition = "Generate a photo"inputs = { prompt = "ImgGenPrompt" }output = "images.Photo"nb_steps = 8diff --git a/pipelex/libraries/pipelines/meta/meta_pipeline.toml b/pipelex/libraries/pipelines/meta/meta_pipeline.tomlnew file mode 100644index 000000000..21d6d99d0--- /dev/null+++ b/pipelex/libraries/pipelines/meta/meta_pipeline.toml@@ -0,0 +1,64 @@+domain = "meta"+definition = "Meta domain is the domain of pipelines about pipelines"++[concept]+PipelineDraft = "Rough solution to create a pipeline"+PipelineDraftText = "Rough solution to create a pipeline in text format"+PipeDraft = "Rough solution to create a pipe"+PipelineBlueprint = "Details enabling to create a pipeline"+ConceptBlueprint = "Details enabling to create a concept"+PipeBlueprint = "Details enabling to create a pipe"++[pipe]++[pipe.build_blueprint]+type = "PipeSequence"+definition = "Build a pipeline blueprint from natural language requirements"+inputs = { draft_pipeline_rules = "Text", build_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }+output = "PipelineBlueprint"+steps = [+ { pipe = "draft_pipeline_text", result = "pipeline_draft" },+ { pipe = "structure_blueprint", result = "pipeline_blueprint" },+]++[pipe.draft_pipeline_text]+type = "PipeLLM"+definition = "Generate a rough pipeline draft from natural language requirements"+inputs = { draft_pipeline_rules = "Text", requirements = "Text", pipeline_name = "Text", domain = "Text" }+output = "PipelineDraftText"+llm = "llm_to_pipelex"+prompt_template = """+You are a Pipelex pipeline architect. Build a pipeline draft text showing how we solve the problem.++@draft_pipeline_rules++---+Name of the pipeline: $pipeline_name.++Domain: $domain.++Requirements: $requirements+"""++[pipe.structure_blueprint]+type = "PipeLLM"+definition = "Generate a pipeline blueprint from natural language requirements"+inputs = { build_pipeline_rules = "Text", pipeline_draft = "Text", pipeline_name = "Text", domain = "Text", requirements = "Text" }+output = "PipelineBlueprint"+llm = "llm_to_pipelex"+# structuring_method = "preliminary_text"+prompt_template = """+You are a Pipelex pipeline architect. Build a structured pipeline blueprint from the provided pipeline draft text.++@build_pipeline_rules++@pipeline_draft++---+Name of the pipeline: $pipeline_name.++Domain: $domain.++Requirements: $requirements.+"""+diff --git a/pipelex/libraries/pipelines/meta/pipeline_draft.py b/pipelex/libraries/pipelines/meta/pipeline_draft.pynew file mode 100644index 000000000..26e5ab970--- /dev/null+++ b/pipelex/libraries/pipelines/meta/pipeline_draft.py@@ -0,0 +1,27 @@+from typing import Dict++from pydantic import Field++from pipelex.core.stuff_content import StructuredContent+++class PipeDraft(StructuredContent):+ code: str+ type: str+ definition: str+ inputs: Dict[str, str]+ output: str+++class PipelineDraft(StructuredContent):+ """Complete blueprint of a pipeline library TOML file."""++ # Domain information (required)+ domain: str+ definition: str++ # Concepts section - concept_name -> definition (string) or blueprint (dict)+ concept: Dict[str, str] = Field(default_factory=dict)++ # Pipes section - pipe_name -> blueprint dict+ pipe: Dict[str, PipeDraft] = Field(default_factory=dict)diff --git a/pipelex/tools/migrate/__init__.py b/pipelex/migration/__init__.pysimilarity index 100%rename from pipelex/tools/migrate/__init__.pyrename to pipelex/migration/__init__.pydiff --git a/pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.py b/pipelex/migration/migrate_v0_1_0_to_v0_2_0.pysimilarity index 80%rename from pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.pyrename to pipelex/migration/migrate_v0_1_0_to_v0_2_0.pyindex adf889cf8..b11e84fd7 100644--- a/pipelex/tools/migrate/migrate_v0_1_0_to_v0_2_0.py+++ b/pipelex/migration/migrate_v0_1_0_to_v0_2_0.py@@ -5,7 +5,7 @@ import shutilfrom pathlib import Pathfrom typing import Any, ClassVar, Dict, List, Match-from pipelex.tools.migrate.migration_result import MigrationResult+from pipelex.migration.migration_result import MigrationResultclass TomlMigrator:@@ -247,13 +247,68 @@ def migrate_concept_syntax(directory: Path, create_backups: bool = True, dry_run"""Convenience function to migrate TOML files from Concept = to definition = syntax.+ This function accepts either a directory (the historical behavior) or a path to a single TOML file.+Args:- directory: Directory containing TOML files to migrate- create_backups: Whether to create backup files before migration+ directory: Path to a directory containing TOML files to migrate, or a single TOML file+ create_backups: Whether to create backup files before migration (ignored in dry-run)dry_run: If True, only preview changes without applying themReturns:MigrationResult with migration statistics"""migrator = TomlMigrator()- return migrator.migrate_directory(directory, create_backups, dry_run)++ # Support passing a single TOML file path in addition to a directory+ input_path = directory+ if input_path.is_file():+ result = MigrationResult()+ result.files_processed = 1++ if input_path.suffix.lower() != ".toml":+ result.errors.append(f"Provided file is not a .toml file: {input_path}")+ return result++ try:+ with open(input_path, "r", encoding="utf-8") as f:+ content = f.read()+ except Exception as e:+ result.errors.append(f"Failed to read file {input_path}: {e}")+ return result++ if not migrator.needs_migration(content):+ return result++ # Count only concept changes for consistency with directory migration statistics+ def _is_inside_multiline_string(text: str, position: int) -> bool:+ """Local helper to check if a position is inside a TOML multiline string."""+ text_before = text[:position]+ triple_double_quotes = text_before.count('"""')+ triple_single_quotes = text_before.count("'''")+ return (triple_double_quotes % 2 == 1) or (triple_single_quotes % 2 == 1)++ matches = list(migrator.concept_pattern.finditer(content))+ changes_count = 0+ for match in matches:+ if not _is_inside_multiline_string(content, match.start()):+ changes_count += 1++ if dry_run:+ result.files_modified = 1+ result.total_changes = changes_count+ result.modified_files.append(input_path)+ return result++ # Perform actual migration+ try:+ actual_changes = migrator.migrate_file(input_path, create_backup=create_backups)+ result.files_modified = 1+ result.total_changes = actual_changes+ result.modified_files.append(input_path)+ except Exception as e:+ result.errors.append(f"Error processing {input_path}: {e}")++ return result++ # Default: treat as directory (existing behavior)+ return migrator.migrate_directory(input_path, create_backups, dry_run)diff --git a/pipelex/tools/migrate/migration_result.py b/pipelex/migration/migration_result.pysimilarity index 100%rename from pipelex/tools/migrate/migration_result.pyrename to pipelex/migration/migration_result.pydiff --git a/pipelex/pipe_operators/pipe_llm.py b/pipelex/pipe_operators/pipe_llm.pyindex f92f8ac05..62c2294f8 100644--- a/pipelex/pipe_operators/pipe_llm.py+++ b/pipelex/pipe_operators/pipe_llm.py@@ -11,6 +11,7 @@ from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingChoicefrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstractfrom pipelex.config import StaticValidationReaction, get_config+from pipelex.core.concept import Conceptfrom pipelex.core.concept_code_factory import ConceptCodeFactoryfrom pipelex.core.concept_native import NativeConcept, NativeConceptClassfrom pipelex.core.domain import Domain, SpecialDomain@@ -79,7 +80,11 @@ class PipeLLM(PipeOperator):if self.structuring_method is not None:output_concept = get_required_concept(concept_code=self.output_concept_code)if output_concept.structure_class_name == NativeConceptClass.TEXT:- raise PipeDefinitionError(f"Output concept '{self.output_concept_code}' is a Text concept, so it cannot be structured")+ concept_name = Concept.extract_concept_name_from_str(concept_str=self.output_concept_code)+ raise PipeDefinitionError(+ f"Output concept '{self.output_concept_code}' is considered a Text concept, "+ f"so it cannot be structured. Maybe you forgot to add '{concept_name}' to the class registry?"+ )return self@override@@ -305,9 +310,13 @@ class PipeLLM(PipeOperator):))+ is_with_preliminary_text = (+ self.structuring_method == StructuringMethod.PRELIMINARY_TEXT+ ) or get_config().pipelex.structure_config.is_default_text_then_structurellm_prompt_run_params = PipeRunParams.copy_by_injecting_multiplicity(pipe_run_params=pipe_run_params,applied_output_multiplicity=applied_output_multiplicity,+ is_with_preliminary_text=is_with_preliminary_text,)# llm_prompt_1: LLMPrompt = (# await self.pipe_llm_prompt.run_pipe(@@ -340,7 +349,8 @@ class PipeLLM(PipeOperator):log.debug(f"PipeLLM generating {fixed_nb_output} output(s)" if fixed_nb_output else "PipeLLM generating a list of output(s)")llm_prompt_2_factory: Optional[LLMPromptFactoryAbstract]- if structuring_method := self.structuring_method:+ if self.structuring_method:+ structuring_method = cast(StructuringMethod, self.structuring_method)log.debug(f"PipeLLM pipe_code is '{self.code}' and structuring_method is '{structuring_method}'")match structuring_method:case StructuringMethod.DIRECT:diff --git a/pipelex/pipe_operators/pipe_llm_factory.py b/pipelex/pipe_operators/pipe_llm_factory.pyindex d5c3ce043..69d6d420c 100644--- a/pipelex/pipe_operators/pipe_llm_factory.py+++ b/pipelex/pipe_operators/pipe_llm_factory.py@@ -14,6 +14,7 @@ from pipelex.pipe_operators.pipe_jinja2_factory import PipeJinja2Factoryfrom pipelex.pipe_operators.pipe_llm import PipeLLM, StructuringMethodfrom pipelex.pipe_operators.pipe_llm_prompt import PipeLLMPromptfrom pipelex.tools.templating.jinja2_errors import Jinja2TemplateError+from pipelex.tools.templating.template_provider_abstract import TemplateNotFoundErrorfrom pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_lists@@ -100,12 +101,16 @@ class PipeLLMFactory(PipeSpecificFactoryProtocol[PipeLLMBlueprint, PipeLLM]):error_msg += "The prompt template is not provided."raise PipeDefinitionError(error_msg) from excelif pipe_blueprint.prompt is None and pipe_blueprint.prompt_name is None:- # no jinja2 provided, no verbatim name, no fixed text, let's use the pipe code as jinja2 name- user_pipe_jinja2 = PipeJinja2(- code="adhoc_for_user_prompt",- domain=domain_code,- jinja2_name=pipe_code,- )+ # no jinja2 provided, no verbatim name, no fixed text, let's try and use the pipe code as jinja2 name+ try:+ user_pipe_jinja2 = PipeJinja2(+ code="adhoc_for_user_prompt",+ domain=domain_code,+ jinja2_name=pipe_code,+ )+ except TemplateNotFoundError as exc:+ error_msg = f"Jinja2 template not found for pipe '{pipe_code}' in domain '{domain_code}': {exc}."+ raise PipeDefinitionError(error_msg) from excuser_images: List[str] = []if pipe_blueprint.inputs:diff --git a/pipelex/pipe_operators/pipe_llm_prompt.py b/pipelex/pipe_operators/pipe_llm_prompt.pyindex 218bc40c2..7f65b58e4 100644--- a/pipelex/pipe_operators/pipe_llm_prompt.py+++ b/pipelex/pipe_operators/pipe_llm_prompt.py@@ -171,9 +171,15 @@ class PipeLLMPrompt(PipeOperator):# Append output structure prompt if neededif pipe_run_params.dynamic_output_concept_code:- user_text += PipeLLMPrompt.get_output_structure_prompt(output_concept=pipe_run_params.dynamic_output_concept_code)+ user_text += PipeLLMPrompt.get_output_structure_prompt(+ output_concept=pipe_run_params.dynamic_output_concept_code,+ is_with_preliminary_text=pipe_run_params.is_with_preliminary_text or False,+ )else:- user_text += PipeLLMPrompt.get_output_structure_prompt(output_concept=self.output_concept_code)+ user_text += PipeLLMPrompt.get_output_structure_prompt(+ output_concept=self.output_concept_code,+ is_with_preliminary_text=pipe_run_params.is_with_preliminary_text or False,+ )log.verbose(f"User text with {self.output_concept_code=}:\n {user_text}")@@ -231,7 +237,7 @@ class PipeLLMPrompt(PipeOperator):)@staticmethod- def get_output_structure_prompt(output_concept: str) -> str:+ def get_output_structure_prompt(output_concept: str, is_with_preliminary_text: bool) -> str:class_name = Concept.extract_concept_name_from_str(concept_str=output_concept)output_class = get_class_registry().get_class(class_name)if not output_class:@@ -242,14 +248,26 @@ class PipeLLMPrompt(PipeOperator):if not class_structure:return ""- output_structure_prompt = (- f"\n\n---\nRequested output format: The output should be the following class: {class_name}\n"- f"{chr(10).join(class_structure)}\n"- "You do NOT need to output a formatted JSON object, another LLM will take care of that. "- "If you cannot find a value that is Optional, output None for that field."- "However, you MUST clearly output the values for each of these fields in your response.\n---\n"- "DO NOT create information. If the information is not present, output None."- )+ class_structure_str = "\n".join(class_structure)++ # TODO: use proper prompt templating for this+ if is_with_preliminary_text:+ output_structure_prompt = (+ f"\n\n---\nRequested output format: The requested output will be used to define the following class: {class_name}\n"+ f"{class_structure_str}\n"+ "You do NOT need to output a formatted JSON object, another LLM will take care of that. "+ "If you cannot find a value that is Optional, output None for that field. "+ "However, you MUST clearly output the values for each of these fields in your response.\n---\n"+ "DO NOT create information. If the information is not present, output None."+ )+ else:+ output_structure_prompt = (+ f"\n\n---\nRequested output format: The output must conform to the following BaseModel: {class_name}\n"+ f"{class_structure_str}\n"+ "If you cannot find a value that is Optional, output None for that field. "+ "However, you MUST clearly output the values for each of these fields in your response.\n---\n"+ "DO NOT create information. If the information is not present, output None."+ )return output_structure_promptasync def _unravel_text(diff --git a/pipelex/pipe_works/pipe_dry.py b/pipelex/pipe_works/pipe_dry.pyindex 65a5537aa..01be2580a 100644--- a/pipelex/pipe_works/pipe_dry.py+++ b/pipelex/pipe_works/pipe_dry.py@@ -44,6 +44,15 @@ async def dry_run_single_pipe(pipe_code: str) -> str:return f"FAILED: {str(exc)}"+async def dry_run_pipe_codes(pipe_codes: List[str]) -> Dict[str, str]:+ """+ Dry run a list of pipe codes.+ """+ pipe_provider = get_pipe_provider()+ pipes = [pipe_provider.get_required_pipe(pipe_code=pipe_code) for pipe_code in pipe_codes]+ return await dry_run_pipes(pipes=pipes)++async def dry_run_pipes(pipes: List[PipeAbstract]) -> Dict[str, str]:"""Dry run all pipes in the library using ThreadPoolExecutor for true parallelism.diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex f97129248..068171963 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -239,7 +239,10 @@ class Pipelex(metaclass=MetaSingleton):# TODO: add kwargs to make() so that subclasses can employ specific parameters@classmethoddef make(- cls, relative_config_folder_path: Optional[str] = None, absolute_config_folder_path: Optional[str] = None, from_file: Optional[bool] = True+ cls,+ relative_config_folder_path: Optional[str] = None,+ absolute_config_folder_path: Optional[str] = None,+ from_file: Optional[bool] = True,) -> Self:"""Create and initialize a Pipelex instance.diff --git a/pipelex/tools/misc/json_utils.py b/pipelex/tools/misc/json_utils.pyindex 630937d93..a1c528128 100644--- a/pipelex/tools/misc/json_utils.py+++ b/pipelex/tools/misc/json_utils.py@@ -1,5 +1,5 @@import json-from typing import Any, Dict, List, Optional, Tuple, Union, cast+from typing import Any, Dict, List, Mapping, Optional, Tuple, Union, castfrom kajson import kajsonfrom pydantic import BaseModel@@ -8,7 +8,7 @@ from pipelex.tools.exceptions import ToolExceptionfrom pipelex.tools.misc.file_utils import save_text_to_pathfrom pipelex.tools.typing.pydantic_utils import CustomBaseModel-JsonContent = Union[Dict[Any, Any], List[Any]]+JsonContent = Union[Dict[str, Any], List[Any]]class ArgumentTypeError(ToolException):@@ -182,7 +182,7 @@ def deep_update(target_dict: Dict[str, Any], updates: Dict[str, Any]):target_dict[key] = value-def remove_none_values(json_content: JsonContent) -> JsonContent:+def remove_none_values(json_content: JsonContent | Any) -> JsonContent | Any:"""Recursively removes all None values from a JSON-compatible data structure.@@ -214,10 +214,25 @@ def remove_none_values(json_content: JsonContent) -> JsonContent:}"""if isinstance(json_content, dict):- json_content = cast(Dict[str, Any], json_content)- return {key: remove_none_values(json_content=value) for key, value in json_content.items() if value is not None}- else:+ json_content = cast(Dict[str, Any], json_content) # pyright: ignore[reportUnnecessaryCast]+ cleaned_dict: Dict[str, Any] = {}+ for key, value in json_content.items():+ if value is not None:+ cleaned_dict[key] = remove_none_values(json_content=value)+ return cleaned_dict+ elif isinstance(json_content, list):+ json_content = cast(List[Any], json_content) # pyright: ignore[reportUnnecessaryCast]return [remove_none_values(item) for item in json_content]+ else:+ return json_content+++def remove_none_values_from_dict(data: Mapping[str, Any]) -> Dict[str, Any]:+ processed = remove_none_values(json_content=data)+ if not isinstance(processed, dict):+ raise JsonTypeError("Removing None values from a dict, we expected a dict in return")+ processed = cast(Dict[str, Any], processed) # pyright: ignore[reportUnnecessaryCast]+ return processeddef purify_json(diff --git a/pipelex/tools/misc/toml_utils.py b/pipelex/tools/misc/toml_utils.pyindex 2316100cf..4dd7d9a79 100644--- a/pipelex/tools/misc/toml_utils.py+++ b/pipelex/tools/misc/toml_utils.py@@ -1,3 +1,5 @@+from __future__ import annotations+from typing import Any, Dict, List, Optionalimport toml@@ -53,7 +55,7 @@ def validate_toml_file(path: str) -> None:_validate_toml_content(content, path)-def _clean_trailing_whitespace(content: str) -> str:+def clean_trailing_whitespace(content: str) -> str:"""Clean trailing whitespace from TOML content.This function:@@ -98,7 +100,7 @@ def load_toml_from_path(path: str) -> Dict[str, Any]:with open(path, "r", encoding="utf-8") as file:content = file.read()- cleaned_content = _clean_trailing_whitespace(content)+ cleaned_content = clean_trailing_whitespace(content)# If content changed, write it backif content != cleaned_content:@@ -120,29 +122,3 @@ def failable_load_toml_from_path(path: str) -> Optional[Dict[str, Any]]:except toml.TomlDecodeError as exc:print(f"Failed to parse TOML file '{path}': {exc}")return None---def save_toml_to_path(data: Dict[str, Any], path: str) -> None:- """Save dictionary as TOML to file path.-- Args:- data: Dictionary to save as TOML- path: Path where to save the TOML file- """- with open(path, "w", encoding="utf-8") as file:- toml_content = toml.dumps(data)- cleaned_content = _clean_trailing_whitespace(toml_content)- file.write(cleaned_content)---def dict_to_toml_string(data: Dict[str, Any]) -> str:- """Convert dictionary to TOML string format.-- Args:- data: Dictionary to convert to TOML-- Returns:- TOML formatted string- """- toml_content = toml.dumps(data)- return _clean_trailing_whitespace(toml_content)diff --git a/pyproject.toml b/pyproject.tomlindex 2c7636cd0..8d2103775 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -39,6 +39,7 @@ dependencies = ["rich>=13.8.1","shortuuid>=1.0.13","toml>=0.10.2",+ "tomlkit>=0.13.2","typer>=0.16.0","typing-extensions>=4.13.2","yattag>=1.15.2",diff --git a/tests/e2e/pipelex/tools/migrate/__init__.py b/tests/e2e/pipelex/migration/__init__.pysimilarity index 100%rename from tests/e2e/pipelex/tools/migrate/__init__.pyrename to tests/e2e/pipelex/migration/__init__.pydiff --git a/tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.py b/tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.pysimilarity index 99%rename from tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.pyrename to tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.pyindex a896b429a..635c55de3 100644--- a/tests/e2e/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0_e2e.py+++ b/tests/e2e/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0_e2e.py@@ -5,7 +5,7 @@ from typing import Any, List, Tupleimport pytest-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import migrate_concept_syntax+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import migrate_concept_syntaxclass TestMigrationE2E:@@ -336,7 +336,7 @@ Concept = "Test concept"import shutil# Copy the test data file to our temp directory- test_data_path = Path(__file__).parent.parent.parent.parent.parent / "data" / "test_migrate_v0_1_0_to_v0_2_0.toml"+ test_data_path = Path(__file__).parent.parent.parent.parent / "data" / "test_migrate_v0_1_0_to_v0_2_0.toml"test_file = tmp_path / "test_migrate.toml"shutil.copy2(test_data_path, test_file)diff --git a/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.pynew file mode 100644index 000000000..1e1805bba--- /dev/null+++ b/tests/integration/pipelex/pipes/pipe_operator/pipe_llm/test_generate_pipeline_blueprint.py@@ -0,0 +1,46 @@+import pytest++from pipelex import pretty_print+from pipelex.core.pipe_run_params import PipeRunMode+from pipelex.create.helpers import get_support_file+from pipelex.libraries.pipeline_blueprint import PipelineBlueprint+from pipelex.pipeline.execute import execute_pipeline++[email protected](loop_scope="class")[email protected]_runnable+ "requirements",+ [+ (+ "Create a domain 'sample' to describe simple demo pipelines. "+ "Define concept ThingDescription = 'A short natural language description of a thing'. "+ "Add a pipe write_thing that takes input { text = Text } and outputs ThingDescription using PipeLLM."+ ),+ (+ "Domain: gadgets. Concept: GadgetSpec = 'Technical description of a gadget'. "+ "Add pipe summarize_gadget: inputs { text = Text } -> output GadgetSpec via PipeLLM."+ ),+ ],+)+async def test_generate_pipeline_blueprint(pipe_run_mode: PipeRunMode, requirements: str):+ pipe_output = await execute_pipeline(+ pipe_code="build_blueprint",+ input_memory={+ "domain": "test_domain",+ "pipeline_name": "test_pipeline",+ "requirements": requirements,+ "draft_pipeline_rules": get_support_file(subpath="create/draft_pipelines.md"),+ "build_pipeline_rules": get_support_file(subpath="create/build_pipelines.md"),+ },+ pipe_run_mode=pipe_run_mode,+ )++ blueprint = pipe_output.main_stuff_as(content_type=PipelineBlueprint)+ assert isinstance(blueprint, PipelineBlueprint)+ # Basic sanity checks+ assert isinstance(blueprint.domain, str) and blueprint.domain != ""++ pretty_print(blueprint, title="PipelineBlueprint")diff --git a/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py b/tests/integration/pipelex/pipes/test_pipe_llm_prompt.pyindex be5872265..7aa68df24 100644--- a/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py+++ b/tests/integration/pipelex/pipes/test_pipe_llm_prompt.py@@ -49,7 +49,9 @@ class TestPipeLLMPrompt:assert pipe_output.main_stuff is not Noneassert pipe_output.llm_prompt is not Noneassert pipe_output.llm_prompt.user_text is not None- assert pipe_output.llm_prompt.user_text.endswith(PipeLLMPrompt.get_output_structure_prompt(NativeConcept.TEXT.code))+ assert pipe_output.llm_prompt.user_text.endswith(+ PipeLLMPrompt.get_output_structure_prompt(NativeConcept.TEXT.code, is_with_preliminary_text=False)+ )async def test_prompt_with_images(self, pipe_run_mode: PipeRunMode):"""Test prompt with image inputs."""@@ -109,7 +111,7 @@ class TestPipeLLMPrompt:assert pipe_output.main_stuff is not None# Verify output structure is appended- output_structure = pipe_llm_prompt.get_output_structure_prompt(concept_code)+ output_structure = pipe_llm_prompt.get_output_structure_prompt(concept_code, is_with_preliminary_text=False)assert pipe_output.llm_prompt is not Noneassert pipe_output.llm_prompt.user_text is not Nonediff --git a/tests/test_pipelines/discord_newsletter.toml b/tests/test_pipelines/discord_newsletter.tomlindex 600f6c780..eb84f8682 100644--- a/tests/test_pipelines/discord_newsletter.toml+++ b/tests/test_pipelines/discord_newsletter.toml@@ -7,7 +7,8 @@ ChannelSummary = "A summarized Discord channel for newsletter inclusion"Newsletter = "The final newsletter content with organized channel summaries"[pipe.write_discord_newsletter]-PipeSequence = "Create a newsletter from Discord articles by summarizing channels and organizing content"+type = "PipeSequence"+definition = "Create a newsletter from Discord articles by summarizing channels and organizing content"inputs = { discord_channel_updates = "DiscordChannelUpdate" }output = "Newsletter"steps = [@@ -15,7 +16,8 @@ steps = [][pipe.summarize_discord_channel_update]-PipeLLM = "Summarize a Discord channel's messages into newsletter-friendly content"+type = "PipeLLM"+definition = "Summarize a Discord channel's messages into newsletter-friendly content"inputs = { discord_channel_update = "DiscordChannelUpdate" }output = "ChannelSummary"system_prompt = "You are a newsletter editor who creates engaging summaries of Discord channel content. You extract key information, preserve important links, and write in a clear, concise style suitable for newsletter readers."diff --git a/tests/test_pipelines/failing_pipelines.toml b/tests/test_pipelines/failing_pipelines.tomlindex ac775a9ff..64323fac3 100644--- a/tests/test_pipelines/failing_pipelines.toml+++ b/tests/test_pipelines/failing_pipelines.toml@@ -4,7 +4,8 @@ domain = "failing_pipelines"[pipe][pipe.dummy]-PipeLLM = "This pipe is a dummy pipe"+type = "PipeLLM"+definition = "This pipe is a dummy pipe"output = "Text"llm = { llm_handle = "gpt-4o-mini", temperature = 1, max_tokens = 50 }prompt_template = """@@ -12,7 +13,8 @@ This is a dummy prompt, do whatever you want. Something funny but not a joke, ok"""[pipe.infinite_loop_1]-PipeSequence = "This pipe will cause an infinite loop"+type = "PipeSequence"+definition = "This pipe will cause an infinite loop"output = "Text"steps = [{ pipe = "dummy", result = "dummy_result" },diff --git a/tests/test_pipelines/misc_tests/flows.toml b/tests/test_pipelines/misc_tests/flows.tomlindex dfb49b264..86674a112 100644--- a/tests/test_pipelines/misc_tests/flows.toml+++ b/tests/test_pipelines/misc_tests/flows.toml@@ -8,7 +8,8 @@ Color = "A color"[pipe][pipe.choose_colors]-PipeLLM = "Choose Colors"+type = "PipeLLM"+definition = "Choose Colors"output = "Color"prompt_template = """Choose {{ _nb_output }} colors.@@ -16,7 +17,8 @@ Choose {{ _nb_output }} colors.llm = "llm_for_creative_writing"[pipe.sequence_for_batch_test]-PipeSequence = "Sequence for parallel test"+type = "PipeSequence"+definition = "Sequence for parallel test"inputs = { color = "Color" }output = "Color"steps = [@@ -25,7 +27,8 @@ steps = [][pipe.capitalize_color]-PipeLLM = "Capitalize Colors"+type = "PipeLLM"+definition = "Capitalize Colors"inputs = { color = "Color" }output = "Color"prompt_template = """@@ -38,7 +41,8 @@ Output only the word, nothing else."""[pipe.capitalize_last_letter]-PipeLLM = "Capitalize Last Letter"+type = "PipeLLM"+definition = "Capitalize Last Letter"inputs = { capitalized_color = "Color" }output = "Color"prompt_template = """diff --git a/tests/test_pipelines/misc_tests/multiplicity.toml b/tests/test_pipelines/misc_tests/multiplicity.tomlindex 91c182b2b..1f8957e2a 100644--- a/tests/test_pipelines/misc_tests/multiplicity.toml+++ b/tests/test_pipelines/misc_tests/multiplicity.toml@@ -9,7 +9,8 @@ ProductOfNature = "Something produced by Nature"FantasyScene = "A fantasy scene"[pipe.original_power_ranger_colors]-PipeLLM = "Recall the colors of the original power rangers"+type = "PipeLLM"+definition = "Recall the colors of the original power rangers"output = "Color"prompt_template = """Remind me of the colors of the {{ _nb_output }} original power rangers.@@ -18,7 +19,8 @@ llm = "llm_for_creative_writing"nb_output = 5[pipe.alltime_power_ranger_colors]-PipeLLM = "Recall the colors of all the power rangers"+type = "PipeLLM"+definition = "Recall the colors of all the power rangers"output = "Color"prompt_template = """Remind me of the colors of all the power rangers over all series.@@ -27,7 +29,8 @@ Don't list the names of the characters or the name of the series: just list thellm = "llm_for_creative_writing"[pipe.imagine_nature_product]-PipeLLM = "Imagine a product of nature"+type = "PipeLLM"+definition = "Imagine a product of nature"inputs = { color = "Color" }output = "ProductOfNature"prompt_template = """@@ -36,7 +39,8 @@ Just state what it is in a single sentence."""[pipe.imagine_fantasy_scene_including_products_of_nature]-PipeLLM = "Imagine a fantasy scene including products of nature"+type = "PipeLLM"+definition = "Imagine a fantasy scene including products of nature"inputs = { product_of_nature = "ProductOfNature" }output = "FantasyScene"prompt_template = """@@ -48,7 +52,8 @@ Keep it short and concise, just one sentence."""[pipe.imagine_nature_scene_of_original_power_rangers_colors]-PipeSequence = "Imagine nature scenes of Power Rangers colors"+type = "PipeSequence"+definition = "Imagine nature scenes of Power Rangers colors"output = "Image"steps = [{ pipe = "original_power_ranger_colors", result = "color" },@@ -58,7 +63,8 @@ steps = [[pipe.imagine_nature_scene_of_alltime_power_rangers_colors]-PipeSequence = "Imagine nature scenes of Power Rangers colors"+type = "PipeSequence"+definition = "Imagine nature scenes of Power Rangers colors"output = "Image"steps = [{ pipe = "alltime_power_ranger_colors", result = "color", multiple_output = true },diff --git a/tests/test_pipelines/misc_tests/subfolder_1/cars.toml b/tests/test_pipelines/misc_tests/subfolder_1/cars.tomlindex e07230d48..47c961a64 100644--- a/tests/test_pipelines/misc_tests/subfolder_1/cars.toml+++ b/tests/test_pipelines/misc_tests/subfolder_1/cars.toml@@ -10,7 +10,8 @@ CarDescription = "A detailed description of a car model"[pipe][pipe.generate_car_description]-PipeLLM = "Generate a description of a car"+type = "PipeLLM"+definition = "Generate a description of a car"inputs = { car = "Car" }output = "CarDescription"prompt_template = """diff --git a/tests/test_pipelines/misc_tests/subfolder_2/animals.toml b/tests/test_pipelines/misc_tests/subfolder_2/animals.tomlindex 02dc21291..f35a6c846 100644--- a/tests/test_pipelines/misc_tests/subfolder_2/animals.toml+++ b/tests/test_pipelines/misc_tests/subfolder_2/animals.toml@@ -10,7 +10,8 @@ AnimalDescription = "A detailed description of an animal"[pipe][pipe.generate_animal_description]-PipeLLM = "Generate a description of an animal"+type = "PipeLLM"+definition = "Generate a description of an animal"inputs = { animal = "Animal" }output = "AnimalDescription"prompt_template = """diff --git a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml b/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.tomlindex dfe99d7ea..2a879f1d9 100644--- a/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml+++ b/tests/test_pipelines/misc_tests/subfolder_2/subfolder/flowers.toml@@ -8,7 +8,8 @@ FlowerDescription = "A detailed description of a flower"[pipe][pipe.generate_flower_description]-PipeLLM = "Generate a description of a flower"+type = "PipeLLM"+definition = "Generate a description of a flower"inputs = { flower = "Flower" }output = "FlowerDescription"prompt_template = """diff --git a/tests/test_pipelines/misc_tests/test_image_inputs.toml b/tests/test_pipelines/misc_tests/test_image_inputs.tomlindex 0acd25588..329418215 100644--- a/tests/test_pipelines/misc_tests/test_image_inputs.toml+++ b/tests/test_pipelines/misc_tests/test_image_inputs.toml@@ -6,7 +6,8 @@ Article = "Article"[pipe][pipe.extract_article_from_image]-PipeLLM = "Describe a page content"+type = "PipeLLM"+definition = "Describe a page content"inputs = { image = "Image" }output = "Article"system_prompt = """@@ -17,7 +18,8 @@ Extract the text and the date of the article in the image."""[pipe.describe_page]-PipeLLM = "Describe a page"+type = "PipeLLM"+definition = "Describe a page"inputs = { "page.page_view" = "Image", page = "Page" }output = "Article"system_prompt = """diff --git a/tests/test_pipelines/misc_tests/test_jinja2.toml b/tests/test_pipelines/misc_tests/test_jinja2.tomlindex 664922f79..9a0d44398 100644--- a/tests/test_pipelines/misc_tests/test_jinja2.toml+++ b/tests/test_pipelines/misc_tests/test_jinja2.toml@@ -2,7 +2,8 @@ domain = "test_jinja2"definition = "This library is intended for testing Jinja2"[pipe.jinja2_test_1]-PipeJinja2 = "Jinja2 test 1"+type = "PipeJinja2"+definition = "Jinja2 test 1"inputs = { text = "Text" }output = "Text"jinja2 = """diff --git a/tests/test_pipelines/misc_tests/tests.toml b/tests/test_pipelines/misc_tests/tests.tomlindex 378121f5a..d5e91c743 100644--- a/tests/test_pipelines/misc_tests/tests.toml+++ b/tests/test_pipelines/misc_tests/tests.toml@@ -9,7 +9,8 @@ ArticleAndCritic = "An article and a critique of it"Complex = "A complex object"[pipe.simple_llm_test_from_image]-PipeLLM = "Simple LLM test from image"+type = "PipeLLM"+definition = "Simple LLM test from image"inputs = { image = "Image" }output = "Text"prompt_template = """@@ -17,7 +18,8 @@ Describe the using rap lyrics, including puns and references to the image."""[pipe.test_no_input]-PipeLLM = "No Input"+type = "PipeLLM"+definition = "No Input"output = "Text"llm = "llm_for_testing_gen_text"prompt_template = """@@ -25,7 +27,8 @@ Explain that this is a test prompt which took no input from the user."""[pipe.test_no_input_that_could_be_long]-PipeLLM = "No Input but generates a text that could be long"+type = "PipeLLM"+definition = "No Input but generates a text that could be long"output = "Text"llm = { llm_handle = "gpt-4o-mini", temperature = 0.5, max_tokens = 100 }prompt_template = """diff --git a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.tomlindex 166ed5844..b13d3ad15 100644--- a/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_batch/uppercase_transformer.toml@@ -7,7 +7,8 @@ TextList = "List of Text"[pipe][pipe.test_pipe_batch]-PipeBatch = "Test pipe batch"+type = "PipeBatch"+definition = "Test pipe batch"inputs = { text_item = "Text", text_list = "TextList" }output = "UppercaseText"branch_pipe_code = "uppercase_transformer"@@ -15,7 +16,8 @@ input_list_name = "text_list"input_item_name = "text_item"[pipe.uppercase_transformer]-PipeLLM = "Transform text to uppercase"+type = "PipeLLM"+definition = "Transform text to uppercase"inputs = { text_item = "Text" }output = "UppercaseText"prompt_template = """diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.tomlindex bef33ccdc..531fafd66 100644--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_1.toml@@ -6,7 +6,8 @@ CategoryInput = "Input with a category field"[pipe][pipe.basic_condition_by_category]-PipeCondition = "Route based on category field"+type = "PipeCondition"+definition = "Route based on category field"inputs = { input_data = "CategoryInput" }output = "native.Text"expression_template = "{{ input_data.category }}"@@ -17,21 +18,24 @@ medium = "process_medium"large = "process_large"[pipe.process_small]-PipeLLM = "Generate random text for small items"+type = "PipeLLM"+definition = "Generate random text for small items"output = "native.Text"prompt_template = """Output this only: "small""""[pipe.process_medium]-PipeLLM = "Generate random text for medium items"+type = "PipeLLM"+definition = "Generate random text for medium items"output = "native.Text"prompt_template = """Output this only: "medium""""[pipe.process_large]-PipeLLM = "Generate random text for large items"+type = "PipeLLM"+definition = "Generate random text for large items"output = "native.Text"prompt_template = """Output this only: "large"diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.tomlindex 0b4050843..a9c21cd11 100644--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_2.toml@@ -6,7 +6,8 @@ CategoryInput = "Input with a category field"[pipe][pipe.basic_condition_by_category_2]-PipeCondition = "Route based on category field using expression"+type = "PipeCondition"+definition = "Route based on category field using expression"inputs = { input_data = "CategoryInput" }output = "native.Text"expression = "input_data.category"@@ -17,21 +18,24 @@ medium = "process_medium_2"large = "process_large_2"[pipe.process_small_2]-PipeLLM = "Generate random text for small items"+type = "PipeLLM"+definition = "Generate random text for small items"output = "native.Text"prompt_template = """Output this only: "small""""[pipe.process_medium_2]-PipeLLM = "Generate random text for medium items"+type = "PipeLLM"+definition = "Generate random text for medium items"output = "native.Text"prompt_template = """Output this only: "medium""""[pipe.process_large_2]-PipeLLM = "Generate random text for large items"+type = "PipeLLM"+definition = "Generate random text for large items"output = "native.Text"prompt_template = """Output this only: "large"diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.tomlindex 9ea535763..54c1ea33d 100644--- a/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/pipe_condition_complex.toml@@ -9,7 +9,8 @@ ProcessingContext = "Combined processing context"[pipe]# Main entry point - routes by document type first[pipe.complex_document_processor]-PipeCondition = "Primary routing by document type"+type = "PipeCondition"+definition = "Primary routing by document type"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"expression_template = "{{ doc_request.document_type }}"@@ -21,7 +22,8 @@ legal = "legal_document_router"# Technical document processing branch[pipe.technical_document_router]-PipeCondition = "Route technical documents by priority and user level"+type = "PipeCondition"+definition = "Route technical documents by priority and user level"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"expression_template = "{% if doc_request.priority == 'urgent' %}urgent_tech{% elif user_profile.user_level == 'expert' and doc_request.complexity == 'high' %}expert_tech{% else %}standard_tech{% endif %}"@@ -33,7 +35,8 @@ standard_tech = "standard_technical_processor"# Business document processing branch[pipe.business_document_router]-PipeCondition = "Route business documents by department and priority"+type = "PipeCondition"+definition = "Route business documents by department and priority"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"expression_template = "{% if doc_request.priority == 'urgent' %}urgent_business{% elif user_profile.department == 'finance' %}finance_business{% elif user_profile.department == 'marketing' %}marketing_business{% else %}general_business{% endif %}"@@ -46,7 +49,8 @@ general_business = "general_business_processor"# Legal document processing branch[pipe.legal_document_router]-PipeCondition = "Route legal documents by complexity and user level"+type = "PipeCondition"+definition = "Route legal documents by complexity and user level"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"expression_template = "{% if doc_request.complexity == 'high' and user_profile.user_level != 'beginner' %}complex_legal{% elif doc_request.language != 'english' %}international_legal{% else %}standard_legal{% endif %}"@@ -58,7 +62,8 @@ standard_legal = "standard_legal_processor"# Leaf processors - Technical[pipe.urgent_technical_processor]-PipeLLM = "Process urgent technical documents with high priority"+type = "PipeLLM"+definition = "Process urgent technical documents with high priority"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -73,7 +78,8 @@ Output: "URGENT_TECHNICAL_PROCESSED""""[pipe.expert_technical_processor]-PipeLLM = "Process complex technical documents for expert users"+type = "PipeLLM"+definition = "Process complex technical documents for expert users"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -86,7 +92,8 @@ Output: "EXPERT_TECHNICAL_PROCESSED""""[pipe.standard_technical_processor]-PipeLLM = "Process standard technical documents"+type = "PipeLLM"+definition = "Process standard technical documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -99,7 +106,8 @@ Output: "STANDARD_TECHNICAL_PROCESSED"# Leaf processors - Business[pipe.urgent_business_processor]-PipeLLM = "Process urgent business documents"+type = "PipeLLM"+definition = "Process urgent business documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -111,7 +119,8 @@ Output: "URGENT_BUSINESS_PROCESSED""""[pipe.finance_business_processor]-PipeLLM = "Process finance business documents"+type = "PipeLLM"+definition = "Process finance business documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -123,7 +132,8 @@ Output: "FINANCE_BUSINESS_PROCESSED""""[pipe.marketing_business_processor]-PipeLLM = "Process marketing business documents"+type = "PipeLLM"+definition = "Process marketing business documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -135,7 +145,8 @@ Output: "MARKETING_BUSINESS_PROCESSED""""[pipe.general_business_processor]-PipeLLM = "Process general business documents"+type = "PipeLLM"+definition = "Process general business documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -148,7 +159,8 @@ Output: "GENERAL_BUSINESS_PROCESSED"# Leaf processors - Legal[pipe.complex_legal_processor]-PipeLLM = "Process complex legal documents"+type = "PipeLLM"+definition = "Process complex legal documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """@@ -161,7 +173,8 @@ Output: "COMPLEX_LEGAL_PROCESSED""""[pipe.international_legal_processor]-PipeLLM = "Process international legal documents"+type = "PipeLLM"+definition = "Process international legal documents"inputs = { doc_request = "DocumentRequest" }output = "native.Text"prompt_template = """@@ -173,7 +186,8 @@ Output: "INTERNATIONAL_LEGAL_PROCESSED""""[pipe.standard_legal_processor]-PipeLLM = "Process standard legal documents"+type = "PipeLLM"+definition = "Process standard legal documents"inputs = { doc_request = "DocumentRequest", user_profile = "UserProfile" }output = "native.Text"prompt_template = """diff --git a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml b/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.tomlindex 46a1c41ab..6410f0f55 100644--- a/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_condition/text_length_condition.toml@@ -4,7 +4,8 @@ definition = "Test pipes for PipeCondition based on text length"[pipe][pipe.capitalize_long_text]-PipeLLM = "Capitalize text for long inputs"+type = "PipeLLM"+definition = "Capitalize text for long inputs"inputs = { input_text = "Text" }output = "Text"prompt_template = """Transform the following LONG text to uppercase and add "LONG: " prefix:@@ -14,7 +15,8 @@ prompt_template = """Transform the following LONG text to uppercase and add "LONReturn only the prefixed uppercase text, nothing else."""[pipe.add_prefix_short_text]-PipeLLM = "Add prefix for short text inputs"+type = "PipeLLM"+definition = "Add prefix for short text inputs"inputs = { input_text = "Text" }output = "Text"prompt_template = """Add the prefix "SHORT: " to the beginning of the following text:diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml b/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.tomlindex a898a83c3..6c359aab5 100644--- a/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_parallel/parallel_text_analysis.toml@@ -4,7 +4,8 @@ definition = "Test pipes for PipeParallel text analysis"[pipe][pipe.analyze_sentiment]-PipeLLM = "Analyze sentiment of text"+type = "PipeLLM"+definition = "Analyze sentiment of text"inputs = { input_text = "Text" }output = "Text"prompt_template = """Analyze the sentiment of the following text and return only one word: positive, negative, or neutral.@@ -14,7 +15,8 @@ prompt_template = """Analyze the sentiment of the following text and return onlySentiment:"""[pipe.count_words]-PipeLLM = "Count words in text"+type = "PipeLLM"+definition = "Count words in text"inputs = { input_text = "Text" }output = "Text"prompt_template = """Count the number of words in the following text and return only the number:@@ -24,7 +26,8 @@ prompt_template = """Count the number of words in the following text and returnWord count:"""[pipe.extract_keywords]-PipeLLM = "Extract keywords from text"+type = "PipeLLM"+definition = "Extract keywords from text"inputs = { input_text = "Text" }output = "Text"prompt_template = """Extract the top 3 keywords from the following text. Return them as a comma-separated list:diff --git a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml b/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.tomlindex 7fb132521..78c8d8619 100644--- a/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_parallel/pipe_parallel_1.toml@@ -9,7 +9,8 @@ CombinedAnalysis = "Combined analysis results from parallel processing"[pipe][pipe.parallel_document_analysis]-PipeParallel = "Analyze document length and content in parallel"+type = "PipeParallel"+definition = "Analyze document length and content in parallel"inputs = { document = "DocumentInput" }output = "CombinedAnalysis"add_each_output = true@@ -20,7 +21,8 @@ parallels = [][pipe.analyze_length]-PipeLLM = "Analyze document length and structure"+type = "PipeLLM"+definition = "Analyze document length and structure"inputs = { document = "DocumentInput" }output = "LengthAnalysis"llm = "llm_for_testing_gen_text"@@ -36,7 +38,8 @@ Provide a brief analysis focusing on:"""[pipe.analyze_content]-PipeLLM = "Analyze document content and themes"+type = "PipeLLM"+definition = "Analyze document content and themes"inputs = { document = "DocumentInput" }output = "ContentAnalysis"llm = "llm_for_testing_gen_text"diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.tomlindex c4d279e63..87694b705 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/capitalize_text.toml@@ -3,7 +3,8 @@ definition = "Test pipe for capitalizing text to uppercase"[pipe][pipe.capitalize_text]-PipeLLM = "Transform text to uppercase"+type = "PipeLLM"+definition = "Transform text to uppercase"inputs = { input_text = "Text" }output = "Text"prompt_template = """Transform the following text to uppercase:@@ -13,7 +14,8 @@ prompt_template = """Transform the following text to uppercase:Return only the uppercase text, nothing else."""[pipe.add_prefix]-PipeLLM = "Add PROCESSED prefix to text"+type = "PipeLLM"+definition = "Add PROCESSED prefix to text"inputs = { capitalized_text = "Text" }output = "Text"prompt_template = """Add the prefix "PROCESSED: " to the beginning of the following text:diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.tomlindex 383388cf5..5a8fdb1d3 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_1.toml@@ -9,7 +9,8 @@ SummaryText = "A summary of the processed text"[pipe][pipe.simple_text_sequence]-PipeSequence = "Process text through cleaning and summarization"+type = "PipeSequence"+definition = "Process text through cleaning and summarization"inputs = { raw_text = "RawText" }output = "SummaryText"steps = [@@ -18,7 +19,8 @@ steps = [][pipe.clean_text]-PipeLLM = "Clean and normalize the input text"+type = "PipeLLM"+definition = "Clean and normalize the input text"inputs = { raw_text = "RawText" }output = "CleanedText"llm = "llm_for_testing_gen_text"@@ -32,7 +34,8 @@ Clean and normalize the following text by:"""[pipe.generate_summary]-PipeLLM = "Generate a summary of the cleaned text"+type = "PipeLLM"+definition = "Generate a summary of the cleaned text"inputs = { cleaned_text = "CleanedText" }output = "SummaryText"llm = "llm_for_testing_gen_text"diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.tomlindex b8613da31..689564787 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_2.toml@@ -10,7 +10,8 @@ Document = "A document containing multiple customer reviews"[pipe][pipe.analyze_reviews_sequence]-PipeSequence = "Process customer reviews with sentiment analysis"+type = "PipeSequence"+definition = "Process customer reviews with sentiment analysis"inputs = { document = "Document" }output = "ProductRating"steps = [@@ -20,7 +21,8 @@ steps = [][pipe.extract_individual_reviews]-PipeLLM = "Extract individual reviews from document"+type = "PipeLLM"+definition = "Extract individual reviews from document"inputs = { document = "Document" }output = "CustomerReview"nb_output = 1@@ -32,7 +34,8 @@ Extract each individual customer review from this document as separate items:"""[pipe.analyze_review_sentiment]-PipeLLM = "Analyze sentiment of a single review"+type = "PipeLLM"+definition = "Analyze sentiment of a single review"inputs = { single_review = "CustomerReview" }output = "SentimentAnalysis"llm = "llm_for_testing_gen_text"@@ -45,7 +48,8 @@ Provide sentiment (positive/negative/neutral) and confidence score."""[pipe.aggregate_review_results]-PipeLLM = "Aggregate all review analyses into final rating"+type = "PipeLLM"+definition = "Aggregate all review analyses into final rating"inputs = { sentiment_analyses = "SentimentAnalysis" }output = "ProductRating"llm = "llm_for_testing_gen_text"diff --git a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.tomlindex bd9f2220e..ee60d0f72 100644--- a/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml+++ b/tests/test_pipelines/pipe_controllers/pipe_sequence/pipe_sequence_3.toml@@ -11,7 +11,8 @@ BestIdea = "The top-ranked creative idea with justification"[pipe][pipe.creative_ideation_sequence]-PipeSequence = "Generate multiple ideas, analyze each individually, then select the best"+type = "PipeSequence"+definition = "Generate multiple ideas, analyze each individually, then select the best"inputs = { topic = "CreativeTopic" }output = "BestIdea"steps = [@@ -22,7 +23,8 @@ steps = [][pipe.generate_multiple_ideas]-PipeLLM = "Generate exactly 5 creative ideas for a topic"+type = "PipeLLM"+definition = "Generate exactly 5 creative ideas for a topic"inputs = { topic = "CreativeTopic" }output = "CreativeIdea"nb_output = 5@@ -41,7 +43,8 @@ Ideas:"""[pipe.brainstorm_concepts]-PipeLLM = "Brainstorm variable number of concepts"+type = "PipeLLM"+definition = "Brainstorm variable number of concepts"inputs = { topic = "CreativeTopic" }output = "CreativeIdea"multiple_output = true@@ -55,7 +58,8 @@ Each concept should be briefly described."""[pipe.analyze_single_idea]-PipeLLM = "Analyze a single idea in detail"+type = "PipeLLM"+definition = "Analyze a single idea in detail"inputs = { single_idea = "CreativeIdea" }output = "IdeaAnalysis"llm = "llm_for_testing_gen_text"@@ -75,7 +79,8 @@ Provide a comprehensive assessment."""[pipe.evaluate_all_ideas]-PipeLLM = "Evaluate and rank all analyzed ideas"+type = "PipeLLM"+definition = "Evaluate and rank all analyzed ideas"inputs = { detailed_analyses = "IdeaAnalysis" }output = "IdeaEvaluation"llm = "llm_for_testing_gen_text"@@ -91,7 +96,8 @@ Provide:"""[pipe.select_best_idea]-PipeLLM = "Select the best idea based on evaluation"+type = "PipeLLM"+definition = "Select the best idea based on evaluation"inputs = { evaluation = "IdeaEvaluation" }output = "BestIdea"llm = "llm_for_testing_gen_text"diff --git a/tests/test_pipelines/tricky_questions.toml b/tests/test_pipelines/tricky_questions.tomlindex f15dcd7e3..04e0648f1 100644--- a/tests/test_pipelines/tricky_questions.toml+++ b/tests/test_pipelines/tricky_questions.toml@@ -14,7 +14,8 @@ refines = "AnswerToAQuestion"[pipe][pipe.analyse_question_tricky]-PipeLLM = "Analyze a question to determine whether it's straightforward or tricky"+type = "PipeLLM"+definition = "Analyze a question to determine whether it's straightforward or tricky"inputs = { question = "Question" }output = "QuestionAnalysis"llm = "llm_to_reason"@@ -29,7 +30,8 @@ If there's an obvious trap, state it without getting into details."""[pipe.answer_after_analysis]-PipeLLM = "Answer knowingly after analyzing a question"+type = "PipeLLM"+definition = "Answer knowingly after analyzing a question"inputs = { question = "Question", question_analysis = "QuestionAnalysis" }output = "ThoughtfulAnswer"llm = "llm_to_reason"@@ -47,14 +49,16 @@ Answer in 4 parts:"""[pipe.conclude_thoughtful_answer]-PipeJinja2 = "Conclude a thoughtful answer"+type = "PipeJinja2"+definition = "Conclude a thoughtful answer"inputs = { thoughtful_answer = "ThoughtfulAnswer" }output = "ThoughtfulAnswerConclusion"jinja2 = "After analyzing the question, here is my answer: $thoughtful_answer.the_answer"[pipe.conclude_tricky_question_by_steps]-PipeSequence = "Answer a tricky question by first analyzing its trickiness and then concluding"+type = "PipeSequence"+definition = "Answer a tricky question by first analyzing its trickiness and then concluding"inputs = { question = "Question" }output = "ThoughtfulAnswerConclusion"steps = [diff --git a/tests/unit/pipelex/tools/migrate/__init__.py b/tests/unit/pipelex/migration/__init__.pysimilarity index 100%rename from tests/unit/pipelex/tools/migrate/__init__.pyrename to tests/unit/pipelex/migration/__init__.pydiff --git a/tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.py b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.pysimilarity index 99%rename from tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.pyrename to tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.pyindex ae61ecfa0..747ecbaa7 100644--- a/tests/unit/pipelex/tools/migrate/test_migrate_v0_1_0_to_v0_2_0.py+++ b/tests/unit/pipelex/migration/test_migrate_v0_1_0_to_v0_2_0.py@@ -2,7 +2,7 @@ from pathlib import Pathimport pytest-from pipelex.tools.migrate.migrate_v0_1_0_to_v0_2_0 import MigrationResult, TomlMigrator, migrate_concept_syntax+from pipelex.migration.migrate_v0_1_0_to_v0_2_0 import MigrationResult, TomlMigrator, migrate_concept_syntaxclass TestTomlMigrator:diff --git a/uv.lock b/uv.lockindex 573525370..ae4ea3189 100644--- a/uv.lock+++ b/uv.lock@@ -1773,6 +1773,7 @@ dependencies = [{ name = "rich" },{ name = "shortuuid" },{ name = "toml" },+ { name = "tomlkit" },{ name = "typer" },{ name = "typing-extensions" },{ name = "yattag" },@@ -1874,6 +1875,7 @@ requires-dist = [{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6.8" },{ name = "shortuuid", specifier = ">=1.0.13" },{ name = "toml", specifier = ">=0.10.2" },+ { name = "tomlkit", specifier = ">=0.13.2" },{ name = "typer", specifier = ">=0.16.0" },{ name = "types-aioboto3", extras = ["bedrock", "bedrock-runtime"], marker = "extra == 'dev'", specifier = ">=13.4.0" },{ name = "types-aiofiles", marker = "extra == 'dev'", specifier = ">=24.1.0.20240626" },@@ -2596,6 +2598,15 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/6e/c2/61d3e0f47e2b74ef40a68b9e6ad5984f6241a942f7cd3bbfbdbd03861ea9/tomli-2.2.1-py3-none-any.whl", hash = "sha256:cb55c73c5f4408779d0cf3eef9f762b9c9f147a77de7b258bef0a5628adc85cc", size = 14257, upload-time = "2024-11-27T22:38:35.385Z" },]+[[package]]+name = "tomlkit"+version = "0.13.3"+source = { registry = "https://pypi.org/simple" }+sdist = { url = "https://files.pythonhosted.org/packages/cc/18/0bbf3884e9eaa38819ebe46a7bd25dcd56b67434402b66a58c4b8e552575/tomlkit-0.13.3.tar.gz", hash = "sha256:430cf247ee57df2b94ee3fbe588e71d362a941ebb545dec29b53961d61add2a1", size = 185207, upload-time = "2025-06-05T07:13:44.947Z" }+wheels = [+ { url = "https://files.pythonhosted.org/packages/bd/75/8539d011f6be8e29f339c42e633aae3cb73bffa95dd0f9adec09b9c58e85/tomlkit-0.13.3-py3-none-any.whl", hash = "sha256:c89c649d79ee40629a9fda55f8ace8c6a1b42deb912b2a8fd8d942ddadb606b0", size = 38901, upload-time = "2025-06-05T07:13:43.546Z" },+]+[[package]]name = "tqdm"version = "4.67.1"