Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Refactor/generalize new inference system (#278)

Louis Choqueld121b39ebf5b
diff --git a/.env.example b/.env.example
index 1d15f1d63..76865ecf5 100644
--- a/.env.example
+++ b/.env.example
@@ -1,14 +1,20 @@
PYRIGHT_PYTHON_FORCE_VERSION=1.1.398
+PIPELEX_INFERENCE_API_KEY=
+
OPENAI_API_KEY=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
-AZURE_OPENAI_API_ENDPOINT=
-AZURE_OPENAI_API_VERSION=2025-04-01-preview # this is the latest API version with previex features
-AZURE_OPENAI_API_KEY=
+AZURE_API_BASE=
+AZURE_API_VERSION=
+AZURE_API_KEY=
+
+GCP_PROJECT_ID=
+GCP_LOCATION=
+GCP_CREDENTIALS_FILE_PATH=path/to/credentials.json
ANTHROPIC_API_KEY=
@@ -17,11 +23,9 @@ MISTRAL_API_KEY=
PERPLEXITY_API_KEY=
PERPLEXITY_API_ENDPOINT=https://api.perplexity.ai
-FAL_API_KEY=
-
-GCP_PROJECT_ID=
-GCP_REGION=
-GCP_CREDENTIALS_FILE_PATH=path/to/credentials.json
-
XAI_API_KEY=
XAI_API_ENDPOINT=https://api.x.ai/v1/
+
+FAL_API_KEY=
+
+BLACKBOX_API_KEY=
diff --git a/.pipelex/create/build_pipelines.md b/.pipelex/create/build_pipelines.md
deleted file mode 100644
index c1cfdc5ef..000000000
--- a/.pipelex/create/build_pipelines.md
+++ /dev/null
@@ -1,317 +0,0 @@
-# Building Pipelex Pipelines from Drafts
-
-This guide explains how to translate a markdown pipeline draft into a formal `PipelexBundleBlueprint` structure that can be validated and executed by Pipelex.
-
-## Overview
-
-1. Convert the draft's narrative flow into a formal domain/concepts/pipes structure
-2. Map draft notation to specific pipe types and configurations
-3. Ensure all working memory references are properly captured in operator pipe inputs (wrapping up input requirements for controller pipes will be done later)
-
-## Translation Workflow
-
-### Step 1: Extract Domain Information
-
-Create the blueprint root:
-```json
-{
- "domain": "domain_name",
- "definition": "What this pipeline does"
-}
-```
-
-### Step 2: Identify and Define Concepts
-
-Review the draft for all data types mentioned:
-
-- **Input variables** → Concepts needed
-- **Intermediate results** → Concepts needed
-- **Final output** → Concept needed
-
-#### Concept Rules
-
-1. **Use native concepts when possible**:
- - `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`
- - Don't redefine these
-
-2. **Use library concepts when applicable**:
- - `documents.TextAndImagesContent`
- - `images.VisualDescription`
- - `images.ImgGenPrompt`
- - `images.Photo`
-
-3. **Create domain concepts for specialized data**:
- - Use PascalCase: `Analysis`, `TechnicalReport`, `UserQuery`
- - Never use plurals: `Document` not `Documents`
- - Never use circumstantial names: `Photo` not `InputPhoto`
- - Avoid adjectives: `Text` not `LongText`
-
-#### Concept Definition Formats
-
-Quick form (for simple text-based concepts):
-```json
-"concept": {
- "UserQuery": "Natural language question from a user",
- "Summary": "Condensed version of content"
-}
-```
-
-Full form (when structure or refinement needed):
-```json
-"concept": {
- "TechnicalAnalysis": {
- "definition": "Technical analysis of a system",
- "refines": "Text"
- }
-}
-```
-
-### Step 3: Translate Flow Steps to Pipes
-
-Map each step from your draft to the appropriate pipe type and configuration.
-
-#### Operator Mappings
-
-| Draft Notation | Pipe Type | Key Configuration |
-|---------------|-----------|-------------------|
-| `LLM[purpose]: (inputs) → output` | `PipeLLM` | Set `prompt_template` or `prompt` |
-| `OCR: (document) → pages` | `PipeOcr` | Input must be named `ocr_input` |
-| `ImgGen: (prompt) → image` | `PipeImgGen` | Configure image generation settings |
-| `Func[name]: (data) → result` | `PipeFunc` | Reference Python function |
-
-#### Controller Mappings
-
-| Draft Pattern | Pipe Type | Structure |
-|--------------|-----------|-----------|
-| Sequential steps (Step 1, Step 2...) | `PipeSequence` | Use `steps` array |
-| Process each item in list | `PipeSequence` step with batch | Add `batch_over` and `batch_as` to step |
-| Parallel branches (Branch A, Branch B...) | `PipeParallel` | Use `parallels` array |
-| Conditional routing (Case "x"...) | `PipeCondition` | Use `expression` and `pipe_map` |
-
-### Step 4: Configure Each Pipe
-
-#### PipeLLM Configuration
-
-From draft step:
-```markdown
-### Step: Analyze document for key insights
-LLM[analyze]: (document, criteria) → analysis
-```
-
-To blueprint:
-```json
-"analyze_document": {
- "type": "PipeLLM",
- "definition": "Analyze document for key insights",
- "inputs": {
- "document": "Text",
- "criteria": "Text"
- },
- "output": "Analysis",
- "prompt_template": "Analyze the following document based on the provided criteria.\n\n@document\n\n@criteria"
-}
-```
-
-**Prompt Template Rules**:
-
-- Use `@variable` for block content (multi-line)
-- Use `$variable` for inline content (within sentences)
-
-#### PipeOcr Configuration
-
-From draft step:
-```markdown
-### Step: Extract content from PDF
-OCR: (document) → pages
-```
-
-To blueprint:
-```json
-"extract_pages": {
- "type": "PipeOcr",
- "definition": "Extract content from PDF",
- "inputs": {
- "ocr_input": "PDF" // Must be named ocr_input
- },
- "output": "Page", // Output is always Page concept
- "page_images": true,
- "page_views": true
- "ocr_handle": "mistral/mistral-ocr-latest"
-}
-```
-
-#### PipeSequence Configuration
-
-From draft sequence:
-```markdown
-Step 1: Extract information
-extract_info(document) → key_info
-
-Step 2: Analyze information
-analyze(key_info) → analysis
-```
-
-To blueprint:
-```json
-"process_document": {
- "type": "PipeSequence",
- "definition": "Extract and analyze document",
- "inputs": {
- "document": "Text"
- },
- "output": "Analysis",
- "steps": [
- {
- "pipe": "extract_info",
- "result": "key_info"
- },
- {
- "pipe": "analyze_info",
- "result": "analysis"
- }
- ]
-}
-```
-
-#### Batch Processing Configuration
-
-From draft batch:
-```markdown
-Process each item in documents:
-analyze_document(document) → analysis
-```
-
-To blueprint step within PipeSequence:
-```json
-{
- "pipe": "analyze_single",
- "batch_over": "documents",
- "batch_as": "document",
- "result": "analyses"
-}
-```
-Using the above, the required input used is `documents`, and by looping on the items of that list, each one will be named `document`, and that's how it will be received in the `analyze_single` pipe.
-
-#### PipeCondition Configuration
-
-From draft conditional:
-```markdown
-Route based on category:
-Case "technical": analyze_technical(document) → analysis
-Case "legal": analyze_legal(document) → analysis
-```
-
-To blueprint:
-```json
-"route_analysis": {
- "type": "PipeCondition",
- "definition": "Route document to appropriate analyzer",
- "inputs": {
- "document": "Text",
- "category": "Text"
- },
- "output": "Analysis",
- "expression": "category",
- "pipe_map": {
- "technical": "analyze_technical",
- "legal": "analyze_legal"
- }
-}
-```
-
-## Complete Example: Draft to Blueprint
-
-### Draft (Markdown)
-
-```markdown
-# Pipeline: document_insights
-Domain: analysis
-Purpose: Extract insights from documents based on criteria
-
-## Inputs
-- `document` : Document to analyze
-- `criteria` : Analysis criteria
-
-## Flow
-
-### Step 1: Extract key information
-LLM[extract]: (document) → key_points
-
-### Step 2: Analyze against criteria
-LLM[analyze]: (key_points, criteria) → insights
-
-## Output
-Returns: `insights`
-```
-
-### Blueprint (JSON)
-
-```json
-{
- "domain": "analysis",
- "definition": "Extract insights from documents based on criteria",
- "concept": {
- "KeyPoints": "Important information extracted from document",
- "Insights": "Analysis results based on specific criteria"
- },
- "pipe": {
- "document_insights": {
- "type": "PipeSequence",
- "definition": "Extract and analyze document insights",
- "output": "Insights",
- "steps": [
- {
- "pipe": "extract_key_points",
- "result": "key_points"
- },
- {
- "pipe": "analyze_points",
- "result": "insights"
- }
- ]
- },
- "extract_key_points": {
- "type": "PipeLLM",
- "definition": "Extract key information",
- "inputs": {
- "document": "Text"
- },
- "output": "KeyPoints",
- "prompt_template": "Extract the key points from this document.\n\n@document"
- },
- "analyze_points": {
- "type": "PipeLLM",
- "definition": "Analyze against criteria",
- "inputs": {
- "key_points": "KeyPoints",
- "criteria": "Text"
- },
- "output": "Insights",
- "prompt_template": "Analyze these key points against the provided criteria.\n\n@key_points\n\n@criteria"
- }
- }
-}
-```
-
-## Validation Checklist
-
-Before finalizing your blueprint:
-
-- [ ] Domain is defined with a single word or snake_case code
-- [ ] All concepts follow naming conventions (PascalCase, no plurals, no adjectives, no circumstantial names)
-- [ ] Native concepts are not redefined
-- [ ] Each pipe has `type`, `definition`, `inputs`, and `output`
-- [ ] Controller pipes don't need to declare inputs, they will be inferred from the blueprint
-- [ ] OCR pipes use `ocr_input` as the input variable name
-- [ ] Batch operations specify both `batch_over` and `batch_as`
-- [ ] Variable names are semantic (describe what they are, not their role)
-- [ ] All referenced sub-pipes exist in the blueprint
-- [ ] Prompt templates use `@` for blocks and `$` for inline content
-
-## Common Pitfalls to Avoid
-
-2. **Wrong OCR input name**: Must be `ocr_input`, not `document` or `pdf`
-3. **Redefining native concepts**: Use `Text` directly, don't create `MyText`
-4. **Circumstantial naming**: Use `Photo` not `InputPhoto`, `Analysis` not `FinalAnalysis`
-5. **Missing result names**: Each step in a sequence should specify its `result` name
-6. **Inconsistent variable names**: The same data should have the same name throughout
diff --git a/.pipelex/create/design_pipelines.md b/.pipelex/create/design_pipelines.md
deleted file mode 100644
index 240a5ed00..000000000
--- a/.pipelex/create/design_pipelines.md
+++ /dev/null
@@ -1,270 +0,0 @@
-## Designing Pipelex pipelines
-
-This section explains how to design pipelines for Pipelex.
-
-### Workflow for a designer agent
-
-- Plan the pipeline in natural language (goal, inputs, steps, outputs).
-
-- Translate the plan into a `PipelexBundleBlueprint` object graph: domain, concepts, pipes.
-
-- Define controllers first (`PipeSequence`, `PipeParallel`, `PipeCondition`), then operators (`PipeLLM`, `PipeOcr`, etc.).
-
-- For each new pipe, detail its `type` and `definition`.
-
-- Ensure each pipe’s `inputs` shows all required variables, including those used by nested steps or conditional branches. `inputs` is a dict mapping variable names to concept codes.
-
-- Choose appropriate native or structured output concepts.
-
-### Root blueprint object: PipelexBundleBlueprint
-
-`PipelexBundleBlueprint` fields:
-
-- `domain: str` — required. Single word or snake_case code for the domain.
-
-- `definition: Optional[str]` — optional textual description of the domain.
-
-- `system_prompt: Optional[str]` — default system prompt applied by LLM pipes in this domain if they don’t provide one.
-
-- `system_prompt_to_structure: Optional[str]` — domain-level instructions to structure outputs.
-
-- `prompt_template_to_structure: Optional[str]` — domain-level instructions to structure prompts.
-
-- `concept: Dict[str, Union[str, ConceptBlueprint]]` — concept section. Keys are concept names (PascalCase). Values are either:
-
- - a string (quick form): the concept definition; or
-
- - a `ConceptBlueprint` object (full form).
-
-- `pipe: Dict[str, Dict[str, Any]]` — pipes section. Keys are pipe codes (snake_case). Values are per-pipe details dicts that will be validated into concrete `PipeBlueprint` subclasses based on their `type`.
-
-### Concepts
-
-- Conceiving and naming concepts:
- - Use PascalCase for concept names.
-
- - DO NOT use plurals in concept names (use `Item` not `Items`).
-
- - DO NOT use adjectives in concept names (use `Text`, not `LargeText`).
-
- - When a concept or concept reference is unqualified (no `domain.` prefix), it will be prefixed with the current blueprint `domain` during validation/expansion, unless its name corresponds to one of the native concepts.
-
- - Do not redefine native concepts (use them directly): `Text`, `Image`, `PDF`, `TextAndImages`, `Number`, `Page`, `LLMPrompt`.
-
- - Textual concepts refining `native.Text` are acceptable too, and you don't need to specify their structure: by default it'll be text.
-
- - Reuse existing concepts whenever possible. For instance if you take a `Poem` and make it dark, the result is a `Poem` too: you MUST NOT create a new `DarkPoem` concept in this kind of situation.
-
- - Concepts from the base library are also provided and must be used whenever possible:
-
- - `documents.TextAndImagesContent` = "A content that comprises text and images where the text can include local links to the images"
-
- - `images.VisualDescription` = "Visual description of something"
-
- - `images.ImgGenPrompt`:
- - definition = "Prompt to generate an image"
- - refines = "Text"
-
- - `images.Photo`:
- - definition = "Photo"
- - structure = "ImageContent"
- - refines = "Image"
-
- - DO NOT create a concept qualified with circumstances like `InputPhoto`: it's just a `Photo`, it so happens that we're using it as input, OK, but that will be convened by the rest of the structure.
-
-- Quick form (string):
-
- - `concept["ConceptName"] = "Definition of the concept"`
-
- - If no `structure` is specified and the name isn’t a known structure class, it implicitly refines `native.Text`.
-
-- Full form (`ConceptBlueprint` from `pipelex/core/concept_factory.py`):
-
- - `definition: str` — required.
-
- - `structure: Optional[str]` — name of a registered `StructuredContent` subclass (typically defined in `pipelex_libraries/pipelines/<domain>.py`). If provided, it must exist in the class registry.
-
- - `refines: Union[str, List[str]]` — concept codes or names it refines. May be a single string or a list. Unqualified names are prefixed with the current `domain` automatically.
-
- - `domain: Optional[str]` — not required when provided via the library loader.
-
-### Pipes: overall rules
-
-Each entry under `pipe` is a dict that will be converted into a typed `PipeBlueprint` (see `pipelex/core/blueprint.py` for the base and factories in `pipe_controllers/` and `pipe_operators/`). Use the new format:
-
-- `type: str` — the pipe class name, e.g., `PipeLLM`, `PipeOcr`, `PipeSequence`, `PipeParallel`, `PipeCondition`, `PipeFunc`, `PipeImgGen`.
-
-- `definition: str` — human-readable description of the pipe’s responsibility.
-
-- `inputs: Optional[Dict[str, str]]` — map of variable name -> concept code or name. Unqualified names are prefixed with `domain`.
-
-- `output: str` — target concept code or name. Unqualified names are prefixed with `domain`.
-
-Important:
-
-- For controller pipes (`PipeSequence`, `PipeParallel`, `PipeCondition`), the parent pipe’s `inputs` must include all variables required by its sub-pipes or conditional branches that come from the initial working memory (i.e., not produced by previous steps).
-
-- Multiplicity: some pipes and sub-steps can produce multiple outputs (see below). Plan result names accordingly.
-
-### Pipe-specific blueprint schemas
-
-Below are the additional fields expected per type, on top of the base fields above (`type`, `definition`, `inputs`, `output`). All fields are validated by their blueprint classes.
-
-#### PipeLLM (operators/pipe_llm_factory.py)
-
-- Prompt sources (choose at most one per group):
-
- - System prompt: one of `system_prompt`, `system_prompt_name`, `system_prompt_template`, `system_prompt_template_name`.
-
- - User prompt: one of `prompt`, `prompt_name`, `prompt_template`, `template_name`.
-
-- LLM selection:
-
- - `llm: Optional[LLMSettingOrPresetId]` — model for text.
-
- - `llm_to_structure: Optional[LLMSettingOrPresetId]` — model for object structuring.
-
-- Structuring:
-
- - `structuring_method: Optional[StructuringMethod]`
-
- - `prompt_template_to_structure: Optional[str]`
-
- - `system_prompt_to_structure: Optional[str]`
-
-- Multiplicity:
-
- - `nb_output: Optional[int]` — exact number of outputs; mutually exclusive with `multiple_output`.
-
- - `multiple_output: Optional[bool]` — allow variable number of outputs.
-
-Behavioral notes:
-
-- If no system prompt is provided, the domain’s `system_prompt` (from the root blueprint) is used if available.
-
-- Inputs that are `Image` concepts are automatically exposed as user images to the LLM; other inputs become text blocks.
-
-- Prompt template insertion style for the agent:
-
- - Use block insertion (equivalent to `@name`) for long, multi-line stuff.
-
- - Use inline insertion (equivalent to `$name`) for short tokens embedded in sentences.
-
- - Do not synthesize literal markers; provide structured prompt fields and let the runtime render appropriately.
-
-#### PipeOcr (operators/pipe_ocr_factory.py)
-
-- Required input variable name: `ocr_input`.
-
-- The `ocr_input` concept must be `Image` or `PDF`.
-
-- Additional options:
-
- - `ocr_platform: Optional[OcrPlatform]` — defaults to Mistral.
-
- - `page_images: bool` — include detected images.
-
- - `page_image_captions: bool` — caption images.
-
- - `page_views: bool` — include rendered page views.
-
- - `page_views_dpi: Optional[int]` — DPI for page views; defaults to config.
-
-- Output concept should be `Page`. The operational result is a list of `Page` items.
-
-#### PipeSequence (controllers/pipe_sequence_factory.py)
-
-- `steps: List[SubPipeBlueprint]`
-
-- `SubPipeBlueprint` fields (controllers/sub_pipe_factory.py):
-
- - `pipe: str` — referenced pipe code to execute.
-
- - `result: Optional[str]` — name to assign to this step’s output in working memory.
-
- - Multiplicity (choose at most one): `nb_output: Optional[int]`, `multiple_output: Optional[bool]`.
-
- - Batch (both or none): `batch_over: Union[bool, str]` and `batch_as: Optional[str]`.
-
-Rules:
-
-- If `batch_over` is provided, `batch_as` is required; if `batch_as` is provided, `batch_over` is required.
-
-- When batching, the step runs in parallel over each element of the specified list, and the `result` becomes a list.
-
-#### PipeParallel (controllers/pipe_parallel_factory.py)
-
-- `parallels: List[SubPipeBlueprint]` — each must specify `result`.
-
-- `add_each_output: bool = True` — whether to add each parallel result separately to memory.
-
-- `combined_output: Optional[str]` — optional combined output concept code; if provided without a domain prefix, it’s prefixed automatically.
-
-Rules:
-
-- At least one of `add_each_output` or `combined_output` must be set appropriately (cannot disable both behaviors).
-
-#### PipeCondition (controllers/pipe_condition_factory.py)
-
-- Choose exactly one of:
-
- - `expression: Optional[str]` — direct expression.
-
- - `expression_template: Optional[str]` — Jinja2 template expression.
-
-- `pipe_map: Dict[str, str]` — map of evaluated expression values to pipe codes to execute.
-
-- `default_pipe_code: Optional[str]` — optional fallback pipe code if no key matches.
-
-- `add_alias_from_expression_to: Optional[str]` — if set, store the evaluated expression under this variable name in working memory.
-
-### Input and output concepts resolution
-
-- Unqualified concept names in `inputs` and `output` are automatically prefixed with the current `domain` during blueprint validation (see `PipeBlueprint.add_domain_prefix`).
-
-- For `PipeParallel.combined_output`, the same domain-prefixing rule applies.
-
-### Checklist for the agent
-
-- Define `domain` and optional domain-level prompts (`system_prompt`, `system_prompt_to_structure`, `prompt_template_to_structure`).
-
-- Create `concept` entries using either quick string form or full `ConceptBlueprint` with `definition`, optional `structure`, and `refines`.
-
-- For each pipe under `pipe`:
-
- - Provide `type`, `definition`, `inputs`, `output`.
-
- - Add type-specific fields (see sections above).
-
- - Ensure controller pipes include all upstream inputs needed by nested steps/branches.
-
- - Set multiplicity (`nb_output`/`multiple_output`) and batch settings where needed.
-
- - For OCR, use `ocr_input` with `Image` or `PDF`, and set `output` to `Page`.
-
-- Validate mentally that every referenced sub-pipe exists and every input it requires is either provided by the parent `inputs` or produced by a previous step.
-
-### Minimal skeleton (object form)
-
-The agent should generate a structure similar to the following (illustrative, not exhaustive):
-
-```json
-{
- "domain": "my_domain",
- "definition": "What this domain covers",
- "system_prompt": "Global system prompt for LLM pipes (optional)",
- "concept": {
- "UserQuery": "Natural language question from a user",
- "Answer": { "definition": "Answer text" }
- },
- "pipe": {
- "answer_question": {
- "type": "PipeLLM",
- "definition": "Produce an answer from a question and context",
- "inputs": { "question": "UserQuery", "context": "Text" },
- "output": "Answer",
- "prompt_template": "Analyze the question and context, then answer concisely.\n\n@question\n\n@context"
- }
- }
-}
-```
diff --git a/.pipelex/create/draft_pipelines.md b/.pipelex/create/draft_pipelines.md
deleted file mode 100644
index c1cabf032..000000000
--- a/.pipelex/create/draft_pipelines.md
+++ /dev/null
@@ -1,259 +0,0 @@
-# Drafting Pipelex Pipelines
-
-This guide explains how to draft pipelines in natural language using markdown notation before translating them into formal Pipelex definitions.
-
-## Core Concept: WorkingMemory
-
-Every pipeline execution maintains a `WorkingMemory` that acts as a shared data space. When a pipeline starts, the initial inputs are placed in this memory. As each pipe executes, its output is added to the memory with a specific name (the `result` name). Subsequent pipes can access any data in the memory by referencing these names.
-
-**Key Rules**:
-
-- When a pipe produces an output with name `X`, any later pipe can use `X` as an input. The names must match exactly.
-- Controller pipes (PipeSequence, PipeParallel, PipeCondition) automatically inherit the input requirements of their nested operators - you don't need to specify their inputs explicitly.
-
-## Drafting Workflow
-
-1. **Start with the goal**: What does this pipeline accomplish?
-2. **Define inputs**: What data does it need to start?
-3. **Narrate the flow**: Describe each step and how data flows through WorkingMemory
-4. **Specify outputs**: What final result emerges?
-
-## Pipeline Draft Structure
-
-### Header
-```markdown
-# Pipeline: [pipeline_name]
-Domain: [domain_name]
-Purpose: [What this pipeline does]
-```
-
-### Inputs Section
-```markdown
-## Inputs
-- `document` : PDF document to analyze
-- `criteria` : Analysis criteria to apply
-```
-
-### Flow Section
-
-Describe the pipeline flow using concise step notation:
-
-```markdown
-## Flow
-
-WorkingMemory starts with: {document, criteria}
-
-### Step 1: Process document content
-LLM[process]: (document) → processed_content
-
-WorkingMemory now contains: {document, criteria, processed_content}
-
-### Step 2: Apply analysis criteria to processed content
-LLM[synthesize]: (processed_content, criteria) → analysis_report
-
-WorkingMemory now contains: {document, criteria, processed_content, analysis_report}
-```
-
-### Output Section
-```markdown
-## Output
-Returns: `analysis_report` - Comprehensive analysis based on criteria
-```
-
-## Pipe Types and Notation
-
-## Operator Notations
-
-**Important**: During drafting, DO NOT write actual prompts or prompt templates. Focus on describing what each operator does and how data flows. Prompts will be written during implementation.
-
-### LLM Operations (including vision, to generate text from text and images)
-```markdown
-LLM[purpose]: (inputs) → output
-
-Examples:
-LLM[summarize]: (long_text) → summary
-LLM[analyze]: (document, criteria) → analysis
-```
-
-### OCR Operations (to extract text and images from PDF pages, including full page views)
-```markdown
-OCR: (pdf_document) → pages_list
-```
-
-### Image Generation (to generate images from text prompts)
-```markdown
-ImgGen: (prompt_text) → generated_image
-```
-
-### Function Operations (to call external python functions)
-```markdown
-Func[function_name]: (data) → result
-```
-
-### Sequential Processing (to run steps in sequence, use it when steps need results from previous steps)
-```markdown
-Step 1: Extract key information from document
-extract_info(document) → key_info
-
-Step 2: Analyze extracted information
-analyze(key_info) → analysis
-
-Step 3: Generate report with original document context
-generate_report(analysis, document) → report
-```
-
-### Batch Processing (to process each item in a list independently and in parallel)
-```markdown
-Process each item in documents:
-analyze_document(document) → analysis
-
-Collect all results → document_analyses
-```
-
-### Parallel Processing (to run different pipes in parallel from the same memory, use it when steps can be run independently, each with a copy of the working memory)
-```markdown
-Parallel execution:
-
-Branch A: Extract technical details
-extract_technical(document) → technical_info
-
-Branch B: Extract business insights
-extract_business(document) → business_info
-
-Combine: Merge analysis results
-merge_insights(technical_info, business_info) → comprehensive_analysis
-```
-
-### Conditional Processing (to run different pipes based on a condition)
-```markdown
-Route based on expression(document_type):
-
-Case "technical": Handle technical documents
-analyze_technical(document) → analysis
-
-Case "legal": Handle legal documents
-analyze_legal(document) → analysis
-
-Default: Handle general documents
-analyze_general(document) → analysis
-```
-
-## Memory Flow Examples
-
-### Example 1: Document Analysis Pipeline
-```markdown
-# Pipeline: analyze_document
-Domain: document_analysis
-Purpose: Extract and analyze key information from documents
-
-## Inputs
-- `document` : PDF document to analyze
-
-## Flow
-
-WorkingMemory: {document}
-
-### Step 1: Extract content from PDF
-OCR: (document) → pages
-
-WorkingMemory: {document, pages}
-
-### Step 2: Summarize each page in parallel
-Process each page in pages:
-LLM[summarize]: (page) → summary
-
-Collect all results → summaries
-
-WorkingMemory: {document, pages, summaries}
-
-### Step 3: Create comprehensive analysis from summaries
-LLM[synthesize]: (summaries) → analysis
-
-WorkingMemory: {document, pages, summaries, analysis}
-
-## Output
-Returns: `analysis`
-```
-
-### Example 2: Conditional Processing
-```markdown
-# Pipeline: smart_responder
-Domain: customer_service
-Purpose: Route and respond to customer queries
-
-## Inputs
-- `query` : Customer question
-- `context` : Available context information
-
-## Flow
-
-WorkingMemory: {query, context}
-
-### Step 1: Classify query into support category
-LLM[classify]: (query) → category
-
-WorkingMemory: {query, context, category}
-
-### Step 2: Route to appropriate support handler
-Route based on category:
-
-Case "technical": Technical support handler
-LLM[technical_support]: (query, context) → response
-
-Case "billing": Billing support handler
-LLM[billing_support]: (query, context) → response
-
-Default: General support handler
-LLM[general_support]: (query) → response
-
-WorkingMemory: {query, context, category, response}
-
-## Output
-Returns: `response`
-```
-
-## Best Practices
-
-1. **Use semantic variable names**: Name variables based on what they represent, not how they're used. Use `document` instead of `input_document`, `text` instead of `input_text`, `analysis` instead of `result_1`.
-
-2. **Name consistently**: Once you name a result (e.g., `analysis`), use that exact name when referencing it later.
-
-3. **Show memory state**: After complex steps, show what's in WorkingMemory to track data flow.
-
-4. **Be explicit about inputs**: For each operator step, clearly indicate which items from WorkingMemory are being used.
-
-5. **Use descriptive names**: Choose names that clearly describe the content: `summaries`, `technical_info`, `user_preferences`.
-
-6. **Concise step descriptions**: Each step should have a clear title that describes what it accomplishes. Avoid repeating the same information in multiple places.
-
-7. **No prompts during drafting**: Focus on describing WHAT each step does, not HOW (the actual prompts). Prompts are implementation details added later.
-
-8. **Block vs Inline references**:
- - Use `@variable_name` notation for block insertions (multi-line content)
- - Use `$variable_name` notation for inline insertions (single values in sentences)
-
-9. **Batch operations**: When processing lists, clearly indicate:
- - What list is being iterated (`batch_over`)
- - What each item is called during processing (`batch_as`)
- - What the collected results are named
-
-## Validation Checklist
-
-Before finalizing your draft:
-
-- [ ] Every input used by an operator pipe exists in WorkingMemory (either from initial inputs or previous steps)
-- [ ] Every result name is unique (no overwriting unless intentional)
-- [ ] The final output exists in WorkingMemory
-- [ ] Names are consistent throughout (no typos or variations)
-- [ ] Variable names are semantic (describe what they are, not how they're used)
-- [ ] The flow clearly shows data transformation from inputs to outputs
-- [ ] Each step has a concise, non-repetitive description
-
-## From Draft to Implementation
-
-Once your markdown draft is complete, it can be translated into:
-1. A formal PLX pipeline definition with explicit pipe declarations
-2. Python structure classes for complex data types (when needed)
-3. Validated, executable Pipelex pipeline code
-
-The narrative draft serves as both documentation and a blueprint for implementation, making the pipeline's logic transparent and maintainable.
diff --git a/.pipelex/create/structures.md b/.pipelex/create/structures.md
deleted file mode 100644
index fe785a954..000000000
--- a/.pipelex/create/structures.md
+++ /dev/null
@@ -1,396 +0,0 @@
-# Defining Structures for Pipelex Pipelines
-
-This guide explains how to define schemas and enums for structured data generation in multi-step workflows. This step occurs between drafting pipelines (see `draft_pipelines.md`) and building them (see `build_pipelines.md`).
-
-## When to Define Structures
-
-After drafting your pipeline but before building it, review your draft for:
-
-1. **Complex data that needs validation**: When your pipeline generates structured information that must follow specific formats
-2. **Multi-field outputs**: When a single "result" actually contains multiple related pieces of information
-3. **Constrained values**: When fields can only have specific allowed values
-4. **Reusable data types**: When the same structure appears in multiple places
-
-## Structure Definition Workflow
-
-### Step 1: Identify Structure Needs from Draft
-
-Review your pipeline draft and look for:
-
-```markdown
-# Example draft excerpt
-LLM[analyze]: (document) → analysis_report
-LLM[extract]: (page) → contact_info
-LLM[classify]: (query) → support_category
-```
-
-Ask yourself:
-- What specific fields does `analysis_report` contain?
-- What format should `contact_info` follow?
-- What are the valid values for `support_category`?
-
-### Step 2: Design Schema Definitions
-
-Create a PLX file defining your structures using the syntax from `structured_output_generator.py`.
-
-## Structure Definition Syntax
-
-### Basic Structure Definition
-
-```plx
-[structure.ModelName]
-definition = "Clear description of what this represents"
-
-[structure.ModelName.fields]
-field_name = "Field description"
-another_field = { type = "text", definition = "More detailed field", required = true }
-```
-
-### Field Types
-
-| Type | Description | Example |
-|------|-------------|---------|
-| `text` | String content | `name = "Person's full name"` |
-| `number` | Floating point | `score = { type = "number", definition = "Confidence score" }` |
-| `integer` | Whole numbers | `count = { type = "integer", definition = "Number of items" }` |
-| `boolean` | True/false | `active = { type = "boolean", definition = "Is active" }` |
-| `list` | Array of items | `tags = { type = "list", item_type = "text", definition = "List of tags" }` |
-| `dict` | Key-value pairs | `metadata = { type = "dict", key_type = "text", value_type = "text" }` |
-
-### Field Configuration Options
-
-```plx
-# Required field (must be provided)
-title = { type = "text", definition = "Document title", required = true }
-
-# Optional field with default value
-priority = { type = "integer", definition = "Priority level", default = 1 }
-
-# List with specific item type
-categories = { type = "list", item_type = "text", definition = "Document categories" }
-```
-
-## Enum Definition Syntax
-
-### Simple Enum (List of Values)
-
-```plx
-[enum.Priority]
-definition = "Task priority levels"
-values = ["low", "medium", "high", "urgent"]
-```
-
-### Descriptive Enum (Key-Value Pairs)
-
-```plx
-[enum.DocumentType]
-definition = "Types of documents we can process"
-
-[enum.DocumentType.values]
-technical = "Technical documentation and manuals"
-legal = "Legal contracts and agreements"
-financial = "Financial reports and statements"
-marketing = "Marketing materials and content"
-```
-
-### Using Enums in Structures
-
-```plx
-# Reference defined enum
-status = { type = "Priority", definition = "Current priority level" }
-
-# Inline choices for simple cases
-size = { choices = ["S", "M", "L", "XL"], definition = "Available sizes" }
-```
-
-## Complete Example: Document Analysis Pipeline
-
-### From Draft to Structures
-
-**Original Draft Step:**
-```markdown
-### Step 2: Extract structured information from each page
-LLM[extract]: (page) → page_info
-
-### Step 3: Classify document type and priority
-LLM[classify]: (document_content) → document_metadata
-
-### Step 4: Generate comprehensive analysis
-LLM[analyze]: (page_info_list, document_metadata) → final_analysis
-```
-
-**Structure Definition (document_analysis_structures.plx):**
-
-```plx
-# Enums for constrained values
-[enum.DocumentType]
-definition = "Types of documents we can process"
-values = ["technical", "legal", "financial", "marketing", "general"]
-
-[enum.Priority]
-definition = "Document processing priority"
-values = ["low", "medium", "high", "urgent"]
-
-[enum.ConfidenceLevel]
-definition = "AI confidence in extraction"
-
-[enum.ConfidenceLevel.values]
-low = "Low confidence - manual review recommended"
-medium = "Medium confidence - spot check suggested"
-high = "High confidence - likely accurate"
-
-# Structures for structured data
-[structure.PageInfo]
-definition = "Structured information extracted from a document page"
-
-[structure.PageInfo.fields]
-page_number = { type = "integer", definition = "Page number in document", required = true }
-main_content = "Primary text content of the page"
-key_points = { type = "list", item_type = "text", definition = "Important points found on page" }
-entities = { type = "list", item_type = "text", definition = "Named entities (people, organizations, etc.)" }
-confidence = { type = "ConfidenceLevel", definition = "Confidence in extraction quality" }
-
-[structure.DocumentMetadata]
-definition = "High-level document classification and properties"
-
-[structure.DocumentMetadata.fields]
-document_type = { type = "DocumentType", definition = "Classified document type", required = true }
-priority = { type = "Priority", definition = "Processing priority", required = true }
-title = "Inferred document title"
-summary = "Brief document summary"
-page_count = { type = "integer", definition = "Total number of pages" }
-language = { choices = ["en", "fr", "es", "de"], definition = "Document language" }
-
-[structure.DocumentAnalysis]
-definition = "Comprehensive analysis combining all extracted information"
-
-[structure.DocumentAnalysis.fields]
-document_metadata = { type = "DocumentMetadata", definition = "Document classification info", required = true }
-page_analyses = { type = "list", item_type = "PageInfo", definition = "Analysis for each page", required = true }
-overall_insights = { type = "list", item_type = "text", definition = "Key insights across entire document" }
-recommendations = { type = "list", item_type = "text", definition = "Recommended actions based on analysis" }
-confidence_score = { type = "number", definition = "Overall confidence in analysis (0.0-1.0)" }
-```
-
-## Best Practices
-
-### 1. Semantic Naming
-- Use **PascalCase** for schema and enum names: `DocumentAnalysis`, `Priority`
-- Use **snake_case** for field names: `page_number`, `confidence_score`
-- Avoid plurals in schema names: `PageInfo` not `PageInfos`
-- Avoid circumstantial names: `Analysis` not `FinalAnalysis`
-
-### 2. Clear Definitions
-- Every schema and enum should have a `definition`
-- Field descriptions should explain what the field contains, not how it's used
-- Use complete sentences for definitions
-
-### 3. Appropriate Constraints
-- Mark fields as `required = true` only when they're always expected
-- Use enums for fields with limited, known values
-- Use inline `choices` for simple, one-off constraints
-- Set reasonable defaults where applicable
-
-### 4. Logical Grouping
-- Group related fields into the same schema
-- Don't create overly granular schemas for simple data
-- Consider if a field should be separate or part of a larger structure
-
-### 5. Reusability
-- Design schemas to be reusable across different pipes
-- Create enums for values that appear in multiple contexts
-- Avoid pipeline-specific naming in reusable structures
-
-## Integration with Pipeline Building
-
-### Updating Concept Definitions
-
-After defining structures, update your pipeline blueprint concepts to reference the structure definitions:
-
-**Simple approach (structure name matches concept name):**
-```json
-"concept": {
- "PageInfo": "Information extracted from a page",
- "DocumentMetadata": "Document classification data",
- "DocumentAnalysis": "Complete document analysis"
-}
-```
-
-**Structure reference approaches:**
-```json
-"concept": {
- "PageInfo": {
- "definition": "Structured information extracted from a document page",
- "structure": "PageInfo"
- },
- "DocumentMetadata": {
- "definition": "High-level document classification and properties",
- "structure": {
- "document_type": {
- "type": "DocumentType",
- "definition": "Classified document type",
- "required": true
- },
- "priority": {
- "type": "Priority",
- "definition": "Processing priority",
- "required": true
- },
- "title": "Inferred document title",
- "summary": "Brief document summary",
- "page_count": {
- "type": "integer",
- "definition": "Total number of pages"
- },
- "language": {
- "choices": ["en", "fr", "es", "de"],
- "definition": "Document language"
- }
- }
- },
- "DocumentAnalysis": {
- "definition": "Comprehensive analysis combining all extracted information",
- "structure": "DocumentAnalysis"
- }
-}
-```
-
-**Structure attribute options:**
-- **String reference**: `"structure": "ClassName"` - References a predefined structure class
-- **Inline definition**: `"structure": { ... }` - Defines fields directly within the concept
-- **Auto-detection**: Omit `structure` entirely - System looks for StructuredContent subclass with matching concept name or falls back on `Text` (i.e. not structured)
-
-### When to Use Each Approach
-
-**Use string reference when:**
-- Structure is reusable across multiple concepts or pipelines
-- Structure is complex and benefits from being defined in a separate PLX file
-- You want to maintain separation between concept definitions and structure details
-
-**Use inline definition when:**
-- Structure is specific to this one concept
-- Structure is simple and doesn't warrant a separate definition
-- You want to keep concept and structure together for clarity
-- Rapid prototyping where you don't want to manage separate files
-
-**Use auto-detection when:**
-- Concept name exactly matches an existing StructuredContent class
-- Working with simple, standard structures that are already registered
-
-### Generating Python Classes
-
-**From PLX files:**
-```python
-from pipelex.create.structured_output_generator import generate_structured_outputs_from_plx_file
-
-# Generate Python classes from PLX definitions
-generate_structured_outputs_from_plx_file(
- "document_analysis_structures.plx",
- "document_analysis.py"
-)
-```
-
-**From inline definitions (for concept integration):**
-```python
-from pipelex.create.structured_output_generator import generate_structured_output_from_inline_definition
-
-# Generate from inline field definitions (as used in concept.structure)
-fields_def = {
- "document_type": {
- "type": "DocumentType",
- "definition": "Classified document type",
- "required": True
- },
- "priority": {
- "type": "Priority",
- "definition": "Processing priority",
- "required": True
- },
- "title": "Inferred document title",
- "page_count": {
- "type": "integer",
- "definition": "Total number of pages"
- },
- "language": {
- "choices": ["en", "fr", "es", "de"],
- "definition": "Document language"
- }
-}
-
-enums = {
- "DocumentType": {
- "definition": "Types of documents we can process",
- "values": ["technical", "legal", "financial", "marketing", "general"]
- },
- "Priority": {
- "definition": "Document processing priority",
- "values": ["low", "medium", "high", "urgent"]
- }
-}
-
-python_code = generate_structured_output_from_inline_definition(
- "DocumentMetadata",
- fields_def,
- enums
-)
-```
-
-## Validation Checklist
-
-Before finalizing your structure definitions:
-
-- [ ] All structure names use PascalCase and are singular
-- [ ] All field names use snake_case
-- [ ] Every structure and enum has a clear `definition`
-- [ ] Required fields are marked appropriately
-- [ ] Enums are used for constrained values with known options
-- [ ] Field types match the expected data format
-- [ ] Structures group related fields logically
-- [ ] Structure names are semantic (describe what they are, not how they're used)
-- [ ] No native concepts (Text, Image, PDF) are redefined
-- [ ] List item types are specified correctly
-- [ ] Dictionary key and value types are appropriate
-
-## Common Patterns
-
-### Contact Information
-```plx
-[structure.ContactInfo]
-definition = "Contact information extracted from document"
-
-[structure.ContactInfo.fields]
-name = "Full name of contact"
-email = "Email address"
-phone = "Phone number"
-company = "Company or organization name"
-address = "Physical address"
-```
-
-### Analysis Results
-```plx
-[enum.Sentiment]
-definition = "Sentiment analysis results"
-values = ["positive", "negative", "neutral"]
-
-[structure.TextAnalysis]
-definition = "Results of text analysis"
-
-[structure.TextAnalysis.fields]
-sentiment = { type = "Sentiment", definition = "Overall sentiment", required = true }
-key_themes = { type = "list", item_type = "text", definition = "Main themes identified" }
-confidence = { type = "number", definition = "Analysis confidence (0.0-1.0)" }
-word_count = { type = "integer", definition = "Total word count" }
-```
-
-### Classification Results
-```plx
-[structure.Classification]
-definition = "Multi-class classification result"
-
-[structure.Classification.fields]
-predicted_class = { type = "text", definition = "Most likely class", required = true }
-confidence_scores = { type = "dict", key_type = "text", value_type = "number", definition = "Confidence for each class" }
-threshold_met = { type = "boolean", definition = "Whether prediction meets confidence threshold" }
-```
-
-This structured approach ensures your pipeline generates well-validated, type-safe data that can be reliably processed by downstream systems.
diff --git a/.pipelex/inference/backends.toml b/.pipelex/inference/backends.toml
index ebf1509b5..957ac68ca 100644
--- a/.pipelex/inference/backends.toml
+++ b/.pipelex/inference/backends.toml
@@ -36,7 +36,7 @@ enabled = true
aws_region = "${AWS_REGION}"
[vertexai]
-enabled = true
+enabled = false
gcp_project_id = "${GCP_PROJECT_ID}"
gcp_location = "${GCP_LOCATION}"
gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
@@ -45,3 +45,6 @@ gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
enabled = true
api_key = "${MISTRAL_API_KEY}"
+[internal] # software-only backend, runs internally, without AI
+enabled = true
+
diff --git a/.pipelex/inference/backends/internal.toml b/.pipelex/inference/backends/internal.toml
new file mode 100644
index 000000000..54dba5a0a
--- /dev/null
+++ b/.pipelex/inference/backends/internal.toml
@@ -0,0 +1,10 @@
+# software-only backend, runs internally, without AI
+
+[pypdfium2-extract-text]
+model_type = "text_extractor"
+sdk = "pypdfium2"
+model_id = "extract-text"
+inputs = ["pdf"]
+outputs = ["pages"]
+costs = {}
+
diff --git a/.pipelex/inference/backends/mistral.toml b/.pipelex/inference/backends/mistral.toml
index 13f61e4a3..b452cef8b 100644
--- a/.pipelex/inference/backends/mistral.toml
+++ b/.pipelex/inference/backends/mistral.toml
@@ -92,3 +92,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.4, output = 2.0 }
+[mistral-ocr]
+model_type = "text_extractor"
+model_id = "mistral-ocr-latest"
+max_tokens = 131072
+inputs = ["pdf", "image"]
+outputs = ["pages"]
+costs = { input = 0.4, output = 2.0 }
+
diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.toml
index e5ef1aa66..643a21e68 100644
--- a/.pipelex/pipelex.toml
+++ b/.pipelex/pipelex.toml
@@ -11,7 +11,6 @@ api_key_method = "env"
####################################################################################################
[cogt.ocr_config]
-ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
[pipelex.feature_config]
@@ -19,4 +18,3 @@ page_output_text_file_name = "page_text.md"
is_pipeline_tracking_enabled = false
is_activity_tracking_enabled = false
is_reporting_enabled = true
-
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8c2e502ac..d188ab875 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,32 @@
# Changelog
+## Unreleased
+
+### Added
+
+- Unified OCR system using model handles instead of separate OcrHandle enum
+- ModelType enum supporting LLM and TEXT_EXTRACTOR types
+- Enhanced error handling in library loading with better validation messages
+- Config template management with `config-template` and `cft` Makefile targets
+
+### Changed
+
+- ⚠️ Breaking change: Renamed `ocr_handle` to `ocr_model` in `PipeOcrBlueprint`
+- OCR system now uses InferenceModelSpec with unified model handles
+- Renamed `get_llm_deck()` to `get_model_deck()` and updated parameter names from `llm_handle` to `model_handle`
+- Simplified OCR worker factory using plugin SDK matching
+- Enhanced plugin system compatibility with InferenceModelSpec
+- Improved error messages throughout system
+- Updated .env.example file with slightly modified key names (more standard).
+- Improved management of placeholder environment variables for unit tests.
+
+### Removed
+
+- Legacy OCR classes: OcrHandle, OcrPlatform, OcrEngine, OcrEngineFactory
+- Obsolete configuration fields and setup methods
+- PipelexFileError exception class
+
+
## [v0.10.1] - 2025-09-17
### Changed
diff --git a/Makefile b/Makefile
index 5f85e0565..0c3379060 100644
--- a/Makefile
+++ b/Makefile
@@ -51,6 +51,9 @@ make lint - lint with ruff check
make pyright - Check types with pyright
make mypy - Check types with mypy
+make config-template - Update config template from .pipelex/
+make cft - Shorthand -> config-template
+
make cleanenv - Remove virtual env and lock files
make cleanderived - Remove extraneous compiled files, caches, logs, etc.
make cleanlibraries - Remove pipelex_libraries
@@ -105,7 +108,8 @@ export HELP
run-all-tests run-manual-trigger-gha-tests run-gha_disabled-tests \
validate v check c cc \
merge-check-ruff-lint merge-check-ruff-format merge-check-mypy merge-check-pyright \
- li check-unused-imports fix-unused-imports check-uv check-TODOs docs docs-check docs-deploy
+ li check-unused-imports fix-unused-imports check-uv check-TODOs docs docs-check docs-deploy \
+ config-template cft
all help:
@echo "$$HELP"
@@ -159,6 +163,14 @@ build: env
$(call PRINT_TITLE,"Building the wheels")
@uv build
+config-template:
+ $(call PRINT_TITLE,"Updating config template from .pipelex/")
+ @rm -rf pipelex/config_template/*
+ @cp -r .pipelex/* pipelex/config_template/
+
+cft: config-template
+ @echo "> done: cft = config-template"
+
##############################################################################################
############################ Cleaning ############################
##############################################################################################
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
index f844109d8..ba06aa3e1 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-controllers/PipeSequence.md
@@ -44,7 +44,7 @@ Let's imagine a pipeline that first extracts text from an image, then summarizes
type = "PipeOcr"
definition = "Extract text from an image"
output = "Text"
-ocr_handle = "mistral/mistral-ocr-latest"
+ocr_model = "mistral-ocr"
[pipe.summarize_text]
type = "PipeLLM"
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
index 59e912b4c..4c9fcc692 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/pipe-operators/PipeOcr.md
@@ -53,7 +53,7 @@ inputs = { ocr_input = "ScannedDocument" }
output = "Page"
page_views = true
page_views_dpi = 200
-ocr_handle = "mistral/mistral-ocr-latest"
+ocr_model = "mistral-ocr"
```
The output of the PipeOcr must be exactly the native `Page` concept.
diff --git a/docs/pages/configuration/config-technical/cogt-config.md b/docs/pages/configuration/config-technical/cogt-config.md
index 467526e2a..8c71605a3 100644
--- a/docs/pages/configuration/config-technical/cogt-config.md
+++ b/docs/pages/configuration/config-technical/cogt-config.md
@@ -109,7 +109,6 @@ Configuration for Optical Character Recognition:
```toml
[cogt.ocr_config]
-ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
default_page_views_dpi = 72
```
@@ -187,6 +186,5 @@ output_format = "jpg"
seed = "auto"
[cogt.ocr_config]
-ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
```
diff --git a/pipelex/cogt/config_cogt.py b/pipelex/cogt/config_cogt.py
index 3418d7bfe..4f11f37b3 100644
--- a/pipelex/cogt/config_cogt.py
+++ b/pipelex/cogt/config_cogt.py
@@ -11,7 +11,6 @@ from pipelex.tools.misc.file_utils import find_files_in_dir
class OcrConfig(ConfigModel):
- ocr_handles: List[str]
page_output_text_file_name: str
default_page_views_dpi: int
diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.py
index e28e3045f..2ae27021f 100644
--- a/pipelex/cogt/content_generation/assignment_models.py
+++ b/pipelex/cogt/content_generation/assignment_models.py
@@ -12,7 +12,6 @@ from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.hub import get_class_registry
@@ -141,7 +140,7 @@ class Jinja2Assignment(BaseModel):
class OcrAssignment(BaseModel):
job_metadata: JobMetadata
- ocr_handle: OcrHandle
+ ocr_handle: str
ocr_input: OcrInput
ocr_job_params: OcrJobParams
ocr_job_config: OcrJobConfig
diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.py
index e4f6a722b..fabff4d6e 100644
--- a/pipelex/cogt/content_generation/content_generator.py
+++ b/pipelex/cogt/content_generation/content_generator.py
@@ -25,7 +25,6 @@ from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.cogt.ocr.ocr_output import OcrOutput
@@ -266,7 +265,7 @@ class ContentGenerator(ContentGeneratorProtocol):
self,
job_metadata: JobMetadata,
ocr_input: OcrInput,
- ocr_handle: OcrHandle,
+ ocr_handle: str,
ocr_job_params: Optional[OcrJobParams] = None,
ocr_job_config: Optional[OcrJobConfig] = None,
) -> OcrOutput:
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index d2917302f..0eac3b24f 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -12,7 +12,6 @@ from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.cogt.ocr.ocr_output import ExtractedImageFromPage, OcrOutput, Page
@@ -204,7 +203,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
self,
job_metadata: JobMetadata,
ocr_input: OcrInput,
- ocr_handle: OcrHandle,
+ ocr_handle: str,
ocr_job_params: Optional[OcrJobParams] = None,
ocr_job_config: Optional[OcrJobConfig] = None,
) -> OcrOutput:
diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.py
index dc516bc1f..be11adba0 100644
--- a/pipelex/cogt/content_generation/content_generator_protocol.py
+++ b/pipelex/cogt/content_generation/content_generator_protocol.py
@@ -8,7 +8,6 @@ from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.cogt.ocr.ocr_output import OcrOutput
@@ -121,7 +120,7 @@ class ContentGeneratorProtocol(Protocol):
self,
job_metadata: JobMetadata,
ocr_input: OcrInput,
- ocr_handle: OcrHandle,
+ ocr_handle: str,
ocr_job_params: OcrJobParams,
ocr_job_config: OcrJobConfig,
) -> OcrOutput: ...
diff --git a/pipelex/cogt/inference/inference_manager.py b/pipelex/cogt/inference/inference_manager.py
index 4dc2e648c..cabc2a26e 100644
--- a/pipelex/cogt/inference/inference_manager.py
+++ b/pipelex/cogt/inference/inference_manager.py
@@ -12,7 +12,6 @@ from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.llm.llm_worker_factory import LLMWorkerFactory
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.ocr.ocr_engine_factory import OcrEngineFactory
from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
from pipelex.cogt.ocr.ocr_worker_factory import OcrWorkerFactory
from pipelex.config import get_config
@@ -81,7 +80,7 @@ class InferenceManager(InferenceManagerProtocol):
f"No LLM worker for '{llm_handle}', set it up or enable cogt.inference_manager_config.is_auto_setup_preset_llm"
)
- inference_model = get_models_manager().get_inference_model(llm_handle=llm_handle)
+ inference_model = get_models_manager().get_inference_model(model_handle=llm_handle)
llm_worker = self._setup_one_internal_llm_worker(
inference_model=inference_model,
llm_handle=llm_handle,
@@ -105,16 +104,6 @@ class InferenceManager(InferenceManagerProtocol):
# Manage IMGG Workers
####################################################################################################
- @override
- def setup_imgg_workers(self):
- log.verbose("Setting up Imgg Workers...")
- imgg_handles = get_config().cogt.imgg_config.imgg_handles
- log.verbose(f"{len(imgg_handles)} Imgg handles found")
- for imgg_handle in imgg_handles:
- self._setup_one_imgg_worker(imgg_handle=imgg_handle)
-
- log.debug("Done setting up Imgg Workers (async)")
-
def _setup_one_imgg_worker(self, imgg_handle: str) -> ImggWorkerAbstract:
imgg_engine = ImggEngineFactory.make_imgg_engine(imgg_handle=imgg_handle)
log.verbose(imgg_engine.desc, title=f"Setting up ImgEngine for '{imgg_handle}'")
@@ -141,34 +130,30 @@ class InferenceManager(InferenceManagerProtocol):
# Manage OCR Workers
####################################################################################################
- @override
- def setup_ocr_workers(self):
- log.verbose("Setting up OCR Workers...")
- ocr_handles = get_config().cogt.ocr_config.ocr_handles
- log.verbose(f"{len(ocr_handles)} OCR handles found")
- for ocr_handle in ocr_handles:
- self._setup_one_ocr_worker(ocr_handle=ocr_handle)
-
- log.debug("Done setting up OCR Workers (async)")
-
- def _setup_one_ocr_worker(self, ocr_handle: str) -> OcrWorkerAbstract:
- ocr_engine = OcrEngineFactory.make_ocr_engine(ocr_handle=ocr_handle)
- log.verbose(ocr_engine.desc, title=f"Setting up OcrEngine for '{ocr_handle}'")
+ def _setup_one_ocr_worker(
+ self,
+ inference_model: InferenceModelSpec,
+ model_handle: str,
+ ) -> OcrWorkerAbstract:
ocr_worker = self.ocr_worker_factory.make_ocr_worker(
- ocr_engine=ocr_engine,
+ inference_model=inference_model,
reporting_delegate=get_report_delegate(),
)
- self.ocr_workers[ocr_handle] = ocr_worker
+ self.ocr_workers[model_handle] = ocr_worker
return ocr_worker
@override
- def get_ocr_worker(self, ocr_handle: str) -> OcrWorkerAbstract:
- ocr_worker = self.ocr_workers.get(ocr_handle)
- if ocr_worker is None:
- if not get_config().cogt.inference_manager_config.is_auto_setup_preset_ocr:
- raise InferenceManagerWorkerSetupError(
- f"Found no OCR worker for '{ocr_handle}', set it up or enable cogt.inference_manager_config.is_auto_setup_preset_ocr"
- )
+ def get_ocr_worker(self, model_handle: str) -> OcrWorkerAbstract:
+ if ocr_worker := self.ocr_workers.get(model_handle):
+ return ocr_worker
+ if not get_config().cogt.inference_manager_config.is_auto_setup_preset_ocr:
+ raise InferenceManagerWorkerSetupError(
+ f"Found no OCR worker for '{model_handle}', set it up or enable cogt.inference_manager_config.is_auto_setup_preset_ocr"
+ )
- ocr_worker = self._setup_one_ocr_worker(ocr_handle=ocr_handle)
+ inference_model = get_models_manager().get_inference_model(model_handle=model_handle)
+ ocr_worker = self._setup_one_ocr_worker(
+ inference_model=inference_model,
+ model_handle=model_handle,
+ )
return ocr_worker
diff --git a/pipelex/cogt/inference/inference_manager_protocol.py b/pipelex/cogt/inference/inference_manager_protocol.py
index 848f6cbca..53fcfd2ec 100644
--- a/pipelex/cogt/inference/inference_manager_protocol.py
+++ b/pipelex/cogt/inference/inference_manager_protocol.py
@@ -30,14 +30,10 @@ class InferenceManagerProtocol(Protocol):
# IMG Generation Workers
####################################################################################################
- def setup_imgg_workers(self): ...
-
def get_imgg_worker(self, imgg_handle: str) -> ImggWorkerAbstract: ...
####################################################################################################
# OCR Workers
####################################################################################################
- def setup_ocr_workers(self): ...
-
- def get_ocr_worker(self, ocr_handle: str) -> OcrWorkerAbstract: ...
+ def get_ocr_worker(self, model_handle: str) -> OcrWorkerAbstract: ...
diff --git a/pipelex/cogt/model_backends/model_spec.py b/pipelex/cogt/model_backends/model_spec.py
index e9e803334..891fcb641 100644
--- a/pipelex/cogt/model_backends/model_spec.py
+++ b/pipelex/cogt/model_backends/model_spec.py
@@ -3,6 +3,7 @@ from typing import List, Optional
from pydantic import Field
from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.model_backends.prompting_target import PromptingTarget
from pipelex.cogt.usage.cost_category import CostsByCategoryDict
from pipelex.tools.config.config_model import ConfigModel
@@ -12,6 +13,7 @@ class InferenceModelSpec(ConfigModel):
backend_name: str
name: str
sdk: str
+ model_type: ModelType = Field(strict=False)
model_id: str
inputs: List[str] = Field(default_factory=list)
outputs: List[str] = Field(default_factory=list)
diff --git a/pipelex/cogt/model_backends/model_spec_factory.py b/pipelex/cogt/model_backends/model_spec_factory.py
index 7afb2c443..7566885b9 100644
--- a/pipelex/cogt/model_backends/model_spec_factory.py
+++ b/pipelex/cogt/model_backends/model_spec_factory.py
@@ -5,6 +5,7 @@ from pydantic import BaseModel, Field, field_validator
from pipelex.cogt.exceptions import InferenceModelSpecError
from pipelex.cogt.model_backends.model_constraints import ModelConstraints
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.model_backends.prompting_target import PromptingTarget
from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.tools.config.config_model import ConfigModel
@@ -13,6 +14,7 @@ from pipelex.tools.config.config_model import ConfigModel
class InferenceModelSpecBlueprint(ConfigModel):
enabled: bool = True
sdk: Optional[str] = None
+ model_type: ModelType = Field(default=ModelType.LLM, strict=False)
model_id: str
inputs: List[str] = Field(default_factory=list)
outputs: List[str] = Field(default_factory=list)
@@ -55,6 +57,7 @@ class InferenceModelSpecFactory(BaseModel):
backend_name=backend_name,
name=name,
sdk=sdk,
+ model_type=blueprint.model_type,
model_id=blueprint.model_id,
inputs=blueprint.inputs,
outputs=blueprint.outputs,
diff --git a/pipelex/cogt/model_backends/model_type.py b/pipelex/cogt/model_backends/model_type.py
new file mode 100644
index 000000000..02e01ecf0
--- /dev/null
+++ b/pipelex/cogt/model_backends/model_type.py
@@ -0,0 +1,6 @@
+from pipelex.types import StrEnum
+
+
+class ModelType(StrEnum):
+ LLM = "llm"
+ TEXT_EXTRACTOR = "text_extractor"
diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.py
index c78ca8391..a8fd66b62 100644
--- a/pipelex/cogt/models/model_deck.py
+++ b/pipelex/cogt/models/model_deck.py
@@ -60,7 +60,7 @@ class ModelDeck(ConfigModel):
@classmethod
def final_validate(cls, deck: Self): # pyright: ignore[reportIncompatibleMethodOverride]
for llm_preset_id, llm_setting in deck.llm_presets.items():
- inference_model = deck.get_required_inference_model(llm_handle=llm_setting.llm_handle)
+ inference_model = deck.get_required_inference_model(model_handle=llm_setting.llm_handle)
try:
cls._validate_llm_setting(llm_setting=llm_setting, inference_model=inference_model)
except ConfigValidationError as exc:
@@ -120,25 +120,25 @@ class ModelDeck(ConfigModel):
self.check_llm_setting(llm_setting_or_preset_id=llm_setting)
return
- def get_optional_inference_model(self, llm_handle: str) -> Optional[InferenceModelSpec]:
- if inference_model := self.inference_models.get(llm_handle):
+ def get_optional_inference_model(self, model_handle: str) -> Optional[InferenceModelSpec]:
+ if inference_model := self.inference_models.get(model_handle):
return inference_model
- elif redirection := self.aliases.get(llm_handle):
- log.debug(f"Redirection for '{llm_handle}': {redirection}")
+ elif redirection := self.aliases.get(model_handle):
+ log.debug(f"Redirection for '{model_handle}': {redirection}")
if isinstance(redirection, str):
alias_list = [redirection]
else:
alias_list = redirection
for alias in alias_list:
- if inference_model := self.get_optional_inference_model(llm_handle=alias):
+ if inference_model := self.get_optional_inference_model(model_handle=alias):
return inference_model
- log.warning(f"Skipping LLM handle '{llm_handle}' because it's not found in deck")
+ log.warning(f"Skipping model handle '{model_handle}' because it's not found in deck")
return None
- def get_required_inference_model(self, llm_handle: str) -> InferenceModelSpec:
- inference_model = self.get_optional_inference_model(llm_handle=llm_handle)
+ def get_required_inference_model(self, model_handle: str) -> InferenceModelSpec:
+ inference_model = self.get_optional_inference_model(model_handle=model_handle)
if inference_model is None:
- raise LLMHandleNotFoundError(f"LLM handle '{llm_handle}' not found in deck")
- if llm_handle not in self.inference_models:
- log.dev(f"LLM handle '{llm_handle}' is an alias which resolves to '{inference_model.name}'")
+ raise LLMHandleNotFoundError(f"Model handle '{model_handle}' not found in deck")
+ if model_handle not in self.inference_models:
+ log.dev(f"Model handle '{model_handle}' is an alias which resolves to '{inference_model.name}'")
return inference_model
diff --git a/pipelex/cogt/models/model_deck_check.py b/pipelex/cogt/models/model_deck_check.py
index 9d72278f5..ce9990940 100644
--- a/pipelex/cogt/models/model_deck_check.py
+++ b/pipelex/cogt/models/model_deck_check.py
@@ -7,7 +7,7 @@ def check_llm_setting_with_deck(llm_setting_or_preset_id: LLMSettingOrPresetId):
if isinstance(llm_setting_or_preset_id, LLMSetting):
return
preset_id: str = llm_setting_or_preset_id
- llm_deck = get_models_manager().get_llm_deck()
+ llm_deck = get_models_manager().get_model_deck()
if preset_id in llm_deck.llm_presets:
return
raise LLMPresetNotFoundError(f"llm preset id '{preset_id}' not found in deck")
diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.py
index 01459f753..8beb112b9 100644
--- a/pipelex/cogt/models/model_manager.py
+++ b/pipelex/cogt/models/model_manager.py
@@ -22,13 +22,13 @@ class ModelManager(ModelManagerAbstract):
def __init__(self) -> None:
self.routing_profile_library = RoutingProfileLibrary.make_empty()
self.inference_backend_library = InferenceBackendLibrary.make_empty()
- self.llm_deck: Optional[ModelDeck] = None
+ self.model_deck: Optional[ModelDeck] = None
@override
- def get_llm_deck(self) -> ModelDeck:
- if self.llm_deck is None:
- raise RuntimeError("LLM deck is not initialized")
- return self.llm_deck
+ def get_model_deck(self) -> ModelDeck:
+ if self.model_deck is None:
+ raise RuntimeError("Model deck is not initialized")
+ return self.model_deck
@override
def teardown(self) -> None:
@@ -39,8 +39,8 @@ class ModelManager(ModelManagerAbstract):
def setup(self) -> None:
self.routing_profile_library.load()
self.inference_backend_library.load()
- llm_deck_blueprint = self.load_deck_blueprint()
- self.llm_deck = self.build_deck(llm_deck_blueprint=llm_deck_blueprint)
+ deck_blueprint = self.load_deck_blueprint()
+ self.model_deck = self.build_deck(model_deck_blueprint=deck_blueprint)
@classmethod
def _validate_toml_files(cls) -> None:
@@ -77,9 +77,9 @@ class ModelManager(ModelManagerAbstract):
llm_deck_blueprint = ModelDeckBlueprint.model_validate(full_llm_deck_dict)
return llm_deck_blueprint
- def build_deck(self, llm_deck_blueprint: ModelDeckBlueprint) -> ModelDeck:
+ def build_deck(self, model_deck_blueprint: ModelDeckBlueprint) -> ModelDeck:
all_models_and_possible_backends = self.inference_backend_library.get_all_models_and_possible_backends()
- llm_handles: Dict[str, InferenceModelSpec] = {}
+ model_handles: Dict[str, InferenceModelSpec] = {}
for model_name, available_backends in all_models_and_possible_backends.items():
backend_match_for_model = self.routing_profile_library.get_backend_match_for_model_from_active_routing_profile(
@@ -127,22 +127,22 @@ class ModelManager(ModelManagerAbstract):
f"Model spec '{model_name}' not found in any of the available backends '{available_backends}' "
f"which was set as default in routing profile '{backend_match_for_model.routing_profile_name}'"
)
- llm_handles[model_name] = model_spec
-
- llm_deck = ModelDeck(
- inference_models=llm_handles,
- aliases=llm_deck_blueprint.aliases,
- llm_presets=llm_deck_blueprint.llm_presets,
- llm_choice_defaults=llm_deck_blueprint.llm_choice_defaults,
- llm_choice_overrides=llm_deck_blueprint.llm_choice_overrides,
+ model_handles[model_name] = model_spec
+
+ model_deck = ModelDeck(
+ inference_models=model_handles,
+ aliases=model_deck_blueprint.aliases,
+ llm_presets=model_deck_blueprint.llm_presets,
+ llm_choice_defaults=model_deck_blueprint.llm_choice_defaults,
+ llm_choice_overrides=model_deck_blueprint.llm_choice_overrides,
)
- return llm_deck
+ return model_deck
@override
- def get_inference_model(self, llm_handle: str) -> InferenceModelSpec:
- if self.llm_deck is None:
- raise RuntimeError("LLM deck is not initialized")
- return self.llm_deck.get_required_inference_model(llm_handle=llm_handle)
+ def get_inference_model(self, model_handle: str) -> InferenceModelSpec:
+ if self.model_deck is None:
+ raise RuntimeError("Model deck is not initialized")
+ return self.model_deck.get_required_inference_model(model_handle=model_handle)
@override
def get_required_inference_backend(self, backend_name: str) -> InferenceBackend:
diff --git a/pipelex/cogt/models/model_manager_abstract.py b/pipelex/cogt/models/model_manager_abstract.py
index e6ae5dc2b..b0cc94593 100644
--- a/pipelex/cogt/models/model_manager_abstract.py
+++ b/pipelex/cogt/models/model_manager_abstract.py
@@ -15,11 +15,11 @@ class ModelManagerAbstract(ABC):
pass
@abstractmethod
- def get_inference_model(self, llm_handle: str) -> InferenceModelSpec:
+ def get_inference_model(self, model_handle: str) -> InferenceModelSpec:
pass
@abstractmethod
- def get_llm_deck(self) -> ModelDeck:
+ def get_model_deck(self) -> ModelDeck:
pass
@abstractmethod
diff --git a/pipelex/cogt/ocr/ocr_engine.py b/pipelex/cogt/ocr/ocr_engine.py
deleted file mode 100644
index a57b19475..000000000
--- a/pipelex/cogt/ocr/ocr_engine.py
+++ /dev/null
@@ -1,12 +0,0 @@
-from pydantic import BaseModel
-
-from pipelex.cogt.ocr.ocr_platform import OcrPlatform
-
-
-class OcrEngine(BaseModel):
- ocr_platform: OcrPlatform
- ocr_model_name: str
-
- @property
- def desc(self) -> str:
- return f"Ocr Engine {self.ocr_platform}/{self.ocr_model_name}"
diff --git a/pipelex/cogt/ocr/ocr_engine_factory.py b/pipelex/cogt/ocr/ocr_engine_factory.py
deleted file mode 100644
index ce30a5206..000000000
--- a/pipelex/cogt/ocr/ocr_engine_factory.py
+++ /dev/null
@@ -1,27 +0,0 @@
-from pipelex.cogt.exceptions import CogtError
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
-from pipelex.cogt.ocr.ocr_platform import OcrPlatform
-
-
-class OcrEngineFactoryError(CogtError):
- pass
-
-
-class OcrEngineFactory:
- @classmethod
- def make_ocr_engine(
- cls,
- ocr_handle: str,
- ) -> OcrEngine:
- parts = ocr_handle.split("/")
- if len(parts) != 2:
- raise OcrEngineFactoryError(f"Invalid Ocr handle: {ocr_handle}")
-
- try:
- ocr_platform = OcrPlatform(parts[0])
- except ValueError:
- raise OcrEngineFactoryError(f"Invalid Ocr platform: {parts[0]}")
-
- ocr_model_name = parts[1]
-
- return OcrEngine(ocr_platform=ocr_platform, ocr_model_name=ocr_model_name)
diff --git a/pipelex/cogt/ocr/ocr_handle.py b/pipelex/cogt/ocr/ocr_handle.py
deleted file mode 100644
index f949fe6aa..000000000
--- a/pipelex/cogt/ocr/ocr_handle.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from pipelex.types import StrEnum
-
-
-class OcrHandle(StrEnum):
- BASIC_OCR = "basic/pypdfium2"
- MISTRAL_OCR = "mistral/mistral-ocr-latest"
diff --git a/pipelex/cogt/ocr/ocr_job.py b/pipelex/cogt/ocr/ocr_job.py
index 12f6bdaf9..8a2edd17a 100644
--- a/pipelex/cogt/ocr/ocr_job.py
+++ b/pipelex/cogt/ocr/ocr_job.py
@@ -3,7 +3,7 @@ from datetime import datetime
from typing_extensions import override
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams, OcrJobReport
@@ -18,7 +18,7 @@ class OcrJob(InferenceJobAbstract):
def validate_before_execution(self):
pass
- def ocr_job_before_start(self, ocr_engine: OcrEngine):
+ def ocr_job_before_start(self, inference_model: InferenceModelSpec):
# Reset metadata
self.job_metadata.started_at = datetime.now()
diff --git a/pipelex/cogt/ocr/ocr_platform.py b/pipelex/cogt/ocr/ocr_platform.py
deleted file mode 100644
index 5a6baa1f2..000000000
--- a/pipelex/cogt/ocr/ocr_platform.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from pipelex.types import StrEnum
-
-
-class OcrPlatform(StrEnum):
- BASIC = "basic"
- MISTRAL = "mistral"
diff --git a/pipelex/cogt/ocr/ocr_worker_abstract.py b/pipelex/cogt/ocr/ocr_worker_abstract.py
index d7211b4e0..42570da61 100644
--- a/pipelex/cogt/ocr/ocr_worker_abstract.py
+++ b/pipelex/cogt/ocr/ocr_worker_abstract.py
@@ -1,11 +1,11 @@
from abc import abstractmethod
-from typing import Optional
+from typing import Any, Dict, Optional
from typing_extensions import override
from pipelex import log
from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_job import OcrJob
from pipelex.cogt.ocr.ocr_output import OcrOutput
from pipelex.pipeline.job_metadata import UnitJobId
@@ -15,11 +15,13 @@ from pipelex.reporting.reporting_protocol import ReportingProtocol
class OcrWorkerAbstract(InferenceWorkerAbstract):
def __init__(
self,
- ocr_engine: OcrEngine,
+ extra_config: Dict[str, Any],
+ inference_model: InferenceModelSpec,
reporting_delegate: Optional[ReportingProtocol] = None,
):
InferenceWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)
- self.ocr_engine = ocr_engine
+ self.extra_config = extra_config
+ self.inference_model = inference_model
#########################################################
# Instance methods
@@ -28,7 +30,7 @@ class OcrWorkerAbstract(InferenceWorkerAbstract):
@property
@override
def desc(self) -> str:
- return f"Ocr Worker using:\n{self.ocr_engine.desc}"
+ return f"OCR-Worker:{self.inference_model.tag}"
def _check_can_perform_job(self, ocr_job: OcrJob):
# This can be overridden by subclasses for specific checks
@@ -38,7 +40,7 @@ class OcrWorkerAbstract(InferenceWorkerAbstract):
self,
ocr_job: OcrJob,
) -> OcrOutput:
- log.debug(f"OCR Worker ocr_extract_pages:\n{self.ocr_engine.desc}")
+ log.debug(f"OCR Worker ocr_extract_pages:\n{self.inference_model.desc}")
# Verify that the job is valid
ocr_job.validate_before_execution()
@@ -51,7 +53,7 @@ class OcrWorkerAbstract(InferenceWorkerAbstract):
ocr_job.job_metadata.unit_job_id = UnitJobId.OCR_EXTRACT_PAGES
# Prepare job
- ocr_job.ocr_job_before_start(ocr_engine=self.ocr_engine)
+ ocr_job.ocr_job_before_start(inference_model=self.inference_model)
# Execute job
result = await self._ocr_extract_pages(ocr_job=ocr_job)
diff --git a/pipelex/cogt/ocr/ocr_worker_factory.py b/pipelex/cogt/ocr/ocr_worker_factory.py
index 0c61b5ff7..718d50417 100644
--- a/pipelex/cogt/ocr/ocr_worker_factory.py
+++ b/pipelex/cogt/ocr/ocr_worker_factory.py
@@ -1,8 +1,7 @@
from typing import Optional
from pipelex.cogt.exceptions import MissingDependencyError
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
-from pipelex.cogt.ocr.ocr_platform import OcrPlatform
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
from pipelex.hub import get_models_manager, get_plugin_manager
from pipelex.plugins.plugin_sdk_registry import Plugin
@@ -12,15 +11,15 @@ from pipelex.reporting.reporting_protocol import ReportingProtocol
class OcrWorkerFactory:
def make_ocr_worker(
self,
- ocr_engine: OcrEngine,
+ inference_model: InferenceModelSpec,
reporting_delegate: Optional[ReportingProtocol] = None,
) -> OcrWorkerAbstract:
- backend = get_models_manager().get_required_inference_backend("mistral")
- ocr_plugin = Plugin.make_for_ocr_engine(ocr_platform=ocr_engine.ocr_platform)
+ plugin = Plugin.make_for_inference_model(inference_model=inference_model)
+ backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)
plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
ocr_worker: OcrWorkerAbstract
- match ocr_engine.ocr_platform:
- case OcrPlatform.MISTRAL:
+ match plugin.sdk:
+ case "mistral":
try:
import mistralai # noqa: F401
except ImportError as exc:
@@ -33,23 +32,27 @@ class OcrWorkerFactory:
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.plugins.mistral.mistral_ocr_worker import MistralOcrWorker
- ocr_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=ocr_plugin) or plugin_sdk_registry.set_sdk_instance(
- plugin=ocr_plugin,
+ ocr_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=plugin,
sdk_instance=MistralFactory.make_mistral_client(backend=backend),
)
ocr_worker = MistralOcrWorker(
sdk_instance=ocr_sdk_instance,
- ocr_engine=ocr_engine,
+ extra_config=backend.extra_config,
+ inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
- case OcrPlatform.BASIC:
+ case "pypdfium2":
from pipelex.plugins.pypdfium2.pypdfium2_worker import Pypdfium2Worker
ocr_worker = Pypdfium2Worker(
sdk_instance=None,
- ocr_engine=ocr_engine,
+ extra_config=backend.extra_config,
+ inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
+ case _:
+ raise NotImplementedError(f"Plugin '{plugin}' is not supported")
return ocr_worker
diff --git a/pipelex/config_template/inference/backends.toml b/pipelex/config_template/inference/backends.toml
index 64c393f41..7b68e6d4d 100644
--- a/pipelex/config_template/inference/backends.toml
+++ b/pipelex/config_template/inference/backends.toml
@@ -44,3 +44,6 @@ gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
[mistral]
enabled = false
api_key = "${MISTRAL_API_KEY}"
+
+[internal] # software-only backend, runs internally, without AI
+enabled = true
diff --git a/pipelex/config_template/inference/backends/internal.toml b/pipelex/config_template/inference/backends/internal.toml
new file mode 100644
index 000000000..da42bf7d1
--- /dev/null
+++ b/pipelex/config_template/inference/backends/internal.toml
@@ -0,0 +1,9 @@
+
+[pypdfium2-extract-text]
+model_type = "text_extractor"
+sdk = "pypdfium2"
+model_id = "extract-text"
+inputs = ["pdf"]
+outputs = ["pages"]
+costs = {}
+
diff --git a/pipelex/config_template/inference/backends/mistral.toml b/pipelex/config_template/inference/backends/mistral.toml
index 13f61e4a3..b452cef8b 100644
--- a/pipelex/config_template/inference/backends/mistral.toml
+++ b/pipelex/config_template/inference/backends/mistral.toml
@@ -92,3 +92,11 @@ inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.4, output = 2.0 }
+[mistral-ocr]
+model_type = "text_extractor"
+model_id = "mistral-ocr-latest"
+max_tokens = 131072
+inputs = ["pdf", "image"]
+outputs = ["pages"]
+costs = { input = 0.4, output = 2.0 }
+
diff --git a/pipelex/config_template/inference/deck/base_deck.toml b/pipelex/config_template/inference/deck/base_deck.toml
index 76e436a87..51d646c9a 100644
--- a/pipelex/config_template/inference/deck/base_deck.toml
+++ b/pipelex/config_template/inference/deck/base_deck.toml
@@ -89,3 +89,4 @@ llm_to_extract_tables = { llm_handle = "base-claude", temperature = 0.1 }
[llm_choice_defaults]
for_text = "cheap_llm_for_text"
for_object = "cheap_llm_for_object"
+
diff --git a/pipelex/config_template/inference/routing_profiles.toml b/pipelex/config_template/inference/routing_profiles.toml
index 9a26b8f4c..be9b51d2c 100644
--- a/pipelex/config_template/inference/routing_profiles.toml
+++ b/pipelex/config_template/inference/routing_profiles.toml
@@ -82,3 +82,4 @@ default = "pipelex_inference"
# - Exact names: "gpt-4o-mini"
# - Wildcards: "claude-*" (matches all models starting with claude-)
# - Partial wildcards: "*-sonnet" (matches all sonnet variants)
+
diff --git a/pipelex/config_template/pipelex.toml b/pipelex/config_template/pipelex.toml
index e5ef1aa66..b8f8b129f 100644
--- a/pipelex/config_template/pipelex.toml
+++ b/pipelex/config_template/pipelex.toml
@@ -11,7 +11,6 @@ api_key_method = "env"
####################################################################################################
[cogt.ocr_config]
-ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
[pipelex.feature_config]
diff --git a/pipelex/core/exceptions.py b/pipelex/core/exceptions.py
index ab615cb16..62f41fe28 100644
--- a/pipelex/core/exceptions.py
+++ b/pipelex/core/exceptions.py
@@ -13,12 +13,6 @@ class PipelexConfigurationError(PipelexInterpreterError):
pass
-class PipelexFileError(PipelexInterpreterError):
- """Raised when there are file-related issues in PipelexInterpreter."""
-
- pass
-
-
class PipelexUnknownPipeError(PipelexInterpreterError):
"""Raised when encountering an unknown pipe blueprint type."""
diff --git a/pipelex/core/interpreter.py b/pipelex/core/interpreter.py
index ba20c7049..2d2993351 100644
--- a/pipelex/core/interpreter.py
+++ b/pipelex/core/interpreter.py
@@ -7,18 +7,31 @@ from typing_extensions import Self
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
-from pipelex.core.exceptions import PipelexConfigurationError, PipelexFileError, PipelexUnknownPipeError
+from pipelex.core.exceptions import (
+ PipelexConfigurationError,
+ PipelexUnknownPipeError,
+)
from pipelex.core.pipes.pipe_input_spec_blueprint import InputRequirementBlueprint
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
-from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint
-from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint
-from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import (
+ PipeConditionBlueprint,
+)
+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import (
+ PipeParallelBlueprint,
+)
+from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import (
+ PipeSequenceBlueprint,
+)
from pipelex.pipe_operators.func.pipe_func_blueprint import PipeFuncBlueprint
from pipelex.pipe_operators.img_gen.pipe_img_gen_blueprint import PipeImgGenBlueprint
from pipelex.pipe_operators.jinja2.pipe_jinja2_blueprint import PipeJinja2Blueprint
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_operators.ocr.pipe_ocr_blueprint import PipeOcrBlueprint
-from pipelex.tools.misc.toml_utils import clean_trailing_whitespace, validate_toml_content, validate_toml_file
+from pipelex.tools.misc.toml_utils import (
+ clean_trailing_whitespace,
+ validate_toml_content,
+ validate_toml_file,
+)
class PLXDecodeError(toml.TomlDecodeError):
@@ -63,23 +76,19 @@ class PipelexInterpreter(BaseModel):
validate_toml_content(content=self.file_content, file_path=str(self.file_path))
return self
- def get_file_content(self) -> str:
+ def _read_and_reformat(self) -> str:
"""Load PLX content from file_path or use file_content directly."""
if self.file_path:
- try:
- with open(self.file_path, "r", encoding="utf-8") as file:
- file_content = file.read()
-
- # Clean trailing whitespace and write back if needed
- cleaned_content = clean_trailing_whitespace(file_content)
- if file_content != cleaned_content:
- with open(self.file_path, "w", encoding="utf-8") as file:
- file.write(cleaned_content)
- return cleaned_content
- return file_content
-
- except Exception as exc:
- raise PipelexFileError(f"Failed to read PLX file '{self.file_path}': {exc}") from exc
+ with open(self.file_path, "r", encoding="utf-8") as file:
+ file_content = file.read()
+
+ # Clean trailing whitespace and write back if needed
+ cleaned_content = clean_trailing_whitespace(file_content)
+ if file_content != cleaned_content:
+ with open(self.file_path, "w", encoding="utf-8") as file:
+ file.write(cleaned_content)
+ return cleaned_content
+ return file_content
elif self.file_content is None:
raise PipelexConfigurationError("file_content must be provided if file_path is not provided")
return self.file_content
@@ -128,7 +137,7 @@ class PipelexInterpreter(BaseModel):
def make_pipelex_bundle_blueprint(self) -> PipelexBundleBlueprint:
"""Make a PipelexBundleBlueprint from the file_path or file_content"""
- file_content = self.get_file_content()
+ file_content = self._read_and_reformat()
plx_data = self._parse_plx_content(file_content)
return PipelexBundleBlueprint.model_validate(plx_data)
@@ -335,7 +344,7 @@ class PipelexInterpreter(BaseModel):
PipelexInterpreter.add_inputs_to_lines_if_exist(lines, pipe.inputs)
lines.append(f'output = "{PipelexInterpreter.escape_plx_string(pipe.output_concept_string_or_concept_code)}"')
- lines.append(f'ocr_handle = "{pipe.ocr_handle.value}"')
+ lines.append(f'ocr_model = "{pipe.ocr_model}"')
return "\n".join(lines)
@@ -534,7 +543,9 @@ class PipelexInterpreter(BaseModel):
return field_value
@staticmethod
- def serialize_concept_structure(structure: Union[str, Dict[str, Any]]) -> Union[str, Dict[str, Any]]:
+ def serialize_concept_structure(
+ structure: Union[str, Dict[str, Any]],
+ ) -> Union[str, Dict[str, Any]]:
"""Serialize concept structure field."""
if isinstance(structure, str):
return structure
@@ -545,7 +556,9 @@ class PipelexInterpreter(BaseModel):
return result
@staticmethod
- def serialize_single_concept(concept_blueprint: Union[ConceptBlueprint, str]) -> Union[str, Dict[str, Any]]:
+ def serialize_single_concept(
+ concept_blueprint: Union[ConceptBlueprint, str],
+ ) -> Union[str, Dict[str, Any]]:
"""Serialize a single concept blueprint."""
if isinstance(concept_blueprint, str):
return concept_blueprint
@@ -560,7 +573,10 @@ class PipelexInterpreter(BaseModel):
return concept_data
elif concept_blueprint.refines is not None:
# Concept with refines
- concept_data = {"definition": concept_blueprint.definition, "refines": concept_blueprint.refines}
+ concept_data = {
+ "definition": concept_blueprint.definition,
+ "refines": concept_blueprint.refines,
+ }
return concept_data
else:
# Simple concept with just definition
@@ -746,11 +762,9 @@ class PipelexInterpreter(BaseModel):
result["output"] = pipe.output_concept_string_or_concept_code
# Add required fields
- result["ocr_handle"] = pipe.ocr_handle
+ result["ocr_model"] = pipe.ocr_model
# Add optional fields
- if pipe.ocr_platform:
- result["ocr_platform"] = pipe.ocr_platform
if pipe.page_images:
result["page_images"] = pipe.page_images
if pipe.page_image_captions:
@@ -950,7 +964,9 @@ class PipelexInterpreter(BaseModel):
return result
@staticmethod
- def serialize_input_requirement(input_req: InputRequirementBlueprint) -> Dict[str, Any]:
+ def serialize_input_requirement(
+ input_req: InputRequirementBlueprint,
+ ) -> Dict[str, Any]:
"""Serialize a single InputRequirementBlueprint to PLX format."""
result: Dict[str, Any] = {"concept": input_req.concept}
if input_req.multiplicity is not None:
@@ -958,7 +974,9 @@ class PipelexInterpreter(BaseModel):
return result
@staticmethod
- def serialize_inputs(inputs: Mapping[str, Union[str, InputRequirementBlueprint]]) -> Dict[str, Any]:
+ def serialize_inputs(
+ inputs: Mapping[str, Union[str, InputRequirementBlueprint]],
+ ) -> Dict[str, Any]:
"""Convert InputRequirementBlueprint objects to proper PLX format for serialization."""
result: Dict[str, Any] = {}
for key, value in inputs.items():
@@ -970,7 +988,9 @@ class PipelexInterpreter(BaseModel):
return result
@staticmethod
- def inputs_to_plx_string(inputs: Mapping[str, Union[str, InputRequirementBlueprint]]) -> str:
+ def inputs_to_plx_string(
+ inputs: Mapping[str, Union[str, InputRequirementBlueprint]],
+ ) -> str:
"""Convert inputs dictionary to PLX string format."""
inputs_dict = PipelexInterpreter.serialize_inputs(inputs)
inputs_parts: list[str] = []
@@ -997,7 +1017,10 @@ class PipelexInterpreter(BaseModel):
return f"{{ {', '.join(inputs_parts)} }}"
@staticmethod
- def add_inputs_to_lines_if_exist(lines: list[str], pipe_inputs: Optional[Mapping[str, Union[str, InputRequirementBlueprint]]]) -> None:
+ def add_inputs_to_lines_if_exist(
+ lines: list[str],
+ pipe_inputs: Optional[Mapping[str, Union[str, InputRequirementBlueprint]]],
+ ) -> None:
"""Add inputs line to PLX lines if inputs exist."""
if pipe_inputs:
inputs_dict = PipelexInterpreter.serialize_inputs(pipe_inputs)
diff --git a/pipelex/hub.py b/pipelex/hub.py
index f63039f1d..e2727e294 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -353,7 +353,7 @@ def get_imgg_worker(
def get_ocr_worker(
ocr_handle: str,
) -> OcrWorkerAbstract:
- return get_inference_manager().get_ocr_worker(ocr_handle=ocr_handle)
+ return get_inference_manager().get_ocr_worker(model_handle=ocr_handle)
def get_report_delegate() -> ReportingProtocol:
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index ad8b5c887..724c39b1e 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -2,6 +2,7 @@ import os
from pathlib import Path
from typing import ClassVar, List, Optional, Type
+from pydantic import ValidationError
from typing_extensions import override
from pipelex import log
@@ -22,6 +23,7 @@ from pipelex.core.pipes.pipe_library import PipeLibrary
from pipelex.exceptions import (
ConceptLibraryError,
LibraryError,
+ PipelexSetupError,
PipeLibraryError,
)
from pipelex.libraries.library_config import LibraryConfig
@@ -31,6 +33,7 @@ from pipelex.tools.func_registry_utils import FuncRegistryUtils
from pipelex.tools.misc.file_utils import find_files_in_dir
from pipelex.tools.misc.toml_utils import TOMLValidationError, validate_toml_file
from pipelex.tools.runtime_manager import runtime_manager
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
from pipelex.types import StrEnum
@@ -206,7 +209,11 @@ class LibraryManager(LibraryManagerAbstract):
return pipes
@override
- def load_libraries(self, library_dirs: Optional[List[Path]] = None, library_file_paths: Optional[List[Path]] = None) -> None:
+ def load_libraries(
+ self,
+ library_dirs: Optional[List[Path]] = None,
+ library_file_paths: Optional[List[Path]] = None,
+ ) -> None:
dirs_to_use: List[Path] = self._get_pipeline_library_dirs()
all_plx_paths: List[Path] = self._get_pipelex_plx_files_from_dirs(dirs_to_use)
@@ -229,7 +236,13 @@ class LibraryManager(LibraryManagerAbstract):
# Parse all blueprints first
blueprints: List[PipelexBundleBlueprint] = []
for plx_file_path in all_plx_paths:
- blueprint = PipelexInterpreter(file_path=plx_file_path).make_pipelex_bundle_blueprint()
+ try:
+ blueprint = PipelexInterpreter(file_path=plx_file_path).make_pipelex_bundle_blueprint()
+ except FileNotFoundError as exc:
+ raise PipelexSetupError(f"Could not find PLX blueprint at '{plx_file_path}'") from exc
+ except ValidationError as exc:
+ error_msg = format_pydantic_validation_error(exc)
+ raise PipelexSetupError(f"Could not load PLX blueprint from '{plx_file_path}' because of: {error_msg}") from exc
blueprints.append(blueprint)
# Load all domains first
diff --git a/pipelex/libraries/pipelines/documents.plx b/pipelex/libraries/pipelines/documents.plx
index 6f38360ca..0f8b290ef 100644
--- a/pipelex/libraries/pipelines/documents.plx
+++ b/pipelex/libraries/pipelines/documents.plx
@@ -17,7 +17,7 @@ inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
page_views = false
-ocr_handle = "mistral/mistral-ocr-latest"
+ocr_model = "mistral-ocr"
[pipe.ocr_page_contents_and_views_from_pdf]
type = "PipeOcr"
@@ -26,5 +26,5 @@ inputs = { ocr_input = "PDF" }
output = "Page"
page_images = true
page_views = true
-ocr_handle = "mistral/mistral-ocr-latest"
+ocr_model = "mistral-ocr"
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index 1432ab151..efa908d35 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -265,7 +265,7 @@ class PipeLLM(PipeOperator):
llm_for_text_choice = self.llm_choices.for_text
llm_for_object_choice = self.llm_choices.for_object
- llm_deck = get_models_manager().get_llm_deck()
+ llm_deck = get_models_manager().get_model_deck()
# Choice of main LLM for text first from this PipeLLM setting (self.llm_choices)
# or from the llm_choice_overrides or fallback on the llm_choice_defaults
@@ -283,7 +283,7 @@ class PipeLLM(PipeOperator):
llm_setting_for_object: LLMSetting = llm_deck.get_llm_setting(llm_setting_or_preset_id=llm_setting_or_preset_id_for_object)
if (not self.llm_prompt_spec.prompting_style) and (
- inference_model := llm_deck.get_optional_inference_model(llm_handle=llm_setting_main.llm_handle)
+ inference_model := llm_deck.get_optional_inference_model(model_handle=llm_setting_main.llm_handle)
):
# Note: the case where we don't get an inference model corresponds to the use of an external LLM Plugin
# TODO: improve this by making it possible to get the inference model for external LLM Plugins
diff --git a/pipelex/pipe_operators/ocr/pipe_ocr.py b/pipelex/pipe_operators/ocr/pipe_ocr.py
index 4b15742c7..5961c30b5 100644
--- a/pipelex/pipe_operators/ocr/pipe_ocr.py
+++ b/pipelex/pipe_operators/ocr/pipe_ocr.py
@@ -6,7 +6,6 @@ from typing_extensions import Self, override
from pipelex import log
from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.config import StaticValidationReaction, get_config
@@ -41,7 +40,7 @@ PIPE_OCR_INPUT_NAME = "ocr_input"
class PipeOcr(PipeOperator):
- ocr_handle: OcrHandle
+ ocr_handle: str
should_caption_images: bool
should_include_images: bool
should_include_page_views: bool
diff --git a/pipelex/pipe_operators/ocr/pipe_ocr_blueprint.py b/pipelex/pipe_operators/ocr/pipe_ocr_blueprint.py
index a9865946f..410a9c969 100644
--- a/pipelex/pipe_operators/ocr/pipe_ocr_blueprint.py
+++ b/pipelex/pipe_operators/ocr/pipe_ocr_blueprint.py
@@ -1,14 +1,11 @@
from typing import Literal, Optional
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
-from pipelex.cogt.ocr.ocr_platform import OcrPlatform
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
class PipeOcrBlueprint(PipeBlueprint):
type: Literal["PipeOcr"] = "PipeOcr"
- ocr_handle: OcrHandle
- ocr_platform: Optional[OcrPlatform] = None
+ ocr_model: str
page_images: Optional[bool] = None
page_image_captions: Optional[bool] = None
page_views: Optional[bool] = None
diff --git a/pipelex/pipe_operators/ocr/pipe_ocr_factory.py b/pipelex/pipe_operators/ocr/pipe_ocr_factory.py
index 158bfe64d..3c93930b1 100644
--- a/pipelex/pipe_operators/ocr/pipe_ocr_factory.py
+++ b/pipelex/pipe_operators/ocr/pipe_ocr_factory.py
@@ -39,7 +39,7 @@ class PipeOcrFactory(PipeFactoryProtocol[PipeOcrBlueprint, PipeOcr]):
inputs=PipeInputSpecFactory.make_from_blueprint(
domain=domain, blueprint=blueprint.inputs or {}, concept_codes_from_the_same_domain=concept_codes_from_the_same_domain
),
- ocr_handle=blueprint.ocr_handle,
+ ocr_handle=blueprint.ocr_model,
should_include_images=blueprint.page_images or False,
should_caption_images=blueprint.page_image_captions or False,
should_include_page_views=blueprint.page_views or False,
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index 0256c600a..6505f8e56 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -127,7 +127,6 @@ sdxl_lightning_map_quality_to_steps = { "low" = 2, "medium" = 4, "high" = 8 }
####################################################################################################
[cogt.ocr_config]
-ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
default_page_views_dpi = 72
diff --git a/pipelex/plugins/mistral/mistral_ocr_worker.py b/pipelex/plugins/mistral/mistral_ocr_worker.py
index 59611dcca..07f8389c8 100644
--- a/pipelex/plugins/mistral/mistral_ocr_worker.py
+++ b/pipelex/plugins/mistral/mistral_ocr_worker.py
@@ -1,11 +1,11 @@
-from typing import Any, Optional
+from typing import Any, Dict, Optional
from mistralai import Mistral
from typing_extensions import override
from pipelex import log
from pipelex.cogt.exceptions import OcrCapabilityError, SdkTypeError
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_input import OcrInputError
from pipelex.cogt.ocr.ocr_job import OcrJob
from pipelex.cogt.ocr.ocr_output import OcrOutput
@@ -22,10 +22,15 @@ class MistralOcrWorker(OcrWorkerAbstract):
def __init__(
self,
sdk_instance: Any,
- ocr_engine: OcrEngine,
+ extra_config: Dict[str, Any],
+ inference_model: InferenceModelSpec,
reporting_delegate: Optional[ReportingProtocol] = None,
):
- super().__init__(ocr_engine=ocr_engine, reporting_delegate=reporting_delegate)
+ super().__init__(
+ extra_config=extra_config,
+ inference_model=inference_model,
+ reporting_delegate=reporting_delegate,
+ )
if not isinstance(sdk_instance, Mistral):
raise SdkTypeError(f"Provided OCR sdk_instance for {self.__class__.__name__} is not of type Mistral: it's a '{type(sdk_instance)}'")
@@ -108,7 +113,7 @@ class MistralOcrWorker(OcrWorkerAbstract):
image_url: str,
) -> OcrOutput:
ocr_response = await self.mistral_client.ocr.process_async(
- model=self.ocr_engine.ocr_model_name,
+ model=self.inference_model.model_id,
document={
"type": "image_url",
"image_url": image_url,
@@ -129,7 +134,7 @@ class MistralOcrWorker(OcrWorkerAbstract):
mime_type = file_type.mime
ocr_response = await self.mistral_client.ocr.process_async(
- model=self.ocr_engine.ocr_model_name,
+ model=self.inference_model.model_id,
document={"type": "image_url", "image_url": f"data:{mime_type};base64,{b64.decode('utf-8')}"},
)
ocr_output = await MistralFactory.make_ocr_output_from_mistral_response(
@@ -143,7 +148,7 @@ class MistralOcrWorker(OcrWorkerAbstract):
should_include_images: bool = False,
) -> OcrOutput:
ocr_response = await self.mistral_client.ocr.process_async(
- model=self.ocr_engine.ocr_model_name,
+ model=self.inference_model.model_id,
document={
"type": "document_url",
"document_url": pdf_url,
diff --git a/pipelex/plugins/plugin_sdk_registry.py b/pipelex/plugins/plugin_sdk_registry.py
index c8643f208..aca4bae8b 100644
--- a/pipelex/plugins/plugin_sdk_registry.py
+++ b/pipelex/plugins/plugin_sdk_registry.py
@@ -4,7 +4,6 @@ from pydantic import BaseModel, Field, RootModel
from pipelex.cogt.imgg.imgg_platform import ImggPlatform
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.ocr.ocr_platform import OcrPlatform
class Plugin(BaseModel):
@@ -15,14 +14,6 @@ class Plugin(BaseModel):
def sdk_handle(self) -> str:
return f"{self.sdk}@{self.backend}"
- @classmethod
- def make_for_ocr_engine(cls, ocr_platform: OcrPlatform) -> "Plugin":
- match ocr_platform:
- case OcrPlatform.MISTRAL:
- return Plugin(sdk="mistral", backend="mistral")
- case OcrPlatform.BASIC:
- return Plugin(sdk="basic", backend="none")
-
@classmethod
def make_for_imgg_engine(cls, imgg_platform: ImggPlatform) -> "Plugin":
match imgg_platform:
diff --git a/pipelex/plugins/pypdfium2/pypdfium2_worker.py b/pipelex/plugins/pypdfium2/pypdfium2_worker.py
index 1193c4c3f..399cd55c8 100644
--- a/pipelex/plugins/pypdfium2/pypdfium2_worker.py
+++ b/pipelex/plugins/pypdfium2/pypdfium2_worker.py
@@ -2,7 +2,7 @@ from typing import Any, Dict, Optional
from typing_extensions import override
-from pipelex.cogt.ocr.ocr_engine import OcrEngine
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_input import OcrInputError
from pipelex.cogt.ocr.ocr_job import OcrJob
from pipelex.cogt.ocr.ocr_output import OcrOutput, Page
@@ -16,10 +16,11 @@ class Pypdfium2Worker(OcrWorkerAbstract):
def __init__(
self,
sdk_instance: Any,
- ocr_engine: OcrEngine,
+ extra_config: Dict[str, Any],
+ inference_model: InferenceModelSpec,
reporting_delegate: Optional[ReportingProtocol] = None,
):
- super().__init__(ocr_engine=ocr_engine, reporting_delegate=reporting_delegate)
+ super().__init__(extra_config=extra_config, inference_model=inference_model, reporting_delegate=reporting_delegate)
@override
async def _ocr_extract_pages(
diff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.py
index c42082988..84cc0af95 100644
--- a/pipelex/test_extras/shared_pytest_plugins.py
+++ b/pipelex/test_extras/shared_pytest_plugins.py
@@ -1,7 +1,9 @@
import pytest
from pytest import FixtureRequest, Parser
+from rich import print
from pipelex.core.pipes.pipe_run_params import PipeRunMode
+from pipelex.tools.environment import ENV_DUMMY_PLACEHOLDER_VALUE, is_env_set, set_env
from pipelex.tools.runtime_manager import RunMode, runtime_manager
@@ -24,3 +26,56 @@ def pytest_addoption(parser: Parser):
def pipe_run_mode(request: FixtureRequest) -> PipeRunMode:
mode_str = request.config.getoption("--pipe-run-mode")
return PipeRunMode(mode_str)
+
+
+def _setup_env_var_placeholders():
+ """Set placeholder environment variables when running in CI to prevent import failures.
+
+ These placeholders allow the code to import successfully, while actual inference tests
+ remain skipped via pytest markers.
+ """
+
+ # Define placeholder values for all inference-related env vars
+ env_var_placeholders = {
+ "PIPELEX_API_TOKEN": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "PIPELEX_API_BASE_URL": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "PIPELEX_INFERENCE_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "OPENAI_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AWS_ACCESS_KEY_ID": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AWS_SECRET_ACCESS_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AWS_REGION": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AZURE_API_BASE": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AZURE_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "AZURE_API_VERSION": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "GCP_PROJECT_ID": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "GCP_LOCATION": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "GCP_CREDENTIALS_FILE_PATH": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "ANTHROPIC_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "MISTRAL_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "PERPLEXITY_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "PERPLEXITY_API_ENDPOINT": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "XAI_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "XAI_API_ENDPOINT": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "FAL_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ "BLACKBOX_API_KEY": ENV_DUMMY_PLACEHOLDER_VALUE,
+ }
+
+ # Set placeholders for env vars who's presence is required for the code to run properly
+ # even if their value is not used in the test
+ substitutions_counter = 0
+ for key, placeholder_value in env_var_placeholders.items():
+ if not is_env_set(key):
+ set_env(key, placeholder_value)
+ substitutions_counter += 1
+
+ if substitutions_counter > 0:
+ print(f"[yellow]Set {substitutions_counter} placeholder environment variables[/yellow]")
+
+
[email protected](scope="session", autouse=True)
+def setup_ci_environment():
+ """Set up CI environment variables and configuration before any tests run."""
+ # Check if we're running in CI (GitHub Actions or generic CI environment)
+ if runtime_manager.is_unit_testing:
+ _setup_env_var_placeholders()
+ yield
diff --git a/pipelex/tools/environment.py b/pipelex/tools/environment.py
index 99ff29270..9b31d8bf3 100644
--- a/pipelex/tools/environment.py
+++ b/pipelex/tools/environment.py
@@ -1,10 +1,12 @@
import os
-from typing import Optional
+from typing import Iterable, Optional
from dotenv import load_dotenv
from pipelex.tools.exceptions import ToolException
+ENV_DUMMY_PLACEHOLDER_VALUE = "env-dummy-placeholder"
+
load_dotenv(dotenv_path=".env", override=True)
@@ -24,6 +26,24 @@ def get_optional_env(key: str) -> Optional[str]:
return value
+def is_env_set(key: str | Iterable[str]) -> bool:
+ if isinstance(key, str):
+ return os.getenv(key) is not None
+ for each_key in key:
+ if os.getenv(each_key) is None:
+ return False
+ return True
+
+
+def any_is_placeholder_env(key: str | Iterable[str]) -> bool:
+ if isinstance(key, str):
+ return os.getenv(key) == ENV_DUMMY_PLACEHOLDER_VALUE
+ for each_key in key:
+ if os.getenv(each_key) == ENV_DUMMY_PLACEHOLDER_VALUE:
+ return True
+ return True
+
+
def set_env(key: str, value: str) -> None:
os.environ[key] = value
return None
diff --git a/pyproject.toml b/pyproject.toml
index e46fe8a0c..39cd1c99c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "pipelex"
-version = "0.10.1"
+version = "0.10.2"
description = "Pipelex is an open-source dev tool based on a simple declarative language that lets you define replicable, structured, composable LLM pipelines."
authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]
maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]
diff --git a/tests/cases/documents.py b/tests/cases/documents.py
index 6d38c9bbf..841a1942f 100644
--- a/tests/cases/documents.py
+++ b/tests/cases/documents.py
@@ -10,9 +10,11 @@ class PDFTestCases:
TEST_DOCUMENT_DIRECTORY = "tests/data/documents"
# Local file paths
+ PDF_FILE_PATH_1 = f"{TEST_DOCUMENT_DIRECTORY}/solar_system.pdf"
+ PDF_FILE_PATH_2 = f"{TEST_DOCUMENT_DIRECTORY}/illustrated_train_article.pdf"
DOCUMENT_FILE_PATHS: ClassVar[List[str]] = [
- f"{TEST_DOCUMENT_DIRECTORY}/solar_system.pdf",
- f"{TEST_DOCUMENT_DIRECTORY}/illustrated_train_article.pdf",
+ PDF_FILE_PATH_1,
+ PDF_FILE_PATH_2,
]
# Remote URLs
diff --git a/tests/conftest.py b/tests/conftest.py
index 3ab87df43..15ed2ca5c 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,8 +1,4 @@
-import atexit
-import os
-
import pytest
-from rich import print
from rich.console import Console
from rich.traceback import Traceback
@@ -21,98 +17,6 @@ pytest_plugins = [
TEST_OUTPUTS_DIR = "temp/test_outputs"
-def _setup_ci_env_vars():
- """Set placeholder environment variables when running in CI to prevent import failures.
-
- These placeholders allow the code to import successfully, while actual inference tests
- remain skipped via pytest markers.
- """
- # Check if we're running in CI (GitHub Actions or generic CI environment)
- if os.getenv("CI") or os.getenv("GITHUB_ACTIONS"):
- print("[yellow]CI environment detected - setting placeholder API keys[/yellow]")
-
- # Define placeholder values for all inference-related env vars
- ci_placeholders = {
- "PIPELEX_API_TOKEN": "ci-placeholder-token",
- "PIPELEX_API_BASE_URL": "https://app.pipelex.ai/api/v1",
- "PIPELEX_INFERENCE_API_KEY": "ci-placeholder-inference-key",
- "OPENAI_API_KEY": "sk-ci-placeholder-key",
- "AWS_ACCESS_KEY_ID": "ci-placeholder-aws-key",
- "AWS_SECRET_ACCESS_KEY": "ci-placeholder-aws-secret",
- "AWS_REGION": "us-east-1",
- "AZURE_OPENAI_API_ENDPOINT": "https://ci-placeholder.openai.azure.com",
- "AZURE_OPENAI_API_VERSION": "2025-04-01-preview",
- "AZURE_OPENAI_API_KEY": "ci-placeholder-azure-key",
- "AZURE_API_BASE": "https://ci-placeholder.openai.azure.com",
- "AZURE_API_KEY": "ci-placeholder-azure-api-key",
- "AZURE_API_VERSION": "2025-04-01-preview",
- "ANTHROPIC_API_KEY": "sk-ant-ci-placeholder-key",
- "MISTRAL_API_KEY": "ci-placeholder-mistral-key",
- "PERPLEXITY_API_KEY": "ci-placeholder-perplexity-key",
- "PERPLEXITY_API_ENDPOINT": "https://api.perplexity.ai",
- "FAL_API_KEY": "ci-placeholder-fal-key",
- "GCP_PROJECT_ID": "ci-placeholder-project",
- "GCP_REGION": "us-central1",
- "GCP_LOCATION": "us-central1",
- "GCP_CREDENTIALS_FILE_PATH": "/tmp/ci-placeholder-gcp-credentials.json",
- "XAI_API_KEY": "ci-placeholder-xai-key",
- "XAI_API_ENDPOINT": "https://api.x.ai/v1/",
- "BLACKBOX_API_KEY": "ci-placeholder-blackbox-key",
- "CUSTOM_ENDPOINT_BASE_URL": "http://localhost:11434/v1/",
- "CUSTOM_ENDPOINT_API_KEY": "ci-placeholder-custom-key",
- }
-
- # Set placeholders, overriding any existing values in CI
- # This ensures tests won't accidentally use real API keys in CI
- for key, value in ci_placeholders.items():
- os.environ[key] = value
-
- # Temporarily modify backends.toml to disable problematic backends in CI
- backends_path = ".pipelex/inference/backends.toml"
- backup_path = ".pipelex/inference/backends.toml.backup"
-
- # Create backup if it doesn't exist
- if not os.path.exists(backup_path):
- with open(backends_path, "r") as f:
- original_content = f.read()
- with open(backup_path, "w") as f:
- f.write(original_content)
-
- # Modify the original file to disable vertexai
- with open(backends_path, "r") as f:
- content = f.read()
-
- # Disable vertexai backend for CI
- content = content.replace("[vertexai]\nenabled = true", "[vertexai]\nenabled = false")
-
- with open(backends_path, "w") as f:
- f.write(content)
-
- print(f"[yellow]Set {len(ci_placeholders)} placeholder environment variables[/yellow]")
-
-
-def _restore_ci_backends_file():
- """Restore original backends.toml file if we're in CI and a backup exists."""
- if os.getenv("CI") or os.getenv("GITHUB_ACTIONS"):
- backends_path = ".pipelex/inference/backends.toml"
- backup_path = ".pipelex/inference/backends.toml.backup"
-
- if os.path.exists(backup_path):
- with open(backup_path, "r") as f:
- original_content = f.read()
- with open(backends_path, "w") as f:
- f.write(original_content)
- os.remove(backup_path)
- print("[yellow]Restored original backends.toml file[/yellow]")
-
-
-# Set up CI environment variables before any imports that might need them
-_setup_ci_env_vars()
-
-# Register cleanup function to restore backends file
-atexit.register(_restore_ci_backends_file)
-
-
@pytest.fixture(scope="module", autouse=True)
def reset_pipelex_config_fixture():
# Code to run before each test
diff --git a/tests/integration/pipelex/cogt/test_content_generator.py b/tests/integration/pipelex/cogt/test_content_generator.py
index 1fcb3c305..da3a313bf 100644
--- a/tests/integration/pipelex/cogt/test_content_generator.py
+++ b/tests/integration/pipelex/cogt/test_content_generator.py
@@ -10,13 +10,12 @@ from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.cogt.ocr.ocr_output import OcrOutput
from pipelex.hub import get_content_generator, get_models_manager
from pipelex.pipeline.job_metadata import JobMetadata
-from tests.cases import ImageTestCases
+from tests.cases import ImageTestCases, PDFTestCases
from tests.integration.pipelex.cogt.test_data import Employee
USER_TEXT_FOR_BASE = """
@@ -61,7 +60,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_llm_text_only(self, request: FixtureRequest):
- llm_setting_main = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_text")
+ llm_setting_main = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_text")
text: str = await get_content_generator().make_llm_text(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
@@ -75,7 +74,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_object_direct(self, request: FixtureRequest):
- llm_setting_for_object = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
+ llm_setting_for_object = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
person_direct: Employee = await get_content_generator().make_object_direct(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
@@ -90,7 +89,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_object_list_direct(self, request: FixtureRequest):
- llm_setting_for_object = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
+ llm_setting_for_object = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
person_list_direct: List[Employee] = await get_content_generator().make_object_list_direct(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
@@ -130,10 +129,10 @@ class TestContentGenerator:
@pytest.mark.ocr
@pytest.mark.inference
- async def test_make_ocr_extract_pages(self, request: FixtureRequest):
+ async def test_make_ocr_extract_pages_from_image(self, ocr_handle_from_image: str, request: FixtureRequest):
ocr_output = await get_content_generator().make_ocr_extract_pages(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
- ocr_handle=OcrHandle.MISTRAL_OCR,
+ ocr_handle=ocr_handle_from_image,
ocr_input=OcrInput(image_uri=ImageTestCases.IMAGE_FILE_PATH_PNG),
ocr_job_params=OcrJobParams.make_default_ocr_job_params(),
ocr_job_config=OcrJobConfig(),
@@ -141,6 +140,19 @@ class TestContentGenerator:
pretty_print(ocr_output, title="ocr_extract_pages")
assert isinstance(ocr_output, OcrOutput)
+ @pytest.mark.ocr
+ @pytest.mark.inference
+ async def test_make_ocr_extract_pages_from_pdf(self, ocr_handle: str, request: FixtureRequest):
+ ocr_output = await get_content_generator().make_ocr_extract_pages(
+ job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
+ ocr_handle=ocr_handle,
+ ocr_input=OcrInput(pdf_uri=PDFTestCases.PDF_FILE_PATH_1),
+ ocr_job_params=OcrJobParams.make_default_ocr_job_params(),
+ ocr_job_config=OcrJobConfig(),
+ )
+ pretty_print(ocr_output, title="ocr_extract_pages")
+ assert isinstance(ocr_output, OcrOutput)
+
@pytest.mark.llm
@pytest.mark.inference
async def test_make_llm_text_with_error(self, request: FixtureRequest):
diff --git a/tests/integration/pipelex/cogt/test_llm_gen_object.py b/tests/integration/pipelex/cogt/test_llm_gen_object.py
index ce6f29c8e..d25125b5b 100644
--- a/tests/integration/pipelex/cogt/test_llm_gen_object.py
+++ b/tests/integration/pipelex/cogt/test_llm_gen_object.py
@@ -12,7 +12,7 @@ from tests.integration.pipelex.cogt.test_data import LLMTestCases
def get_async_worker_and_job(llm_preset_id: str, user_text: str):
- llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(user_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
diff --git a/tests/integration/pipelex/cogt/test_llm_gen_text.py b/tests/integration/pipelex/cogt/test_llm_gen_text.py
index 2dfa3fa1a..78dbf5dc4 100644
--- a/tests/integration/pipelex/cogt/test_llm_gen_text.py
+++ b/tests/integration/pipelex/cogt/test_llm_gen_text.py
@@ -15,7 +15,7 @@ from tests.integration.pipelex.cogt.test_data import LLMTestCases
def get_worker_and_job(llm_preset_id: str, user_text: str) -> Tuple[LLMWorkerAbstract, LLMJob]:
- llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(user_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
diff --git a/tests/integration/pipelex/cogt/test_llm_report.py b/tests/integration/pipelex/cogt/test_llm_report.py
index 04d2d62ae..da319b870 100644
--- a/tests/integration/pipelex/cogt/test_llm_report.py
+++ b/tests/integration/pipelex/cogt/test_llm_report.py
@@ -27,7 +27,7 @@ class TestLLMReport:
get_report_delegate().generate_report()
def _get_async_worker_and_job(self, llm_preset_id: str, prompt_text: str):
- llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_model_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(prompt_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
diff --git a/tests/integration/pipelex/cogt/test_ocr.py b/tests/integration/pipelex/cogt/test_ocr.py
index e2150a8d6..22464ff60 100644
--- a/tests/integration/pipelex/cogt/test_ocr.py
+++ b/tests/integration/pipelex/cogt/test_ocr.py
@@ -1,7 +1,6 @@
import pytest
from pipelex import pretty_print
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobParams
from pipelex.cogt.ocr.ocr_job_factory import OcrJobFactory
@@ -16,8 +15,8 @@ from tests.cases import ImageTestCases, PDFTestCases
@pytest.mark.asyncio(loop_scope="class")
class TestOcr:
@pytest.mark.parametrize("file_path", PDFTestCases.DOCUMENT_FILE_PATHS)
- async def test_ocr_pdf_path(self, file_path: str):
- ocr_worker = get_ocr_worker(ocr_handle=OcrHandle.MISTRAL_OCR)
+ async def test_ocr_pdf_path(self, ocr_handle: str, file_path: str):
+ ocr_worker = get_ocr_worker(ocr_handle=ocr_handle)
ocr_job = OcrJobFactory.make_ocr_job(
ocr_input=OcrInput(pdf_uri=file_path),
)
@@ -27,8 +26,8 @@ class TestOcr:
assert ocr_output.pages
@pytest.mark.parametrize("url", PDFTestCases.DOCUMENT_URLS)
- async def test_ocr_pdf_url(self, url: str):
- ocr_worker = get_ocr_worker(ocr_handle=OcrHandle.MISTRAL_OCR)
+ async def test_ocr_pdf_url(self, ocr_handle: str, url: str):
+ ocr_worker = get_ocr_worker(ocr_handle=ocr_handle)
ocr_job = OcrJobFactory.make_ocr_job(
ocr_input=OcrInput(pdf_uri=url),
)
@@ -37,8 +36,8 @@ class TestOcr:
assert ocr_output.pages
@pytest.mark.parametrize("file_path", ImageTestCases.IMAGE_FILE_PATHS)
- async def test_ocr_image_file(self, file_path: str):
- ocr_worker = get_ocr_worker(ocr_handle=OcrHandle.MISTRAL_OCR)
+ async def test_ocr_image_file(self, ocr_handle_from_image: str, file_path: str):
+ ocr_worker = get_ocr_worker(ocr_handle=ocr_handle_from_image)
ocr_job = OcrJobFactory.make_ocr_job(
ocr_input=OcrInput(image_uri=file_path),
)
@@ -47,8 +46,8 @@ class TestOcr:
assert ocr_output.pages
@pytest.mark.parametrize("url", ImageTestCases.IMAGE_URLS)
- async def test_ocr_image_url(self, url: str):
- ocr_worker = get_ocr_worker(ocr_handle=OcrHandle.MISTRAL_OCR)
+ async def test_ocr_image_url(self, ocr_handle_from_image: str, url: str):
+ ocr_worker = get_ocr_worker(ocr_handle=ocr_handle_from_image)
ocr_job = OcrJobFactory.make_ocr_job(
ocr_input=OcrInput(image_uri=url),
)
@@ -57,8 +56,8 @@ class TestOcr:
assert ocr_output.pages
@pytest.mark.parametrize("file_path", PDFTestCases.DOCUMENT_FILE_PATHS)
- async def test_ocr_image_save(self, file_path: str):
- ocr_worker = get_ocr_worker(ocr_handle=OcrHandle.MISTRAL_OCR)
+ async def test_ocr_image_save(self, ocr_handle_from_image: str, file_path: str):
+ ocr_worker = get_ocr_worker(ocr_handle=ocr_handle_from_image)
ocr_job_params = OcrJobParams(
should_include_images=True,
should_caption_images=False,
diff --git a/tests/integration/pipelex/cogt/test_setup_inference_workers.py b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
index 92fa56e71..54cb856b3 100644
--- a/tests/integration/pipelex/cogt/test_setup_inference_workers.py
+++ b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
@@ -2,20 +2,33 @@ import pytest
from pipelex import log
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.config import get_config
from pipelex.hub import get_inference_manager, get_models_manager
@pytest.mark.gha_disabled
@pytest.mark.codex_disabled
class TestSetupInferenceWorkers:
- def test_setup_inference_manager(self):
+ def test_setup_inference_workers(self):
inference_manager = get_inference_manager()
- llm_handle_to_inference_model = get_models_manager().get_llm_deck().inference_models
- log.verbose(f"{len(llm_handle_to_inference_model)} LLM engine_cards found")
- for llm_handle, inference_model in llm_handle_to_inference_model.items():
- llm_worker = inference_manager.get_llm_worker(llm_handle=llm_handle)
- if isinstance(llm_worker, LLMWorkerInternalAbstract):
- assert inference_model == llm_worker.inference_model
- log.debug("Done setting up LLM Workers (async)")
+ inference_models = get_models_manager().get_model_deck().inference_models
+ log.verbose(f"{len(inference_models)} LLM engine_cards found")
+ for model_handle, inference_model in inference_models.items():
+ match inference_model.model_type:
+ case ModelType.LLM:
+ llm_worker = inference_manager.get_llm_worker(llm_handle=model_handle)
+ if isinstance(llm_worker, LLMWorkerInternalAbstract):
+ assert inference_model == llm_worker.inference_model
+ case ModelType.TEXT_EXTRACTOR:
+ _ = inference_manager.get_ocr_worker(model_handle=model_handle)
- get_inference_manager().setup_imgg_workers()
+ def test_setup_imgg_workers(self):
+ inference_manager = get_inference_manager()
+ imgg_handles = get_config().cogt.imgg_config.imgg_handles
+ log.verbose(f"{len(imgg_handles)} Imgg handles found")
+ for imgg_handle in imgg_handles:
+ imgg_worker = inference_manager.get_imgg_worker(imgg_handle=imgg_handle)
+ assert imgg_worker is not None
+ assert imgg_worker.imgg_engine is not None
+ log.debug("Done setting up Imgg Workers (async)")
diff --git a/tests/integration/pipelex/conftest.py b/tests/integration/pipelex/conftest.py
index 62c76226e..0bda0d816 100644
--- a/tests/integration/pipelex/conftest.py
+++ b/tests/integration/pipelex/conftest.py
@@ -4,7 +4,6 @@ import pytest
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.llm.llm_job_components import LLMJobParams
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.plugins.plugin_sdk_registry import Plugin
@@ -221,10 +220,20 @@ def imgg_handle(request: pytest.FixtureRequest) -> ImggHandle:
@pytest.fixture(
params=[
- OcrHandle.BASIC_OCR,
- OcrHandle.MISTRAL_OCR,
+ "pypdfium2-extract-text",
+ "mistral-ocr",
]
)
-def ocr_handle(request: pytest.FixtureRequest) -> OcrHandle:
- assert isinstance(request.param, OcrHandle)
+def ocr_handle(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
+
+
+ params=[
+ "mistral-ocr",
+ ]
+)
+def ocr_handle_from_image(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
return request.param
diff --git a/tests/integration/pipelex/conftest.py.orig b/tests/integration/pipelex/conftest.py.orig
deleted file mode 100644
index c0a4a4a55..000000000
--- a/tests/integration/pipelex/conftest.py.orig
+++ /dev/null
@@ -1,235 +0,0 @@
-from typing import Optional
-
-import pytest
-
-from pipelex.cogt.imgg.imgg_handle import ImggHandle
-from pipelex.cogt.llm.llm_job_components import LLMJobParams
-<<<<<<< HEAD
-from pipelex.plugins.plugin_sdk_registry import Plugin
-=======
-from pipelex.cogt.llm.llm_models.llm_family import LLMCreator, LLMFamily
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
->>>>>>> dev
-
-
- params=[
- # "llm_for_testing_gen_text",
- # "llm_for_testing_gen_object",
- "llm_for_creative_writing",
- ]
-)
-def llm_preset_id(request: pytest.FixtureRequest) -> str:
- assert isinstance(request.param, str)
- return request.param
-
-
-# TODO: make it efficient to also test multiple platforms like openai/azure and mistral/anthropic/bedrock
- params=[
- # "o1",
- # "gpt-4o",
- # "gpt-4o-mini",
- # "gpt-4-5-preview",
- # "o1-mini",
- # "o3-mini",
- # "gpt-5-mini", # TODO: fix this
- # "gpt-5-nano", # TODO: fix this
- # "best-gpt",
- # "claude-3-haiku",
- # "claude-3-5-sonnet",
- # "claude-3-7-sonnet",
- # "best-claude",
- # "mistral-large",
- # "ministral-3b",
- # "ministral-8b",
- "mistral-medium",
- "mistral-medium-2508",
- # "pixtral-12b",
- # "pixtral-large",
- # "gemini-1-5-pro",
- # "gemini-1-5-flash",
- # "gemini-2-flash",
- # "gemini-2-pro",
- # "gemini-2-5-flash",
- # "gemini-2-5-pro",
- # "bedrock-mistral-large",
- # "bedrock-claude-3-7-sonnet",
- # "bedrock-meta-llama-3-3-70b-instruct",
- # "bedrock-nova-pro",
- # "sonar",
- # "pipelex/gpt-4o-mini",
- # "pipelex/claude-3.7-sonnet",
- # "pipelex/gemini-2.0-flash-vertex",
- # "pipelex/gemini-2.0-flash",
- # "best-claude",
- # "llm_to_engineer",
- # "gpt-5-nano",
- # "gpt-4o-mini",
- # "gpt-5-mini",
- # "gpt-5-chat",
- # "claude-4-sonnet",
- # "claude-4.1-opus",
- # "claude-3.5-sonnet",
- # "claude-3.5-sonnet-v2"
- # "claude-3.7-sonnet",
- # "gemini-2.5-flash-lite",
- # "gemini-2.5-flash",
- # "gemini-2.5-pro",
- # "grok-3",
- # "grok-3-mini",
- ]
-)
-def llm_handle(request: pytest.FixtureRequest) -> str:
- assert isinstance(request.param, str)
- return request.param
-
-
- params=[
- # "o1",
- # "o3-mini",
- # "gpt-4o",
- # "gpt-4o-mini",
- # "gpt-5-mini",
- # "gpt-5-nano",
- # "gpt-5-chat",
- # "gpt-4-5-preview",
- # "claude-3-haiku",
- # "claude-3.5-sonnet",
- # "claude-3.7-sonnet",
- # "claude-4.1-opus",
- # "pixtral-12b",
- # "pixtral-large",
- # "gemini-2.5-pro",
- # "gemini-2.5-flash",
- # "mistral-small3.1",
- "mistral-medium",
- "mistral-medium-2508",
- # "qwen3:8b",
- ]
-)
-def llm_handle_for_vision(request: pytest.FixtureRequest) -> str:
- assert isinstance(request.param, str)
- return request.param
-
-
- params=[
- Plugin(sdk="openai", backend="openai"),
- Plugin(sdk="azure_openai", backend="azure_openai"),
- ]
-)
-def plugin_for_openai(request: pytest.FixtureRequest) -> Plugin:
- assert isinstance(request.param, Plugin)
- return request.param
-
-
- params=[
- Plugin(sdk="anthropic", backend="anthropic"),
- Plugin(sdk="bedrock_anthropic", backend="bedrock_anthropic"),
- ]
-)
-def plugin_for_anthropic(request: pytest.FixtureRequest) -> Plugin:
- assert isinstance(request.param, Plugin)
- return request.param
-
-
- params=[
- # None,
- "https://inference.pipelex.com/v1",
- ]
-)
-def openai_endpoint(request: pytest.FixtureRequest) -> Optional[str]:
- assert isinstance(request.param, str) or request.param is None
- return request.param
-
-
- params=[
- "gpt-5-mini-2025-08-07",
- # "gpt-5-nano-2025-08-07",
- # "gpt-5-chat-2025-08-07",
- # "gpt-5-mini",
- # "gpt-5-nano",
- # "gpt-5-chat-latest",
- # "gpt-5",
- "gpt-4o-mini",
- # "open-mixtral-8x7b",
- # "google/gemini-2.0-flash",
- # "google/gemini-2.5-pro-preview-05-06",
- # "google/gemini-2.5-pro-preview-06-05", # not yet on VertexAI
- # "google/gemini-2.5-flash-preview-04-17",
- # "google/gemini-2.5-flash-preview-05-20",
- # "o1",
- # "o4-mini",
- # "bedrock-mistral-large",
- # "sonar",
- # "claude-3-7-sonnet-20250219",
- # "claude-sonnet-4-20250514",
- # "claude-opus-4-20250514",
- # "claude-opus-4-1-20250805",
- # "us.anthropic.claude-sonnet-4-20250514-v1:0",
- # "us.anthropic.claude-opus-4-20250514-v1:0",
- # "us.anthropic.claude-opus-4-1-20250805-v1:0",
- # "sonar",
- # "sonar-pro",
- # "gemma3:4b",
- # "llama4:scout",
- # "mistral-small3.1:24b",
- # "qwen3:8b",
- # "blackboxai/openai/gpt-4o-mini",
- # "pipelex/openai/gpt-4o-mini",
- # "openai/gpt-4o-mini",
- # "grok-3",
- # "grok-3-mini",
- # "pipelex/gpt-4o-mini",
- # "pipelex/claude-3.7-sonnet",
- # "vertex_ai/gemini-2.0-flash",
- ]
-)
-def llm_id(request: pytest.FixtureRequest) -> str:
- assert isinstance(request.param, str)
- return request.param
-
-
- params=[
- LLMJobParams(
- temperature=0.5,
- max_tokens=None,
- seed=None,
- ),
- ]
-)
-def llm_job_params(request: pytest.FixtureRequest) -> LLMJobParams:
- assert isinstance(request.param, LLMJobParams)
- return request.param
-
-
- params=[
- # ImggHandle.FLUX_1_PRO_LEGACY,
- # ImggHandle.FLUX_1_1_PRO,
- # ImggHandle.FLUX_1_1_ULTRA,
- ImggHandle.SDXL_LIGHTNING,
- # ImggHandle.OPENAI_GPT_IMAGE_1,
- ]
-)
-def imgg_handle(request: pytest.FixtureRequest) -> ImggHandle:
- assert isinstance(request.param, ImggHandle)
- return request.param
-
-
- params=[
- OcrHandle.BASIC_OCR,
- # OcrHandle.MISTRAL_OCR,
- ]
-)
-def ocr_handle(request: pytest.FixtureRequest) -> OcrHandle:
- assert isinstance(request.param, OcrHandle)
- return request.param
diff --git a/tests/integration/pipelex/pipes/operator/pipe_ocr/test_pipe_ocr.py b/tests/integration/pipelex/pipes/operator/pipe_ocr/test_pipe_ocr.py
index 64bbf79fe..424af565f 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_ocr/test_pipe_ocr.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_ocr/test_pipe_ocr.py
@@ -3,7 +3,6 @@ from typing import Any
import pytest
from pipelex import pretty_print
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptEnum
@@ -43,14 +42,11 @@ class TestPipeOCR:
@pytest.mark.parametrize("image_url", PipeOcrTestCases.PIPE_OCR_IMAGE_TEST_CASES)
async def test_pipe_ocr_image(
self,
- ocr_handle: OcrHandle,
+ ocr_handle_from_image: str,
pipe_run_mode: PipeRunMode,
image_url: str,
setup: Any,
):
- if ocr_handle == OcrHandle.BASIC_OCR:
- pytest.skip("Basic OCR is not supported for image processing")
-
pipe_ocr_blueprint = PipeOcrBlueprint(
definition="OCR test for image processing",
inputs={"page_scan": InputRequirementBlueprint(concept=NativeConceptEnum.IMAGE.value)},
@@ -59,7 +55,7 @@ class TestPipeOCR:
page_image_captions=False,
page_views=True,
page_views_dpi=72,
- ocr_handle=ocr_handle,
+ ocr_model=ocr_handle_from_image,
)
pipe_job = PipeJobFactory.make_pipe_job(
@@ -83,7 +79,7 @@ class TestPipeOCR:
@pytest.mark.parametrize("pdf_url", PipeOcrTestCases.PIPE_OCR_PDF_TEST_CASES)
async def test_pipe_ocr_pdf(
self,
- ocr_handle: OcrHandle,
+ ocr_handle: str,
pipe_run_mode: PipeRunMode,
pdf_url: str,
):
@@ -91,7 +87,7 @@ class TestPipeOCR:
definition="OCR test for PDF processing",
inputs={PIPE_OCR_INPUT_NAME: InputRequirementBlueprint(concept=NativeConceptEnum.PDF.value)},
output=NativeConceptEnum.TEXT_AND_IMAGES.value,
- ocr_handle=ocr_handle,
+ ocr_model=ocr_handle,
page_images=True,
page_image_captions=False,
page_views=True,
diff --git a/tests/integration/pipelex/plugins/test_anthropic.py b/tests/integration/pipelex/plugins/test_anthropic.py
index fa1c4d979..d66c51094 100644
--- a/tests/integration/pipelex/plugins/test_anthropic.py
+++ b/tests/integration/pipelex/plugins/test_anthropic.py
@@ -8,6 +8,9 @@ from pipelex.hub import get_models_manager
from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedError
from pipelex.plugins.anthropic.anthropic_llms import anthropic_list_anthropic_models
from pipelex.plugins.plugin_sdk_registry import Plugin
+from pipelex.tools.environment import any_is_placeholder_env, is_env_set
+
+REQUIRED_ENV_VARS = ["ANTHROPIC_API_KEY"]
# TODO: fix this: test works for Anthropic but not if you set peferred platform for Anthropic is Bedrock
@@ -23,6 +26,10 @@ class TestAnthropic:
pytestconfig: pytest.Config,
plugin_for_anthropic: Plugin,
):
+ if not is_env_set(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")
+ if any_is_placeholder_env(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")
try:
backend = get_models_manager().get_required_inference_backend("anthropic")
anthropic_models_list = await anthropic_list_anthropic_models(
diff --git a/tests/integration/pipelex/plugins/test_bedrock.py b/tests/integration/pipelex/plugins/test_bedrock.py
index 0b4d10d42..2d14f7d9a 100644
--- a/tests/integration/pipelex/plugins/test_bedrock.py
+++ b/tests/integration/pipelex/plugins/test_bedrock.py
@@ -7,6 +7,8 @@ from rich import box
from rich.console import Console
from rich.table import Table
+from pipelex.tools.environment import any_is_placeholder_env, is_env_set
+
warnings.filterwarnings(
"ignore",
message=r".*datetime\.datetime\.utcnow\(\).*",
@@ -15,6 +17,8 @@ warnings.filterwarnings(
# Apply pytest-level filter to ensure the warning is suppressed during test collection and execution
pytestmark = pytest.mark.filterwarnings("ignore:.*datetime\\.datetime\\.utcnow\\(\\).*:DeprecationWarning")
+REQUIRED_ENV_VARS = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_REGION"]
+
# make t VERBOSE=2 TEST=TestBedrock
@pytest.mark.gha_disabled
@@ -27,6 +31,10 @@ class TestBedrock:
bedrock_provider: str,
bedrock_region_name: str,
):
+ if not is_env_set(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")
+ if any_is_placeholder_env(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")
client = boto3.client("bedrock", region_name=bedrock_region_name) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
response: Dict[str, Any] = client.list_foundation_models(byProvider=bedrock_provider) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
bedrock_models_list: List[Dict[str, Any]] = response["modelSummaries"] # pyright: ignore[reportUnknownVariableType]
@@ -59,6 +67,10 @@ class TestBedrock:
pytestconfig: pytest.Config,
bedrock_region_name: str,
):
+ if not is_env_set(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")
+ if any_is_placeholder_env(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")
client = boto3.client("bedrock", region_name=bedrock_region_name) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
response: Dict[str, Any] = client.list_inference_profiles() # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
inference_profiles_list: List[Dict[str, Any]] = response["inferenceProfileSummaries"] # pyright: ignore[reportUnknownVariableType]
diff --git a/tests/integration/pipelex/plugins/test_mistral.py b/tests/integration/pipelex/plugins/test_mistral.py
index b9af1c796..829cd9b1d 100644
--- a/tests/integration/pipelex/plugins/test_mistral.py
+++ b/tests/integration/pipelex/plugins/test_mistral.py
@@ -5,6 +5,9 @@ from rich.table import Table
from pipelex import pretty_print
from pipelex.plugins.mistral.mistral_llms import list_mistral_models
+from pipelex.tools.environment import any_is_placeholder_env, is_env_set
+
+REQUIRED_ENV_VARS = ["MISTRAL_API_KEY"]
# make t VERBOSE=2 TEST=TestMistral
@@ -14,6 +17,10 @@ class TestMistral:
# pytest -k test_mistral_list_models -s -vv
# make t VERBOSE=2 TEST=test_mistral_list_models
def test_mistral_list_models(self, pytestconfig: pytest.Config):
+ if not is_env_set(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")
+ if any_is_placeholder_env(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")
mistral_models_list = list_mistral_models()
if pytestconfig.get_verbosity() >= 2:
# Create and configure the table
@@ -42,6 +49,10 @@ class TestMistral:
# pytest -k test_mistral_list_model_ids -s -vv
# make t VERBOSE=2 TEST=test_mistral_list_model_ids
def test_mistral_list_model_ids(self, pytestconfig: pytest.Config):
+ if not is_env_set(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")
+ if any_is_placeholder_env(REQUIRED_ENV_VARS):
+ pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")
mistral_models_list = list_mistral_models()
mistral_model_ids = [{"id": model.id, "aliases": model.aliases} for model in mistral_models_list]
if pytestconfig.get_verbosity() >= 2:
diff --git a/tests/integration/pipelex/plugins/test_openai.py b/tests/integration/pipelex/plugins/test_openai.py
index 8d01566c1..f76593d02 100644
--- a/tests/integration/pipelex/plugins/test_openai.py
+++ b/tests/integration/pipelex/plugins/test_openai.py
@@ -4,6 +4,7 @@ from pipelex import pretty_print
from pipelex.hub import get_models_manager
from pipelex.plugins.openai.openai_llms import openai_list_available_models
from pipelex.plugins.plugin_sdk_registry import Plugin
+from pipelex.tools.environment import any_is_placeholder_env, is_env_set
# make t VERBOSE=2 TEST=TestOpenAI
@@ -17,6 +18,17 @@ class TestOpenAI:
pytestconfig: pytest.Config,
plugin_for_openai: Plugin,
):
+ match plugin_for_openai.backend:
+ case "openai":
+ required_env_vars = ["OPENAI_API_KEY"]
+ case "azure_openai":
+ required_env_vars = ["AZURE_API_KEY", "AZURE_API_BASE", "AZURE_API_VERSION"]
+ case _:
+ raise ValueError(f"Plugin {plugin_for_openai} is not supported in this test")
+ if not is_env_set(required_env_vars):
+ pytest.skip(f"Some key(s) missing amongst {required_env_vars}")
+ if any_is_placeholder_env(required_env_vars):
+ pytest.skip(f"Some key(s) among {required_env_vars} are a placeholder, can't be used to test listing models")
backend = get_models_manager().get_required_inference_backend(plugin_for_openai.backend)
openai_models_list = await openai_list_available_models(
plugin=plugin_for_openai,
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck.py b/tests/unit/pipelex/cogt/models/test_model_deck.py
index ee199cbef..2ff662841 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck.py
@@ -7,6 +7,7 @@ from pytest_mock import MockerFixture
from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaults
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import ModelDeck
from pipelex.cogt.usage.cost_category import CostCategory
@@ -20,6 +21,7 @@ class TestModelDeckGetOptionalInferenceModel:
backend_name="test_backend",
name=name,
sdk="test_sdk",
+ model_type=ModelType.LLM,
model_id=f"test_model_{name}",
costs={CostCategory.INPUT: 0.001, CostCategory.OUTPUT: 0.002},
max_tokens=1000,
@@ -65,7 +67,7 @@ class TestModelDeckGetOptionalInferenceModel:
# Assert
assert result is None
- mock_log.warning.assert_called_once_with("Skipping LLM handle 'nonexistent-model' because it's not found in deck")
+ mock_log.warning.assert_called_once_with("Skipping model handle 'nonexistent-model' because it's not found in deck")
def test_simple_string_alias_resolution_success(self, mocker: MockerFixture):
"""Test successful resolution of a simple string alias."""
@@ -94,7 +96,7 @@ class TestModelDeckGetOptionalInferenceModel:
assert result is None
mock_log.debug.assert_called_once_with("Redirection for 'best-gpt': nonexistent-model")
# The final warning is about the original alias, not the target
- mock_log.warning.assert_called_with("Skipping LLM handle 'best-gpt' because it's not found in deck")
+ mock_log.warning.assert_called_with("Skipping model handle 'best-gpt' because it's not found in deck")
def test_list_alias_resolution_first_success(self, mocker: MockerFixture):
"""Test list alias resolution where first model in list exists."""
@@ -186,7 +188,7 @@ class TestModelDeckGetOptionalInferenceModel:
assert result is None
# Empty list evaluates to False, so no debug log is called
mock_log.debug.assert_not_called()
- mock_log.warning.assert_called_once_with("Skipping LLM handle 'empty-alias' because it's not found in deck")
+ mock_log.warning.assert_called_once_with("Skipping model handle 'empty-alias' because it's not found in deck")
def test_circular_alias_prevention(self, mocker: MockerFixture):
"""Test that circular aliases don't cause infinite loops."""
@@ -267,7 +269,7 @@ class TestModelDeckGetOptionalInferenceModel:
# Assert
assert result is None
- mock_log.warning.assert_called_once_with(f"Skipping LLM handle '{llm_handle}' because it's not found in deck")
+ mock_log.warning.assert_called_once_with(f"Skipping model handle '{llm_handle}' because it's not found in deck")
def test_logging_behavior(self, mocker: MockerFixture):
"""Test that logging is called with correct messages."""
@@ -283,4 +285,4 @@ class TestModelDeckGetOptionalInferenceModel:
mock_log.debug.assert_called_with("Redirection for 'test-alias': target-model")
# The final warning is about the original alias, not the target
- mock_log.warning.assert_called_with("Skipping LLM handle 'test-alias' because it's not found in deck")
+ mock_log.warning.assert_called_with("Skipping model handle 'test-alias' because it's not found in deck")
diff --git a/tests/unit/pipelex/core/interpreter/test_interpreter_serialize.py b/tests/unit/pipelex/core/interpreter/test_interpreter_serialize.py
index edadd04b5..e41aee383 100644
--- a/tests/unit/pipelex/core/interpreter/test_interpreter_serialize.py
+++ b/tests/unit/pipelex/core/interpreter/test_interpreter_serialize.py
@@ -2,7 +2,6 @@
from typing import Any, Dict
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint, ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
from pipelex.core.interpreter import PipelexInterpreter
from pipelex.core.pipes.pipe_input_spec_blueprint import InputRequirementBlueprint
@@ -288,10 +287,10 @@ class TestSerializePipeBlueprints:
def test_serialize_ocr_pipe_basic(self):
"""Test serializing basic PipeOcr blueprint."""
- pipe = PipeOcrBlueprint(type="PipeOcr", definition="Extract text from PDF", output="Page", ocr_handle=OcrHandle.BASIC_OCR)
+ pipe = PipeOcrBlueprint(type="PipeOcr", definition="Extract text from PDF", output="Page", ocr_model="pypdfium2-extract-text")
result = PipelexInterpreter.serialize_ocr_pipe(pipe, "test_domain")
- expected = {"type": "PipeOcr", "definition": "Extract text from PDF", "output": "Page", "ocr_handle": "basic/pypdfium2"}
+ expected = {"type": "PipeOcr", "definition": "Extract text from PDF", "output": "Page", "ocr_model": "pypdfium2-extract-text"}
assert result == expected
def test_serialize_sequence_pipe_basic(self):
diff --git a/tests/unit/pipelex/core/interpreter/test_pipelex_interpreter_ocr_plx.py b/tests/unit/pipelex/core/interpreter/test_pipelex_interpreter_ocr_plx.py
index 2a558c7ee..4c46c5d46 100644
--- a/tests/unit/pipelex/core/interpreter/test_pipelex_interpreter_ocr_plx.py
+++ b/tests/unit/pipelex/core/interpreter/test_pipelex_interpreter_ocr_plx.py
@@ -1,6 +1,5 @@
import pytest
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.core.interpreter import PipelexInterpreter
from pipelex.pipe_operators.ocr.pipe_ocr_blueprint import PipeOcrBlueprint
@@ -18,13 +17,13 @@ class TestPipelexInterpreterOcrPLX:
type="PipeOcr",
definition="Extract text from document",
output="Page",
- ocr_handle=OcrHandle.BASIC_OCR,
+ ocr_model="pypdfium2-extract-text",
),
"""[pipe.extract_text]
type = "PipeOcr"
definition = "Extract text from document"
output = "Page"
-ocr_handle = "basic/pypdfium2\"""",
+ocr_model = "pypdfium2-extract-text\"""",
),
# OCR pipe with inputs
(
@@ -34,14 +33,14 @@ ocr_handle = "basic/pypdfium2\"""",
definition="Extract text from PDF",
inputs={"ocr_input": "PDF"},
output="Page",
- ocr_handle=OcrHandle.BASIC_OCR,
+ ocr_model="pypdfium2-extract-text",
),
"""[pipe.extract_with_input]
type = "PipeOcr"
definition = "Extract text from PDF"
inputs = { ocr_input = "PDF" }
output = "Page"
-ocr_handle = "basic/pypdfium2\"""",
+ocr_model = "pypdfium2-extract-text\"""",
),
],
)
diff --git a/tests/unit/pipelex/core/test_data/pipes/operators/ocr/pipe_ocr.py b/tests/unit/pipelex/core/test_data/pipes/operators/ocr/pipe_ocr.py
index e4e252c37..810e2f5f7 100644
--- a/tests/unit/pipelex/core/test_data/pipes/operators/ocr/pipe_ocr.py
+++ b/tests/unit/pipelex/core/test_data/pipes/operators/ocr/pipe_ocr.py
@@ -1,6 +1,5 @@
"""PipeOcr test cases."""
-from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.concepts.concept_native import NativeConceptEnum
from pipelex.pipe_operators.ocr.pipe_ocr_blueprint import PipeOcrBlueprint
@@ -14,7 +13,7 @@ definition = "Domain with OCR pipe"
type = "PipeOcr"
definition = "Extract text from document"
output = "Page"
-ocr_handle = "basic/pypdfium2"
+ocr_model = "pypdfium2-extract-text"
""",
PipelexBundleBlueprint(
domain="test_pipes",
@@ -24,7 +23,7 @@ ocr_handle = "basic/pypdfium2"
type="PipeOcr",
definition="Extract text from document",
output=NativeConceptEnum.PAGE.value,
- ocr_handle=OcrHandle.BASIC_OCR,
+ ocr_model="pypdfium2-extract-text",
),
},
),