Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Feature/epic inference providers (#269)

Louis Choquel944bce8d8673
diff --git a/.cursor/rules/llms.mdc b/.cursor/rules/llms.mdc
index e814eb827..7f1d75f7b 100644
--- a/.cursor/rules/llms.mdc
+++ b/.cursor/rules/llms.mdc
@@ -4,32 +4,30 @@ alwaysApply: false
---
# Rules to choose LLM models used in PipeLLMs.
-## LLM Handles
+## LLM Configuration System
-In order to use it in a pipe, an LLM is referenced by its llm_handle and possibly by an llm_preset.
-Both llm_handles and llm_presets are defined in this toml config file: [base_llm_deck.toml](mdc:pipelex/libraries/llm_deck/base_llm_deck.toml)
+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-## LLM Handles
+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
+- **Routing**: `.pipelex/inference/routing_profiles.toml`
-An llm_handle matches the handle (an id of sorts) with the full specification of the LLM to use, i.e.:
-- llm_name
-- llm_version
-- llm_platform_choice
+## LLM Handles
-The declaration of llm_handles looks like this in toml syntax:
-```toml
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
-```
+An llm_handle can be either:
+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-In mosty cases, we only want to use version "latest" and llm_platform_choice "default" in which case the declaration is simply a match of the llm_handle to the llm_name, like this:
```toml
-best-claude = "claude-4-opus"
+[aliases]
+best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
best-mistral = "mistral-large"
+base-gpt = "gpt-5"
```
-And of course, llm_handles are automatically assigned for all models by their name, with version "latest" and llm_platform_choice "default".
+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
## Using an LLM Handle in a PipeLLM
@@ -80,4 +78,4 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_
You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).
+You can override the predefined llm presets in `.pipelex/inference/deck/overrides.toml`.
diff --git a/.cursor/rules/pipelex.mdc b/.cursor/rules/pipelex.mdc
index a706db3b9..3020740a7 100644
--- a/.cursor/rules/pipelex.mdc
+++ b/.cursor/rules/pipelex.mdc
@@ -1,5 +1,5 @@
---
-alwaysApply: true
+alwaysApply: false
---
# Pipeline Guide
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
index 33452fcbd..f7e79caf1 100644
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -660,32 +660,30 @@ But don't write documentation unless asked explicitly to.
# Rules to choose LLM models used in PipeLLMs.
-## LLM Handles
+## LLM Configuration System
-In order to use it in a pipe, an LLM is referenced by its llm_handle and possibly by an llm_preset.
-Both llm_handles and llm_presets are defined in this toml config file: [base_llm_deck.toml](mdc:pipelex/libraries/llm_deck/base_llm_deck.toml)
+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-## LLM Handles
+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
+- **Routing**: `.pipelex/inference/routing_profiles.toml`
-An llm_handle matches the handle (an id of sorts) with the full specification of the LLM to use, i.e.:
-- llm_name
-- llm_version
-- llm_platform_choice
+## LLM Handles
-The declaration of llm_handles looks like this in toml syntax:
-```toml
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
-```
+An llm_handle can be either:
+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-In mosty cases, we only want to use version "latest" and llm_platform_choice "default" in which case the declaration is simply a match of the llm_handle to the llm_name, like this:
```toml
-best-claude = "claude-4-opus"
+[aliases]
+best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
best-mistral = "mistral-large"
+base-gpt = "gpt-5"
```
-And of course, llm_handles are automatically assigned for all models by their name, with version "latest" and llm_platform_choice "default".
+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
## Using an LLM Handle in a PipeLLM
@@ -736,7 +734,7 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_
You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).
+You can override the predefined llm presets in `.pipelex/inference/deck/overrides.toml`.
These rules apply when writing unit tests.
- Always use pytest
diff --git a/.pipelex/inference/backends.toml b/.pipelex/inference/backends.toml
new file mode 100644
index 000000000..ebf1509b5
--- /dev/null
+++ b/.pipelex/inference/backends.toml
@@ -0,0 +1,47 @@
+[pipelex_inference]
+endpoint = "https://inference.pipelex.com/v1"
+api_key = "${PIPELEX_INFERENCE_API_KEY}"
+
+[blackboxai]
+enabled = true
+endpoint = "https://api.blackbox.ai/v1"
+api_key = "${BLACKBOX_API_KEY}"
+
+[openai]
+enabled = true
+api_key = "${OPENAI_API_KEY}"
+
+[azure_openai]
+enabled = true
+endpoint = "${AZURE_API_BASE}"
+api_key = "${AZURE_API_KEY}"
+api_version = "${AZURE_API_VERSION}"
+
+[anthropic]
+enabled = true
+api_key = "${ANTHROPIC_API_KEY}"
+claude_4_tokens_limit = 8192
+
+[ollama]
+enabled = true
+endpoint = "http://localhost:11434/v1"
+
+[xai]
+enabled = true
+endpoint = "https://api.x.ai/v1"
+api_key = "${XAI_API_KEY}"
+
+[bedrock]
+enabled = true
+aws_region = "${AWS_REGION}"
+
+[vertexai]
+enabled = true
+gcp_project_id = "${GCP_PROJECT_ID}"
+gcp_location = "${GCP_LOCATION}"
+gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
+
+[mistral]
+enabled = true
+api_key = "${MISTRAL_API_KEY}"
+
diff --git a/.pipelex/inference/backends/anthropic.toml b/.pipelex/inference/backends/anthropic.toml
new file mode 100644
index 000000000..0dc85517c
--- /dev/null
+++ b/.pipelex/inference/backends/anthropic.toml
@@ -0,0 +1,67 @@
+default_sdk = "anthropic"
+default_prompting_target = "anthropic"
+
+[claude-3-haiku]
+model_id = "claude-3-haiku-20240307"
+max_tokens = 4096
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 0.25, output = 1.25 }
+
+[claude-3-opus]
+model_id = "claude-3-opus-20240229"
+max_tokens = 4096
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 15.0, output = 75.0 }
+
+["claude-3.5-sonnet"]
+model_id = "claude-3-5-sonnet-20240620"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.5-sonnet-v2"]
+model_id = "claude-3-5-sonnet-20241022"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.7-sonnet"]
+model_id = "claude-3-7-sonnet-20250219"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-sonnet]
+model_id = "claude-sonnet-4-20250514"
+max_tokens = 64000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-opus]
+model_id = "claude-opus-4-20250514"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-4.1-opus"]
+model_id = "claude-opus-4-1-20250805"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
diff --git a/.pipelex/inference/backends/azure_openai.toml b/.pipelex/inference/backends/azure_openai.toml
new file mode 100644
index 000000000..b26178854
--- /dev/null
+++ b/.pipelex/inference/backends/azure_openai.toml
@@ -0,0 +1,69 @@
+default_sdk = "azure_openai"
+default_prompting_target = "openai"
+
+[gpt-4o]
+model_id = "gpt-4o-2024-11-20"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o-mini]
+model_id = "gpt-4o-mini-2024-07-18"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+["gpt-4.1"]
+model_id = "gpt-4.1-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "gpt-4.1-mini-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "gpt-4.1-nano-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[o1-mini]
+model_id = "o1-mini-2024-09-12"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 3.0, output = 12.0 }
+
+[o1]
+model_id = "o1-2024-12-17"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.0, output = 60.0 }
+
+[o3-mini]
+model_id = "o3-mini-2025-01-31"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+
+[gpt-5-mini]
+model_id = "gpt-5-mini-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.0 }
+
+[gpt-5-nano]
+model_id = "gpt-5-nano-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.4 }
+
+[gpt-5-chat]
+model_id = "gpt-5-chat-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+
diff --git a/.pipelex/inference/backends/bedrock.toml b/.pipelex/inference/backends/bedrock.toml
new file mode 100644
index 000000000..447c76f03
--- /dev/null
+++ b/.pipelex/inference/backends/bedrock.toml
@@ -0,0 +1,84 @@
+default_sdk = "bedrock_aioboto3"
+default_prompting_target = "anthropic"
+
+[bedrock-mistral-large]
+model_id = "mistral.mistral-large-2407-v1:0"
+max_tokens = 8192
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 4.0, output = 12.0 }
+
+[bedrock-meta-llama-3-3-70b-instruct]
+model_id = "us.meta.llama3-3-70b-instruct-v1:0"
+max_tokens = 8192
+inputs = ["text"]
+outputs = ["text"]
+# TODO: find out the actual cost per million tokens for llama3 on bedrock
+costs = { input = 3.0, output = 15.0 }
+
+[bedrock-nova-pro]
+model_id = "us.amazon.nova-pro-v1:0"
+max_tokens = 5120
+inputs = ["text"]
+outputs = ["text"]
+# TODO: find out the actual cost per million tokens for nova on bedrock
+costs = { input = 3.0, output = 15.0 }
+
+################################################################################
+# Anthropic Models
+################################################################################
+
+["claude-3.5-sonnet"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-3-5-sonnet-20240620-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.5-sonnet-v2"]
+sdk = "bedrock_anthropic"
+model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.7-sonnet"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-sonnet]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-sonnet-4-20250514-v1:0"
+max_tokens = 64000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-opus]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-opus-4-20250514-v1:0"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-4.1-opus"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-opus-4-1-20250805-v1:0"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
diff --git a/.pipelex/inference/backends/blackboxai.toml b/.pipelex/inference/backends/blackboxai.toml
new file mode 100644
index 000000000..14ca78283
--- /dev/null
+++ b/.pipelex/inference/backends/blackboxai.toml
@@ -0,0 +1,156 @@
+
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+# OpenAI Models
+[gpt-4o-mini]
+model_id = "blackboxai/openai/gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.60 }
+
+[gpt-4o]
+model_id = "blackboxai/openai/gpt-4o"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.50, output = 10.00 }
+
+[o1-mini]
+model_id = "blackboxai/openai/o1-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.10, output = 4.40 }
+
+[o4-mini]
+model_id = "blackboxai/openai/o4-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.10, output = 4.40 }
+
+["gpt-4.5-preview"]
+model_id = "blackboxai/openai/gpt-4.5-preview"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 75.00, output = 150.00 }
+
+# Anthropic Models
+["claude-3.5-haiku"]
+model_id = "blackboxai/anthropic/claude-3.5-haiku"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.80, output = 4.00 }
+
+["claude-3.5-sonnet"]
+model_id = "blackboxai/anthropic/claude-3.5-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+["claude-3.7-sonnet"]
+model_id = "blackboxai/anthropic/claude-3.7-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+[claude-opus-4]
+model_id = "blackboxai/anthropic/claude-opus-4"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.00, output = 75.00 }
+
+[claude-sonnet-4]
+model_id = "blackboxai/anthropic/claude-sonnet-4"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+# Google Models
+["gemini-2.5-flash"]
+model_id = "blackboxai/google/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.30, output = 2.50 }
+
+["gemini-2.5-pro"]
+model_id = "blackboxai/google/gemini-2.5-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
+["gemini-flash-1.5-8b"]
+model_id = "blackboxai/google/gemini-flash-1.5-8b"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.15 }
+
+# Free Models
+[deepseek-chat-free]
+model_id = "blackboxai/deepseek/deepseek-chat:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+[deepseek-r1-free]
+model_id = "blackboxai/deepseek/deepseek-r1:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+["llama-3.3-70b-instruct-free"]
+model_id = "blackboxai/meta-llama/llama-3.3-70b-instruct:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+# Mistral Models
+[mistral-large]
+model_id = "blackboxai/mistralai/mistral-large"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 2.00, output = 6.00 }
+
+[pixtral-large-2411]
+model_id = "blackboxai/mistralai/pixtral-large-2411"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.00, output = 6.00 }
+
+# Cost-Effective Models
+["llama-3.3-70b-instruct"]
+model_id = "blackboxai/meta-llama/llama-3.3-70b-instruct"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.12 }
+
+["qwen-2.5-72b-instruct"]
+model_id = "blackboxai/qwen/qwen-2.5-72b-instruct"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.12, output = 0.39 }
+
+# Vision Models
+["llama-3.2-11b-vision-instruct"]
+model_id = "blackboxai/meta-llama/llama-3.2-11b-vision-instruct"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.05 }
+
+["qwen2.5-vl-72b-instruct"]
+model_id = "blackboxai/qwen/qwen2.5-vl-72b-instruct"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 0.75 }
+
+# Amazon Nova Models
+[nova-micro-v1]
+model_id = "blackboxai/amazon/nova-micro-v1"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.14 }
+
+[nova-lite-v1]
+model_id = "blackboxai/amazon/nova-lite-v1"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.06, output = 0.24 }
+
diff --git a/.pipelex/inference/backends/mistral.toml b/.pipelex/inference/backends/mistral.toml
new file mode 100644
index 000000000..13f61e4a3
--- /dev/null
+++ b/.pipelex/inference/backends/mistral.toml
@@ -0,0 +1,94 @@
+default_sdk = "mistral"
+default_prompting_target = "mistral"
+
+[ministral-3b]
+model_id = "ministral-3b-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.04 }
+
+[ministral-8b]
+model_id = "ministral-8b-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.1 }
+
+[mistral-7b-2312]
+model_id = "mistral-large-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 0.25 }
+
+[mistral-8x7b-2312]
+model_id = "open-mixtral-8x7b"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.7, output = 0.7 }
+
+[mistral-codestral-2405]
+model_id = "codestral-2405"
+max_tokens = 262144
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 1.0, output = 3.0 }
+
+[mistral-large-2402]
+model_id = "mistral-large-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 4.0, output = 12.0 }
+
+[mistral-large]
+model_id = "mistral-large-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 4.0, output = 12.0 }
+
+[mistral-small-2402]
+model_id = "mistral-small-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.0, output = 3.0 }
+
+[mistral-small]
+model_id = "mistral-small-latest"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.0, output = 3.0 }
+
+[pixtral-12b]
+model_id = "pixtral-12b-latest"
+max_tokens = 131072
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.15 }
+
+[pixtral-large]
+model_id = "pixtral-large-latest"
+max_tokens = 131072
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.0, output = 6.0 }
+
+[mistral-medium]
+model_id = "mistral-medium-latest"
+max_tokens = 128000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 2.0 }
+
+[mistral-medium-2508]
+model_id = "mistral-medium-2508"
+max_tokens = 128000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 2.0 }
+
diff --git a/.pipelex/inference/backends/ollama.toml b/.pipelex/inference/backends/ollama.toml
new file mode 100644
index 000000000..2b66dee18
--- /dev/null
+++ b/.pipelex/inference/backends/ollama.toml
@@ -0,0 +1,31 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[gemma3-4b]
+model_id = "gemma3:4b"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+[llama4-scout]
+model_id = "llama4:scout"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+["mistral-small3.1-24b"]
+model_id = "mistral-small3.1:24b"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+[qwen3-8b]
+model_id = "qwen3:8b"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0, output = 0 }
+# TODO: support <think> tokens
+
diff --git a/.pipelex/inference/backends/openai.toml b/.pipelex/inference/backends/openai.toml
new file mode 100644
index 000000000..6e3619499
--- /dev/null
+++ b/.pipelex/inference/backends/openai.toml
@@ -0,0 +1,124 @@
+default_sdk = "openai"
+default_prompting_target = "openai"
+
+["gpt-3.5-turbo"]
+model_id = "gpt-3.5-turbo-1106"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.5, output = 1.5 }
+
+[gpt-4]
+model_id = "gpt-4"
+costs = { input = 30.0, output = 60.0 }
+
+[gpt-4-turbo]
+model_id = "gpt-4-turbo"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 10.0, output = 30.0 }
+
+[gpt-4o-2024-11-20]
+model_id = "gpt-4o-2024-11-20"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o]
+model_id = "gpt-4o"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o-mini-2024-07-18]
+model_id = "gpt-4o-mini-2024-07-18"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+[gpt-4o-mini]
+model_id = "gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+["gpt-4.1"]
+model_id = "gpt-4.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "gpt-4.1-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "gpt-4.1-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[o1-mini]
+model_id = "o1-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3.0, output = 12.0 }
+constraints = ["temperature_must_be_1"]
+
+[o1]
+model_id = "o1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.0, output = 60.0 }
+constraints = ["temperature_must_be_1"]
+
+[o3-mini]
+model_id = "o3-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+constraints = ["temperature_must_be_1"]
+
+[o3]
+model_id = "o3"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 10.0, output = 40.0 }
+constraints = ["temperature_must_be_1"]
+
+[o4-mini]
+model_id = "o4-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5]
+model_id = "gpt-5"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-mini]
+model_id = "gpt-5-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.0 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-nano]
+model_id = "gpt-5-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.4 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-chat]
+model_id = "gpt-5-chat-latest"
+inputs = ["text", "images"]
+outputs = ["text"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
diff --git a/.pipelex/inference/backends/perplexity.toml b/.pipelex/inference/backends/perplexity.toml
new file mode 100644
index 000000000..4eb004db5
--- /dev/null
+++ b/.pipelex/inference/backends/perplexity.toml
@@ -0,0 +1,32 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[sonar-pro]
+model_id = "sonar-pro"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar]
+model_id = "sonar"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-deep-research]
+model_id = "sonar-deep-research"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-reasoning-pro]
+model_id = "sonar-reasoning-pro"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-reasoning]
+model_id = "sonar-reasoning"
+inputs = ["text"]
+outputs = ["text"]
+
+[perplexity-deepseek-r1]
+model_id = "r1-1776"
+inputs = ["text"]
+outputs = ["text"]
diff --git a/.pipelex/inference/backends/pipelex_inference.toml b/.pipelex/inference/backends/pipelex_inference.toml
new file mode 100644
index 000000000..22016adc7
--- /dev/null
+++ b/.pipelex/inference/backends/pipelex_inference.toml
@@ -0,0 +1,104 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+# OpenAI Models
+[gpt-4o-mini]
+model_id = "pipelex/gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.17, output = 0.66 }
+
+["gpt-4.1"]
+model_id = "pipelex/gpt-4.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "pipelex/gpt-4.1-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "pipelex/gpt-4.1-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[gpt-5-nano]
+model_id = "pipelex/gpt-5-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.40 }
+
+[gpt-5-mini]
+model_id = "pipelex/gpt-5-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.00 }
+
+[gpt-5-chat]
+model_id = "pipelex/gpt-5-chat"
+inputs = ["text", "images"]
+outputs = ["text"]
+costs = { input = 1.25, output = 10.00 }
+
+[gpt-image-1]
+model_id = "pipelex/gpt-image-1"
+inputs = ["text"]
+outputs = ["images"]
+costs = { input = 5, output = 0.01 }
+
+# Anthropic Models
+["claude-4-sonnet"]
+model_id = "pipelex/claude-4-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3, output = 15 }
+
+["claude-4.1-opus"]
+model_id = "pipelex/claude-4.1-opus"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15, output = 75 }
+
+# Google Models using Google API
+["gemini-2.0-flash"]
+model_id = "gemini/gemini-2.0-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.10, output = 0.40 }
+
+["gemini-2.5-flash"]
+model_id = "gemini/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.30, output = 2.50 }
+
+["gemini-2.5-flash-lite"]
+model_id = "gemini/gemini-2.5-flash-lite"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.10, output = 0.40 }
+
+# XAI Models
+[grok-3]
+model_id = "grok-3"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3, output = 15 }
+
+[grok-3-mini]
+model_id = "grok-3-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.3, output = 0.5 }
+
+# Flux Models
+[fast-lightning-sdxl]
+model_id = "pipelex/fast-lightning-sdxl"
+inputs = ["text"]
+outputs = ["images"]
+costs = { input = 0.01, output = 0.01 }
+
diff --git a/.pipelex/inference/backends/vertexai.toml b/.pipelex/inference/backends/vertexai.toml
new file mode 100644
index 000000000..e476cfcb1
--- /dev/null
+++ b/.pipelex/inference/backends/vertexai.toml
@@ -0,0 +1,24 @@
+default_sdk = "openai"
+default_prompting_target = "gemini"
+
+["gemini-2.0-flash"]
+model_id = "google/gemini-2.0-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 0.1, output = 0.4 }
+
+["gemini-2.5-pro"]
+model_id = "google/gemini-2.5-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 1.25, output = 10.0 }
+
+["gemini-2.5-flash"]
+model_id = "google/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 0.30, output = 2.50 }
+
diff --git a/.pipelex/inference/backends/xai.toml b/.pipelex/inference/backends/xai.toml
new file mode 100644
index 000000000..ea8fb58f7
--- /dev/null
+++ b/.pipelex/inference/backends/xai.toml
@@ -0,0 +1,27 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[grok-3]
+model_id = "grok-3"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3, output = 15 }
+
+[grok-3-mini]
+model_id = "grok-3-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.3, output = 0.5 }
+
+[grok-3-fast]
+model_id = "grok-3-fast-latest"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 5, output = 25 }
+
+[grok-3-mini-fast]
+model_id = "grok-3-mini-fast-latest"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.15, output = 4 }
+
diff --git a/pipelex/libraries/llm_deck/base_llm_deck.toml b/.pipelex/inference/deck/base_deck.toml
similarity index 89%
rename from pipelex/libraries/llm_deck/base_llm_deck.toml
rename to .pipelex/inference/deck/base_deck.toml
index cddfbc0d0..0736f0347 100644
--- a/pipelex/libraries/llm_deck/base_llm_deck.toml
+++ b/.pipelex/inference/deck/base_deck.toml
@@ -2,24 +2,19 @@
# LLM Deck base
####################################################################################################
-# LLM Handles
+# Aliases
####################################################################################################
-llm_external_handles = []
-
-# Match a llm_handle with either just and llm_name
-# or a complete blueprint including llm_version (defaulting to "latest") and llm_platform_choice (defaulting to "default").
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
+[aliases]
base-claude = "claude-4-sonnet"
base-gpt = "gpt-5"
base-gemini = "gemini-2.5-flash"
base-mistral = "mistral-medium"
best-gpt = "gpt-5"
-best-claude = "claude-4-1-opus"
+best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
-best-mistral = "mistral-large"
+best-mistral = "mistral-medium"
best-grok = "grok-3"
cheap_llm_for_text = "gpt-4o-mini"
@@ -28,6 +23,8 @@ cheap_llm_for_object = "gpt-4o-mini"
base_llm_for_text = "gpt-4o-mini"
base_llm_for_object = "gpt-4o-mini"
+llm_to_engineer = ["claude-4.1-opus", "gemini-2.5-pro"]
+llm_for_large_codebase = ["gemini-2.5-pro", "claude-4-sonnet", "gpt-5"]
####################################################################################################
# LLM Presets
@@ -54,7 +51,7 @@ llm_cheap = { llm_handle = "base-gemini", temperature = 0.1 }
# Generation skills
llm_for_factual_writing = { llm_handle = "base-gpt", temperature = 0.1 }
-llm_for_creative_writing = { llm_handle = "best-claude", temperature = 0.9 }
+llm_for_creative_writing = { llm_handle = "claude-4-sonnet", temperature = 0.9 }
# Reasoning skills
llm_to_reason_short = { llm_handle = "base-claude", temperature = 0.5, max_tokens = 500 }
diff --git a/pipelex/libraries/llm_deck/overrides.toml b/.pipelex/inference/deck/overrides.toml
similarity index 100%
rename from pipelex/libraries/llm_deck/overrides.toml
rename to .pipelex/inference/deck/overrides.toml
diff --git a/.pipelex/inference/routing_profiles.toml b/.pipelex/inference/routing_profiles.toml
new file mode 100644
index 000000000..be9b51d2c
--- /dev/null
+++ b/.pipelex/inference/routing_profiles.toml
@@ -0,0 +1,85 @@
+# Routing profile library - Routes models to their backends
+# ================================================
+# This file controls which backend serves which model.
+# Simply change the 'active' field to switch profiles,
+# or you can add your own custom profiles.
+# ================================================
+
+# Which profile to use (change this to switch routing)
+active = "all_pipelex"
+
+# We recommend using the "all_pipelex" profile to get a head start with all models.
+# The Pipelex Inference backend is currently not recommended for production use,
+# but it's great for development and testing.
+# To use the Pipelex Inference backend (all_pipelex profile):
+# 1. Join our Discord community to get your free API key (no credit card required):
+# Visit https://go.pipelex.com/discord and request your key in the appropriate channel
+# 2. Set the environment variable: export PIPELEX_INFERENCE_API_KEY="your-api-key"
+# 3. The .pipelex/inference/backends.toml is already configured with api_key = "${PIPELEX_INFERENCE_API_KEY}"
+# which will get the key from the environment variable.
+
+# ============================================
+# Routing Profiles
+# ============================================
+
+[profiles.all_pipelex]
+description = "Use Pipelex Inference backend for all models"
+default = "pipelex_inference"
+
+[profiles.all_anthropic]
+description = "Use Anthropic backend for all its supported models"
+default = "anthropic"
+
+[profiles.all_azure_openai]
+description = "Use Azure OpenAI backend for all its supported models"
+default = "azure_openai"
+
+[profiles.to_test_various_backends]
+description = "Route models to various backends"
+
+[profiles.custom_routing]
+description = "Custom routing"
+default = "pipelex_inference"
+
+[profiles.custom_routing.routes]
+# Pattern matching: "model-pattern" = "backend-name"
+"gpt-*" = "pipelex_inference"
+"claude-*" = "pipelex_inference"
+"gemini-*" = "pipelex_inference"
+"grok-*" = "pipelex_inference"
+"*-sdxl" = "pipelex_inference"
+
+[profiles.to_test_various_backends.routes]
+"gpt-5-nano" = "pipelex_inference"
+"gpt-4o-mini" = "blackboxai"
+"gpt-5-mini" = "openai"
+"gpt-5-chat" = "azure_openai"
+
+"claude-4-sonnet" = "pipelex_inference"
+"claude-3.5-sonnet" = "bedrock"
+"claude-3.5-sonnet-v2" = "anthropic"
+"claude-3.7-sonnet" = "blackboxai"
+
+"gemini-2.5-flash-lite" = "pipelex_inference"
+"gemini-2.5-flash" = "blackboxai"
+"gemini-2.5-pro" = "vertexai"
+
+"grok-3" = "pipelex_inference"
+"grok-3-mini" = "xai"
+
+# ============================================
+# Custom Profiles
+# ============================================
+# Add your own profiles below following the same pattern:
+#
+# [profiles.your_profile_name]
+# description = "What this profile does"
+# default = "backend-name" # Where to route models by default
+# [profiles.your_profile_name.routes]
+# "model-pattern" = "backend-name" # Specific routing rules
+#
+# Pattern matching supports:
+# - Exact names: "gpt-4o-mini"
+# - Wildcards: "claude-*" (matches all models starting with claude-)
+# - Partial wildcards: "*-sonnet" (matches all sonnet variants)
+
diff --git a/.vscode/launch.json b/.vscode/launch.json
index e7753f589..35afe5a40 100644
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -16,15 +16,27 @@
"justMyCode": false
},
{
- "name": "Debug test_pipe_ocr_pdf",
+ "name": "Debug test without inference",
"type": "debugpy",
"request": "launch",
"module": "pytest",
"args": [
"-s",
"-vv",
- "-m ocr",
- "-k test_pipe_ocr_pdf"
+ "-k test_pipe_llm_with_external_llm_handle"
+ ],
+ "console": "integratedTerminal"
+ },
+ {
+ "name": "Debug test inference",
+ "type": "debugpy",
+ "request": "launch",
+ "module": "pytest",
+ "args": [
+ "-s",
+ "-vv",
+ "-m inference",
+ "-k test_simple_gen"
],
"console": "integratedTerminal"
},
diff --git a/AGENTS.md b/AGENTS.md
index 54ef028ea..d7446120c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -659,32 +659,30 @@ But don't write documentation unless asked explicitly to.
# Rules to choose LLM models used in PipeLLMs.
-## LLM Handles
+## LLM Configuration System
-In order to use it in a pipe, an LLM is referenced by its llm_handle and possibly by an llm_preset.
-Both llm_handles and llm_presets are defined in this toml config file: [base_llm_deck.toml](mdc:pipelex/libraries/llm_deck/base_llm_deck.toml)
+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-## LLM Handles
+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
+- **Routing**: `.pipelex/inference/routing_profiles.toml`
-An llm_handle matches the handle (an id of sorts) with the full specification of the LLM to use, i.e.:
-- llm_name
-- llm_version
-- llm_platform_choice
+## LLM Handles
-The declaration of llm_handles looks like this in toml syntax:
-```toml
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
-```
+An llm_handle can be either:
+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-In mosty cases, we only want to use version "latest" and llm_platform_choice "default" in which case the declaration is simply a match of the llm_handle to the llm_name, like this:
```toml
-best-claude = "claude-4-opus"
+[aliases]
+best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
best-mistral = "mistral-large"
+base-gpt = "gpt-5"
```
-And of course, llm_handles are automatically assigned for all models by their name, with version "latest" and llm_platform_choice "default".
+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
## Using an LLM Handle in a PipeLLM
@@ -735,7 +733,7 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_
You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).
+You can override the predefined llm presets in `.pipelex/inference/deck/overrides.toml`.
These rules apply when writing unit tests.
- Always use pytest
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e4c080884..190413fec 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,12 +1,59 @@
# Changelog
-## [v0.9.6] - 2025-09-16
+## Unreleased
+
+### Highlight: New Inference Backend Configuration System
+
+We've completely redesigned how LLMs are configured and accessed in Pipelex, making it more flexible and easier to get started:
+
+- **Get started in seconds** with [Pipelex Inference](pages/configuration/config-technical/inference-backend-config.md#pipelex-inference): Use a single API key to access all major LLM providers (OpenAI, Anthropic, Google, Mistral, and more)
+- **Flexible backend configuration**: Configure multiple inference backends (Azure OpenAI, AWS Bedrock, Vertex AI, etc.) through simple TOML files in `.pipelex/inference/`
+- **Smart model routing**: Automatically route models to the right backend using [routing profiles](pages/configuration/config-technical/inference-backend-config.md#routing-profiles) with pattern matching
+- **User-friendly aliases**: Define shortcuts like `best-claude` → `claude-4.1-opus` with optional fallback chains
+- **Cost-aware model specs**: Each model includes detailed pricing, capabilities, and constraints for better cost management
+
+For complete details, see the [Inference Backend Configuration](pages/configuration/config-technical/inference-backend-config.md) documentation.
### Added
+- New inference backend configuration system in `.pipelex/inference/` directory
+- Support for 10+ inference backends: OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Mistral, Vertex AI, XAI, BlackboxAI, Perplexity, Ollama, and **Pipelex Inference**
+- Model routing profiles with pattern matching (`*model*`, `model*`, `*model`)
+- Model aliases with waterfall fallback chains
+- Environment variable and secret substitution in TOML configs (`${VAR}` and `${secret:KEY}`)
+- Comprehensive model specifications with detailed cost categories
+- Unified plugin SDK registry for all backends
+- CI environment detection with automatic placeholder API keys for testing
+- Improved `pipelex init config` command to copy entire configuration template directory structure to `.pipelex/` with smart file handling (skips existing files, shows clear progress messages)
- Added `FuncRegistryUtils` to register functions in a pipelex folder that have a specific signature.
-- Added `mistral-medium` and `mistral-medium-2508` to the LLM deck.
-- Added handle for `gemini-2.5-flash` to the LLM deck.
+- Added `mistral-medium` and `mistral-medium-2508` to the Mistral backend configuration.
+- Added `gemini-2.5-flash` to the VertexAI backend configuration.
+
+### Changed
+
+- LLM configuration moved from `pipelex_libraries/llm_deck/` to `.pipelex/inference/deck/`
+- LLM handles simplified to direct model names or user-defined aliases
+- Model deck completely redesigned with inference models, aliases, and presets
+- Plugin system refactored to use backend-specific TOML configuration
+- Token categories renamed to cost categories with expanded types
+
+### Fixed
+
+- Improved error messages for missing environment variables
+- Enhanced TOML configuration validation
+- More robust model routing and backend selection
+
+### Removed
+
+- Legacy LLM model library system (`llm_integrations/` directory)
+- Platform-specific configuration classes (AnthropicConfig, OpenAIConfig, etc.)
+- Deprecated LLM engine blueprint and factory classes
+- Old LLM platform and family enumerations
+
+### Security
+
+- Enhanced secret management with secure fallback patterns
+- Improved API key handling through centralized backend configuration
## [v0.9.5] - 2025-09-12
diff --git a/CLAUDE.md b/CLAUDE.md
index 593243fa8..315ab1b1c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -660,32 +660,30 @@ But don't write documentation unless asked explicitly to.
# Rules to choose LLM models used in PipeLLMs.
-## LLM Handles
+## LLM Configuration System
-In order to use it in a pipe, an LLM is referenced by its llm_handle and possibly by an llm_preset.
-Both llm_handles and llm_presets are defined in this toml config file: [base_llm_deck.toml](mdc:pipelex/libraries/llm_deck/base_llm_deck.toml)
+In order to use it in a pipe, an LLM is referenced by its llm_handle (alias) and possibly by an llm_preset.
+LLM configurations are managed through the new inference backend system with files located in `.pipelex/inference/`:
-## LLM Handles
+- **Model Deck**: `.pipelex/inference/deck/base_deck.toml` and `.pipelex/inference/deck/overrides.toml`
+- **Backends**: `.pipelex/inference/backends.toml` and `.pipelex/inference/backends/*.toml`
+- **Routing**: `.pipelex/inference/routing_profiles.toml`
-An llm_handle matches the handle (an id of sorts) with the full specification of the LLM to use, i.e.:
-- llm_name
-- llm_version
-- llm_platform_choice
+## LLM Handles
-The declaration of llm_handles looks like this in toml syntax:
-```toml
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
-```
+An llm_handle can be either:
+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-In mosty cases, we only want to use version "latest" and llm_platform_choice "default" in which case the declaration is simply a match of the llm_handle to the llm_name, like this:
```toml
-best-claude = "claude-4-opus"
+[aliases]
+best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
best-mistral = "mistral-large"
+base-gpt = "gpt-5"
```
-And of course, llm_handles are automatically assigned for all models by their name, with version "latest" and llm_platform_choice "default".
+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
## Using an LLM Handle in a PipeLLM
@@ -736,7 +734,7 @@ The setting here `llm = "llm_to_extract_invoice"` works because "llm_to_extract_
You must not use an LLM preset in a PipeLLM that does not exist in the deck. If needed, you can add llm presets.
-You can override the predefined llm presets in [overrides.toml](../../pipelex/libraries/llm_deck/overrides.toml).
+You can override the predefined llm presets in `.pipelex/inference/deck/overrides.toml`.
These rules apply when writing unit tests.
- Always use pytest
diff --git a/Makefile b/Makefile
index 7401f0ffe..e4bd1d57e 100644
--- a/Makefile
+++ b/Makefile
@@ -45,7 +45,6 @@ make install - Create local virtualenv & install all dependenci
make update - Upgrade dependencies via uv
make validate - Run the setup sequence to validate the config and libraries
make build - Build the wheels
-make doc - Serve documentation locally with mkdocs
make format - format with ruff format
make lint - lint with ruff check
@@ -85,7 +84,7 @@ make fix-unused-imports - Fix unused imports with ruff
make fui - Shorthand -> fix-unused-imports
make check-TODOs - Check for TODOs
-make docs - Serve documentation with mkdocs
+make docs - Serve documentation locally with mkdocs
make docs-check - Check documentation build with mkdocs
make docs-deploy - Deploy documentation with mkdocs
diff --git a/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md b/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md
new file mode 100644
index 000000000..eed439a74
--- /dev/null
+++ b/NEW_INFERENCE_BACKEND_CONFIG_SYSTEM.md
@@ -0,0 +1,73 @@
+# New Inference Backend Configuration System
+
+This document provides an overview of the new system for configuring LLM inference backends, models, and routing within Pipelex. The system is designed to be highly flexible and is managed through a series of TOML configuration files.
+
+The core components of this system reside in the following directories:
+- `pipelex/cogt/model_backends/`
+- `pipelex/cogt/model_routing/`
+- `pipelex/cogt/models/`
+
+## Core Concepts
+
+The new system is built around a few key concepts: Inference Backends, Model Specs, Routing Profiles, and the Model Deck.
+
+### 1. Inference Backends
+
+An **Inference Backend** represents a provider of LLM services, such as OpenAI, Anthropic, Google Vertex AI, etc. Each backend is defined by its name, API endpoint, and authentication details (e.g., API key).
+
+- **Implementation**: `pipelex.cogt.model_backends.backend.InferenceBackend`
+- **Configuration**: Backends are defined in a central TOML file (e.g., `backends.toml`), specified by `cogt.inference_config.backends_library_path` in the main configuration.
+
+### 2. Model Specs
+
+An **Inference Model Spec** (`InferenceModelSpec`) provides detailed information about a specific model available through a backend. This includes:
+
+- `model_id`: The identifier used by the backend's API.
+- `sdk`: The Pipelex SDK to use for communication (e.g., `openai`, `anthropic`).
+- Capabilities: Supported `inputs` (e.g., `text`, `images`) and `outputs` (e.g., `text`, `structured`).
+- Pricing: `costs` for input and output tokens.
+- Constraints: `max_tokens`, `max_prompt_images`, and other model-specific limitations.
+
+- **Implementation**: `pipelex.cogt.model_backends.model_spec.InferenceModelSpec`
+- **Configuration**: Model specs for each backend are defined in their own dedicated TOML file (e.g., `openai.toml`, `anthropic.toml`). The path to these files is determined by `cogt.inference_config.model_specs_path(backend_name)`.
+
+### 3. Routing Profiles
+
+A **Routing Profile** (`RoutingProfile`) defines the rules for selecting which backend should be used for a given model name. This allows for flexible and environment-specific model routing. A routing profile consists of:
+
+- A `default` backend to use if no other rule matches.
+- A set of `routes` that map a model name (or a pattern) to a specific backend.
+
+The routing logic supports exact matches, as well as prefix, suffix, and "contains" wildcard (`*`) matching.
+
+- **Implementation**: `pipelex.cogt.model_routing.routing_profile.RoutingProfile`
+- **Configuration**: Multiple routing profiles can be defined in a single TOML file (e.g., `routing.toml`), specified by `cogt.inference_config.routing_profile_library_path`. One profile is designated as `active`.
+
+### 4. The Model Deck
+
+The **Model Deck** (`ModelDeck`) is the final, unified collection of all configured and available models in the system. It is constructed at runtime by the `ModelManager` and serves as the single source of truth for the rest of the application. The Model Deck contains:
+
+- **Inference Models**: A resolved map of model names (`llm_handle`) to their corresponding `InferenceModelSpec`.
+- **Aliases**: User-friendly names that map to one or more model names, creating a fallback chain (e.g., `best-claude` -> `claude-3-opus-20240229`).
+- **LLM Presets**: Pre-defined configurations (`LLMSetting`) for specific tasks, combining an `llm_handle` with parameters like `temperature` and `max_tokens` (e.g., `llm_to_reason`).
+- **Default Choices**: Default model presets to use for text generation (`for_text`) and structured data extraction (`for_object`).
+
+- **Implementation**: `pipelex.cogt.models.model_deck.ModelDeck`
+- **Configuration**: Aliases, presets, and defaults are loaded from one or more `llm_deck.toml` files, specified by `cogt.inference_config.get_llm_deck_paths()`.
+
+## How It All Works Together: The Loading Process
+
+The entire system is orchestrated by the `ModelManager` during application startup. Here is a step-by-step breakdown of the process:
+
+1. **Load Backends**: The `InferenceBackendLibrary` reads the `backends.toml` file to get a list of all defined backends. For each enabled backend, it proceeds to load its associated models.
+2. **Load Model Specs**: For each backend, the library reads the corresponding `<backend_name>.toml` file to load the `InferenceModelSpec` for every model that backend provides.
+3. **Load Routing Profiles**: The `RoutingProfileLibrary` reads the `routing.toml` file to load all defined routing profiles and identifies the `active` profile.
+4. **Build the Deck**: The `ModelManager` begins to build the `ModelDeck`.
+ a. It gets the list of all models from all loaded backends.
+ b. For each model, it consults the `active` `RoutingProfile` to determine which backend should be used.
+ c. It retrieves the `InferenceModelSpec` for that model from the chosen backend.
+ d. This resolved model spec is added to the `ModelDeck`'s list of `inference_models`.
+5. **Load Deck Configuration**: The `ModelManager` reads the `llm_deck.toml` files and loads the `aliases`, `llm_presets`, and default choices into the `ModelDeck`.
+6. **Finalization**: The `ModelDeck` is now fully constructed and ready to be used by the rest of Pipelex to get model specifications and settings via `get_models_manager().get_llm_deck()`.
+
+This layered configuration system provides a powerful way to manage models from various providers, route them according to specific needs, and define reusable presets for common tasks, all through a set of clear and manageable TOML files.
diff --git a/docs/pages/advanced-customization/index.md b/docs/pages/advanced-customization/index.md
index f68371a1d..ceb53e5d6 100644
--- a/docs/pages/advanced-customization/index.md
+++ b/docs/pages/advanced-customization/index.md
@@ -16,8 +16,6 @@ from pipelex import Pipelex
pipelex = Pipelex(
config_dir_path="./pipelex_libraries",
template_provider=MyTemplateProvider(),
- llm_model_provider=MyLLMProvider(),
- inference_manager=MyInferenceManager(),
pipeline_tracker=MyPipelineTracker(),
activity_manager=MyActivityManager(),
reporting_delegate=MyReportingDelegate()
@@ -37,7 +35,6 @@ from pipelex.hub import PipelexHub
hub = PipelexHub()
hub.set_template_provider(MyTemplateProvider())
-hub.set_llm_models_provider(MyLLMProvider())
# ... and so on for other components
```
@@ -81,49 +78,37 @@ Pipelex supports injection of the following components:
- Default: `TemplateLibrary`
- [Details](template-provider-injection.md)
-2. **LLM Model Provider** (`LLMModelLibrary`)
-
- - Protocol: `LLMModelProviderAbstract`
- - Default: `LLMModelLibrary`
- - [Details](llm-model-provider-injection.md)
-
-3. **Inference Manager** (`InferenceManager`)
-
- - Protocol: `InferenceManagerProtocol`
- - Default: `InferenceManager`
- - [Details](inference-manager-injection.md)
-
-4. **Reporting Delegate** (`ReportingManager`)
+2. **Reporting Delegate** (`ReportingManager`)
- Protocol: `ReportingProtocol`
- Default: `ReportingManager` or `ReportingNoOp` if disabled
- [Details](reporting-delegate-injection.md)
-5. **Pipeline Tracker** (`PipelineTracker`)
+3. **Pipeline Tracker** (`PipelineTracker`)
- Protocol: `PipelineTrackerProtocol`
- Default: `PipelineTracker` or `PipelineTrackerNoOp` if disabled
- [Details](pipeline-tracker-injection.md)
-6. **Activity Manager** (`ActivityManager`)
+4. **Activity Manager** (`ActivityManager`)
- Protocol: `ActivityManagerProtocol`
- Default: `ActivityManager` or `ActivityManagerNoOp` if disabled
- [Details](activity-manager-injection.md)
-7. **Secrets Provider** (`EnvSecretsProvider`)
+5. **Secrets Provider** (`EnvSecretsProvider`)
- Protocol: `SecretsProviderAbstract`
- Default: `EnvSecretsProvider`
- [Details](secrets-provider-injection.md)
-8. **Content Generator** (`ContentGenerator`)
+6. **Content Generator** (`ContentGenerator`)
- Protocol: `ContentGeneratorProtocol`
- Default: `ContentGenerator`
- [Details](content-generator-injection.md)
-9. **Pipe Router** (`PipeRouter`)
+7. **Pipe Router** (`PipeRouter`)
- Protocol: `PipeRouterProtocol`
- Default: `PipeRouter`
diff --git a/docs/pages/advanced-customization/inference-manager-injection.md b/docs/pages/advanced-customization/inference-manager-injection.md
deleted file mode 100644
index d051dbc05..000000000
--- a/docs/pages/advanced-customization/inference-manager-injection.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# Inference Manager Injection
-
-⚠️ Under construction
diff --git a/docs/pages/advanced-customization/llm-model-provider-injection.md b/docs/pages/advanced-customization/llm-model-provider-injection.md
deleted file mode 100644
index bc63eeb2a..000000000
--- a/docs/pages/advanced-customization/llm-model-provider-injection.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# LLM Model Provider Injection
-
-⚠️ Under construction
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/ai-plugins-for-multi-llm-workflows.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/ai-plugins-for-multi-llm-workflows.md
deleted file mode 100644
index e07b9484f..000000000
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/ai-plugins-for-multi-llm-workflows.md
+++ /dev/null
@@ -1,238 +0,0 @@
-# Plugins Configuration
-
-The Plugins Configuration manages all external service integrations in Pipelex, including various LLM providers and image generation services.
-
-## Overview
-
-```toml
-[pipelex.plugins]
-# Plugin sections
-[pipelex.plugins.anthropic_config]
-[pipelex.plugins.azure_openai_config]
-[pipelex.plugins.bedrock_config]
-[pipelex.plugins.vertexai_config]
-[pipelex.plugins.mistral_config]
-[pipelex.plugins.openai_config]
-[pipelex.plugins.perplexity_config]
-[pipelex.plugins.xai_config]
-[pipelex.plugins.custom_endpoint_config]
-[pipelex.plugins.fal_config]
-```
-
-## Common Authentication Methods
-
-Most plugins support two authentication methods:
-
-- `ENV`: Read credentials from environment variables
-- `SECRET_PROVIDER`: Read credentials from a secrets provider
-
-## Plugin-Specific Configurations
-
-### 1. Anthropic Configuration
-
-```toml
-[pipelex.plugins.anthropic_config]
-# Use 8192 for better streaming/timeout handling, or "unlimited" for full 32/64K tokens (Opus/Sonnet)
-claude_4_reduced_tokens_limit = 8192
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `ANTHROPIC_API_KEY`: API key for Anthropic services
-
-### 2. Azure OpenAI Configuration
-
-```toml
-[pipelex.plugins.azure_openai_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `AZURE_OPENAI_API_KEY`: API key
-- `AZURE_OPENAI_API_ENDPOINT`: API endpoint URL
-- `AZURE_OPENAI_API_VERSION`: API version
-
-### 3. AWS Bedrock Configuration
-
-```toml
-[pipelex.plugins.bedrock_config]
-client_method = "aioboto3" # or "boto3"
-```
-
-Environment Variables:
-
-- `AWS_REGION`: AWS region for Bedrock services
-
-### 4. Google Vertex AI Configuration
-
-```toml
-[pipelex.plugins.vertexai_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `GCP_PROJECT_ID`: Google Cloud project ID
-- `GCP_REGION`: Google Cloud region
-- `GCP_CREDENTIALS_FILE_PATH`: Path to service account credentials file
-
-Dependencies:
-
-- Requires `google-auth-oauthlib` package (`pip install pipelex[google]`)
-
-### 5. Mistral Configuration
-
-```toml
-[pipelex.plugins.mistral_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `MISTRAL_API_KEY`: API key for Mistral services
-
-### 6. OpenAI Configuration
-
-```toml
-[pipelex.plugins.openai_config]
-image_output_compression = 100 # 1-100
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `OPENAI_API_KEY`: API key for OpenAI services
-
-### 7. Perplexity Configuration
-
-```toml
-[pipelex.plugins.perplexity_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `PERPLEXITY_API_KEY`: API key
-- `PERPLEXITY_API_ENDPOINT`: API endpoint URL
-
-### 8. XAI Configuration
-
-```toml
-[pipelex.plugins.xai_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `XAI_API_KEY`: API key
-- `XAI_API_ENDPOINT`: API endpoint URL
-
-### 9. Custom Endpoint Configuration
-
-For custom OpenAI-compatible endpoints (e.g., Ollama, LM Studio):
-
-```toml
-[pipelex.plugins.custom_endpoint_config]
-api_key_method = "env" # or "secret_provider"
-```
-
-Environment Variables:
-
-- `CUSTOM_ENDPOINT_API_KEY`: Optional API key
-- `CUSTOM_ENDPOINT_BASE_URL`: Base URL for the custom endpoint
-
-### 10. FAL Configuration
-
-Configuration for FAL image generation services:
-
-```toml
-[pipelex.plugins.fal_config]
-# Quality to steps mapping for Flux model
-flux_map_quality_to_steps = {
- "low" = 14,
- "medium" = 28,
- "high" = 56
-}
-
-# Quality to steps mapping for SDXL Lightning model
-sdxl_lightning_map_quality_to_steps = {
- "low" = 2,
- "medium" = 4,
- "high" = 8
-}
-```
-
-## Error Handling
-
-Each plugin has its own error types:
-
-- `AnthropicCredentialsError`
-- `AzureOpenAICredentialsError`
-- `BedrockCredentialsError`
-- `VertexAICredentialsError`
-- `MistralCredentialsError`
-- `OpenAICredentialsError`
-- `PerplexityCredentialsError`
-- `XaiCredentialsError`
-- `CustomEndpointCredentialsError`
-
-## Best Practices
-
-1. **Credentials Management**:
-
- - Use environment variables for local development
- - Use secrets provider for production environments
- - Never commit credentials to version control
-
-2. **Error Handling**:
-
- - Always handle credential errors appropriately
- - Implement proper fallbacks when using multiple providers
- - Check for required dependencies (especially for Google services)
-
-3. **Configuration**:
-
- - Set appropriate quality levels for image generation
- - Configure retry limits and timeouts
- - Use appropriate client methods for async/sync operations
-
-## Example Complete Configuration
-
-```toml
-[pipelex.plugins]
-
-[pipelex.plugins.anthropic_config]
-claude_4_reduced_tokens_limit = 8192 # Use "unlimited" for full 32/64K tokens
-api_key_method = "env"
-
-[pipelex.plugins.azure_openai_config]
-api_key_method = "env"
-
-[pipelex.plugins.bedrock_config]
-client_method = "aioboto3"
-
-[pipelex.plugins.vertexai_config]
-api_key_method = "env"
-
-[pipelex.plugins.mistral_config]
-api_key_method = "env"
-
-[pipelex.plugins.openai_config]
-image_output_compression = 100
-api_key_method = "env"
-
-[pipelex.plugins.perplexity_config]
-api_key_method = "env"
-
-[pipelex.plugins.xai_config]
-api_key_method = "env"
-
-[pipelex.plugins.custom_endpoint_config]
-api_key_method = "env"
-
-[pipelex.plugins.fal_config]
-flux_map_quality_to_steps = { "low" = 14, "medium" = 28, "high" = 56 }
-sdxl_lightning_map_quality_to_steps = { "low" = 2, "medium" = 4, "high" = 8 }
-```
diff --git a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
index 37321a0ac..87191301b 100644
--- a/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
+++ b/docs/pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
@@ -2,37 +2,36 @@
## Overview
-Pipelex provides a flexible way to configure and manage your LLM (Large Language Model) integrations through three main concepts:
+Pipelex provides a flexible way to configure and manage your LLM (Large Language Model) integrations through the inference backend configuration system.
-- LLM Handles
-- LLM Presets
-- LLM Deck
+The system provides three main concepts for LLM configuration:
-Those configuration are present in the `pipelex_libraries/llm_deck` directory.
+- LLM Handles (Aliases)
+- LLM Presets
+- Model Deck
-## LLM Handles
+For complete details about the inference backend configuration system, see the [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) documentation.
-An LLM handle is a unique identifier that maps to a specific LLM configuration. It defines:
+## LLM Handles (Aliases)
-- The LLM name (e.g., `gpt-4o-mini`, `claude-4-sonnet`, `mistral-large`, `gemini-2.5-flash`)
-- The model version
-- The platform-specific settings (OpenAI, Anthropic, etc...)
+An LLM handle can be either:
-### Example Handle Configurations
+1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
+2. **An alias** - user-defined shortcuts that map to model names, defined in the `[aliases]` section:
-```toml
-[llm_handles]
-gpt-4o-2024-11-20 = { llm_name = "gpt-4o", llm_version = "2024-11-20" }
-```
-
-There's a much simpler syntax if you want a handle to the latest version and default platform:
+### Example Alias Configurations
```toml
-[llm_handles]
-best-claude = "claude-4-opus"
+[aliases]
+best-claude = "claude-4.1-opus"
+best-gemini = "gemini-2.5-pro"
+best-mistral = "mistral-large"
+base-gpt = "gpt-5"
```
-💡 Defining a llm_handle is alway meant to describe what model it is. Never define a llm_handle to describe what it is for or what it's good at. LLM Settings are for that.
+The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.
+
+💡 Defining an alias is always meant to describe what model it is. Never define an alias to describe what it is for or what it's good at. LLM Presets are for that.
## LLM Settings & LLM Presets
@@ -92,23 +91,31 @@ Extract the weather data from this text:
"""
```
-## LLM Deck
+## Model Deck
-The LLM deck is your central configuration hub for all LLM-related settings. It's stored in the `pipelex_libraries/llm_deck` directory and consists of:
+The Model Deck is your central configuration hub for all LLM-related settings. It's stored in the `.pipelex/inference/deck/` directory and consists of:
-- `base_llm_deck.toml`: Core LLM configurations
+- `base_deck.toml`: Core LLM configurations including aliases and presets
- `overrides.toml`: Custom overrides for specific use cases
### Directory Structure
```bash
-pipelex_libraries/
-└── llm_deck/
- ├── base_llm_deck.toml
- └── overrides.toml
+.pipelex/
+└── inference/
+ ├── backends.toml # Backend configurations
+ ├── routing_profiles.toml # Model routing rules
+ ├── backends/ # Individual backend model specs
+ │ ├── openai.toml
+ │ ├── anthropic.toml
+ │ └── ...
+ └── deck/ # Model deck configurations
+ ├── base_deck.toml # Aliases and presets
+ └── overrides.toml # Custom overrides
```
+
## Best Practices
1. **Consistent Naming**: Use clear, descriptive names for handles and presets
diff --git a/docs/pages/configuration/config-technical/cogt-config.md b/docs/pages/configuration/config-technical/cogt-config.md
index 4c1cac795..467526e2a 100644
--- a/docs/pages/configuration/config-technical/cogt-config.md
+++ b/docs/pages/configuration/config-technical/cogt-config.md
@@ -32,11 +32,6 @@ Configuration for all Language Model interactions:
[cogt.llm_config]
default_max_images = 100 # Maximum number of images in prompts
-# Platform preferences for different LLMs
-[cogt.llm_config.preferred_platforms]
-gpt-4 = "openai"
-claude-3-opus = "anthropic"
-
# Job configuration
[cogt.llm_config.llm_job_config]
is_streaming_enabled = false
@@ -164,7 +159,6 @@ is_auto_setup_preset_ocr = true
[cogt.llm_config]
default_max_images = 100
-preferred_platforms = { "gpt-4" = "openai", "claude-3-opus" = "anthropic" }
[cogt.llm_config.llm_job_config]
is_streaming_enabled = false
diff --git a/docs/pages/configuration/config-technical/inference-backend-config.md b/docs/pages/configuration/config-technical/inference-backend-config.md
new file mode 100644
index 000000000..092b73271
--- /dev/null
+++ b/docs/pages/configuration/config-technical/inference-backend-config.md
@@ -0,0 +1,278 @@
+# Inference Backend Configuration
+
+The Inference Backend Configuration System manages how Pipelex handles LLM providers, model routing, and inference settings. This system provides a flexible and scalable way to configure multiple inference backends and route models to the appropriate providers.
+
+## Overview
+
+The inference backend system is built around four key concepts:
+
+1. **Inference Backends**: Providers of LLM services (OpenAI, Anthropic, Google Vertex AI, etc.)
+2. **Model Specs**: Detailed information about specific models available through backends
+3. **Routing Profiles**: Rules for selecting which backend should handle specific models
+4. **Model Deck**: Unified collection of configured models, aliases, and presets
+
+## Directory Structure
+
+All inference backend configurations are stored in the `.pipelex/inference/` directory:
+
+```
+.pipelex/
+└── inference/
+ ├── backends.toml # Backend provider configurations
+ ├── routing_profiles.toml # Model routing rules
+ ├── backends/ # Individual backend model specifications
+ │ ├── openai.toml
+ │ ├── anthropic.toml
+ │ ├── bedrock.toml
+ │ ├── mistral.toml
+ │ ├── vertexai.toml
+ │ └── ...
+ └── deck/ # Model deck configurations
+ ├── base_deck.toml # Core aliases and presets
+ └── overrides.toml # Custom overrides
+```
+
+## Pipelex Inference
+
+Pipelex Inference is a unified inference backend that provides access to all major LLM providers through a single API key. This is the recommended approach for getting started quickly with Pipelex.
+
+### Benefits
+
+- **Single API Key**: Access OpenAI, Anthropic, Google, Mistral, and more with one key
+- **Simplified Configuration**: No need to manage multiple provider credentials
+- **Automatic Routing**: Models are automatically routed to their respective providers
+
+### Setup
+
+1. Join our Discord community to get your free Pipelex Inference API key (no credit card required, limited time offer):
+ - Visit [https://go.pipelex.com/discord](https://go.pipelex.com/discord) to join
+ - Request your API key in the appropriate channel once you're in
+2. Set the environment variable:
+ ```bash
+ export PIPELEX_INFERENCE_API_KEY="your-api-key"
+ ```
+3. Configure in `.pipelex/inference/backends.toml`:
+ ```toml
+ [pipelex_inference]
+ enabled = true
+ api_key = "${PIPELEX_INFERENCE_API_KEY}"
+ ```
+
+### Usage
+
+Once configured, all models are available through the unified backend. Use standard model names in your pipelines:
+
+```plx
+[pipe.example]
+type = "PipeLLM"
+llm = { llm_handle = "claude-4-sonnet", temperature = 0.7 }
+# Model automatically routed through Pipelex Inference
+```
+
+## Inference Backends
+
+Backends represent LLM service providers. Each backend is configured with its endpoint and authentication details.
+
+### Backend Configuration
+
+Configure backends in `.pipelex/inference/backends.toml`:
+
+```toml
+[openai]
+enabled = true
+api_key = "${OPENAI_API_KEY}"
+
+[anthropic]
+enabled = true
+api_key = "${ANTHROPIC_API_KEY}"
+
+[azure_openai]
+enabled = true
+endpoint = "${AZURE_API_BASE}"
+api_key = "${AZURE_API_KEY}"
+api_version = "${AZURE_API_VERSION}"
+```
+
+### Model Specifications
+
+Each backend has its own model specification file in `.pipelex/inference/backends/`:
+
+```toml
+# openai.toml
+[models.gpt-4o-mini]
+model_id = "gpt-4o-mini"
+sdk = "openai"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.00015, output = 0.0006 }
+max_tokens = 128000
+max_prompt_images = 100
+
+[models.gpt-4-turbo]
+model_id = "gpt-4-turbo"
+sdk = "openai"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.01, output = 0.03 }
+max_tokens = 128000
+```
+
+## Routing Profiles
+
+Routing profiles determine which backend handles specific models. Configure them in `.pipelex/inference/routing_profiles.toml`:
+
+```toml
+# Which profile to use
+active = "all_pipelex"
+
+[profiles.all_pipelex]
+description = "Use Pipelex Inference backend for all models"
+default = "pipelex_inference"
+
+[profiles.mixed]
+description = "Route models to their native providers"
+default = "openai"
+
+[profiles.mixed.routes]
+"claude-*" = "anthropic"
+"gemini-*" = "vertexai"
+"mistral-*" = "mistral"
+"gpt-*" = "openai"
+```
+
+The routing system supports:
+- Exact matches
+- Wildcard patterns (prefix: `pattern*`, suffix: `*pattern`, contains: `*pattern*`)
+- Default fallback
+
+## Model Deck
+
+The Model Deck is the unified configuration hub for all LLM-related settings.
+
+### Aliases
+
+Define user-friendly names that map to model names in `.pipelex/inference/deck/base_deck.toml`:
+
+```toml
+[aliases]
+best-claude = "claude-4.1-opus"
+best-gemini = "gemini-2.5-pro"
+best-mistral = "mistral-large"
+base-gpt = "gpt-5"
+
+# Aliases can also define fallback chains
+llm_to_engineer = ["claude-4.1-opus", "gemini-2.5-pro"]
+```
+
+### LLM Presets
+
+Presets combine model selection with optimized parameters for specific tasks:
+
+```toml
+[llm_presets]
+# General purpose presets
+cheap_llm_for_text = { llm_handle = "gpt-4o-mini", temperature = 0.5 }
+cheap_llm_for_object = { llm_handle = "gpt-4o-mini", temperature = 0.5 }
+
+# Task-specific presets
+llm_for_creative_writing = { llm_handle = "claude-4-sonnet", temperature = 0.9 }
+llm_to_extract = { llm_handle = "claude-4-sonnet", temperature = 0.1 }
+llm_to_reason = { llm_handle = "o4-mini", temperature = 1, max_tokens = "auto" }
+```
+
+### Default Choices
+
+Set default models for different types of generation:
+
+```toml
+[llm_choice_defaults]
+for_text = "cheap_llm_for_text"
+for_object = "cheap_llm_for_object"
+```
+
+## Customization
+
+### Local Overrides
+
+Use `.pipelex/inference/deck/overrides.toml` for project-specific customizations:
+
+```toml
+# Override specific presets
+[llm_presets]
+llm_to_extract = { llm_handle = "gpt-4o-mini", temperature = 0.2 }
+
+# Add custom aliases
+[aliases]
+my_custom_model = "claude-3-sonnet"
+```
+
+### Adding New Backends
+
+To add a new backend:
+
+1. Add backend configuration to `backends.toml`
+2. Create model specification file in `backends/` directory
+3. Update routing profile if needed
+
+## Loading Process
+
+The system loads configurations in this order:
+
+1. **Load Backends**: Read `backends.toml` to get enabled backends
+2. **Load Model Specs**: For each backend, load model specifications
+3. **Load Routing Profiles**: Read routing rules and identify active profile
+4. **Build Model Deck**:
+ - Apply routing rules to determine backend for each model
+ - Load aliases and presets from deck files
+ - Apply overrides
+5. **Finalization**: Validate complete configuration
+
+## Error Handling
+
+Common error types:
+
+- `ModelDeckNotFoundError`: Missing LLM deck configuration files
+- `ModelsManagerError`: Issues with model management
+- `LLMHandleNotFoundError`: Referenced model or alias not found
+- `LLMPresetNotFoundError`: Referenced preset not found
+
+## Best Practices
+
+1. **Backend Management**:
+ - Keep API keys in environment variables
+ - Enable only the backends you need
+ - Document custom backend configurations
+
+2. **Model Routing**:
+ - Use specific routing profiles for different environments
+ - Test routing rules before production deployment
+ - Consider cost implications when routing models
+
+3. **Presets and Aliases**:
+ - Create task-specific presets for consistency
+ - Use meaningful alias names
+ - Document custom presets and their use cases
+
+4. **Customization**:
+ - Use `overrides.toml` for project-specific settings
+ - Keep base configurations unchanged
+ - Version control your custom configurations
+
+## Validation
+
+Validate your inference configuration:
+
+```bash
+# Validate all configurations
+pipelex validate all
+
+# Check specific configuration
+pipelex validate inference
+```
+
+The validation checks:
+- Backend connectivity
+- Model availability
+- Routing consistency
+- Preset validity
+- Alias resolution
diff --git a/docs/pages/configuration/config-technical/library-config.md b/docs/pages/configuration/config-technical/library-config.md
index 7b7bd9a2c..54f85a6ea 100644
--- a/docs/pages/configuration/config-technical/library-config.md
+++ b/docs/pages/configuration/config-technical/library-config.md
@@ -1,6 +1,6 @@
# Library Configuration
-The Library Configuration manages how Pipelex organizes, loads, and handles libraries in your project. Libraries in Pipelex include pipelines, LLM integrations, LLM decks, and templates.
+The Library Configuration manages how Pipelex organizes, loads, and handles libraries in your project. Libraries in Pipelex include pipelines and templates.
## Directory Structure
@@ -16,8 +16,6 @@ pipelex_libraries/ # Exported library root
├── pipelines/ # Pipeline definitions
│ ├── base_library/ # Base pipelines from Pipelex
│ └── your_pipelines/ # Your custom pipelines
-├── llm_integrations/ # LLM integration configurations
-├── llm_deck/ # LLM model configurations
└── templates/ # Template files
```
@@ -59,29 +57,23 @@ This will:
When exporting libraries to your project:
- Use `overwrite=True` to force update existing files
-- Preserve custom overrides in `llm_deck/overrides.toml`
- Maintain directory structure and initialization files
## Validation
The library manager performs several validation steps:
-1. **LLM Deck Validation**:
-
- - Ensures LLM configurations are complete
- - Validates model settings
-
-2. **Concept Library Validation**:
+1. **Concept Library Validation**:
- Checks concept relationships
- Validates concept definitions
-3. **Pipe Library Validation**:
+2. **Pipe Library Validation**:
- Verifies pipe configurations
- Checks domain relationships
-4. **Domain Library Validation**:
+3. **Domain Library Validation**:
- Ensures domain completeness
- Validates domain relationships
@@ -94,7 +86,6 @@ The library system includes specific error types:
- `LibraryParsingError`: For PLX parsing issues
- `ConceptLibraryError`: For concept-related issues
- `PipeLibraryError`: For pipe-related issues
-- `LLMDeckNotFoundError`: For missing LLM configurations
## Best Practices
@@ -112,6 +103,5 @@ The library system includes specific error types:
3. **Customization**:
- - Use `overrides.toml` for local LLM settings
- Keep custom pipelines separate from base library
- Document domain-specific configurations
diff --git a/docs/pages/quick-start/index.md b/docs/pages/quick-start/index.md
index a06cb5b79..cd2f05ac6 100644
--- a/docs/pages/quick-start/index.md
+++ b/docs/pages/quick-start/index.md
@@ -6,6 +6,47 @@ You can **find more powerful examples** in the [Cookbook Examples](../cookbook-e
---
+## Setting up API Keys
+
+Before you can make LLM calls with Pipelex, you need to configure API keys. You have two options:
+
+### Option 1: Use Pipelex Inference (Recommended for Getting Started)
+
+Get **free access** to all well-known commercial and open-source LLMs with a single API key:
+
+1. **Join our Discord community to get your free Pipelex Inference key** (no credit card required, limited time offer)
+ - Visit [https://go.pipelex.com/discord](https://go.pipelex.com/discord) to join
+ - Request your API key in the appropriate channel once you're in
+
+2. **Set up your environment**:
+ ```bash
+ # Create a .env file in your project root
+ echo "PIPELEX_INFERENCE_API_KEY=your-key-here" > .env
+ ```
+
+With Pipelex Inference, you get instant access to models from OpenAI, Anthropic, Google, Mistral, and more - all through a single API key!
+
+### Option 2: Use Your Own API Keys
+
+If you already have API keys from LLM providers, you can use them directly:
+
+```bash
+# Add to your .env file
+OPENAI_API_KEY=your-openai-key
+ANTHROPIC_API_KEY=your-anthropic-key
+GOOGLE_API_KEY=your-google-key
+# Or for Azure OpenAI
+AZURE_API_KEY=your-azure-key
+AZURE_API_BASE=your-azure-endpoint
+# Or for AWS Bedrock
+AWS_ACCESS_KEY_ID=your-aws-key
+AWS_SECRET_ACCESS_KEY=your-aws-secret
+```
+
+For complete configuration details, see the [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) documentation.
+
+---
+
## Your first LLM call with Pipelex
Let's start by running your very first LLM call using Pipelex.
@@ -87,12 +128,12 @@ llm = "llm_for_creative_writing"
prompt_template = """You are a book writer. Your task is to create a character.
Think of it and then output the character description."""
-# The llm deck above is defined in `pipelex_libraries/llm_deck/base_llm_deck.toml` as:
+# The llm deck above is defined in `.pipelex/inference/deck/base_deck.toml` as:
# llm_for_creative_writing = { llm_handle = "best-claude", temperature = 0.9 }
# it's a base preset that we provide. you can add your own presets, too.
```
-💡 We have a lot of [LLM presets available by default](https://github.com/Pipelex/pipelex/tree/main/pipelex/libraries/llm_deck/base_llm_deck.toml).
+💡 We have a lot of [LLM presets available by default](https://github.com/Pipelex/pipelex/tree/main/.pipelex/inference/deck/base_deck.toml).
Make sure you have credentials for the underlying LLM provider (and added your API key to the `.env`) and select the one you want!
Learn more about LLM presets, LLM handles and LLM deck in our [LLM Configuration Guide](../build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md)
diff --git a/mkdocs.yml b/mkdocs.yml
index 0bb589d3a..42c129323 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -126,7 +126,7 @@ nav:
- Optimize Cost & Quality: pages/build-reliable-ai-workflows-with-pipelex/configure-ai-llm-to-optimize-workflows.md
- LLM Structured Generation: pages/build-reliable-ai-workflows-with-pipelex/llm-structured-generation-config.md
- LLM Prompting Style: pages/build-reliable-ai-workflows-with-pipelex/adapt-to-llm-prompting-style-openai-anthropic-mistral.md
- - LLM Providers & Models: pages/build-reliable-ai-workflows-with-pipelex/ai-plugins-for-multi-llm-workflows.md
+ - LLM Providers & Models: pages/configuration/config-technical/inference-backend-config.md
- Configuration:
- Overview: pages/configuration/index.md
- Pipeline Validation Configuration:
@@ -140,6 +140,7 @@ nav:
- Technical Configuration:
- AWS: pages/configuration/config-technical/aws-config.md
- Cogt: pages/configuration/config-technical/cogt-config.md
+ - Inference Backend: pages/configuration/config-technical/inference-backend-config.md
- Library: pages/configuration/config-technical/library-config.md
- Feature: pages/configuration/config-advanced/feature-config.md
- Development:
@@ -157,10 +158,8 @@ nav:
- Storage Provider: pages/advanced-customization/storage-provider-injection.md
- Template Provider: pages/advanced-customization/template-provider-injection.md
- Content Generator: pages/advanced-customization/content-generator-injection.md
- - LLM Model Provider: pages/advanced-customization/llm-model-provider-injection.md
- Pipe Router: pages/advanced-customization/pipe-router-injection.md
- Plugin Manager: pages/advanced-customization/plugin-manager-injection.md
- - Inference Manager: pages/advanced-customization/inference-manager-injection.md
- License: license.md
- Contributing: contributing.md
- Code of Conduct: CODE_OF_CONDUCT.md
diff --git a/pipelex/cli/commands/common.py b/pipelex/cli/commands/common.py
index f4c948780..103f0b42a 100644
--- a/pipelex/cli/commands/common.py
+++ b/pipelex/cli/commands/common.py
@@ -5,9 +5,6 @@ from typing import Final
REQUIRED_PIPELEX_SUBDIRS: Final[list[str]] = [
"pipelines",
- "llm_deck",
- "llm_integrations",
- "plugins",
"templates",
]
diff --git a/pipelex/cli/commands/init_cmd.py b/pipelex/cli/commands/init_cmd.py
index 5fdf6fe06..84b6fbfbb 100644
--- a/pipelex/cli/commands/init_cmd.py
+++ b/pipelex/cli/commands/init_cmd.py
@@ -2,7 +2,7 @@ from __future__ import annotations
import os
import shutil
-from typing import Annotated
+from typing import Annotated, List
import typer
@@ -30,18 +30,50 @@ def do_init_libraries(directory: str = ".", overwrite: bool = False) -> None:
def do_init_config(reset: bool = False) -> None:
"""Initialize pipelex configuration in the current directory."""
- pipelex_template_path = os.path.join(config_manager.pipelex_root_dir, "pipelex_template.toml")
- target_config_path = os.path.join(config_manager.local_root_dir, "pipelex.toml")
+ config_template_dir = os.path.join(config_manager.pipelex_root_dir, "config_template")
+ target_config_dir = config_manager.pipelex_config_dir
- if os.path.exists(target_config_path) and not reset:
- typer.echo("Warning: pipelex.toml already exists. Use --reset to force creation.")
- return
+ os.makedirs(target_config_dir, exist_ok=True)
try:
- shutil.copy2(pipelex_template_path, target_config_path)
- typer.echo(f"Created pipelex.toml at {target_config_path}")
+ copied_files: List[str] = []
+ existing_files: List[str] = []
+
+ def copy_directory_structure(src_dir: str, dst_dir: str, relative_path: str = "") -> None:
+ """Recursively copy directory structure, handling existing files."""
+ for item in os.listdir(src_dir):
+ src_item = os.path.join(src_dir, item)
+ dst_item = os.path.join(dst_dir, item)
+ relative_item = os.path.join(relative_path, item) if relative_path else item
+
+ if os.path.isdir(src_item):
+ os.makedirs(dst_item, exist_ok=True)
+ copy_directory_structure(src_item, dst_item, relative_item)
+ else:
+ if os.path.exists(dst_item) and not reset:
+ existing_files.append(relative_item)
+ else:
+ shutil.copy2(src_item, dst_item)
+ copied_files.append(relative_item)
+
+ copy_directory_structure(config_template_dir, target_config_dir)
+
+ # Report results
+ if copied_files:
+ typer.echo(f"✅ Copied {len(copied_files)} files to {target_config_dir}:")
+ for file in sorted(copied_files):
+ typer.echo(f" • {file}")
+
+ if existing_files:
+ typer.echo(f"ℹ️ Skipped {len(existing_files)} existing files (use --reset to overwrite):")
+ for file in sorted(existing_files):
+ typer.echo(f" • {file}")
+
+ if not copied_files and not existing_files:
+ typer.echo(f"✅ Configuration directory {target_config_dir} is already up to date")
+
except Exception as exc:
- raise PipelexCLIError(f"Failed to create pipelex.toml: {exc}") from exc
+ raise PipelexCLIError(f"Failed to initialize configuration: {exc}") from exc
# Typer group for init commands
diff --git a/pipelex/cogt/config_cogt.py b/pipelex/cogt/config_cogt.py
index aef295bb3..3418d7bfe 100644
--- a/pipelex/cogt/config_cogt.py
+++ b/pipelex/cogt/config_cogt.py
@@ -1,12 +1,13 @@
-from typing import Dict, List, Optional, cast
+from typing import List
-from pydantic import Field, field_validator
+from pydantic import Field
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job_components import ImggJobConfig, ImggJobParams, ImggJobParamsDefaults
from pipelex.cogt.llm.llm_job_components import LLMJobConfig
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.tools.config.models import ConfigModel
+from pipelex.plugins.fal.fal_config import FalConfig
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.misc.file_utils import find_files_in_dir
class OcrConfig(ConfigModel):
@@ -20,6 +21,7 @@ class ImggConfig(ConfigModel):
imgg_job_config: ImggJobConfig
imgg_param_defaults: ImggJobParamsDefaults
imgg_handles: List[str]
+ fal_config: FalConfig
def make_default_imgg_job_params(self) -> ImggJobParams:
return self.imgg_param_defaults.make_imgg_job_params()
@@ -30,29 +32,11 @@ class InstructorConfig(ConfigModel):
class LLMConfig(ConfigModel):
- preferred_platforms: Dict[str, LLMPlatform]
instructor_config: InstructorConfig
llm_job_config: LLMJobConfig
default_max_images: int
- @field_validator("preferred_platforms", mode="before")
- def validate_preferred_platforms_enums(cls, value: Dict[str, str]) -> Dict[str, LLMPlatform]:
- """
- Transform what we got for preferred_platforms (Dict[str, str]) into what the field requires: Dict[str, LLMPlatform]
- """
- the_dict = cast(
- Dict[str, LLMPlatform],
- ConfigModel.transform_dict_str_to_enum(
- input_dict=value,
- value_enum_cls=LLMPlatform,
- ),
- )
- return the_dict
-
- def get_preferred_platform(self, llm_name: str) -> Optional[LLMPlatform]:
- return self.preferred_platforms.get(llm_name)
-
class InferenceManagerConfig(ConfigModel):
is_auto_setup_preset_llm: bool
@@ -60,7 +44,36 @@ class InferenceManagerConfig(ConfigModel):
is_auto_setup_preset_ocr: bool
+class InferenceConfig(ConfigModel):
+ inference_config_path: str
+
+ @property
+ def routing_profile_library_path(self) -> str:
+ return f"{self.inference_config_path}/routing_profiles.toml"
+
+ @property
+ def backends_library_path(self) -> str:
+ return f"{self.inference_config_path}/backends.toml"
+
+ def model_specs_path(self, backend_name: str) -> str:
+ return f"{self.inference_config_path}/backends/{backend_name}.toml"
+
+ def get_llm_deck_paths(self) -> List[str]:
+ """Get all LLM deck TOML file paths sorted alphabetically."""
+ llm_deck_paths = [
+ str(path)
+ for path in find_files_in_dir(
+ dir_path=f"{self.inference_config_path}/deck",
+ pattern="*.toml",
+ is_recursive=True,
+ )
+ ]
+ llm_deck_paths.sort()
+ return llm_deck_paths
+
+
class Cogt(ConfigModel):
+ inference_config: InferenceConfig
inference_manager_config: InferenceManagerConfig
llm_config: LLMConfig
imgg_config: ImggConfig
diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.py
index a0b9b632f..e28e3045f 100644
--- a/pipelex/cogt/content_generation/assignment_models.py
+++ b/pipelex/cogt/content_generation/assignment_models.py
@@ -9,9 +9,9 @@ from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job_components import ImggJobConfig, ImggJobParams
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
from pipelex.cogt.llm.llm_job_components import LLMJobParams
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.py
index 625ea93ed..e4f6a722b 100644
--- a/pipelex/cogt/content_generation/content_generator.py
+++ b/pipelex/cogt/content_generation/content_generator.py
@@ -21,10 +21,10 @@ from pipelex.cogt.image.generated_image import GeneratedImage
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job_components import ImggJobConfig, ImggJobParams
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index 579486994..d2917302f 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -9,9 +9,9 @@ from pipelex.cogt.image.generated_image import GeneratedImage
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job_components import ImggJobConfig, ImggJobParams
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.py
index 2f1cfd4cf..dc516bc1f 100644
--- a/pipelex/cogt/content_generation/content_generator_protocol.py
+++ b/pipelex/cogt/content_generation/content_generator_protocol.py
@@ -5,9 +5,9 @@ from pipelex.cogt.image.generated_image import GeneratedImage
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job_components import ImggJobConfig, ImggJobParams
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index eb8aeac14..c56100840 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -31,14 +31,6 @@ class LLMWorkerError(CogtError):
pass
-class LLMEngineParameterError(CogtError):
- pass
-
-
-class LLMSDKError(CogtError):
- pass
-
-
class LLMPresetNotFoundError(CogtError):
pass
@@ -47,7 +39,7 @@ class LLMSettingsValidationError(CogtError):
pass
-class LLMDeckValidatonError(CogtError):
+class ModelDeckValidatonError(CogtError):
pass
@@ -55,10 +47,6 @@ class LLMHandleNotFoundError(CogtError):
pass
-class LLMModelProviderError(CogtError):
- pass
-
-
class LLMModelPlatformError(ValueError, CogtError):
pass
@@ -142,3 +130,39 @@ class MissingPluginError(CogtError):
class OcrCapabilityError(CogtError):
pass
+
+
+class RoutingProfileLibraryNotFoundError(CogtError):
+ pass
+
+
+class RoutingProfileLibraryError(CogtError):
+ pass
+
+
+class InferenceModelSpecError(CogtError):
+ pass
+
+
+class InferenceBackendError(CogtError):
+ pass
+
+
+class InferenceBackendCredentialsError(CogtError):
+ pass
+
+
+class InferenceBackendLibraryError(CogtError):
+ pass
+
+
+class RoutingProfileError(CogtError):
+ pass
+
+
+class ModelsManagerError(CogtError):
+ pass
+
+
+class ModelDeckNotFoundError(CogtError):
+ pass
diff --git a/pipelex/cogt/imgg/imgg_job_components.py b/pipelex/cogt/imgg/imgg_job_components.py
index 1987672f3..b5f23e7ae 100644
--- a/pipelex/cogt/imgg/imgg_job_components.py
+++ b/pipelex/cogt/imgg/imgg_job_components.py
@@ -2,7 +2,7 @@ from typing import Literal, Optional, Union
from pydantic import BaseModel, Field
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.types import StrEnum
diff --git a/pipelex/cogt/imgg/imgg_worker_factory.py b/pipelex/cogt/imgg/imgg_worker_factory.py
index 29bc8b72b..f78f2a2a7 100644
--- a/pipelex/cogt/imgg/imgg_worker_factory.py
+++ b/pipelex/cogt/imgg/imgg_worker_factory.py
@@ -4,10 +4,9 @@ from pipelex.cogt.exceptions import CogtError, MissingDependencyError
from pipelex.cogt.imgg.imgg_engine import ImggEngine
from pipelex.cogt.imgg.imgg_platform import ImggPlatform
from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.hub import get_plugin_manager, get_secret
+from pipelex.hub import get_models_manager, get_plugin_manager, get_secret
from pipelex.plugins.openai.openai_imgg_worker import OpenAIImggWorker
-from pipelex.plugins.plugin_sdk_registry import PluginSdkHandle
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
@@ -22,7 +21,8 @@ class ImggWorkerFactory:
imgg_engine: ImggEngine,
reporting_delegate: Optional[ReportingProtocol] = None,
) -> ImggWorkerAbstract:
- imgg_sdk_handle = PluginSdkHandle.get_for_imgg_engine(imgg_platform=imgg_engine.imgg_platform)
+ img_gen_plugin = Plugin.make_for_imgg_engine(imgg_platform=imgg_engine.imgg_platform)
+ backend = get_models_manager().get_required_inference_backend("openai")
plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
imgg_worker: ImggWorkerAbstract
match imgg_engine.imgg_platform:
@@ -41,11 +41,9 @@ class ImggWorkerFactory:
from pipelex.plugins.fal.fal_imgg_worker import FalImggWorker
- imgg_sdk_instance = plugin_sdk_registry.get_imgg_sdk_instance(
- imgg_sdk_handle=imgg_sdk_handle
- ) or plugin_sdk_registry.set_imgg_sdk_instance(
- imgg_sdk_handle=imgg_sdk_handle,
- imgg_sdk_instance=FalAsyncClient(key=fal_api_key),
+ imgg_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=img_gen_plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=img_gen_plugin,
+ sdk_instance=FalAsyncClient(key=fal_api_key),
)
imgg_worker = FalImggWorker(
@@ -56,11 +54,12 @@ class ImggWorkerFactory:
case ImggPlatform.OPENAI:
from pipelex.plugins.openai.openai_factory import OpenAIFactory
- imgg_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=imgg_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=imgg_sdk_handle,
- llm_sdk_instance=OpenAIFactory.make_openai_client(llm_platform=LLMPlatform.OPENAI),
+ imgg_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=img_gen_plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=img_gen_plugin,
+ sdk_instance=OpenAIFactory.make_openai_client(
+ plugin=img_gen_plugin,
+ backend=backend,
+ ),
)
imgg_worker = OpenAIImggWorker(
diff --git a/pipelex/cogt/inference/inference_manager.py b/pipelex/cogt/inference/inference_manager.py
index 26e36fbea..4dc2e648c 100644
--- a/pipelex/cogt/inference/inference_manager.py
+++ b/pipelex/cogt/inference/inference_manager.py
@@ -8,16 +8,15 @@ from pipelex.cogt.imgg.imgg_engine_factory import ImggEngineFactory
from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
from pipelex.cogt.imgg.imgg_worker_factory import ImggWorkerFactory
from pipelex.cogt.inference.inference_manager_protocol import InferenceManagerProtocol
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
-from pipelex.cogt.llm.llm_models.llm_engine_factory import LLMEngineFactory
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.llm.llm_worker_factory import LLMWorkerFactory
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_engine_factory import OcrEngineFactory
from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
from pipelex.cogt.ocr.ocr_worker_factory import OcrWorkerFactory
from pipelex.config import get_config
-from pipelex.hub import get_llm_deck, get_report_delegate
+from pipelex.hub import get_models_manager, get_report_delegate
class InferenceManager(InferenceManagerProtocol):
@@ -61,24 +60,13 @@ class InferenceManager(InferenceManagerProtocol):
# Setup LLM Workers
####################################################################################################
- @override
- def setup_llm_workers(self):
- log.verbose("Setting up LLM Workers...")
- llm_handle_to_llm_engine_blueprint = get_llm_deck().llm_handles
- log.verbose(f"{len(llm_handle_to_llm_engine_blueprint)} LLM engine_cards found")
- for llm_handle, llm_engine_blueprint in llm_handle_to_llm_engine_blueprint.items():
- self._setup_one_internal_llm_worker(llm_engine_blueprint=llm_engine_blueprint, llm_handle=llm_handle)
- log.verbose(f"Setup LLM worker for '{llm_handle}' on {llm_engine_blueprint.llm_platform_choice}")
- log.debug("Done setting up LLM Workers (async)")
-
def _setup_one_internal_llm_worker(
self,
- llm_engine_blueprint: LLMEngineBlueprint,
+ inference_model: InferenceModelSpec,
llm_handle: str,
) -> LLMWorkerInternalAbstract:
- llm_engine = LLMEngineFactory.make_llm_engine(llm_engine_blueprint=llm_engine_blueprint)
llm_worker = LLMWorkerFactory.make_llm_worker(
- llm_engine=llm_engine,
+ inference_model=inference_model,
reporting_delegate=get_report_delegate(),
)
self.llm_workers[llm_handle] = llm_worker
@@ -93,9 +81,9 @@ class InferenceManager(InferenceManagerProtocol):
f"No LLM worker for '{llm_handle}', set it up or enable cogt.inference_manager_config.is_auto_setup_preset_llm"
)
- llm_engine_blueprint = get_llm_deck().get_llm_engine_blueprint(llm_handle=llm_handle)
+ inference_model = get_models_manager().get_inference_model(llm_handle=llm_handle)
llm_worker = self._setup_one_internal_llm_worker(
- llm_engine_blueprint=llm_engine_blueprint,
+ inference_model=inference_model,
llm_handle=llm_handle,
)
diff --git a/pipelex/cogt/inference/inference_manager_protocol.py b/pipelex/cogt/inference/inference_manager_protocol.py
index 3a9a598a7..848f6cbca 100644
--- a/pipelex/cogt/inference/inference_manager_protocol.py
+++ b/pipelex/cogt/inference/inference_manager_protocol.py
@@ -17,8 +17,6 @@ class InferenceManagerProtocol(Protocol):
# LLM Workers
####################################################################################################
- def setup_llm_workers(self): ...
-
def get_llm_worker(self, llm_handle: str) -> LLMWorkerAbstract: ...
def set_llm_worker_from_external_plugin(
diff --git a/pipelex/cogt/llm/llm_job.py b/pipelex/cogt/llm/llm_job.py
index 74cddce94..49217293f 100644
--- a/pipelex/cogt/llm/llm_job.py
+++ b/pipelex/cogt/llm/llm_job.py
@@ -4,9 +4,9 @@ from typing_extensions import override
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
class LLMJob(InferenceJobAbstract):
@@ -23,7 +23,7 @@ class LLMJob(InferenceJobAbstract):
def validate_before_execution(self):
self.llm_prompt.validate_before_execution()
- def llm_job_before_start(self, llm_engine: LLMEngine):
+ def llm_job_before_start(self, inference_model: InferenceModelSpec):
# Reset metadata
self.job_metadata.started_at = datetime.now()
@@ -33,7 +33,9 @@ class LLMJob(InferenceJobAbstract):
# Reset info
self.job_report.llm_tokens_usage = LLMTokensUsage(
job_metadata=self.job_metadata,
- llm_engine=llm_engine,
+ inference_model_name=inference_model.name,
+ unit_costs=inference_model.costs,
+ inference_model_id=inference_model.model_id,
nb_tokens_by_category={},
)
diff --git a/pipelex/cogt/llm/llm_models/__init__.py b/pipelex/cogt/llm/llm_models/__init__.py
deleted file mode 100644
index 8b1378917..000000000
--- a/pipelex/cogt/llm/llm_models/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/pipelex/cogt/llm/llm_models/llm_deck.py b/pipelex/cogt/llm/llm_models/llm_deck.py
deleted file mode 100644
index 0be1ae503..000000000
--- a/pipelex/cogt/llm/llm_models/llm_deck.py
+++ /dev/null
@@ -1,161 +0,0 @@
-from typing import Dict, Optional, Union
-
-from pydantic import field_validator, model_validator
-from typing_extensions import Self, override
-
-from pipelex.cogt.exceptions import LLMDeckValidatonError, LLMHandleNotFoundError, LLMPresetNotFoundError, LLMSettingsValidationError
-from pipelex.cogt.llm.llm_models.llm_deck_abstract import LLMDeckAbstract
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
-from pipelex.cogt.llm.llm_models.llm_family import LLMFamily
-from pipelex.cogt.llm.llm_models.llm_model import LLMModel
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingOrPresetId
-from pipelex.hub import get_llm_models_provider
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.exceptions import ConfigValidationError
-
-LLM_PRESET_DISABLED = "disabled"
-
-
-class LLMDeck(LLMDeckAbstract, ConfigModel):
- ############################################################
- # LLMDeckAbstract overrides
- ############################################################
-
- @override
- def check_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId, is_disabled_allowed: bool = False):
- if isinstance(llm_setting_or_preset_id, LLMSetting):
- return
- preset_id: str = llm_setting_or_preset_id
- if preset_id in self.llm_presets:
- return
- if preset_id == LLM_PRESET_DISABLED and is_disabled_allowed:
- return
- raise LLMPresetNotFoundError(f"llm preset id '{preset_id}' not found in deck")
-
- @override
- def get_llm_engine_blueprint(self, llm_handle: str) -> LLMEngineBlueprint:
- the_engine_blueprint = self.llm_handles.get(llm_handle)
- if not the_engine_blueprint:
- raise LLMHandleNotFoundError(f"LLM Engine blueprint for llm_handle '{llm_handle}' not found in deck's engine blueprints")
- return the_engine_blueprint
-
- @override
- def get_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId) -> LLMSetting:
- if isinstance(llm_setting_or_preset_id, LLMSetting):
- return llm_setting_or_preset_id
- else:
- # it's a preset id
- the_llm_preset = self.llm_presets.get(llm_setting_or_preset_id)
- if not the_llm_preset:
- raise LLMPresetNotFoundError(f"LLM preset '{llm_setting_or_preset_id}' not found in deck")
- return the_llm_preset
-
- @override
- def find_llm_model(self, llm_handle: str) -> LLMModel:
- llm_models_provider = get_llm_models_provider()
- llm_engine_blueprint = self.llm_handles[llm_handle]
- llm_model = llm_models_provider.get_llm_model(
- llm_name=llm_engine_blueprint.llm_name,
- llm_version=llm_engine_blueprint.llm_version,
- llm_platform_choice=llm_engine_blueprint.llm_platform_choice,
- )
- return llm_model
-
- @override
- def find_optional_llm_model(self, llm_handle: str) -> Optional[LLMModel]:
- llm_models_provider = get_llm_models_provider()
- llm_engine_blueprint = self.llm_handles.get(llm_handle)
- if not llm_engine_blueprint:
- return None
- llm_model = llm_models_provider.get_optional_llm_model(
- llm_name=llm_engine_blueprint.llm_name,
- llm_version=llm_engine_blueprint.llm_version,
- llm_platform_choice=llm_engine_blueprint.llm_platform_choice,
- )
- return llm_model
-
- @override
- @classmethod
- def final_validate(cls, deck: Self):
- for llm_preset_id, llm_setting in deck.llm_presets.items():
- llm_model = deck.find_llm_model(llm_handle=llm_setting.llm_handle)
- try:
- cls._validate_llm_setting(llm_setting=llm_setting, llm_model=llm_model)
- except ConfigValidationError as exc:
- raise LLMDeckValidatonError(f"LLM preset '{llm_preset_id}' is invalid: {exc}")
-
- ############################################################
- #### LLMDeck concrete validations
- ############################################################
-
- @classmethod
- def _validate_llm_setting(cls, llm_setting: LLMSetting, llm_model: LLMModel):
- if llm_model.max_tokens is not None and (llm_setting_max_tokens := llm_setting.max_tokens):
- if llm_setting_max_tokens > llm_model.max_tokens:
- raise LLMSettingsValidationError(
- (
- f"LLM setting '{llm_setting.llm_handle}' has a max_tokens of {llm_setting_max_tokens}, "
- f"which is greater than the model's max_tokens of {llm_model.max_tokens}"
- )
- )
- match llm_model.llm_family:
- case LLMFamily.O_SERIES:
- if llm_setting.temperature != 1:
- raise LLMSettingsValidationError(
- f"O-series LLMs like '{llm_setting.llm_handle}' must have a temperature of 1, not {llm_setting.temperature}"
- )
- case _:
- pass
-
- @field_validator("llm_handles", mode="before")
- @classmethod
- def validate_llm_handles(cls, value: Dict[str, Union[LLMEngineBlueprint, str]]) -> Dict[str, LLMEngineBlueprint]:
- the_dict: Dict[str, LLMEngineBlueprint] = {}
- for llm_handle, llm_engine_spec in value.items():
- if isinstance(llm_engine_spec, str):
- the_dict[llm_handle] = LLMEngineBlueprint(llm_name=llm_engine_spec)
- elif isinstance(llm_engine_spec, dict):
- the_dict[llm_handle] = LLMEngineBlueprint.model_validate(llm_engine_spec)
- else:
- raise ConfigValidationError(f"Invalid LLM engine blueprint for '{llm_handle}' is a {type(llm_engine_spec)}: {llm_engine_spec}")
- return the_dict
-
- @field_validator("llm_choice_defaults", mode="after")
- @classmethod
- def validate_llm_choice_defaults(cls, llm_choice_defaults: LLMSettingChoices) -> LLMSettingChoices:
- if llm_choice_defaults.for_text is None:
- raise ConfigValidationError("llm_choice_defaults.for_text cannot be None")
- if llm_choice_defaults.for_object is None:
- raise ConfigValidationError("llm_choice_defaults.for_object cannot be None")
- return llm_choice_defaults
-
- @field_validator("llm_choice_overrides", mode="after")
- @classmethod
- def validate_llm_choice_overrides(cls, value: LLMSettingChoices) -> LLMSettingChoices:
- if value.for_text == LLM_PRESET_DISABLED:
- value.for_text = None
- if value.for_object == LLM_PRESET_DISABLED:
- value.for_object = None
- return value
-
- def add_llm_name_as_handle_with_defaults(self, llm_name: str):
- if llm_name in self.llm_handles:
- raise ConfigValidationError(f"LLM engine blueprint for '{llm_name}' is already defined in llm deck's llm_handles")
- # TODO: sort the defaults by llm family
- self.llm_handles[llm_name] = LLMEngineBlueprint(llm_name=llm_name)
-
- def validate_llm_presets(self) -> Self:
- for llm_preset_id, llm_setting in self.llm_presets.items():
- if llm_setting.llm_handle not in self.llm_handles:
- raise LLMHandleNotFoundError(f"llm_handle '{llm_setting.llm_handle}' for llm_preset '{llm_preset_id}' not found in deck")
- return self
-
- @model_validator(mode="after")
- def validate_llm_setting_overrides(self) -> Self:
- self._validate_llm_choices(llm_choices=self.llm_choice_overrides)
- return self
-
- def _validate_llm_choices(self, llm_choices: LLMSettingChoices):
- for llm_setting in llm_choices.list_used_presets():
- self.check_llm_setting(llm_setting_or_preset_id=llm_setting)
- return
diff --git a/pipelex/cogt/llm/llm_models/llm_deck_abstract.py b/pipelex/cogt/llm/llm_models/llm_deck_abstract.py
deleted file mode 100644
index 2fa6835f1..000000000
--- a/pipelex/cogt/llm/llm_models/llm_deck_abstract.py
+++ /dev/null
@@ -1,45 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import Dict, List, Optional
-
-from pydantic import Field
-from typing_extensions import Self
-
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
-from pipelex.cogt.llm.llm_models.llm_model import LLMModel
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingChoicesDefaults, LLMSettingOrPresetId
-
-
-class LLMDeckAbstract(ABC):
- llm_handles: Dict[str, LLMEngineBlueprint] = Field(default_factory=dict)
- llm_external_handles: List[str] = Field(default_factory=list)
- llm_presets: Dict[str, LLMSetting] = Field(default_factory=dict)
- llm_choice_defaults: LLMSettingChoicesDefaults
- llm_choice_overrides: LLMSettingChoices = LLMSettingChoices(
- for_text=None,
- for_object=None,
- )
-
- @abstractmethod
- def check_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId, is_disabled_allowed: bool = False):
- pass
-
- @abstractmethod
- def get_llm_engine_blueprint(self, llm_handle: str) -> LLMEngineBlueprint:
- pass
-
- @abstractmethod
- def get_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId) -> LLMSetting:
- pass
-
- @abstractmethod
- def find_llm_model(self, llm_handle: str) -> LLMModel:
- pass
-
- @abstractmethod
- def find_optional_llm_model(self, llm_handle: str) -> Optional[LLMModel]:
- pass
-
- @classmethod
- @abstractmethod
- def final_validate(cls, deck: Self):
- pass
diff --git a/pipelex/cogt/llm/llm_models/llm_engine.py b/pipelex/cogt/llm/llm_models/llm_engine.py
deleted file mode 100644
index 383614ce2..000000000
--- a/pipelex/cogt/llm/llm_models/llm_engine.py
+++ /dev/null
@@ -1,38 +0,0 @@
-from pydantic import BaseModel, model_validator
-from typing_extensions import Self
-
-from pipelex.cogt.exceptions import LLMModelPlatformError
-from pipelex.cogt.llm.llm_models.llm_model import LLMModel
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-
-
-class LLMEngine(BaseModel):
- llm_platform: LLMPlatform
- llm_model: LLMModel
-
- @model_validator(mode="after")
- def validate_llm_platform(self) -> Self:
- if self.llm_platform not in self.llm_model.platform_llm_id:
- raise LLMModelPlatformError(
- f"Missing id for {self.llm_model.name_and_version} on platform '{self.llm_platform}', provided ids: {self.llm_model.platform_llm_id}"
- )
- return self
-
- @property
- def llm_id(self) -> str:
- return self.llm_model.platform_llm_id[self.llm_platform]
-
- @property
- def desc(self) -> str:
- return f"""llm name: '{self.llm_model.llm_name}':
-version: '{self.llm_model.version}'
-platform: '{self.llm_platform}'
-platform llm id: '{self.llm_id}'"""
-
- @property
- def tag(self) -> str:
- return f"{self.llm_platform}:{self.llm_model.llm_name}:{self.llm_model.version}:id-[{self.llm_id}]"
-
- @property
- def is_gen_object_supported(self) -> bool:
- return self.llm_model.is_gen_object_supported and self.llm_platform.is_gen_object_supported
diff --git a/pipelex/cogt/llm/llm_models/llm_engine_blueprint.py b/pipelex/cogt/llm/llm_models/llm_engine_blueprint.py
deleted file mode 100644
index 84c747d51..000000000
--- a/pipelex/cogt/llm/llm_models/llm_engine_blueprint.py
+++ /dev/null
@@ -1,18 +0,0 @@
-from pydantic import field_validator
-
-from pipelex.cogt.llm.llm_models.llm_model import LATEST_VERSION_NAME
-from pipelex.cogt.llm.llm_models.llm_platform import DEFAULT_PLATFORM_INDICATOR, LLMPlatform, LLMPlatformChoice
-from pipelex.tools.config.models import ConfigModel
-
-
-class LLMEngineBlueprint(ConfigModel):
- llm_name: str
- llm_version: str = LATEST_VERSION_NAME
- llm_platform_choice: LLMPlatformChoice = "default"
-
- @field_validator("llm_platform_choice", mode="before")
- def validate_llm_platform_choice(cls, value: str) -> LLMPlatformChoice:
- if value == DEFAULT_PLATFORM_INDICATOR:
- # Make it a literal "default"
- return "default"
- return LLMPlatform(value)
diff --git a/pipelex/cogt/llm/llm_models/llm_engine_factory.py b/pipelex/cogt/llm/llm_models/llm_engine_factory.py
deleted file mode 100644
index a23cc6ff5..000000000
--- a/pipelex/cogt/llm/llm_models/llm_engine_factory.py
+++ /dev/null
@@ -1,39 +0,0 @@
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
-from pipelex.cogt.llm.llm_models.llm_platform import DEFAULT_PLATFORM_INDICATOR, LLMPlatform
-from pipelex.config import get_config
-from pipelex.hub import get_llm_models_provider
-
-
-class LLMEngineFactory:
- @classmethod
- def make_llm_engine(
- cls,
- llm_engine_blueprint: LLMEngineBlueprint,
- ) -> LLMEngine:
- """
- Create an instance of LLMEngine based on the parameters provided through the LLMEngineCard.
-
- Args:
- llm_engine_blueprint: LLMEngineBlueprint
-
- """
- llm_platform_choice = llm_engine_blueprint.llm_platform_choice
- llm_model = get_llm_models_provider().get_llm_model(
- llm_name=llm_engine_blueprint.llm_name,
- llm_version=llm_engine_blueprint.llm_version,
- llm_platform_choice=llm_platform_choice,
- )
-
- llm_platform: LLMPlatform
- if isinstance(llm_platform_choice, LLMPlatform):
- llm_platform = llm_platform_choice
- else:
- assert llm_platform_choice == DEFAULT_PLATFORM_INDICATOR
- llm_config = get_config().cogt.llm_config
- llm_platform = llm_config.get_preferred_platform(llm_name=llm_engine_blueprint.llm_name) or llm_model.default_platform
-
- return LLMEngine(
- llm_platform=llm_platform,
- llm_model=llm_model,
- )
diff --git a/pipelex/cogt/llm/llm_models/llm_family.py b/pipelex/cogt/llm/llm_models/llm_family.py
deleted file mode 100644
index d927352f6..000000000
--- a/pipelex/cogt/llm/llm_models/llm_family.py
+++ /dev/null
@@ -1,165 +0,0 @@
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.llm_models.llm_prompting_target import LLMPromptingTarget
-from pipelex.types import StrEnum
-
-
-class LLMCreator(StrEnum):
- AMAZON = "Amazon"
- ANTHROPIC = "Anthropic"
- DEEPSEEK = "DeepSeek"
- GOOGLE = "Google"
- OPENAI = "OpenAI"
- META = "Meta"
- MISTRAL = "Mistral"
- PERPLEXITY = "Perplexity"
- XAI = "XAI"
- ALIBABA = "Alibaba"
-
- @property
- def prompting_target(self) -> LLMPromptingTarget:
- match self:
- case LLMCreator.OPENAI:
- return LLMPromptingTarget.OPENAI
- case LLMCreator.ANTHROPIC:
- return LLMPromptingTarget.ANTHROPIC
- case LLMCreator.MISTRAL:
- return LLMPromptingTarget.MISTRAL
- case LLMCreator.GOOGLE:
- return LLMPromptingTarget.GEMINI
- case LLMCreator.AMAZON | LLMCreator.PERPLEXITY | LLMCreator.META | LLMCreator.DEEPSEEK | LLMCreator.XAI | LLMCreator.ALIBABA:
- return LLMPromptingTarget.OPENAI
-
- @property
- def console_color(self) -> str:
- match self:
- case LLMCreator.OPENAI:
- return LLMPlatform.OPENAI.console_color
- case LLMCreator.ANTHROPIC:
- return LLMPlatform.ANTHROPIC.console_color
- case LLMCreator.MISTRAL:
- return LLMPlatform.MISTRAL.console_color
- case LLMCreator.META:
- return "dodger_blue2"
- case LLMCreator.GOOGLE:
- return "blue"
- case LLMCreator.AMAZON:
- return "orange"
- case LLMCreator.PERPLEXITY:
- return "purple"
- case LLMCreator.DEEPSEEK:
- return "red"
- case LLMCreator.XAI:
- return "green"
- case LLMCreator.ALIBABA:
- return "yellow"
-
-
-class LLMFamily(StrEnum):
- GPT_3 = "gpt-3"
- GPT_3_5 = "gpt-3.5"
- GPT_4 = "gpt-4"
- GPT_4O = "gpt-4o"
- GPT_4_5 = "gpt-4.5"
- GPT_4_1 = "gpt-4.1"
- O_SERIES = "o"
- GPT_5 = "gpt-5"
- GPT_5_CHAT = "gpt-5-chat"
-
- CLAUDE_3 = "claude-3"
- CLAUDE_3_5 = "claude-3.5"
- CLAUDE_3_7 = "claude-3.7"
- CLAUDE_4 = "claude-4"
- CLAUDE_4_1 = "claude-4.1"
-
- MISTRAL_7B = "mistral-7b"
- MISTRAL_8X7B = "mistral-8x7b"
- MISTRAL_LARGE = "mistral-large"
- MISTRAL_SMALL = "mistral-small"
- MISTRAL_MEDIUM = "mistral-medium"
- MISTRAL_CODESTRAL = "mistral-codestral"
- MINISTRAL = "ministral"
- PIXTRAL = "pixtral"
-
- LLAMA_3 = "llama-3"
- LLAMA_3_1 = "llama-3.1"
-
- GEMINI = "gemini"
-
- BEDROCK_MISTRAL_LARGE = "bedrock-mistral-large"
- BEDROCK_ANTHROPIC_CLAUDE = "bedrock-anthropic-claude"
- BEDROCK_META_LLAMA_3 = "bedrock-meta-llama-3"
- BEDROCK_AMAZON_NOVA = "bedrock-amazon-nova"
-
- PERPLEXITY_SEARCH = "perplexity-search"
- PERPLEXITY_RESEARCH = "perplexity-research"
- PERPLEXITY_REASONING = "perplexity-reasoning"
- PERPLEXITY_DEEPSEEK = "perplexity-deepseek"
-
- GROK_3 = "grok-3"
-
- CUSTOM_LLAMA_4 = "custom-llama-4"
- CUSTOM_GEMMA_3 = "custom-gemma-3"
- CUSTOM_MISTRAL_SMALL_3_1 = "custom-mistral-small3.1"
- CUSTOM_QWEN_3 = "custom-qwen3"
- CUSTOM_BLACKBOXAI = "custom-blackboxai"
-
- PIPELEX_INFERENCE = "pipelex-inference"
-
- @property
- def creator(self) -> LLMCreator:
- match self:
- case (
- LLMFamily.GPT_4
- | LLMFamily.GPT_3_5
- | LLMFamily.GPT_3
- | LLMFamily.O_SERIES
- | LLMFamily.GPT_4_5
- | LLMFamily.GPT_4_1
- | LLMFamily.GPT_4O
- | LLMFamily.GPT_5
- | LLMFamily.GPT_5_CHAT
- ):
- return LLMCreator.OPENAI
- case LLMFamily.CLAUDE_3 | LLMFamily.CLAUDE_3_5 | LLMFamily.CLAUDE_3_7 | LLMFamily.CLAUDE_4 | LLMFamily.CLAUDE_4_1:
- return LLMCreator.ANTHROPIC
- case LLMFamily.BEDROCK_ANTHROPIC_CLAUDE:
- return LLMCreator.ANTHROPIC
- case (
- LLMFamily.MISTRAL_7B
- | LLMFamily.MISTRAL_8X7B
- | LLMFamily.MISTRAL_LARGE
- | LLMFamily.MISTRAL_SMALL
- | LLMFamily.MISTRAL_MEDIUM
- | LLMFamily.MISTRAL_CODESTRAL
- | LLMFamily.MINISTRAL
- | LLMFamily.PIXTRAL
- | LLMFamily.CUSTOM_MISTRAL_SMALL_3_1
- ):
- return LLMCreator.MISTRAL
- case LLMFamily.BEDROCK_MISTRAL_LARGE:
- return LLMCreator.MISTRAL
- case LLMFamily.LLAMA_3 | LLMFamily.LLAMA_3_1:
- return LLMCreator.META
- case LLMFamily.BEDROCK_META_LLAMA_3:
- return LLMCreator.META
- case LLMFamily.GEMINI:
- return LLMCreator.GOOGLE
- case LLMFamily.BEDROCK_AMAZON_NOVA:
- return LLMCreator.AMAZON
- case LLMFamily.PERPLEXITY_SEARCH | LLMFamily.PERPLEXITY_REASONING | LLMFamily.PERPLEXITY_RESEARCH:
- return LLMCreator.PERPLEXITY
- case LLMFamily.PERPLEXITY_DEEPSEEK:
- return LLMCreator.DEEPSEEK
- case LLMFamily.GROK_3:
- return LLMCreator.XAI
- case LLMFamily.CUSTOM_LLAMA_4 | LLMFamily.CUSTOM_GEMMA_3:
- return LLMCreator.META
- case LLMFamily.CUSTOM_QWEN_3:
- return LLMCreator.ALIBABA
- case LLMFamily.CUSTOM_BLACKBOXAI | LLMFamily.PIPELEX_INFERENCE:
- # TODO: this doesn't make sense for multi-model providers, needs full refactor
- return LLMCreator.OPENAI
-
- @property
- def prompting_target(self) -> LLMPromptingTarget:
- return self.creator.prompting_target
diff --git a/pipelex/cogt/llm/llm_models/llm_model.py b/pipelex/cogt/llm/llm_models/llm_model.py
deleted file mode 100644
index 8f7a924a7..000000000
--- a/pipelex/cogt/llm/llm_models/llm_model.py
+++ /dev/null
@@ -1,59 +0,0 @@
-from typing import Dict, Optional, Set
-
-from pydantic import BaseModel, Field, model_validator
-from typing_extensions import Self
-
-from pipelex.cogt.exceptions import LLMModelDefinitionError
-from pipelex.cogt.llm.llm_models.llm_family import LLMFamily
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.token_category import TokenCostsByCategoryDict
-
-LATEST_VERSION_NAME = "latest"
-DISABLED_PLATFORM_NAME = "disabled"
-
-
-class LLMModel(BaseModel):
- default_platform: LLMPlatform
- llm_family: LLMFamily
- llm_name: str
- version: str
- is_gen_object_supported: bool
- is_vision_supported: bool = False
- # TODO: add skill regarding live online data access
- cost_per_million_tokens_usd: Optional[TokenCostsByCategoryDict] = None
- platform_llm_id: Dict[LLMPlatform, str] = Field(..., min_length=1)
- max_tokens: Optional[int] = None
-
- max_prompt_images: Optional[int] = Field(None, ge=0)
-
- @model_validator(mode="after")
- def check_vision_and_nb_images(self) -> Self:
- if self.is_vision_supported and self.max_prompt_images and self.max_prompt_images < 1:
- raise LLMModelDefinitionError("If vision is True, max_prompt_images must be >= 1 or not set")
- if not self.is_vision_supported and self.max_prompt_images and self.max_prompt_images > 0:
- raise LLMModelDefinitionError("If vision is False, max_prompt_images must be 0 or not set")
- return self
-
- @property
- def name_and_version(self) -> str:
- return f"{self.llm_name}/{self.version}"
-
- @property
- def name_and_version_and_platform(self) -> str:
- return f"{self.llm_name}/{self.version}/{self.default_platform}"
-
- @property
- def desc(self) -> str:
- return f"{self.llm_family}/{self.llm_name}/{self.version}/{self.default_platform}"
-
- @property
- def is_system_prompt_supported(self) -> bool:
- return self.llm_family != LLMFamily.O_SERIES or self.llm_name == "o1"
-
- @property
- def supported_platforms(self) -> Set[LLMPlatform]:
- return set(self.platform_llm_id.keys())
-
- @property
- def enabled_platforms(self) -> Set[LLMPlatform]:
- return set([key for key, value in self.platform_llm_id.items() if value != DISABLED_PLATFORM_NAME])
diff --git a/pipelex/cogt/llm/llm_models/llm_model_library.py b/pipelex/cogt/llm/llm_models/llm_model_library.py
deleted file mode 100644
index 118c0685f..000000000
--- a/pipelex/cogt/llm/llm_models/llm_model_library.py
+++ /dev/null
@@ -1,163 +0,0 @@
-import os
-from typing import Any, ClassVar, Dict, List, Optional
-
-from pydantic import Field, RootModel
-from typing_extensions import override
-
-from pipelex import log
-from pipelex.cogt.exceptions import LLMModelPlatformError, LLMModelProviderError
-from pipelex.cogt.llm.llm_models.llm_model import LATEST_VERSION_NAME, LLMModel
-from pipelex.cogt.llm.llm_models.llm_model_provider_abstract import LLMModelProviderAbstract
-from pipelex.cogt.llm.llm_models.llm_platform import DEFAULT_PLATFORM_INDICATOR, LLMPlatform, LLMPlatformChoice
-from pipelex.libraries.library_config import LibraryConfig
-from pipelex.tools.exceptions import ToolException
-from pipelex.tools.misc.toml_utils import load_toml_from_path
-
-# Some types to clarify what dictionaries we're loading from the TOML files
-LLMModelDict = Dict[str, Any]
-LLMModelVersion = str
-LLMModelGroup = Dict[LLMModelVersion, LLMModelDict]
-LLMName = str
-LLMModelFamilyGroups = Dict[LLMName, LLMModelGroup]
-LLMFamilyName = str
-LLMModelLibraryDict = Dict[LLMFamilyName, LLMModelFamilyGroups]
-
-
-class LLMModelLibraryError(ToolException):
- pass
-
-
-LLMModelLibraryRoot = List[LLMModel]
-
-
-class LLMModelLibrary(LLMModelProviderAbstract, RootModel[LLMModelLibraryRoot]):
- root: LLMModelLibraryRoot = Field(default_factory=list)
- library_config: ClassVar[LibraryConfig]
-
- @classmethod
- def make_empty(cls, config_dir_path: str) -> "LLMModelLibrary":
- cls.library_config = LibraryConfig(config_dir_path=config_dir_path)
- return cls()
-
- @override
- def setup(self):
- llm_model_library_dict = self.load_llm_model_library_dict()
- for llm_family, llm_family_groups in llm_model_library_dict.items():
- for llm_group_name, llm_group in llm_family_groups.items():
- for version, llm_model_dict in llm_group.items():
- complete_llm_model_dict = {
- **llm_model_dict,
- "llm_family": llm_family,
- "llm_name": llm_group_name,
- "version": version,
- }
- if "default_platform" not in llm_model_dict:
- complete_llm_model_dict["default_platform"] = list(llm_model_dict["platform_llm_id"].keys())[0]
- llm_model = LLMModel.model_validate(complete_llm_model_dict)
- self.root.append(llm_model)
-
- log.debug(f"Loaded {len(self.root)} llm models")
-
- @override
- def teardown(self):
- self.root = []
-
- @property
- @override
- def desc(self) -> str:
- return "TOML-based llm model provider with models in memory"
-
- @classmethod
- def load_llm_model_library_dict(cls) -> LLMModelLibraryDict:
- libraries_path = cls.library_config.llm_integrations_dir_path
- if not os.path.exists(libraries_path):
- raise LLMModelLibraryError(f"LLM model library path `{libraries_path}` not found. Please run `pipelex init-libraries` to create it.")
- llm_library: LLMModelLibraryDict = {}
- library_file_names = os.listdir(libraries_path)
- library_file_names = [file_name for file_name in library_file_names if file_name.endswith(".toml")]
- for library_file_name in sorted(library_file_names):
- library_path = os.path.join(libraries_path, library_file_name)
- try:
- llm_families: LLMModelLibraryDict = load_toml_from_path(library_path)
- except UnicodeDecodeError as exc:
- raise LLMModelLibraryError(f"Failed to load LLM model library file '{library_file_name}' from '{library_path}': {exc}")
- llm_library.update(llm_families)
- return llm_library
-
- @override
- def get_all_llm_models(self) -> List[LLMModel]:
- return self.root
-
- @override
- def get_llm_model(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> LLMModel:
- llm_model = self.get_optional_llm_model(llm_name=llm_name, llm_version=llm_version, llm_platform_choice=llm_platform_choice)
- if not llm_model:
- raise LLMModelProviderError(f"Model '{llm_name}' version '{llm_version}' not found")
- else:
- return llm_model
-
- @override
- def get_optional_llm_model(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> Optional[LLMModel]:
- if llm_version == LATEST_VERSION_NAME:
- return self._get_optional_llm_model_latest_version(llm_name=llm_name, llm_platform_choice=llm_platform_choice)
-
- found_llm_models = [llm_model for llm_model in self.root if llm_model.llm_name == llm_name and llm_model.version == llm_version]
- if not found_llm_models:
- return None
- if len(found_llm_models) > 1:
- raise LLMModelProviderError(f"Model '{llm_name}' version '{llm_version}' has multiple instances in the library")
- llm_model = found_llm_models[0]
-
- if isinstance(llm_platform_choice, LLMPlatform):
- llm_platform: LLMPlatform = llm_platform_choice
- if llm_platform not in llm_model.platform_llm_id:
- raise LLMModelPlatformError(f"Missing llm id for LLM model {llm_model.name_and_version}'s chosen platform='{llm_platform}'")
- return llm_model
-
- def _get_optional_llm_model_latest_version(
- self,
- llm_name: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> Optional[LLMModel]:
- found_llm_models = [llm_model for llm_model in self.root if llm_model.llm_name == llm_name]
- if not found_llm_models:
- return None
- elif len(found_llm_models) == 1:
- # only one so it is the latest we've got
- llm_model = found_llm_models[0]
- elif chosen_model := next((model for model in found_llm_models if model.version == LATEST_VERSION_NAME), None):
- # we've got a version named 'latest' so we use it
- llm_model = chosen_model
- elif isinstance(llm_platform_choice, LLMPlatform):
- llm_platform: LLMPlatform = llm_platform_choice
- llm_models_with_platform = [model for model in found_llm_models if llm_platform in model.platform_llm_id]
- if not llm_models_with_platform:
- raise LLMModelProviderError(f"Model '{llm_name}' has no version for platform '{llm_platform}'")
- llm_model = max(llm_models_with_platform, key=lambda llm_model: llm_model.version)
- else:
- assert llm_platform_choice == DEFAULT_PLATFORM_INDICATOR
- llm_model = max(found_llm_models, key=lambda llm_model: llm_model.version)
- return llm_model
-
- def _get_llm_model_latest_version(
- self,
- llm_name: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> LLMModel:
- llm_model_latest_version = self.get_optional_llm_model(
- llm_name=llm_name, llm_version=LATEST_VERSION_NAME, llm_platform_choice=llm_platform_choice
- )
- if llm_model_latest_version is not None:
- return llm_model_latest_version
- else:
- raise LLMModelProviderError(f"Could not find model named '{llm_name}'")
diff --git a/pipelex/cogt/llm/llm_models/llm_model_provider_abstract.py b/pipelex/cogt/llm/llm_models/llm_model_provider_abstract.py
deleted file mode 100644
index 403d906cd..000000000
--- a/pipelex/cogt/llm/llm_models/llm_model_provider_abstract.py
+++ /dev/null
@@ -1,42 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import List, Optional
-
-from pipelex.cogt.llm.llm_models.llm_model import LLMModel
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatformChoice
-
-
-class LLMModelProviderAbstract(ABC):
- @property
- @abstractmethod
- def desc(self) -> str:
- pass
-
- @abstractmethod
- def get_all_llm_models(self) -> List[LLMModel]:
- pass
-
- @abstractmethod
- def get_llm_model(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> LLMModel:
- pass
-
- @abstractmethod
- def get_optional_llm_model(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ) -> Optional[LLMModel]:
- pass
-
- @abstractmethod
- def setup(self):
- pass
-
- @abstractmethod
- def teardown(self):
- pass
diff --git a/pipelex/cogt/llm/llm_models/llm_platform.py b/pipelex/cogt/llm/llm_models/llm_platform.py
deleted file mode 100644
index e3b1548ba..000000000
--- a/pipelex/cogt/llm/llm_models/llm_platform.py
+++ /dev/null
@@ -1,86 +0,0 @@
-from typing import Literal, Union
-
-from pipelex.types import StrEnum
-
-DEFAULT_PLATFORM_INDICATOR = "default"
-
-
-class LLMPlatform(StrEnum):
- # Naming scheme:
- # single name => it's the name of the platform and we use the standard sdk
- # double name => <PROVIDER>_<SDK> => we use this sdk with this provider
- # TODO: make this separation more "automatic"
- ANTHROPIC = "anthropic"
- AZURE_OPENAI = "azure_openai"
- BEDROCK = "bedrock"
- BEDROCK_ANTHROPIC = "bedrock_anthropic"
- MISTRAL = "mistral"
- OPENAI = "openai"
- PERPLEXITY = "perplexity"
- VERTEXAI = "vertexai"
- XAI = "xai"
- CUSTOM_LLM = "custom_llm"
-
- @staticmethod
- def list_openai_related() -> list["LLMPlatform"]:
- return [
- LLMPlatform.OPENAI,
- LLMPlatform.AZURE_OPENAI,
- LLMPlatform.VERTEXAI,
- LLMPlatform.PERPLEXITY,
- LLMPlatform.XAI,
- LLMPlatform.CUSTOM_LLM,
- ]
-
- @staticmethod
- def list_anthropic_related() -> list["LLMPlatform"]:
- return [LLMPlatform.ANTHROPIC, LLMPlatform.BEDROCK_ANTHROPIC]
-
- @property
- def is_openai_related(self) -> bool:
- return self in LLMPlatform.list_openai_related()
-
- @property
- def is_gen_object_supported(self) -> bool:
- match self:
- case (
- LLMPlatform.OPENAI
- | LLMPlatform.AZURE_OPENAI
- | LLMPlatform.VERTEXAI
- | LLMPlatform.PERPLEXITY
- | LLMPlatform.ANTHROPIC
- | LLMPlatform.MISTRAL
- | LLMPlatform.BEDROCK_ANTHROPIC
- | LLMPlatform.XAI
- | LLMPlatform.CUSTOM_LLM
- ):
- return True
- case LLMPlatform.BEDROCK:
- return False
-
- @property
- def console_color(self) -> str:
- match self:
- case LLMPlatform.OPENAI:
- return "deep_pink1"
- case LLMPlatform.AZURE_OPENAI:
- return "turquoise2"
- case LLMPlatform.ANTHROPIC:
- return "dark_orange"
- case LLMPlatform.MISTRAL:
- return "gold3"
- case LLMPlatform.BEDROCK:
- return "medium_purple1"
- case LLMPlatform.BEDROCK_ANTHROPIC:
- return "spring_green2"
- case LLMPlatform.PERPLEXITY:
- return "gray"
- case LLMPlatform.VERTEXAI:
- return "deep_sky_blue"
- case LLMPlatform.XAI:
- return "green"
- case LLMPlatform.CUSTOM_LLM:
- return "white"
-
-
-LLMPlatformChoice = Union[LLMPlatform, Literal["default"]]
diff --git a/pipelex/cogt/llm/llm_report.py b/pipelex/cogt/llm/llm_report.py
index b373086cb..5febe49a4 100644
--- a/pipelex/cogt/llm/llm_report.py
+++ b/pipelex/cogt/llm/llm_report.py
@@ -2,20 +2,14 @@ from typing import Any, Dict
from pydantic import BaseModel
-from pipelex import log
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_family import LLMFamily
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory, TokenCostsByCategoryDict
+from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.types import StrEnum
class LLMTokenCostReportField(StrEnum):
LLM_NAME = "llm_name"
- LLM_PLATFORM = "llm_platform"
- LLM_FAMILY = "llm_family"
- VERSION = "version"
PLATFORM_LLM_ID = "platform_llm_id"
NB_TOKENS_INPUT = "nb_tokens_input"
NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
@@ -32,30 +26,24 @@ class LLMTokenCostReportField(StrEnum):
return f"nb_tokens_{token_category}"
@staticmethod
- def report_field_for_cost_by_category(token_category: TokenCategory) -> str:
+ def report_field_for_cost_by_category(token_category: CostCategory) -> str:
return f"cost_{token_category}"
class LLMTokenCostReport(BaseModel):
job_metadata: JobMetadata
- llm_name: str
- llm_platform: LLMPlatform
- llm_family: LLMFamily
- version: str
+ inference_model_name: str
platform_llm_id: str
nb_tokens_by_category: NbTokensByCategoryDict
- costs_by_token_category: TokenCostsByCategoryDict
+ costs_by_token_category: CostsByCategoryDict
def as_flat_dictionary(self) -> Dict[str, Any]:
the_dict: Dict[str, Any] = {}
dict_for_job_metadata = self.job_metadata.model_dump(serialize_as_any=True)
the_dict.update(dict_for_job_metadata)
dict_for_llm: Dict[str, Any] = {
- LLMTokenCostReportField.LLM_NAME: self.llm_name,
- LLMTokenCostReportField.LLM_PLATFORM: self.llm_platform,
- LLMTokenCostReportField.LLM_FAMILY: self.llm_family,
- LLMTokenCostReportField.VERSION: self.version,
+ LLMTokenCostReportField.LLM_NAME: self.inference_model_name,
LLMTokenCostReportField.PLATFORM_LLM_ID: self.platform_llm_id,
}
the_dict.update(dict_for_llm)
@@ -72,48 +60,9 @@ class LLMTokenCostReport(BaseModel):
return the_dict
-def model_cost_per_token(llm_engine: LLMEngine, token_type: TokenCategory) -> float:
- # cost_per_million_tokens_usd should be missing only for models that we run on our own GPUs
- if not llm_engine.llm_model.cost_per_million_tokens_usd:
- log.warning(f"cost_per_million_tokens_usd is not set for model {llm_engine.llm_model.name_and_version}")
- return 0.0
- # all token types are not used for all models
- if token_type == TokenCategory.INPUT_CACHED:
- if cost_per_million_tokens := llm_engine.llm_model.cost_per_million_tokens_usd.get(TokenCategory.INPUT_CACHED):
- return cost_per_million_tokens / 1000000
- elif cost_per_million_tokens := llm_engine.llm_model.cost_per_million_tokens_usd.get(TokenCategory.INPUT):
- # according to openai docs, cached input tokens are discounted 50%
- return 0.5 * cost_per_million_tokens / 1000000
- else:
- model = llm_engine.llm_model.name_and_version
- log.warning(f"cost is not set for model {model} neither for {TokenCategory.INPUT} nor {TokenCategory.INPUT_CACHED}")
- return 0.0
- elif token_type == TokenCategory.INPUT_NON_CACHED:
- return model_cost_per_token(llm_engine=llm_engine, token_type=TokenCategory.INPUT)
- elif cost_per_million_tokens := llm_engine.llm_model.cost_per_million_tokens_usd.get(token_type):
- return cost_per_million_tokens / 1000000
- else:
- return 0.0
-
-
class LLMTokensUsage(BaseModel):
job_metadata: JobMetadata
- llm_engine: LLMEngine
+ inference_model_name: str
+ unit_costs: CostsByCategoryDict
+ inference_model_id: str
nb_tokens_by_category: NbTokensByCategoryDict
-
- def compute_cost_report(self) -> LLMTokenCostReport:
- costs_by_token_category: TokenCostsByCategoryDict = {
- token_type: (model_cost_per_token(llm_engine=self.llm_engine, token_type=token_type) * nb_tokens)
- for token_type, nb_tokens in self.nb_tokens_by_category.items()
- }
- token_cost_report = LLMTokenCostReport(
- job_metadata=self.job_metadata,
- llm_name=self.llm_engine.llm_model.llm_name,
- llm_platform=self.llm_engine.llm_platform,
- llm_family=self.llm_engine.llm_model.llm_family,
- version=self.llm_engine.llm_model.version,
- platform_llm_id=self.llm_engine.llm_id,
- nb_tokens_by_category=self.nb_tokens_by_category,
- costs_by_token_category=costs_by_token_category,
- )
- return token_cost_report
diff --git a/pipelex/cogt/llm/llm_models/llm_setting.py b/pipelex/cogt/llm/llm_setting.py
similarity index 92%
rename from pipelex/cogt/llm/llm_models/llm_setting.py
rename to pipelex/cogt/llm/llm_setting.py
index 7fba7e689..9384de8b3 100644
--- a/pipelex/cogt/llm/llm_models/llm_setting.py
+++ b/pipelex/cogt/llm/llm_setting.py
@@ -5,8 +5,8 @@ from typing_extensions import Self
from pipelex.cogt.exceptions import LLMSettingsValidationError
from pipelex.cogt.llm.llm_job_components import LLMJobParams
-from pipelex.cogt.llm.llm_models.llm_prompting_target import LLMPromptingTarget
-from pipelex.tools.config.models import ConfigModel
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.tools.exceptions import ConfigValidationError
@@ -14,7 +14,7 @@ class LLMSetting(ConfigModel):
llm_handle: str
temperature: float = Field(..., ge=0, le=1)
max_tokens: Optional[int] = None
- prompting_target: Optional[LLMPromptingTarget] = Field(default=None, strict=False)
+ prompting_target: Optional[PromptingTarget] = Field(default=None, strict=False)
@field_validator("max_tokens", mode="before")
@classmethod
diff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.py
index 90f9bbbd4..1d95cc3c0 100644
--- a/pipelex/cogt/llm/llm_worker_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_abstract.py
@@ -31,7 +31,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
@property
@override
def desc(self) -> str:
- return "LLM Worker • if you're using an external plugin, override this method to describe your llm worker"
+ return "If you're using an external plugin, override this method to describe your llm worker"
@property
@abstractmethod
diff --git a/pipelex/cogt/llm/llm_worker_factory.py b/pipelex/cogt/llm/llm_worker_factory.py
index 55ef8d4a9..bb1065f2c 100644
--- a/pipelex/cogt/llm/llm_worker_factory.py
+++ b/pipelex/cogt/llm/llm_worker_factory.py
@@ -1,27 +1,27 @@
from typing import Optional
from pipelex.cogt.exceptions import MissingDependencyError
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.config import get_config
-from pipelex.hub import get_plugin_manager
-from pipelex.plugins.plugin_sdk_registry import PluginSdkHandle
+from pipelex.hub import get_models_manager, get_plugin_manager
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.reporting.reporting_protocol import ReportingProtocol
class LLMWorkerFactory:
@staticmethod
def make_llm_worker(
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
reporting_delegate: Optional[ReportingProtocol] = None,
) -> LLMWorkerInternalAbstract:
- llm_sdk_handle = PluginSdkHandle.get_for_llm_platform(llm_platform=llm_engine.llm_platform)
+ plugin = Plugin.make_for_inference_model(inference_model=inference_model)
+ backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)
plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
llm_worker: LLMWorkerInternalAbstract
- match llm_engine.llm_platform:
- case LLMPlatform.OPENAI | LLMPlatform.AZURE_OPENAI | LLMPlatform.PERPLEXITY | LLMPlatform.XAI:
+ match plugin.sdk:
+ case "openai" | "azure_openai":
from pipelex.plugins.openai.openai_factory import OpenAIFactory
structure_method: Optional[StructureMethod] = None
@@ -30,59 +30,21 @@ class LLMWorkerFactory:
from pipelex.plugins.openai.openai_llm_worker import OpenAILLMWorker
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=OpenAIFactory.make_openai_client(llm_platform=llm_engine.llm_platform),
+ sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=plugin,
+ sdk_instance=OpenAIFactory.make_openai_client(
+ plugin=plugin,
+ backend=backend,
+ ),
)
llm_worker = OpenAILLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
+ sdk_instance=sdk_instance,
+ inference_model=inference_model,
structure_method=structure_method,
reporting_delegate=reporting_delegate,
)
- case LLMPlatform.VERTEXAI:
- try:
- import google.auth # noqa: F401
- except ImportError as exc:
- raise MissingDependencyError("google-auth-oauthlib", "google", "This dependency is required to connect to google.") from exc
-
- from pipelex.plugins.openai.openai_factory import OpenAIFactory
- from pipelex.plugins.openai.openai_llm_worker import OpenAILLMWorker
-
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=OpenAIFactory.make_openai_client(llm_platform=llm_engine.llm_platform),
- )
-
- llm_worker = OpenAILLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
- structure_method=StructureMethod.INSTRUCTOR_VERTEX_JSON,
- reporting_delegate=reporting_delegate,
- )
- case LLMPlatform.CUSTOM_LLM:
- from pipelex.plugins.openai.openai_factory import OpenAIFactory
- from pipelex.plugins.openai.openai_llm_worker import OpenAILLMWorker
-
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=OpenAIFactory.make_openai_client(llm_platform=llm_engine.llm_platform),
- )
-
- llm_worker = OpenAILLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
- structure_method=StructureMethod.INSTRUCTOR_OPENAI_STRUCTURED,
- reporting_delegate=reporting_delegate,
- )
- case LLMPlatform.ANTHROPIC | LLMPlatform.BEDROCK_ANTHROPIC:
+ case "anthropic" | "bedrock_anthropic":
try:
import anthropic # noqa: F401
except ImportError as exc:
@@ -99,20 +61,19 @@ class LLMWorkerFactory:
from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory
from pipelex.plugins.anthropic.anthropic_llm_worker import AnthropicLLMWorker
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=AnthropicFactory.make_anthropic_client(llm_platform=llm_engine.llm_platform),
+ sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=plugin,
+ sdk_instance=AnthropicFactory.make_anthropic_client(plugin=plugin, backend=backend),
)
llm_worker = AnthropicLLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
+ sdk_instance=sdk_instance,
+ extra_config=backend.extra_config,
+ inference_model=inference_model,
structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS,
reporting_delegate=reporting_delegate,
)
- case LLMPlatform.MISTRAL:
+ case "mistral":
try:
import mistralai # noqa: F401
except ImportError as exc:
@@ -129,20 +90,18 @@ class LLMWorkerFactory:
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.plugins.mistral.mistral_llm_worker import MistralLLMWorker
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=MistralFactory.make_mistral_client(),
+ sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=plugin,
+ sdk_instance=MistralFactory.make_mistral_client(backend=backend),
)
llm_worker = MistralLLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
+ sdk_instance=sdk_instance,
+ inference_model=inference_model,
structure_method=StructureMethod.INSTRUCTOR_MISTRAL_TOOLS,
reporting_delegate=reporting_delegate,
)
- case LLMPlatform.BEDROCK:
+ case "bedrock_boto3" | "bedrock_aioboto3":
try:
import aioboto3 # noqa: F401
import boto3 # noqa: F401
@@ -154,16 +113,16 @@ class LLMWorkerFactory:
from pipelex.plugins.bedrock.bedrock_factory import BedrockFactory
from pipelex.plugins.bedrock.bedrock_llm_worker import BedrockLLMWorker
- llm_sdk_instance = plugin_sdk_registry.get_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle
- ) or plugin_sdk_registry.set_llm_sdk_instance(
- llm_sdk_handle=llm_sdk_handle,
- llm_sdk_instance=BedrockFactory.make_bedrock_client(),
+ sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=plugin,
+ sdk_instance=BedrockFactory.make_bedrock_client(plugin=plugin, backend=backend),
)
llm_worker = BedrockLLMWorker(
- sdk_instance=llm_sdk_instance,
- llm_engine=llm_engine,
+ sdk_instance=sdk_instance,
+ inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
+ case _:
+ raise NotImplementedError(f"Plugin '{plugin}' is not supported")
return llm_worker
diff --git a/pipelex/cogt/llm/llm_worker_internal_abstract.py b/pipelex/cogt/llm/llm_worker_internal_abstract.py
index afc435c1d..38a8cf696 100644
--- a/pipelex/cogt/llm/llm_worker_internal_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_internal_abstract.py
@@ -4,16 +4,16 @@ from typing_extensions import override
from pipelex.cogt.exceptions import LLMCapabilityError
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.reporting.reporting_protocol import ReportingProtocol
class LLMWorkerInternalAbstract(LLMWorkerAbstract):
def __init__(
self,
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
structure_method: Optional[StructureMethod] = None,
reporting_delegate: Optional[ReportingProtocol] = None,
):
@@ -21,12 +21,12 @@ class LLMWorkerInternalAbstract(LLMWorkerAbstract):
Initialize the LLMWorker.
Args:
- llm_engine (LLMEngine): The LLM engine to be used by the worker.
+ inference_model (InferenceModelSpec): The inference model to be used by the worker.
structure_method (Optional[StructureMethod]): The structure method to be used by the worker.
reporting_delegate (Optional[ReportingProtocol]): An optional report delegate for reporting unit jobs.
"""
LLMWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)
- self.llm_engine = llm_engine
+ self.inference_model = inference_model
self.structure_method = structure_method
#########################################################
@@ -36,12 +36,12 @@ class LLMWorkerInternalAbstract(LLMWorkerAbstract):
@property
@override
def desc(self) -> str:
- return f"LLM-Worker:{self.llm_engine.tag}"
+ return f"LLM-Worker:{self.inference_model.tag}"
@property
@override
def is_gen_object_supported(self) -> bool:
- return self.llm_engine.is_gen_object_supported
+ return self.inference_model.is_gen_object_supported
@override
async def _before_job(
@@ -49,7 +49,7 @@ class LLMWorkerInternalAbstract(LLMWorkerAbstract):
llm_job: LLMJob,
):
await super()._before_job(llm_job=llm_job)
- llm_job.llm_job_before_start(llm_engine=self.llm_engine)
+ llm_job.llm_job_before_start(inference_model=self.inference_model)
@override
def _check_can_perform_job(self, llm_job: LLMJob):
@@ -58,10 +58,10 @@ class LLMWorkerInternalAbstract(LLMWorkerAbstract):
def _check_vision_support(self, llm_job: LLMJob):
if llm_job.llm_prompt.user_images:
- if not self.llm_engine.llm_model.is_vision_supported:
- raise LLMCapabilityError(f"LLM Engine '{self.llm_engine.tag}' does not support vision.")
+ if not self.inference_model.is_vision_supported:
+ raise LLMCapabilityError(f"LLM Engine '{self.inference_model.tag}' does not support vision.")
nb_images = len(llm_job.llm_prompt.user_images)
- max_prompt_images = self.llm_engine.llm_model.max_prompt_images or 5000
+ max_prompt_images = self.inference_model.max_prompt_images or 5000
if nb_images > max_prompt_images:
- raise LLMCapabilityError(f"LLM Engine '{self.llm_engine.tag}' does not accept that many images: {nb_images}.")
+ raise LLMCapabilityError(f"LLM Engine '{self.inference_model.tag}' does not accept that many images: {nb_images}.")
diff --git a/pipelex/cogt/model_backends/__init__.py b/pipelex/cogt/model_backends/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/cogt/model_backends/backend.py b/pipelex/cogt/model_backends/backend.py
new file mode 100644
index 000000000..38046a638
--- /dev/null
+++ b/pipelex/cogt/model_backends/backend.py
@@ -0,0 +1,27 @@
+from typing import Any, Dict, List, Optional
+
+from pydantic import Field
+
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.tools.config.config_model import ConfigModel
+
+
+class InferenceBackend(ConfigModel):
+ name: str
+ endpoint: Optional[str] = None
+ api_key: Optional[str] = None
+ extra_config: Dict[str, Any] = Field(default_factory=dict)
+ model_specs: Dict[str, InferenceModelSpec] = Field(default_factory=dict)
+
+ def list_model_names(self) -> List[str]:
+ """List the names of all models in the backend."""
+ return list(self.model_specs.keys())
+
+ def get_model_spec(self, model_name: str) -> Optional[InferenceModelSpec]:
+ """Get a model spec by name."""
+ model_spec = self.model_specs.get(model_name)
+ return model_spec
+
+ def get_extra_config(self, key: str) -> Optional[Any]:
+ """Get an extra config by key."""
+ return self.extra_config.get(key)
diff --git a/pipelex/cogt/model_backends/backend_factory.py b/pipelex/cogt/model_backends/backend_factory.py
new file mode 100644
index 000000000..8c26a9d34
--- /dev/null
+++ b/pipelex/cogt/model_backends/backend_factory.py
@@ -0,0 +1,42 @@
+from typing import Any, Dict, Optional
+
+from pydantic import Field
+
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.plugins.openai.vertexai_factory import VertexAIFactory
+from pipelex.tools.config.config_model import ConfigModel
+
+
+class InferenceBackendBlueprint(ConfigModel):
+ enabled: bool = True
+ endpoint: Optional[str] = None
+ api_key: Optional[str] = None
+ extra_config: Dict[str, Any] = Field(default_factory=dict)
+
+
+class InferenceBackendFactory:
+ @classmethod
+ def make_inference_backend(
+ cls,
+ name: str,
+ blueprint: InferenceBackendBlueprint,
+ extra_config: Dict[str, Any],
+ model_specs: Dict[str, InferenceModelSpec],
+ ) -> InferenceBackend:
+ endpoint = blueprint.endpoint
+ api_key = blueprint.api_key
+ # api_version = blueprint.api_version
+ # Deal with special authentication for some backends
+ match name:
+ case "vertexai":
+ endpoint, api_key = VertexAIFactory.make_endpoint_and_api_key(extra_config=extra_config)
+ case _:
+ pass
+ return InferenceBackend(
+ name=name,
+ endpoint=endpoint,
+ api_key=api_key,
+ extra_config=extra_config,
+ model_specs=model_specs,
+ )
diff --git a/pipelex/cogt/model_backends/backend_library.py b/pipelex/cogt/model_backends/backend_library.py
new file mode 100644
index 000000000..a4829a9e2
--- /dev/null
+++ b/pipelex/cogt/model_backends/backend_library.py
@@ -0,0 +1,118 @@
+from typing import Any, Dict, List, Optional, Set
+
+from pydantic import Field, RootModel, ValidationError
+from typing_extensions import Self
+
+from pipelex import log
+from pipelex.cogt.exceptions import InferenceBackendCredentialsError, InferenceBackendLibraryError, InferenceModelSpecError
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.backend_factory import InferenceBackendBlueprint, InferenceBackendFactory
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_factory import InferenceModelSpecBlueprint, InferenceModelSpecFactory
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
+from pipelex.config import get_config
+from pipelex.tools.misc.dict_utils import apply_to_strings_recursive
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+from pipelex.tools.secrets.secrets_utils import UnknownVarPrefixError, VarNotFoundError, substitute_vars
+
+InferenceBackendLibraryRoot = Dict[str, InferenceBackend]
+
+
+class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
+ root: InferenceBackendLibraryRoot = Field(default_factory=dict)
+
+ def reset(self):
+ self.root = {}
+
+ @classmethod
+ def make_empty(cls) -> Self:
+ return cls(root={})
+
+ def load(self):
+ backends_library_path = get_config().cogt.inference_config.backends_library_path
+ try:
+ backends_dict = load_toml_from_path(path=backends_library_path)
+ except (FileNotFoundError, InferenceBackendLibraryError) as exc:
+ raise InferenceBackendLibraryError(f"Failed to load inference backend library from file '{backends_library_path}': {exc}") from exc
+ for backend_name, backend_dict in backends_dict.items():
+ # We'll split the read settings into standard fields and extra config
+ standard_fields = InferenceBackendBlueprint.model_fields.keys()
+ extra_config: Dict[str, Any] = {}
+ inference_backend_blueprint_dict_raw = backend_dict.copy()
+ if not inference_backend_blueprint_dict_raw.get("enabled", True):
+ continue
+ try:
+ inference_backend_blueprint_dict = apply_to_strings_recursive(inference_backend_blueprint_dict_raw, substitute_vars)
+ except (VarNotFoundError, UnknownVarPrefixError) as exc:
+ raise InferenceBackendCredentialsError(f"Variable substitution failed in file '{backends_library_path}': {exc}") from exc
+
+ for key in backend_dict.keys():
+ if key not in standard_fields:
+ extra_config[key] = inference_backend_blueprint_dict.pop(key)
+ backend_blueprint = InferenceBackendBlueprint.model_validate(inference_backend_blueprint_dict)
+ # if not backend_blueprint.enabled:
+ # continue
+
+ path_to_model_specs_toml = get_config().cogt.inference_config.model_specs_path(backend_name=backend_name)
+ try:
+ model_specs_dict_raw = load_toml_from_path(
+ path=path_to_model_specs_toml,
+ )
+ try:
+ model_specs_dict = apply_to_strings_recursive(model_specs_dict_raw, substitute_vars)
+ except (VarNotFoundError, UnknownVarPrefixError) as exc:
+ raise InferenceModelSpecError(f"Variable substitution failed in file '{path_to_model_specs_toml}': {exc}") from exc
+ except (FileNotFoundError, InferenceModelSpecError) as exc:
+ raise InferenceBackendLibraryError(f"Failed to load inference model specs from file '{path_to_model_specs_toml}': {exc}") from exc
+ default_sdk: Optional[str] = model_specs_dict.pop("default_sdk", None)
+ default_prompting_target: Optional[PromptingTarget] = model_specs_dict.pop("default_prompting_target", None)
+ backend_model_specs: Dict[str, InferenceModelSpec] = {}
+ for model_spec_name, model_spec_dict in model_specs_dict.items():
+ try:
+ model_spec_blueprint = InferenceModelSpecBlueprint.model_validate(model_spec_dict)
+ model_spec = InferenceModelSpecFactory.make_inference_model_spec(
+ backend_name=backend_name,
+ name=model_spec_name,
+ blueprint=model_spec_blueprint,
+ default_prompting_target=default_prompting_target,
+ fallback_sdk=default_sdk,
+ )
+ backend_model_specs[model_spec_name] = model_spec
+ except (InferenceModelSpecError, ValidationError) as exc:
+ raise InferenceBackendLibraryError(
+ f"Failed to load inference model spec '{model_spec_name}' for backend '{backend_name}' "
+ f"from file '{path_to_model_specs_toml}': {exc}"
+ )
+ backend = InferenceBackendFactory.make_inference_backend(
+ name=backend_name,
+ blueprint=backend_blueprint,
+ extra_config=extra_config,
+ model_specs=backend_model_specs,
+ )
+ self.root[backend_name] = backend
+ log.debug(f"Loaded inference backend '{backend_name}'")
+
+ def list_backend_names(self) -> List[str]:
+ return list(self.root.keys())
+
+ def list_all_model_names(self) -> List[str]:
+ """List the names of all models in all backends."""
+ all_model_names: Set[str] = set()
+ for backend in self.root.values():
+ all_model_names.update(backend.list_model_names())
+ return sorted(all_model_names)
+
+ def get_all_models_and_possible_backends(self) -> Dict[str, List[str]]:
+ """Get a dictionary of all models and their possible backends."""
+ all_models_and_possible_backends: Dict[str, List[str]] = {}
+ for backend in self.root.values():
+ for model_name in backend.list_model_names():
+ if model_name not in all_models_and_possible_backends:
+ all_models_and_possible_backends[model_name] = []
+ all_models_and_possible_backends[model_name].append(backend.name)
+ return all_models_and_possible_backends
+
+ def get_inference_backend(self, backend_name: str) -> Optional[InferenceBackend]:
+ """Get a backend by name."""
+ backend = self.root.get(backend_name)
+ return backend
diff --git a/pipelex/cogt/model_backends/model_constraints.py b/pipelex/cogt/model_backends/model_constraints.py
new file mode 100644
index 000000000..f5ed7881e
--- /dev/null
+++ b/pipelex/cogt/model_backends/model_constraints.py
@@ -0,0 +1,7 @@
+from pipelex.types import StrEnum
+
+
+class ModelConstraints(StrEnum):
+ TEMPERATURE_MUST_BE_1 = "temperature_must_be_1"
+ TEMPERATURE_MUST_BE_MULTIPLIED_BY_2 = "temperature_must_be_multiplied_by_2"
+ MAX_TOKENS_MUST_BE_HIGH_ENOUGH = "max_tokens_must_be_high_enough"
diff --git a/pipelex/cogt/model_backends/model_spec.py b/pipelex/cogt/model_backends/model_spec.py
new file mode 100644
index 000000000..e9e803334
--- /dev/null
+++ b/pipelex/cogt/model_backends/model_spec.py
@@ -0,0 +1,41 @@
+from typing import List, Optional
+
+from pydantic import Field
+
+from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
+from pipelex.cogt.usage.cost_category import CostsByCategoryDict
+from pipelex.tools.config.config_model import ConfigModel
+
+
+class InferenceModelSpec(ConfigModel):
+ backend_name: str
+ name: str
+ sdk: str
+ model_id: str
+ inputs: List[str] = Field(default_factory=list)
+ outputs: List[str] = Field(default_factory=list)
+ costs: CostsByCategoryDict = Field(strict=False)
+ max_tokens: Optional[int]
+ max_prompt_images: Optional[int]
+ prompting_target: Optional[PromptingTarget] = Field(default=None, strict=False)
+ constraints: List[ModelConstraints] = Field(default_factory=list)
+
+ # TODO: investigate if this is needed
+ is_system_prompt_supported: bool = True
+
+ @property
+ def tag(self) -> str:
+ return f"[{self.sdk}@{self.backend_name}]({self.model_id})"
+
+ @property
+ def desc(self) -> str:
+ return f"SDK[{self.sdk}]•Backend[{self.backend_name}]•Model[{self.model_id}]"
+
+ @property
+ def is_gen_object_supported(self) -> bool:
+ return "structured" in self.outputs
+
+ @property
+ def is_vision_supported(self) -> bool:
+ return "images" in self.inputs
diff --git a/pipelex/cogt/model_backends/model_spec_factory.py b/pipelex/cogt/model_backends/model_spec_factory.py
new file mode 100644
index 000000000..7afb2c443
--- /dev/null
+++ b/pipelex/cogt/model_backends/model_spec_factory.py
@@ -0,0 +1,66 @@
+from typing import Dict, List, Optional
+
+from pydantic import BaseModel, Field, field_validator
+
+from pipelex.cogt.exceptions import InferenceModelSpecError
+from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
+from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
+from pipelex.tools.config.config_model import ConfigModel
+
+
+class InferenceModelSpecBlueprint(ConfigModel):
+ enabled: bool = True
+ sdk: Optional[str] = None
+ model_id: str
+ inputs: List[str] = Field(default_factory=list)
+ outputs: List[str] = Field(default_factory=list)
+ costs: CostsByCategoryDict = Field(strict=False)
+ max_tokens: Optional[int] = None
+ max_prompt_images: Optional[int] = None
+ prompting_target: Optional[PromptingTarget] = Field(default=None, strict=False)
+ constraints: List[ModelConstraints] = Field(default_factory=list)
+
+ @field_validator("costs", mode="before")
+ def validate_costs(cls, value: Dict[str, float]) -> CostsByCategoryDict:
+ return ConfigModel.transform_dict_of_floats_str_to_enum(
+ input_dict=value,
+ key_enum_cls=CostCategory,
+ )
+
+ @field_validator("constraints", mode="before")
+ def validate_constraints(cls, value: List[str]) -> List[ModelConstraints]:
+ return ConfigModel.transform_list_of_str_to_enum(
+ input_list=value,
+ enum_cls=ModelConstraints,
+ )
+
+
+class InferenceModelSpecFactory(BaseModel):
+ @classmethod
+ def make_inference_model_spec(
+ cls,
+ backend_name: str,
+ name: str,
+ blueprint: InferenceModelSpecBlueprint,
+ default_prompting_target: Optional[PromptingTarget],
+ fallback_sdk: Optional[str],
+ ) -> InferenceModelSpec:
+ sdk = blueprint.sdk or fallback_sdk
+ if not sdk:
+ raise InferenceModelSpecError("No sdk choice provided")
+ prompting_target = blueprint.prompting_target or default_prompting_target
+ return InferenceModelSpec(
+ backend_name=backend_name,
+ name=name,
+ sdk=sdk,
+ model_id=blueprint.model_id,
+ inputs=blueprint.inputs,
+ outputs=blueprint.outputs,
+ costs=blueprint.costs,
+ max_tokens=blueprint.max_tokens,
+ max_prompt_images=blueprint.max_prompt_images,
+ prompting_target=prompting_target,
+ constraints=blueprint.constraints,
+ )
diff --git a/pipelex/cogt/llm/llm_models/llm_prompting_target.py b/pipelex/cogt/model_backends/prompting_target.py
similarity index 79%
rename from pipelex/cogt/llm/llm_models/llm_prompting_target.py
rename to pipelex/cogt/model_backends/prompting_target.py
index f57026498..a53a68a26 100644
--- a/pipelex/cogt/llm/llm_models/llm_prompting_target.py
+++ b/pipelex/cogt/model_backends/prompting_target.py
@@ -1,7 +1,7 @@
from pipelex.types import StrEnum
-class LLMPromptingTarget(StrEnum):
+class PromptingTarget(StrEnum):
OPENAI = "openai"
ANTHROPIC = "anthropic"
MISTRAL = "mistral"
diff --git a/pipelex/cogt/model_routing/__init__.py b/pipelex/cogt/model_routing/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/cogt/model_routing/routing_models.py b/pipelex/cogt/model_routing/routing_models.py
new file mode 100644
index 000000000..e7826aa28
--- /dev/null
+++ b/pipelex/cogt/model_routing/routing_models.py
@@ -0,0 +1,19 @@
+from typing import Optional
+
+from pydantic import BaseModel
+
+from pipelex.types import StrEnum
+
+
+class BackendMatchingMethod(StrEnum):
+ DEFAULT = "default"
+ EXACT_MATCH = "exact_match"
+ PATTERN_MATCH = "pattern_match"
+
+
+class BackendMatchForModel(BaseModel):
+ model_name: str
+ backend_name: str
+ routing_profile_name: str
+ matching_method: BackendMatchingMethod
+ matched_pattern: Optional[str]
diff --git a/pipelex/cogt/model_routing/routing_profile.py b/pipelex/cogt/model_routing/routing_profile.py
new file mode 100644
index 000000000..fc3081412
--- /dev/null
+++ b/pipelex/cogt/model_routing/routing_profile.py
@@ -0,0 +1,58 @@
+from typing import Dict, Optional
+
+from pydantic import Field
+
+from pipelex.cogt.model_routing.routing_models import BackendMatchForModel, BackendMatchingMethod
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.misc.string_utils import matches_wildcard_pattern
+
+
+class RoutingProfile(ConfigModel):
+ """Configuration for model routing to backends."""
+
+ name: str
+ description: Optional[str] = None
+ default: Optional[str] = None
+ routes: Dict[str, str] = Field(default_factory=dict) # Pattern -> Backend mapping
+
+ def get_backend_match_for_model(self, model_name: str) -> Optional[BackendMatchForModel]:
+ """Get the backend name for a given model name.
+
+ Args:
+ model_name: Name of the model to route
+
+ Returns:
+ Backend name to use for this model
+ """
+ # Check exact matches first
+ if model_name in self.routes:
+ return BackendMatchForModel(
+ model_name=model_name,
+ backend_name=self.routes[model_name],
+ routing_profile_name=self.name,
+ matching_method=BackendMatchingMethod.EXACT_MATCH,
+ matched_pattern=None,
+ )
+
+ # Check pattern matches
+ for pattern, backend in self.routes.items():
+ if matches_wildcard_pattern(model_name, pattern):
+ return BackendMatchForModel(
+ model_name=model_name,
+ backend_name=backend,
+ routing_profile_name=self.name,
+ matching_method=BackendMatchingMethod.PATTERN_MATCH,
+ matched_pattern=pattern,
+ )
+
+ # Return default backend
+ if self.default:
+ return BackendMatchForModel(
+ model_name=model_name,
+ backend_name=self.default,
+ routing_profile_name=self.name,
+ matching_method=BackendMatchingMethod.DEFAULT,
+ matched_pattern=None,
+ )
+ else:
+ return None
diff --git a/pipelex/cogt/model_routing/routing_profile_factory.py b/pipelex/cogt/model_routing/routing_profile_factory.py
new file mode 100644
index 000000000..a08872690
--- /dev/null
+++ b/pipelex/cogt/model_routing/routing_profile_factory.py
@@ -0,0 +1,47 @@
+from typing import Dict, Optional
+
+from pydantic import Field
+
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from pipelex.tools.config.config_model import ConfigModel
+
+
+class RoutingProfileBlueprint(ConfigModel):
+ """Blueprint for creating RoutingProfile instances."""
+
+ description: str
+ default: Optional[str] = None
+ routes: Dict[str, str] = Field(default_factory=dict)
+
+
+class RoutingProfileLibraryBlueprint(ConfigModel):
+ """Blueprint for the entire routing profile library."""
+
+ active: str
+ profiles: Dict[str, RoutingProfileBlueprint] = Field(default_factory=dict)
+
+
+class RoutingProfileFactory:
+ """Factory for creating routing profile configurations."""
+
+ @classmethod
+ def make_routing_profile(
+ cls,
+ name: str,
+ blueprint: RoutingProfileBlueprint,
+ ) -> RoutingProfile:
+ """Create a RoutingProfile from a blueprint.
+
+ Args:
+ name: Name of the routing profile
+ blueprint: Blueprint containing configuration data
+
+ Returns:
+ RoutingProfile instance
+ """
+ return RoutingProfile(
+ name=name,
+ description=blueprint.description,
+ default=blueprint.default,
+ routes=blueprint.routes,
+ )
diff --git a/pipelex/cogt/model_routing/routing_profile_library.py b/pipelex/cogt/model_routing/routing_profile_library.py
new file mode 100644
index 000000000..8e7ec0473
--- /dev/null
+++ b/pipelex/cogt/model_routing/routing_profile_library.py
@@ -0,0 +1,103 @@
+from typing import Dict, Optional
+
+from pydantic import Field, RootModel, ValidationError
+from typing_extensions import Self
+
+from pipelex import log
+from pipelex.cogt.exceptions import RoutingProfileLibraryError, RoutingProfileLibraryNotFoundError
+from pipelex.cogt.model_routing.routing_models import BackendMatchForModel
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from pipelex.cogt.model_routing.routing_profile_factory import (
+ RoutingProfileFactory,
+ RoutingProfileLibraryBlueprint,
+)
+from pipelex.config import get_config
+from pipelex.tools.misc.toml_utils import TOMLValidationError, load_toml_from_path
+
+RoutingProfileLibraryRoot = Dict[str, RoutingProfile]
+
+
+class RoutingProfileLibrary(RootModel[RoutingProfileLibraryRoot]):
+ """Library for managing routing profile configurations."""
+
+ root: RoutingProfileLibraryRoot = Field(default_factory=dict)
+ _active_config: Optional[str] = None
+
+ @property
+ def active_profile(self) -> RoutingProfile:
+ if not self._active_config:
+ raise RoutingProfileLibraryError("No active routing profile loaded")
+ if self._active_config not in self.root:
+ raise RoutingProfileLibraryError(f"Active routing profile '{self._active_config}' not found in loaded routing profile library")
+ return self.root[self._active_config]
+
+ @classmethod
+ def make_empty(cls) -> Self:
+ return cls(root={})
+
+ def reset(self) -> None:
+ self.root = {}
+
+ def load(self) -> None:
+ """Load the routing profile library configuration from TOML file."""
+ routing_profile_library_path = get_config().cogt.inference_config.routing_profile_library_path
+
+ try:
+ catalog_dict = load_toml_from_path(path=routing_profile_library_path)
+ except FileNotFoundError as not_found_exc:
+ raise RoutingProfileLibraryNotFoundError(
+ f"Failed to load routing profile library from file '{routing_profile_library_path}': {not_found_exc}"
+ ) from not_found_exc
+ except TOMLValidationError as toml_validation_exc:
+ raise RoutingProfileLibraryError(
+ f"Failed to load routing profile library from file '{routing_profile_library_path}': {toml_validation_exc}"
+ ) from toml_validation_exc
+
+ try:
+ catalog_blueprint = RoutingProfileLibraryBlueprint.model_validate(catalog_dict)
+ except ValidationError as exc:
+ raise RoutingProfileLibraryError(f"Invalid routing profile library configuration in '{routing_profile_library_path}': {exc}") from exc
+
+ # Validate that the active config exists
+ if catalog_blueprint.active not in catalog_blueprint.profiles:
+ raise RoutingProfileLibraryError(
+ f"Active profile '{catalog_blueprint.active}' not found in library. Available profiles: {list(catalog_blueprint.profiles.keys())}"
+ )
+
+ # Load all profiles
+ self.root = {}
+ for config_name, config_blueprint in catalog_blueprint.profiles.items():
+ self.root[config_name] = RoutingProfileFactory.make_routing_profile(
+ name=config_name,
+ blueprint=config_blueprint,
+ )
+ self._active_config = catalog_blueprint.active
+
+ log.debug(f"Loaded routing profile library with active profile: '{self._active_config}'")
+ log.debug(f"Available profiles: {list(self.root.keys())}")
+
+ def get_backend_match_for_model_from_active_routing_profile(self, model_name: str) -> Optional[BackendMatchForModel]:
+ """Get the backend name for a given model.
+
+ Args:
+ model_name: Name of the model to route
+
+ Returns:
+ Backend name to use for this model
+
+ Raises:
+ RoutingProfileLibraryError: If no active profile is set or config not found
+ """
+ profile = self.active_profile
+ backend_match_for_model = profile.get_backend_match_for_model(model_name)
+ return backend_match_for_model
+
+ def list_routing_profile_names(self) -> list[str]:
+ """Get a list of all available routing profile names."""
+ return list(self.root.keys())
+
+ def get_required_routing_profile(self, routing_profile_name: str) -> RoutingProfile:
+ routing_profile = self.root.get(routing_profile_name)
+ if not routing_profile:
+ raise RoutingProfileLibraryError(f"Routing profile '{routing_profile_name}' not found in loaded routing profile library")
+ return routing_profile
diff --git a/pipelex/cogt/models/__init__.py b/pipelex/cogt/models/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.py
new file mode 100644
index 000000000..c78ca8391
--- /dev/null
+++ b/pipelex/cogt/models/model_deck.py
@@ -0,0 +1,144 @@
+from typing import Dict, List, Optional, Union
+
+from pydantic import Field, field_validator, model_validator
+from typing_extensions import Self
+
+from pipelex import log
+from pipelex.cogt.exceptions import LLMHandleNotFoundError, LLMPresetNotFoundError, LLMSettingsValidationError, ModelDeckValidatonError
+from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingChoicesDefaults, LLMSettingOrPresetId
+from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.exceptions import ConfigValidationError
+
+LLM_PRESET_DISABLED = "disabled"
+
+Waterfall = Union[str, List[str]]
+
+
+class ModelDeckBlueprint(ConfigModel):
+ aliases: Dict[str, Waterfall] = Field(default_factory=dict)
+ llm_presets: Dict[str, LLMSetting] = Field(default_factory=dict)
+ llm_choice_defaults: LLMSettingChoicesDefaults
+ llm_choice_overrides: LLMSettingChoices = LLMSettingChoices(
+ for_text=None,
+ for_object=None,
+ )
+
+
+class ModelDeck(ConfigModel):
+ inference_models: Dict[str, InferenceModelSpec] = Field(default_factory=dict)
+ aliases: Dict[str, Waterfall] = Field(default_factory=dict)
+
+ llm_presets: Dict[str, LLMSetting] = Field(default_factory=dict)
+ llm_choice_defaults: LLMSettingChoicesDefaults
+ llm_choice_overrides: LLMSettingChoices = LLMSettingChoices(
+ for_text=None,
+ for_object=None,
+ )
+
+ def check_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId, is_disabled_allowed: bool = False):
+ if isinstance(llm_setting_or_preset_id, LLMSetting):
+ return
+ preset_id: str = llm_setting_or_preset_id
+ if preset_id in self.llm_presets:
+ return
+ if preset_id == LLM_PRESET_DISABLED and is_disabled_allowed:
+ return
+ raise LLMPresetNotFoundError(f"llm preset id '{preset_id}' not found in deck")
+
+ def get_llm_setting(self, llm_setting_or_preset_id: LLMSettingOrPresetId) -> LLMSetting:
+ if isinstance(llm_setting_or_preset_id, LLMSetting):
+ return llm_setting_or_preset_id
+ else:
+ # it's a preset id
+ the_llm_preset = self.llm_presets.get(llm_setting_or_preset_id)
+ if not the_llm_preset:
+ raise LLMPresetNotFoundError(f"LLM preset '{llm_setting_or_preset_id}' not found in deck")
+ return the_llm_preset
+
+ @classmethod
+ def final_validate(cls, deck: Self): # pyright: ignore[reportIncompatibleMethodOverride]
+ for llm_preset_id, llm_setting in deck.llm_presets.items():
+ inference_model = deck.get_required_inference_model(llm_handle=llm_setting.llm_handle)
+ try:
+ cls._validate_llm_setting(llm_setting=llm_setting, inference_model=inference_model)
+ except ConfigValidationError as exc:
+ raise ModelDeckValidatonError(f"LLM preset '{llm_preset_id}' is invalid: {exc}")
+
+ ############################################################
+ #### ModelDeck validations
+ ############################################################
+
+ @classmethod
+ def _validate_llm_setting(cls, llm_setting: LLMSetting, inference_model: InferenceModelSpec):
+ if inference_model.max_tokens is not None and (llm_setting_max_tokens := llm_setting.max_tokens):
+ if llm_setting_max_tokens > inference_model.max_tokens:
+ raise LLMSettingsValidationError(
+ (
+ f"LLM setting '{llm_setting.llm_handle}' has a max_tokens of {llm_setting_max_tokens}, "
+ f"which is greater than the model's max_tokens of {inference_model.max_tokens}"
+ )
+ )
+ if ModelConstraints.TEMPERATURE_MUST_BE_1 in inference_model.constraints and llm_setting.temperature != 1:
+ raise LLMSettingsValidationError(
+ f"LLM setting '{llm_setting.llm_handle}' has a temperature of {llm_setting.temperature}, "
+ f"which is not allowed by the model's constraints: it must be 1"
+ )
+
+ @field_validator("llm_choice_defaults", mode="after")
+ @classmethod
+ def validate_llm_choice_defaults(cls, llm_choice_defaults: LLMSettingChoices) -> LLMSettingChoices:
+ if llm_choice_defaults.for_text is None:
+ raise ConfigValidationError("llm_choice_defaults.for_text cannot be None")
+ if llm_choice_defaults.for_object is None:
+ raise ConfigValidationError("llm_choice_defaults.for_object cannot be None")
+ return llm_choice_defaults
+
+ @field_validator("llm_choice_overrides", mode="after")
+ @classmethod
+ def validate_llm_choice_overrides(cls, value: LLMSettingChoices) -> LLMSettingChoices:
+ if value.for_text == LLM_PRESET_DISABLED:
+ value.for_text = None
+ if value.for_object == LLM_PRESET_DISABLED:
+ value.for_object = None
+ return value
+
+ def validate_llm_presets(self) -> Self:
+ for llm_preset_id, llm_setting in self.llm_presets.items():
+ if llm_setting.llm_handle not in self.inference_models:
+ raise LLMHandleNotFoundError(f"llm_handle '{llm_setting.llm_handle}' for llm_preset '{llm_preset_id}' not found in deck")
+ return self
+
+ @model_validator(mode="after")
+ def validate_llm_setting_overrides(self) -> Self:
+ self._validate_llm_choices(llm_choices=self.llm_choice_overrides)
+ return self
+
+ def _validate_llm_choices(self, llm_choices: LLMSettingChoices):
+ for llm_setting in llm_choices.list_used_presets():
+ self.check_llm_setting(llm_setting_or_preset_id=llm_setting)
+ return
+
+ def get_optional_inference_model(self, llm_handle: str) -> Optional[InferenceModelSpec]:
+ if inference_model := self.inference_models.get(llm_handle):
+ return inference_model
+ elif redirection := self.aliases.get(llm_handle):
+ log.debug(f"Redirection for '{llm_handle}': {redirection}")
+ if isinstance(redirection, str):
+ alias_list = [redirection]
+ else:
+ alias_list = redirection
+ for alias in alias_list:
+ if inference_model := self.get_optional_inference_model(llm_handle=alias):
+ return inference_model
+ log.warning(f"Skipping LLM handle '{llm_handle}' because it's not found in deck")
+ return None
+
+ def get_required_inference_model(self, llm_handle: str) -> InferenceModelSpec:
+ inference_model = self.get_optional_inference_model(llm_handle=llm_handle)
+ if inference_model is None:
+ raise LLMHandleNotFoundError(f"LLM handle '{llm_handle}' not found in deck")
+ if llm_handle not in self.inference_models:
+ log.dev(f"LLM handle '{llm_handle}' is an alias which resolves to '{inference_model.name}'")
+ return inference_model
diff --git a/pipelex/cogt/llm/llm_models/llm_deck_check.py b/pipelex/cogt/models/model_deck_check.py
similarity index 60%
rename from pipelex/cogt/llm/llm_models/llm_deck_check.py
rename to pipelex/cogt/models/model_deck_check.py
index e8889f06a..9d72278f5 100644
--- a/pipelex/cogt/llm/llm_models/llm_deck_check.py
+++ b/pipelex/cogt/models/model_deck_check.py
@@ -1,12 +1,13 @@
from pipelex.cogt.exceptions import LLMPresetNotFoundError
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingOrPresetId
-from pipelex.hub import get_llm_deck
+from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingOrPresetId
+from pipelex.hub import get_models_manager
-def check_llm_setting_with_deck(llm_setting_or_preset_id: LLMSettingOrPresetId, is_disabled_allowed: bool = False):
+def check_llm_setting_with_deck(llm_setting_or_preset_id: LLMSettingOrPresetId):
if isinstance(llm_setting_or_preset_id, LLMSetting):
return
preset_id: str = llm_setting_or_preset_id
- if preset_id in get_llm_deck().llm_presets:
+ llm_deck = get_models_manager().get_llm_deck()
+ if preset_id in llm_deck.llm_presets:
return
raise LLMPresetNotFoundError(f"llm preset id '{preset_id}' not found in deck")
diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.py
new file mode 100644
index 000000000..01459f753
--- /dev/null
+++ b/pipelex/cogt/models/model_manager.py
@@ -0,0 +1,152 @@
+import os
+from typing import Any, Dict, Optional
+
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.cogt.exceptions import ModelDeckNotFoundError, ModelsManagerError
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_routing.routing_models import BackendMatchingMethod
+from pipelex.cogt.model_routing.routing_profile_library import RoutingProfileLibrary
+from pipelex.cogt.models.model_deck import ModelDeck, ModelDeckBlueprint
+from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
+from pipelex.config import get_config
+from pipelex.exceptions import LibraryError
+from pipelex.tools.misc.json_utils import deep_update
+from pipelex.tools.misc.toml_utils import TOMLValidationError, load_toml_from_path, validate_toml_file
+
+
+class ModelManager(ModelManagerAbstract):
+ def __init__(self) -> None:
+ self.routing_profile_library = RoutingProfileLibrary.make_empty()
+ self.inference_backend_library = InferenceBackendLibrary.make_empty()
+ self.llm_deck: Optional[ModelDeck] = None
+
+ @override
+ def get_llm_deck(self) -> ModelDeck:
+ if self.llm_deck is None:
+ raise RuntimeError("LLM deck is not initialized")
+ return self.llm_deck
+
+ @override
+ def teardown(self) -> None:
+ self.routing_profile_library.reset()
+ self.inference_backend_library.reset()
+
+ @override
+ def setup(self) -> None:
+ self.routing_profile_library.load()
+ self.inference_backend_library.load()
+ llm_deck_blueprint = self.load_deck_blueprint()
+ self.llm_deck = self.build_deck(llm_deck_blueprint=llm_deck_blueprint)
+
+ @classmethod
+ def _validate_toml_files(cls) -> None:
+ log.debug("LibraryManager deck TOML file formatting")
+
+ # Validation of LLM deck paths
+ llm_deck_paths = get_config().cogt.inference_config.get_llm_deck_paths()
+ for llm_deck_path in llm_deck_paths:
+ if os.path.exists(llm_deck_path):
+ try:
+ validate_toml_file(llm_deck_path)
+ except TOMLValidationError as exc:
+ log.error(f"TOML formatting issues in LLM deck file '{llm_deck_path}': {exc}")
+ raise LibraryError(f"TOML validation failed for LLM deck file '{llm_deck_path}': {exc}") from exc
+
+ @classmethod
+ def load_deck_blueprint(cls) -> ModelDeckBlueprint:
+ llm_deck_paths = get_config().cogt.inference_config.get_llm_deck_paths()
+ full_llm_deck_dict: Dict[str, Any] = {}
+ if not llm_deck_paths:
+ raise ModelDeckNotFoundError("No LLM deck paths found. Please run `pipelex init-libraries` to create it.")
+
+ for llm_deck_path in llm_deck_paths:
+ if not os.path.exists(llm_deck_path):
+ raise ModelDeckNotFoundError(f"LLM deck path `{llm_deck_path}` not found. Please run `pipelex init-libraries` to create it.")
+ try:
+ llm_deck_dict = load_toml_from_path(path=llm_deck_path)
+ log.debug(f"Loaded LLM deck from {llm_deck_path}")
+ deep_update(full_llm_deck_dict, llm_deck_dict)
+ except Exception as exc:
+ log.error(f"Failed to load LLM deck file '{llm_deck_path}': {exc}")
+ raise
+
+ llm_deck_blueprint = ModelDeckBlueprint.model_validate(full_llm_deck_dict)
+ return llm_deck_blueprint
+
+ def build_deck(self, llm_deck_blueprint: ModelDeckBlueprint) -> ModelDeck:
+ all_models_and_possible_backends = self.inference_backend_library.get_all_models_and_possible_backends()
+ llm_handles: Dict[str, InferenceModelSpec] = {}
+
+ for model_name, available_backends in all_models_and_possible_backends.items():
+ backend_match_for_model = self.routing_profile_library.get_backend_match_for_model_from_active_routing_profile(
+ model_name=model_name,
+ )
+ if backend_match_for_model is None:
+ # raise ModelsManagerError(f"No backend match found for model '{model_name}'")
+ log.verbose(f"No backend match found for model '{model_name}'")
+ continue
+ matched_backend_name = backend_match_for_model.backend_name
+ backend = self.inference_backend_library.get_inference_backend(backend_name=matched_backend_name)
+ if backend is None:
+ raise ModelsManagerError(f"Backend '{matched_backend_name}', requested for model '{model_name}', could not be found")
+ model_spec = backend.get_model_spec(model_name)
+ if model_spec is None:
+ # Not finding the model spec can be an error or not according to the matching method
+ match backend_match_for_model.matching_method:
+ case BackendMatchingMethod.EXACT_MATCH:
+ raise ModelsManagerError(
+ f"Model spec '{model_name}' not found in backend '{matched_backend_name}' "
+ f"which was matched exactly in routing profile '{backend_match_for_model.routing_profile_name}'"
+ )
+ case BackendMatchingMethod.PATTERN_MATCH:
+ log.verbose(
+ f"Model spec '{model_name}' not found in backend '{matched_backend_name}' but it's OK because "
+ f"it was only matched by pattern in routing profile '{backend_match_for_model.routing_profile_name}'"
+ )
+ # We can skip it because it was only a pattern match
+ continue
+ case BackendMatchingMethod.DEFAULT:
+ # We could not find the model spec, but it was a default match,
+ # so we can look for it in the other available backends
+ # TODO: enable to set the order or priority of the available backends
+ for available_backend in available_backends:
+ if available_backend == matched_backend_name:
+ continue
+ backend = self.inference_backend_library.get_inference_backend(backend_name=available_backend)
+ if backend is None:
+ raise ModelsManagerError(f"Backend '{available_backend}' not found for model '{model_name}'")
+ model_spec = backend.get_model_spec(model_name)
+ if model_spec is not None:
+ break
+ if model_spec is None:
+ raise ModelsManagerError(
+ f"Model spec '{model_name}' not found in any of the available backends '{available_backends}' "
+ f"which was set as default in routing profile '{backend_match_for_model.routing_profile_name}'"
+ )
+ llm_handles[model_name] = model_spec
+
+ llm_deck = ModelDeck(
+ inference_models=llm_handles,
+ aliases=llm_deck_blueprint.aliases,
+ llm_presets=llm_deck_blueprint.llm_presets,
+ llm_choice_defaults=llm_deck_blueprint.llm_choice_defaults,
+ llm_choice_overrides=llm_deck_blueprint.llm_choice_overrides,
+ )
+ return llm_deck
+
+ @override
+ def get_inference_model(self, llm_handle: str) -> InferenceModelSpec:
+ if self.llm_deck is None:
+ raise RuntimeError("LLM deck is not initialized")
+ return self.llm_deck.get_required_inference_model(llm_handle=llm_handle)
+
+ @override
+ def get_required_inference_backend(self, backend_name: str) -> InferenceBackend:
+ backend = self.inference_backend_library.get_inference_backend(backend_name)
+ if backend is None:
+ raise ModelsManagerError(f"Inference backend '{backend_name}' not found")
+ return backend
diff --git a/pipelex/cogt/models/model_manager_abstract.py b/pipelex/cogt/models/model_manager_abstract.py
new file mode 100644
index 000000000..e6ae5dc2b
--- /dev/null
+++ b/pipelex/cogt/models/model_manager_abstract.py
@@ -0,0 +1,27 @@
+from abc import ABC, abstractmethod
+
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.models.model_deck import ModelDeck
+
+
+class ModelManagerAbstract(ABC):
+ @abstractmethod
+ def teardown(self) -> None:
+ pass
+
+ @abstractmethod
+ def setup(self) -> None:
+ pass
+
+ @abstractmethod
+ def get_inference_model(self, llm_handle: str) -> InferenceModelSpec:
+ pass
+
+ @abstractmethod
+ def get_llm_deck(self) -> ModelDeck:
+ pass
+
+ @abstractmethod
+ def get_required_inference_backend(self, backend_name: str) -> InferenceBackend:
+ pass
diff --git a/pipelex/cogt/ocr/ocr_job_components.py b/pipelex/cogt/ocr/ocr_job_components.py
index f74f322e8..d7b0cd0d5 100644
--- a/pipelex/cogt/ocr/ocr_job_components.py
+++ b/pipelex/cogt/ocr/ocr_job_components.py
@@ -2,7 +2,7 @@ from typing import Optional
from pydantic import BaseModel
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
class OcrJobParams(BaseModel):
diff --git a/pipelex/cogt/ocr/ocr_worker_factory.py b/pipelex/cogt/ocr/ocr_worker_factory.py
index eea412ec3..0c61b5ff7 100644
--- a/pipelex/cogt/ocr/ocr_worker_factory.py
+++ b/pipelex/cogt/ocr/ocr_worker_factory.py
@@ -4,8 +4,8 @@ from pipelex.cogt.exceptions import MissingDependencyError
from pipelex.cogt.ocr.ocr_engine import OcrEngine
from pipelex.cogt.ocr.ocr_platform import OcrPlatform
from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
-from pipelex.hub import get_plugin_manager
-from pipelex.plugins.plugin_sdk_registry import PluginSdkHandle
+from pipelex.hub import get_models_manager, get_plugin_manager
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.reporting.reporting_protocol import ReportingProtocol
@@ -15,7 +15,8 @@ class OcrWorkerFactory:
ocr_engine: OcrEngine,
reporting_delegate: Optional[ReportingProtocol] = None,
) -> OcrWorkerAbstract:
- ocr_sdk_handle = PluginSdkHandle.get_for_ocr_engine(ocr_platform=ocr_engine.ocr_platform)
+ backend = get_models_manager().get_required_inference_backend("mistral")
+ ocr_plugin = Plugin.make_for_ocr_engine(ocr_platform=ocr_engine.ocr_platform)
plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry
ocr_worker: OcrWorkerAbstract
match ocr_engine.ocr_platform:
@@ -32,11 +33,9 @@ class OcrWorkerFactory:
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.plugins.mistral.mistral_ocr_worker import MistralOcrWorker
- ocr_sdk_instance = plugin_sdk_registry.get_ocr_sdk_instance(
- ocr_sdk_handle=ocr_sdk_handle
- ) or plugin_sdk_registry.set_ocr_sdk_instance(
- ocr_sdk_handle=ocr_sdk_handle,
- ocr_sdk_instance=MistralFactory.make_mistral_client(),
+ ocr_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=ocr_plugin) or plugin_sdk_registry.set_sdk_instance(
+ plugin=ocr_plugin,
+ sdk_instance=MistralFactory.make_mistral_client(backend=backend),
)
ocr_worker = MistralOcrWorker(
diff --git a/pipelex/cogt/usage/cost_category.py b/pipelex/cogt/usage/cost_category.py
new file mode 100644
index 000000000..3330090aa
--- /dev/null
+++ b/pipelex/cogt/usage/cost_category.py
@@ -0,0 +1,19 @@
+from typing import Dict
+
+from pipelex.types import StrEnum
+
+
+class CostCategory(StrEnum):
+ INPUT = "input"
+ INPUT_CACHED = "input_cached"
+ INPUT_NON_CACHED = "input_non_cached"
+ INPUT_JOINED = "input_joined" # joined = cached + non-cached
+ INPUT_AUDIO = "input_audio"
+ OUTPUT = "output"
+ OUTPUT_AUDIO = "output_audio"
+ OUTPUT_REASONING = "output_reasoning"
+ OUTPUT_ACCEPTED_PREDICTION = "output_accepted_prediction"
+ OUTPUT_REJECTED_PREDICTION = "output_rejected_prediction"
+
+
+CostsByCategoryDict = Dict[CostCategory, float]
diff --git a/pipelex/cogt/inference/cost_registry.py b/pipelex/cogt/usage/cost_registry.py
similarity index 82%
rename from pipelex/cogt/inference/cost_registry.py
rename to pipelex/cogt/usage/cost_registry.py
index 69a27e96a..18b807c21 100644
--- a/pipelex/cogt/inference/cost_registry.py
+++ b/pipelex/cogt/usage/cost_registry.py
@@ -8,8 +8,10 @@ from rich.table import Table
from pipelex import log
from pipelex.cogt.exceptions import CostRegistryError
-from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage, model_cost_per_token
-from pipelex.cogt.llm.token_category import TokenCategory
+from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
+from pipelex.cogt.usage.costs_per_token import model_cost_per_token
+from pipelex.cogt.usage.token_category import TokenCategory
CostRegistryRoot = List[LLMTokenCostReport]
@@ -149,15 +151,33 @@ class CostRegistry(RootModel[CostRegistryRoot]):
def compute_total_cost(cls, input_non_cached_cost: float, input_cached_cost: float, output_cost: float) -> float:
return input_non_cached_cost + input_cached_cost + output_cost
+ @classmethod
+ def compute_cost_report(cls, llm_tokens_usage: LLMTokensUsage) -> LLMTokenCostReport:
+ costs_by_token_category: CostsByCategoryDict = {}
+ for token_type, nb_tokens in llm_tokens_usage.nb_tokens_by_category.items():
+ cost_per_token = model_cost_per_token(
+ costs=llm_tokens_usage.unit_costs,
+ cost_category=token_type.to_cost_category,
+ )
+ costs_by_token_category[token_type.to_cost_category] = cost_per_token * nb_tokens
+ token_cost_report = LLMTokenCostReport(
+ job_metadata=llm_tokens_usage.job_metadata,
+ inference_model_name=llm_tokens_usage.inference_model_name,
+ platform_llm_id=llm_tokens_usage.inference_model_id,
+ nb_tokens_by_category=llm_tokens_usage.nb_tokens_by_category,
+ costs_by_token_category=costs_by_token_category,
+ )
+ return token_cost_report
+
@classmethod
def complete_cost_report(cls, llm_tokens_usage: LLMTokensUsage) -> LLMTokenCostReport:
- cost_report = llm_tokens_usage.compute_cost_report()
+ cost_report = cls.compute_cost_report(llm_tokens_usage=llm_tokens_usage)
# compute the input_non_cached tokens
if cost_report.nb_tokens_by_category.get(TokenCategory.INPUT_NON_CACHED) is not None:
- raise CostRegistryError("TokenCategory.INPUT_NON_CACHED already exists in the cost report")
+ raise CostRegistryError("CostCategory.INPUT_NON_CACHED already exists in the cost report")
# we use pop to remove input tokens which will be replaced by "input joined"
nb_tokens_input_joined = cost_report.nb_tokens_by_category.pop(TokenCategory.INPUT, 0)
- cost_report.costs_by_token_category.pop(TokenCategory.INPUT, None)
+ cost_report.costs_by_token_category.pop(CostCategory.INPUT, None)
nb_tokens_input_cached = cost_report.nb_tokens_by_category.get(TokenCategory.INPUT_CACHED, 0)
nb_tokens_input_non_cached = nb_tokens_input_joined - nb_tokens_input_cached
@@ -165,12 +185,12 @@ class CostRegistry(RootModel[CostRegistryRoot]):
cost_report.nb_tokens_by_category[TokenCategory.INPUT_NON_CACHED] = nb_tokens_input_non_cached
cost_report.nb_tokens_by_category[TokenCategory.INPUT_CACHED] = nb_tokens_input_cached
- cost_report.costs_by_token_category[TokenCategory.INPUT_NON_CACHED] = nb_tokens_input_non_cached * model_cost_per_token(
- llm_engine=llm_tokens_usage.llm_engine, token_type=TokenCategory.INPUT_NON_CACHED
+ cost_report.costs_by_token_category[CostCategory.INPUT_NON_CACHED] = nb_tokens_input_non_cached * model_cost_per_token(
+ costs=llm_tokens_usage.unit_costs, cost_category=CostCategory.INPUT_NON_CACHED
)
- costs_input_cached = cost_report.costs_by_token_category.get(TokenCategory.INPUT_CACHED, 0)
- cost_report.costs_by_token_category[TokenCategory.INPUT_CACHED] = costs_input_cached
- cost_report.costs_by_token_category[TokenCategory.INPUT_JOINED] = (
- costs_input_cached + cost_report.costs_by_token_category[TokenCategory.INPUT_NON_CACHED]
+ costs_input_cached = cost_report.costs_by_token_category.get(CostCategory.INPUT_CACHED, 0)
+ cost_report.costs_by_token_category[CostCategory.INPUT_CACHED] = costs_input_cached
+ cost_report.costs_by_token_category[CostCategory.INPUT_JOINED] = (
+ costs_input_cached + cost_report.costs_by_token_category[CostCategory.INPUT_NON_CACHED]
)
return cost_report
diff --git a/pipelex/cogt/usage/costs_per_token.py b/pipelex/cogt/usage/costs_per_token.py
new file mode 100644
index 000000000..6486e5a51
--- /dev/null
+++ b/pipelex/cogt/usage/costs_per_token.py
@@ -0,0 +1,20 @@
+from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
+
+
+def model_cost_per_token(costs: CostsByCategoryDict, cost_category: CostCategory) -> float:
+ # cost_per_million_tokens_usd should be missing only for models that we run on our own GPUs
+ # all token types are not used for all models
+ if cost_category == CostCategory.INPUT_CACHED:
+ if cost_per_million_tokens := costs.get(CostCategory.INPUT_CACHED):
+ return cost_per_million_tokens / 1000000
+ elif cost_per_million_tokens := costs.get(CostCategory.INPUT):
+ # according to openai docs, cached input tokens are discounted 50%
+ return 0.5 * cost_per_million_tokens / 1000000
+ else:
+ return 0.0
+ elif cost_category == CostCategory.INPUT_NON_CACHED:
+ return model_cost_per_token(costs=costs, cost_category=CostCategory.INPUT)
+ elif cost_per_million_tokens := costs.get(cost_category):
+ return cost_per_million_tokens / 1000000
+ else:
+ return 0.0
diff --git a/pipelex/cogt/llm/token_category.py b/pipelex/cogt/usage/token_category.py
similarity index 77%
rename from pipelex/cogt/llm/token_category.py
rename to pipelex/cogt/usage/token_category.py
index 39ea15bc2..e5bac8964 100644
--- a/pipelex/cogt/llm/token_category.py
+++ b/pipelex/cogt/usage/token_category.py
@@ -1,5 +1,6 @@
from typing import Dict
+from pipelex.cogt.usage.cost_category import CostCategory
from pipelex.types import StrEnum
@@ -15,6 +16,9 @@ class TokenCategory(StrEnum):
OUTPUT_ACCEPTED_PREDICTION = "output_accepted_prediction"
OUTPUT_REJECTED_PREDICTION = "output_rejected_prediction"
+ @property
+ def to_cost_category(self) -> CostCategory:
+ return CostCategory(self.value)
+
NbTokensByCategoryDict = Dict[TokenCategory, int]
-TokenCostsByCategoryDict = Dict[TokenCategory, float]
diff --git a/pipelex/config.py b/pipelex/config.py
index 630fea8ee..3e1fc5630 100644
--- a/pipelex/config.py
+++ b/pipelex/config.py
@@ -4,13 +4,14 @@ import shortuuid
from pydantic import Field, field_validator
from pipelex.cogt.config_cogt import Cogt
-from pipelex.cogt.llm.llm_models.llm_prompting_target import LLMPromptingTarget
+from pipelex.cogt.model_backends.prompting_target import PromptingTarget
from pipelex.exceptions import PipelexConfigError, StaticValidationErrorType
from pipelex.hub import get_required_config
from pipelex.libraries.library_config import LibraryConfig
from pipelex.pipeline.track.tracker_config import TrackerConfig
from pipelex.tools.aws.aws_config import AwsConfig
-from pipelex.tools.config.models import ConfigModel, ConfigRoot
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.config.config_root import ConfigRoot
from pipelex.tools.log.log_config import LogConfig
from pipelex.tools.templating.templating_models import PromptingStyle
from pipelex.types import StrEnum
@@ -71,7 +72,7 @@ class PromptingConfig(ConfigModel):
default_prompting_style: PromptingStyle
prompting_styles: Dict[str, PromptingStyle]
- def get_prompting_style(self, prompting_target: Optional[LLMPromptingTarget] = None) -> Optional[PromptingStyle]:
+ def get_prompting_style(self, prompting_target: Optional[PromptingTarget] = None) -> Optional[PromptingStyle]:
if prompting_target:
return self.prompting_styles.get(prompting_target, self.default_prompting_style)
else:
diff --git a/pipelex/config_template/inference/backends.toml b/pipelex/config_template/inference/backends.toml
new file mode 100644
index 000000000..64c393f41
--- /dev/null
+++ b/pipelex/config_template/inference/backends.toml
@@ -0,0 +1,46 @@
+[pipelex_inference]
+endpoint = "https://inference.pipelex.com/v1"
+api_key = "${PIPELEX_INFERENCE_API_KEY}"
+
+[blackboxai]
+enabled = false
+endpoint = "https://api.blackbox.ai/v1"
+api_key = "${BLACKBOX_API_KEY}"
+
+[openai]
+enabled = false
+api_key = "${OPENAI_API_KEY}"
+
+[azure_openai]
+enabled = false
+endpoint = "${AZURE_API_BASE}"
+api_key = "${AZURE_API_KEY}"
+api_version = "${AZURE_API_VERSION}"
+
+[anthropic]
+enabled = false
+api_key = "${ANTHROPIC_API_KEY}"
+claude_4_tokens_limit = 8192
+
+[ollama]
+enabled = false
+endpoint = "http://localhost:11434/v1"
+
+[xai]
+enabled = false
+endpoint = "https://api.x.ai/v1"
+api_key = "${XAI_API_KEY}"
+
+[bedrock]
+enabled = false
+aws_region = "${AWS_REGION}"
+
+[vertexai]
+enabled = false
+gcp_project_id = "${GCP_PROJECT_ID}"
+gcp_location = "${GCP_LOCATION}"
+gcp_credentials_file_path = "${GCP_CREDENTIALS_FILE_PATH}"
+
+[mistral]
+enabled = false
+api_key = "${MISTRAL_API_KEY}"
diff --git a/pipelex/config_template/inference/backends/anthropic.toml b/pipelex/config_template/inference/backends/anthropic.toml
new file mode 100644
index 000000000..0dc85517c
--- /dev/null
+++ b/pipelex/config_template/inference/backends/anthropic.toml
@@ -0,0 +1,67 @@
+default_sdk = "anthropic"
+default_prompting_target = "anthropic"
+
+[claude-3-haiku]
+model_id = "claude-3-haiku-20240307"
+max_tokens = 4096
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 0.25, output = 1.25 }
+
+[claude-3-opus]
+model_id = "claude-3-opus-20240229"
+max_tokens = 4096
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 15.0, output = 75.0 }
+
+["claude-3.5-sonnet"]
+model_id = "claude-3-5-sonnet-20240620"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.5-sonnet-v2"]
+model_id = "claude-3-5-sonnet-20241022"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.7-sonnet"]
+model_id = "claude-3-7-sonnet-20250219"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-sonnet]
+model_id = "claude-sonnet-4-20250514"
+max_tokens = 64000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-opus]
+model_id = "claude-opus-4-20250514"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-4.1-opus"]
+model_id = "claude-opus-4-1-20250805"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
diff --git a/pipelex/config_template/inference/backends/azure_openai.toml b/pipelex/config_template/inference/backends/azure_openai.toml
new file mode 100644
index 000000000..b26178854
--- /dev/null
+++ b/pipelex/config_template/inference/backends/azure_openai.toml
@@ -0,0 +1,69 @@
+default_sdk = "azure_openai"
+default_prompting_target = "openai"
+
+[gpt-4o]
+model_id = "gpt-4o-2024-11-20"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o-mini]
+model_id = "gpt-4o-mini-2024-07-18"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+["gpt-4.1"]
+model_id = "gpt-4.1-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "gpt-4.1-mini-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "gpt-4.1-nano-2025-04-14"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[o1-mini]
+model_id = "o1-mini-2024-09-12"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 3.0, output = 12.0 }
+
+[o1]
+model_id = "o1-2024-12-17"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.0, output = 60.0 }
+
+[o3-mini]
+model_id = "o3-mini-2025-01-31"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+
+[gpt-5-mini]
+model_id = "gpt-5-mini-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.0 }
+
+[gpt-5-nano]
+model_id = "gpt-5-nano-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.4 }
+
+[gpt-5-chat]
+model_id = "gpt-5-chat-2025-08-07"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+
diff --git a/pipelex/config_template/inference/backends/bedrock.toml b/pipelex/config_template/inference/backends/bedrock.toml
new file mode 100644
index 000000000..447c76f03
--- /dev/null
+++ b/pipelex/config_template/inference/backends/bedrock.toml
@@ -0,0 +1,84 @@
+default_sdk = "bedrock_aioboto3"
+default_prompting_target = "anthropic"
+
+[bedrock-mistral-large]
+model_id = "mistral.mistral-large-2407-v1:0"
+max_tokens = 8192
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 4.0, output = 12.0 }
+
+[bedrock-meta-llama-3-3-70b-instruct]
+model_id = "us.meta.llama3-3-70b-instruct-v1:0"
+max_tokens = 8192
+inputs = ["text"]
+outputs = ["text"]
+# TODO: find out the actual cost per million tokens for llama3 on bedrock
+costs = { input = 3.0, output = 15.0 }
+
+[bedrock-nova-pro]
+model_id = "us.amazon.nova-pro-v1:0"
+max_tokens = 5120
+inputs = ["text"]
+outputs = ["text"]
+# TODO: find out the actual cost per million tokens for nova on bedrock
+costs = { input = 3.0, output = 15.0 }
+
+################################################################################
+# Anthropic Models
+################################################################################
+
+["claude-3.5-sonnet"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-3-5-sonnet-20240620-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.5-sonnet-v2"]
+sdk = "bedrock_anthropic"
+model_id = "anthropic.claude-3-5-sonnet-20241022-v2:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-3.7-sonnet"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-3-7-sonnet-20250219-v1:0"
+max_tokens = 8192
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-sonnet]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-sonnet-4-20250514-v1:0"
+max_tokens = 64000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+[claude-4-opus]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-opus-4-20250514-v1:0"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
+["claude-4.1-opus"]
+sdk = "bedrock_anthropic"
+model_id = "us.anthropic.claude-opus-4-1-20250805-v1:0"
+max_tokens = 32000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 100
+costs = { input = 3.0, output = 15.0 }
+
diff --git a/pipelex/config_template/inference/backends/blackboxai.toml b/pipelex/config_template/inference/backends/blackboxai.toml
new file mode 100644
index 000000000..14ca78283
--- /dev/null
+++ b/pipelex/config_template/inference/backends/blackboxai.toml
@@ -0,0 +1,156 @@
+
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+# OpenAI Models
+[gpt-4o-mini]
+model_id = "blackboxai/openai/gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.60 }
+
+[gpt-4o]
+model_id = "blackboxai/openai/gpt-4o"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.50, output = 10.00 }
+
+[o1-mini]
+model_id = "blackboxai/openai/o1-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.10, output = 4.40 }
+
+[o4-mini]
+model_id = "blackboxai/openai/o4-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.10, output = 4.40 }
+
+["gpt-4.5-preview"]
+model_id = "blackboxai/openai/gpt-4.5-preview"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 75.00, output = 150.00 }
+
+# Anthropic Models
+["claude-3.5-haiku"]
+model_id = "blackboxai/anthropic/claude-3.5-haiku"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.80, output = 4.00 }
+
+["claude-3.5-sonnet"]
+model_id = "blackboxai/anthropic/claude-3.5-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+["claude-3.7-sonnet"]
+model_id = "blackboxai/anthropic/claude-3.7-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+[claude-opus-4]
+model_id = "blackboxai/anthropic/claude-opus-4"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.00, output = 75.00 }
+
+[claude-sonnet-4]
+model_id = "blackboxai/anthropic/claude-sonnet-4"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3.00, output = 15.00 }
+
+# Google Models
+["gemini-2.5-flash"]
+model_id = "blackboxai/google/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.30, output = 2.50 }
+
+["gemini-2.5-pro"]
+model_id = "blackboxai/google/gemini-2.5-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.00 }
+
+["gemini-flash-1.5-8b"]
+model_id = "blackboxai/google/gemini-flash-1.5-8b"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.15 }
+
+# Free Models
+[deepseek-chat-free]
+model_id = "blackboxai/deepseek/deepseek-chat:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+[deepseek-r1-free]
+model_id = "blackboxai/deepseek/deepseek-r1:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+["llama-3.3-70b-instruct-free"]
+model_id = "blackboxai/meta-llama/llama-3.3-70b-instruct:free"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.00, output = 0.00 }
+
+# Mistral Models
+[mistral-large]
+model_id = "blackboxai/mistralai/mistral-large"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 2.00, output = 6.00 }
+
+[pixtral-large-2411]
+model_id = "blackboxai/mistralai/pixtral-large-2411"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.00, output = 6.00 }
+
+# Cost-Effective Models
+["llama-3.3-70b-instruct"]
+model_id = "blackboxai/meta-llama/llama-3.3-70b-instruct"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.12 }
+
+["qwen-2.5-72b-instruct"]
+model_id = "blackboxai/qwen/qwen-2.5-72b-instruct"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.12, output = 0.39 }
+
+# Vision Models
+["llama-3.2-11b-vision-instruct"]
+model_id = "blackboxai/meta-llama/llama-3.2-11b-vision-instruct"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.05 }
+
+["qwen2.5-vl-72b-instruct"]
+model_id = "blackboxai/qwen/qwen2.5-vl-72b-instruct"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 0.75 }
+
+# Amazon Nova Models
+[nova-micro-v1]
+model_id = "blackboxai/amazon/nova-micro-v1"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.14 }
+
+[nova-lite-v1]
+model_id = "blackboxai/amazon/nova-lite-v1"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.06, output = 0.24 }
+
diff --git a/pipelex/config_template/inference/backends/mistral.toml b/pipelex/config_template/inference/backends/mistral.toml
new file mode 100644
index 000000000..13f61e4a3
--- /dev/null
+++ b/pipelex/config_template/inference/backends/mistral.toml
@@ -0,0 +1,94 @@
+default_sdk = "mistral"
+default_prompting_target = "mistral"
+
+[ministral-3b]
+model_id = "ministral-3b-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.04, output = 0.04 }
+
+[ministral-8b]
+model_id = "ministral-8b-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.1 }
+
+[mistral-7b-2312]
+model_id = "mistral-large-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 0.25 }
+
+[mistral-8x7b-2312]
+model_id = "open-mixtral-8x7b"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.7, output = 0.7 }
+
+[mistral-codestral-2405]
+model_id = "codestral-2405"
+max_tokens = 262144
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 1.0, output = 3.0 }
+
+[mistral-large-2402]
+model_id = "mistral-large-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 4.0, output = 12.0 }
+
+[mistral-large]
+model_id = "mistral-large-latest"
+max_tokens = 131072
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 4.0, output = 12.0 }
+
+[mistral-small-2402]
+model_id = "mistral-small-2402"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.0, output = 3.0 }
+
+[mistral-small]
+model_id = "mistral-small-latest"
+max_tokens = 32768
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.0, output = 3.0 }
+
+[pixtral-12b]
+model_id = "pixtral-12b-latest"
+max_tokens = 131072
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.15 }
+
+[pixtral-large]
+model_id = "pixtral-large-latest"
+max_tokens = 131072
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.0, output = 6.0 }
+
+[mistral-medium]
+model_id = "mistral-medium-latest"
+max_tokens = 128000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 2.0 }
+
+[mistral-medium-2508]
+model_id = "mistral-medium-2508"
+max_tokens = 128000
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 2.0 }
+
diff --git a/pipelex/config_template/inference/backends/ollama.toml b/pipelex/config_template/inference/backends/ollama.toml
new file mode 100644
index 000000000..2b66dee18
--- /dev/null
+++ b/pipelex/config_template/inference/backends/ollama.toml
@@ -0,0 +1,31 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[gemma3-4b]
+model_id = "gemma3:4b"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+[llama4-scout]
+model_id = "llama4:scout"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+["mistral-small3.1-24b"]
+model_id = "mistral-small3.1:24b"
+inputs = ["text"]
+outputs = ["text"]
+max_prompt_images = 3000
+costs = { input = 0, output = 0 }
+
+[qwen3-8b]
+model_id = "qwen3:8b"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0, output = 0 }
+# TODO: support <think> tokens
+
diff --git a/pipelex/config_template/inference/backends/openai.toml b/pipelex/config_template/inference/backends/openai.toml
new file mode 100644
index 000000000..6e3619499
--- /dev/null
+++ b/pipelex/config_template/inference/backends/openai.toml
@@ -0,0 +1,124 @@
+default_sdk = "openai"
+default_prompting_target = "openai"
+
+["gpt-3.5-turbo"]
+model_id = "gpt-3.5-turbo-1106"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 0.5, output = 1.5 }
+
+[gpt-4]
+model_id = "gpt-4"
+costs = { input = 30.0, output = 60.0 }
+
+[gpt-4-turbo]
+model_id = "gpt-4-turbo"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 10.0, output = 30.0 }
+
+[gpt-4o-2024-11-20]
+model_id = "gpt-4o-2024-11-20"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o]
+model_id = "gpt-4o"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2.5, output = 10.0 }
+
+[gpt-4o-mini-2024-07-18]
+model_id = "gpt-4o-mini-2024-07-18"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+[gpt-4o-mini]
+model_id = "gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.15, output = 0.6 }
+
+["gpt-4.1"]
+model_id = "gpt-4.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "gpt-4.1-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "gpt-4.1-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[o1-mini]
+model_id = "o1-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3.0, output = 12.0 }
+constraints = ["temperature_must_be_1"]
+
+[o1]
+model_id = "o1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15.0, output = 60.0 }
+constraints = ["temperature_must_be_1"]
+
+[o3-mini]
+model_id = "o3-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+constraints = ["temperature_must_be_1"]
+
+[o3]
+model_id = "o3"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 10.0, output = 40.0 }
+constraints = ["temperature_must_be_1"]
+
+[o4-mini]
+model_id = "o4-mini"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = { input = 1.1, output = 4.4 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5]
+model_id = "gpt-5"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-mini]
+model_id = "gpt-5-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.0 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-nano]
+model_id = "gpt-5-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.4 }
+constraints = ["temperature_must_be_1"]
+
+[gpt-5-chat]
+model_id = "gpt-5-chat-latest"
+inputs = ["text", "images"]
+outputs = ["text"]
+costs = { input = 1.25, output = 10.0 }
+constraints = ["temperature_must_be_1"]
+
diff --git a/pipelex/config_template/inference/backends/perplexity.toml b/pipelex/config_template/inference/backends/perplexity.toml
new file mode 100644
index 000000000..4eb004db5
--- /dev/null
+++ b/pipelex/config_template/inference/backends/perplexity.toml
@@ -0,0 +1,32 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[sonar-pro]
+model_id = "sonar-pro"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar]
+model_id = "sonar"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-deep-research]
+model_id = "sonar-deep-research"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-reasoning-pro]
+model_id = "sonar-reasoning-pro"
+inputs = ["text"]
+outputs = ["text"]
+
+[sonar-reasoning]
+model_id = "sonar-reasoning"
+inputs = ["text"]
+outputs = ["text"]
+
+[perplexity-deepseek-r1]
+model_id = "r1-1776"
+inputs = ["text"]
+outputs = ["text"]
diff --git a/pipelex/config_template/inference/backends/pipelex_inference.toml b/pipelex/config_template/inference/backends/pipelex_inference.toml
new file mode 100644
index 000000000..22016adc7
--- /dev/null
+++ b/pipelex/config_template/inference/backends/pipelex_inference.toml
@@ -0,0 +1,104 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+# OpenAI Models
+[gpt-4o-mini]
+model_id = "pipelex/gpt-4o-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.17, output = 0.66 }
+
+["gpt-4.1"]
+model_id = "pipelex/gpt-4.1"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 2, output = 8 }
+
+["gpt-4.1-mini"]
+model_id = "pipelex/gpt-4.1-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.4, output = 1.6 }
+
+["gpt-4.1-nano"]
+model_id = "pipelex/gpt-4.1-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.4 }
+
+[gpt-5-nano]
+model_id = "pipelex/gpt-5-nano"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.05, output = 0.40 }
+
+[gpt-5-mini]
+model_id = "pipelex/gpt-5-mini"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.25, output = 2.00 }
+
+[gpt-5-chat]
+model_id = "pipelex/gpt-5-chat"
+inputs = ["text", "images"]
+outputs = ["text"]
+costs = { input = 1.25, output = 10.00 }
+
+[gpt-image-1]
+model_id = "pipelex/gpt-image-1"
+inputs = ["text"]
+outputs = ["images"]
+costs = { input = 5, output = 0.01 }
+
+# Anthropic Models
+["claude-4-sonnet"]
+model_id = "pipelex/claude-4-sonnet"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 3, output = 15 }
+
+["claude-4.1-opus"]
+model_id = "pipelex/claude-4.1-opus"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 15, output = 75 }
+
+# Google Models using Google API
+["gemini-2.0-flash"]
+model_id = "gemini/gemini-2.0-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.10, output = 0.40 }
+
+["gemini-2.5-flash"]
+model_id = "gemini/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.30, output = 2.50 }
+
+["gemini-2.5-flash-lite"]
+model_id = "gemini/gemini-2.5-flash-lite"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.10, output = 0.40 }
+
+# XAI Models
+[grok-3]
+model_id = "grok-3"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3, output = 15 }
+
+[grok-3-mini]
+model_id = "grok-3-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.3, output = 0.5 }
+
+# Flux Models
+[fast-lightning-sdxl]
+model_id = "pipelex/fast-lightning-sdxl"
+inputs = ["text"]
+outputs = ["images"]
+costs = { input = 0.01, output = 0.01 }
+
diff --git a/pipelex/config_template/inference/backends/vertexai.toml b/pipelex/config_template/inference/backends/vertexai.toml
new file mode 100644
index 000000000..e476cfcb1
--- /dev/null
+++ b/pipelex/config_template/inference/backends/vertexai.toml
@@ -0,0 +1,24 @@
+default_sdk = "openai"
+default_prompting_target = "gemini"
+
+["gemini-2.0-flash"]
+model_id = "google/gemini-2.0-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 0.1, output = 0.4 }
+
+["gemini-2.5-pro"]
+model_id = "google/gemini-2.5-pro"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 1.25, output = 10.0 }
+
+["gemini-2.5-flash"]
+model_id = "google/gemini-2.5-flash"
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+max_prompt_images = 3000
+costs = { input = 0.30, output = 2.50 }
+
diff --git a/pipelex/config_template/inference/backends/xai.toml b/pipelex/config_template/inference/backends/xai.toml
new file mode 100644
index 000000000..ea8fb58f7
--- /dev/null
+++ b/pipelex/config_template/inference/backends/xai.toml
@@ -0,0 +1,27 @@
+default_sdk = "openai"
+default_prompting_target = "anthropic"
+
+[grok-3]
+model_id = "grok-3"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 3, output = 15 }
+
+[grok-3-mini]
+model_id = "grok-3-mini"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.3, output = 0.5 }
+
+[grok-3-fast]
+model_id = "grok-3-fast-latest"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 5, output = 25 }
+
+[grok-3-mini-fast]
+model_id = "grok-3-mini-fast-latest"
+inputs = ["text"]
+outputs = ["text"]
+costs = { input = 0.15, output = 4 }
+
diff --git a/pipelex/config_template/inference/deck/base_deck.toml b/pipelex/config_template/inference/deck/base_deck.toml
new file mode 100644
index 000000000..0736f0347
--- /dev/null
+++ b/pipelex/config_template/inference/deck/base_deck.toml
@@ -0,0 +1,93 @@
+
+# LLM Deck base
+
+####################################################################################################
+# Aliases
+####################################################################################################
+
+[aliases]
+base-claude = "claude-4-sonnet"
+base-gpt = "gpt-5"
+base-gemini = "gemini-2.5-flash"
+base-mistral = "mistral-medium"
+
+best-gpt = "gpt-5"
+best-claude = "claude-4.1-opus"
+best-gemini = "gemini-2.5-pro"
+best-mistral = "mistral-medium"
+best-grok = "grok-3"
+
+cheap_llm_for_text = "gpt-4o-mini"
+cheap_llm_for_object = "gpt-4o-mini"
+
+base_llm_for_text = "gpt-4o-mini"
+base_llm_for_object = "gpt-4o-mini"
+
+llm_to_engineer = ["claude-4.1-opus", "gemini-2.5-pro"]
+llm_for_large_codebase = ["gemini-2.5-pro", "claude-4-sonnet", "gpt-5"]
+
+####################################################################################################
+# LLM Presets
+####################################################################################################
+
+[llm_presets]
+
+####################################################################################################
+# LLM Presets — General purpose
+
+cheap_llm_for_text = { llm_handle = "cheap_llm_for_text", temperature = 0.5 }
+cheap_llm_for_short_text = { llm_handle = "cheap_llm_for_text", temperature = 0.5, max_tokens = 50 }
+cheap_llm_for_object = { llm_handle = "cheap_llm_for_object", temperature = 0.5 }
+cheap_llm_to_structure = { llm_handle = "cheap_llm_for_object", temperature = 0.1 }
+
+llm_for_testing_gen_text = { llm_handle = "base_llm_for_text", temperature = 0.5 }
+llm_for_testing_gen_object = { llm_handle = "base_llm_for_object", temperature = 0.5 }
+
+####################################################################################################
+# LLM Presets — Specific skills
+
+# Cheap llm
+llm_cheap = { llm_handle = "base-gemini", temperature = 0.1 }
+
+# Generation skills
+llm_for_factual_writing = { llm_handle = "base-gpt", temperature = 0.1 }
+llm_for_creative_writing = { llm_handle = "claude-4-sonnet", temperature = 0.9 }
+
+# Reasoning skills
+llm_to_reason_short = { llm_handle = "base-claude", temperature = 0.5, max_tokens = 500 }
+llm_to_reason = { llm_handle = "base-claude", temperature = 1 }
+llm_to_reason_on_diagram = { llm_handle = "best-claude", temperature = 0.5 }
+
+# Search and answer skills
+llm_to_answer = { llm_handle = "best-claude", temperature = 0.1 }
+llm_to_retrieve = { llm_handle = "best-gemini", temperature = 0.1 }
+llm_for_enrichment = { llm_handle = "gpt-4o", temperature = 0.1 }
+llm_to_enrich = { llm_handle = "best-claude", temperature = 0.1 }
+llm_for_question_and_excerpt_reformulation = { llm_handle = "gpt-4o", temperature = 0.9 }
+
+# Engineering skills
+llm_to_engineer = { llm_handle = "best-claude", temperature = 0.2 }
+llm_to_pipelex = { llm_handle = "best-claude", temperature = 0.2 }
+# llm_to_pipelex = { llm_handle = "gpt-5-mini", temperature = 0.2 }
+
+# Image skills
+llm_to_write_imgg_prompt = { llm_handle = "best-claude", temperature = 0.2 }
+llm_to_describe_img = { llm_handle = "best-claude", temperature = 0.5 }
+llm_to_design_fashion = { llm_handle = "best-claude", temperature = 0.7 }
+llm_for_img_to_text = { llm_handle = "best-claude", temperature = 0.1 }
+
+# Extraction skills
+llm_to_extract_diagram = { llm_handle = "best-claude", temperature = 0.5 }
+llm_to_extract_invoice = { llm_handle = "claude-3-7-sonnet", temperature = 0.1 }
+llm_to_extract_invoice_from_scan = { llm_handle = "best-claude", temperature = 0.5 }
+llm_to_extract_legal_terms = { llm_handle = "best-claude", temperature = 0.1 }
+llm_to_extract_tables = { llm_handle = "best-claude", temperature = 0.1 }
+
+####################################################################################################
+# LLM Choices
+####################################################################################################
+
+[llm_choice_defaults]
+for_text = "cheap_llm_for_text"
+for_object = "cheap_llm_for_object"
+
diff --git a/pipelex/config_template/inference/deck/overrides.toml b/pipelex/config_template/inference/deck/overrides.toml
new file mode 100644
index 000000000..999cc4d1a
--- /dev/null
+++ b/pipelex/config_template/inference/deck/overrides.toml
@@ -0,0 +1,10 @@
+
+
+####################################################################################################
+# LLM Deck overrides
+####################################################################################################
+
+[llm_choice_overrides]
+for_text = "disabled"
+for_object = "disabled"
+
diff --git a/pipelex/config_template/inference/routing_profiles.toml b/pipelex/config_template/inference/routing_profiles.toml
new file mode 100644
index 000000000..9a26b8f4c
--- /dev/null
+++ b/pipelex/config_template/inference/routing_profiles.toml
@@ -0,0 +1,84 @@
+# Routing profile library - Routes models to their backends
+# ================================================
+# This file controls which backend serves which model.
+# Simply change the 'active' field to switch profiles,
+# or you can add your own custom profiles.
+# ================================================
+
+# Which profile to use (change this to switch routing)
+active = "all_pipelex"
+
+# We recommend using the "all_pipelex" profile to get a head start with all models.
+# The Pipelex Inference backend is currently not recommended for production use,
+# but it's great for development and testing.
+# To use the Pipelex Inference backend (all_pipelex profile):
+# 1. Join our Discord community to get your free API key (no credit card required):
+# Visit https://go.pipelex.com/discord and request your key in the appropriate channel
+# 2. Set the environment variable: export PIPELEX_INFERENCE_API_KEY="your-api-key"
+# 3. The .pipelex/inference/backends.toml is already configured with api_key = "${PIPELEX_INFERENCE_API_KEY}"
+# which will get the key from the environment variable.
+
+# ============================================
+# Routing Profiles
+# ============================================
+
+[profiles.all_pipelex]
+description = "Use Pipelex Inference backend for all models"
+default = "pipelex_inference"
+
+[profiles.all_anthropic]
+description = "Use Anthropic backend for all its supported models"
+default = "anthropic"
+
+[profiles.all_azure_openai]
+description = "Use Azure OpenAI backend for all its supported models"
+default = "azure_openai"
+
+[profiles.to_test_various_backends]
+description = "Route models to various backends"
+
+[profiles.custom_routing]
+description = "Custom routing"
+default = "pipelex_inference"
+
+[profiles.custom_routing.routes]
+# Pattern matching: "model-pattern" = "backend-name"
+"gpt-*" = "pipelex_inference"
+"claude-*" = "pipelex_inference"
+"gemini-*" = "pipelex_inference"
+"grok-*" = "pipelex_inference"
+"*-sdxl" = "pipelex_inference"
+
+[profiles.to_test_various_backends.routes]
+"gpt-5-nano" = "pipelex_inference"
+"gpt-4o-mini" = "blackboxai"
+"gpt-5-mini" = "openai"
+"gpt-5-chat" = "azure_openai"
+
+"claude-4-sonnet" = "pipelex_inference"
+"claude-3.5-sonnet" = "bedrock"
+"claude-3.5-sonnet-v2" = "anthropic"
+"claude-3.7-sonnet" = "blackboxai"
+
+"gemini-2.5-flash-lite" = "pipelex_inference"
+"gemini-2.5-flash" = "blackboxai"
+"gemini-2.5-pro" = "vertexai"
+
+"grok-3" = "pipelex_inference"
+"grok-3-mini" = "xai"
+
+# ============================================
+# Custom Profiles
+# ============================================
+# Add your own profiles below following the same pattern:
+#
+# [profiles.your_profile_name]
+# description = "What this profile does"
+# default = "backend-name" # Where to route models by default
+# [profiles.your_profile_name.routes]
+# "model-pattern" = "backend-name" # Specific routing rules
+#
+# Pattern matching supports:
+# - Exact names: "gpt-4o-mini"
+# - Wildcards: "claude-*" (matches all models starting with claude-)
+# - Partial wildcards: "*-sonnet" (matches all sonnet variants)
diff --git a/pipelex/pipelex_template.toml b/pipelex/config_template/pipelex.toml
similarity index 78%
rename from pipelex/pipelex_template.toml
rename to pipelex/config_template/pipelex.toml
index 030f8bc14..e5ef1aa66 100644
--- a/pipelex/pipelex_template.toml
+++ b/pipelex/config_template/pipelex.toml
@@ -1,10 +1,3 @@
-[pipelex]
-[pipelex.feature_config]
-# WIP/Experimental feature flags
-is_pipeline_tracking_enabled = true
-is_activity_tracking_enabled = true
-is_reporting_enabled = true
-
[pipelex.aws_config]
api_key_method = "env"
# The possible values are "env" and "secret_provider".
@@ -12,9 +5,6 @@ api_key_method = "env"
# "secret_provider" means that the env var are stored in your Secret Manager (See the doc for injecting a secret provider).
[cogt]
-[cogt.llm_config.preferred_platforms]
-# This overrrides the defaults set for any llm handle
-"gpt-4o-mini" = "openai"
####################################################################################################
# OCR config
@@ -24,3 +14,9 @@ api_key_method = "env"
ocr_handles = ["mistral/mistral-ocr-latest"]
page_output_text_file_name = "page_text.md"
+[pipelex.feature_config]
+# WIP/Experimental feature flags
+is_pipeline_tracking_enabled = false
+is_activity_tracking_enabled = false
+is_reporting_enabled = true
+
diff --git a/pipelex/core/concepts/concept_library.py b/pipelex/core/concepts/concept_library.py
index a5c04c426..9748c480e 100644
--- a/pipelex/core/concepts/concept_library.py
+++ b/pipelex/core/concepts/concept_library.py
@@ -1,7 +1,7 @@
from typing import Any, Dict, List, Optional, Type
from pydantic import Field, RootModel
-from typing_extensions import override
+from typing_extensions import Self, override
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
@@ -44,7 +44,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptProviderAbstract):
self.root = {}
@classmethod
- def make_empty(cls):
+ def make_empty(cls) -> Self:
return cls(root={})
@override
diff --git a/pipelex/core/domains/domain_library.py b/pipelex/core/domains/domain_library.py
index f0d72015e..52ac5501d 100644
--- a/pipelex/core/domains/domain_library.py
+++ b/pipelex/core/domains/domain_library.py
@@ -1,7 +1,7 @@
from typing import Dict, Optional
from pydantic import RootModel
-from typing_extensions import override
+from typing_extensions import Self, override
from pipelex.core.domains.domain import Domain
from pipelex.core.domains.domain_provider_abstract import DomainProviderAbstract
@@ -18,7 +18,7 @@ class DomainLibrary(RootModel[DomainLibraryRoot], DomainProviderAbstract):
self.root = {}
@classmethod
- def make_empty(cls):
+ def make_empty(cls) -> Self:
return cls(root={})
def add_domain(self, domain: Domain):
diff --git a/pipelex/core/pipes/pipe_library.py b/pipelex/core/pipes/pipe_library.py
index 912c9594a..cccb95632 100644
--- a/pipelex/core/pipes/pipe_library.py
+++ b/pipelex/core/pipes/pipe_library.py
@@ -4,7 +4,7 @@ from typing import Dict, List, Optional
from pydantic import RootModel
from rich import box
from rich.table import Table
-from typing_extensions import override
+from typing_extensions import Self, override
from pipelex import pretty_print
from pipelex.core.pipes.pipe_abstract import PipeAbstract
@@ -36,7 +36,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeProviderAbstract):
raise PipeLibraryError(f"Missing dependency for pipe '{pipe.code}': {not_found_error}") from not_found_error
@classmethod
- def make_empty(cls):
+ def make_empty(cls) -> Self:
return cls(root={})
@override
diff --git a/pipelex/hub.py b/pipelex/hub.py
index 11adbcd5b..f63039f1d 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -8,11 +8,8 @@ from pipelex.cogt.content_generation.content_generator_protocol import (
)
from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
from pipelex.cogt.inference.inference_manager_protocol import InferenceManagerProtocol
-from pipelex.cogt.llm.llm_models.llm_deck_abstract import LLMDeckAbstract
-from pipelex.cogt.llm.llm_models.llm_model_provider_abstract import (
- LLMModelProviderAbstract,
-)
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
+from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
from pipelex.cogt.ocr.ocr_worker_abstract import OcrWorkerAbstract
from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.concept_provider_abstract import ConceptProviderAbstract
@@ -28,8 +25,8 @@ from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract
from pipelex.pipeline.track.pipeline_tracker_protocol import PipelineTrackerProtocol
from pipelex.plugins.plugin_manager import PluginManager
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.tools.config.config_root import ConfigRoot
from pipelex.tools.config.manager import config_manager
-from pipelex.tools.config.models import ConfigRoot
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
from pipelex.tools.templating.template_provider_abstract import TemplateProviderAbstract
@@ -52,8 +49,7 @@ class PipelexHub:
self._class_registry: Optional[ClassRegistryAbstract] = None
self._storage_provider: Optional[StorageProviderAbstract] = None
# cogt
- self._llm_models_provider: Optional[LLMModelProviderAbstract] = None
- self._llm_deck_provider: Optional[LLMDeckAbstract] = None
+ self._models_manager: Optional[ModelManagerAbstract] = None
self._plugin_manager: Optional[PluginManager] = None
self._inference_manager: InferenceManagerProtocol
self._report_delegate: ReportingProtocol
@@ -129,11 +125,8 @@ class PipelexHub:
# cogt
- def set_llm_models_provider(self, llm_models_provider: LLMModelProviderAbstract):
- self._llm_models_provider = llm_models_provider
-
- def set_llm_deck_provider(self, llm_deck_provider: LLMDeckAbstract):
- self._llm_deck_provider = llm_deck_provider
+ def set_models_manager(self, models_manager: ModelManagerAbstract):
+ self._models_manager = models_manager
def set_plugin_manager(self, plugin_manager: PluginManager):
self._plugin_manager = plugin_manager
@@ -217,18 +210,10 @@ class PipelexHub:
# cogt
- def get_required_llm_models_provider(self) -> LLMModelProviderAbstract:
- if self._llm_models_provider is None:
- raise RuntimeError("LLMModelProvider is not initialized")
- return self._llm_models_provider
-
- def get_optional_llm_models_provider(self) -> Optional[LLMModelProviderAbstract]:
- return self._llm_models_provider
-
- def get_required_llm_deck(self) -> LLMDeckAbstract:
- if self._llm_deck_provider is None:
- raise RuntimeError("LLMDeck is not initialized")
- return self._llm_deck_provider
+ def get_required_models_manager(self) -> ModelManagerAbstract:
+ if self._models_manager is None:
+ raise RuntimeError("ModelsManager is not initialized")
+ return self._models_manager
def get_plugin_manager(self) -> PluginManager:
if self._plugin_manager is None:
@@ -341,12 +326,8 @@ def get_class_registry() -> ClassRegistryAbstract:
# cogt
-def get_llm_models_provider() -> LLMModelProviderAbstract:
- return get_pipelex_hub().get_required_llm_models_provider()
-
-
-def get_llm_deck() -> LLMDeckAbstract:
- return get_pipelex_hub().get_required_llm_deck()
+def get_models_manager() -> ModelManagerAbstract:
+ return get_pipelex_hub().get_required_models_manager()
def get_plugin_manager() -> PluginManager:
diff --git a/pipelex/libraries/library_config.py b/pipelex/libraries/library_config.py
index b4b5f016c..3bc048d19 100644
--- a/pipelex/libraries/library_config.py
+++ b/pipelex/libraries/library_config.py
@@ -1,6 +1,6 @@
from typing import ClassVar, List
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.tools.misc.file_utils import copy_file_from_package, copy_folder_from_package, find_files_in_dir
PIPELEX_LIBRARIES_PATH = "libraries"
@@ -10,10 +10,6 @@ class LibraryConfig(ConfigModel):
package_name: ClassVar[str] = "pipelex"
config_dir_path: str = "pipelex_libraries"
- @property
- def config_file_path(self) -> str:
- return f"{self.config_dir_path}/config.toml"
-
@property
def pipelines_dir_path(self) -> str:
return f"{self.config_dir_path}/pipelines"
@@ -22,22 +18,10 @@ class LibraryConfig(ConfigModel):
def base_pipelines_dir_path(self) -> str:
return f"{self.config_dir_path}/pipelines/base_library"
- @property
- def llm_integrations_dir_path(self) -> str:
- return f"{self.config_dir_path}/llm_integrations"
-
- @property
- def llm_deck_dir_path(self) -> str:
- return f"{self.config_dir_path}/llm_deck"
-
@property
def templates_dir_path(self) -> str:
return f"{self.config_dir_path}/templates"
- @property
- def plugins_dir_path(self) -> str:
- return f"{self.config_dir_path}/plugins"
-
@property
def test_pipelines_dir_path(self) -> str:
return "tests/test_pipelines"
@@ -46,60 +30,30 @@ class LibraryConfig(ConfigModel):
def failing_pipelines_file_paths(self) -> List[str]:
return ["tests/test_pipelines/failing_pipelines.plx"]
- def get_llm_deck_paths(self) -> List[str]:
- llm_deck_paths = [str(path) for path in find_files_in_dir(dir_path=self.llm_deck_dir_path, pattern="*.toml", is_recursive=True)]
- llm_deck_paths.sort()
- return llm_deck_paths
-
def get_templates_paths(self) -> List[str]:
return [str(path) for path in find_files_in_dir(dir_path=self.templates_dir_path, pattern="*.toml", is_recursive=True)]
- def get_default_plugin_config_path(self) -> str:
- """Get the default plugin config path."""
- return f"{self.plugins_dir_path}/plugin_config.toml"
-
def export_libraries(self, overwrite: bool = False) -> None:
"""Duplicate pipelex libraries files in the client project, preserving directory structure."""
- # pipelines
- copy_folder_from_package(
- package_name=self.package_name,
- folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines",
- target_dir=self.base_pipelines_dir_path,
- overwrite=overwrite,
- )
copy_file_from_package(
package_name=self.package_name,
file_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/__init__.py",
target_path=f"{self.config_dir_path}/__init__.py",
overwrite=overwrite,
)
- copy_file_from_package(
+
+ # pipelines
+ copy_folder_from_package(
package_name=self.package_name,
- file_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines/__init__.py",
- target_path=f"{self.pipelines_dir_path}/__init__.py",
+ folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines",
+ target_dir=self.base_pipelines_dir_path,
overwrite=overwrite,
)
copy_file_from_package(
package_name=self.package_name,
file_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/pipelines/__init__.py",
- target_path=f"{self.base_pipelines_dir_path}/__init__.py",
- overwrite=overwrite,
- )
- # llm_integrations
- copy_folder_from_package(
- package_name=self.package_name,
- folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/llm_integrations",
- target_dir=self.llm_integrations_dir_path,
- overwrite=overwrite,
- )
-
- # llm_deck
- copy_folder_from_package(
- package_name=self.package_name,
- folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/llm_deck",
- target_dir=self.llm_deck_dir_path,
+ target_path=f"{self.pipelines_dir_path}/__init__.py",
overwrite=overwrite,
- non_overwrite_files=["overrides.toml"],
)
# templates
@@ -109,11 +63,3 @@ class LibraryConfig(ConfigModel):
target_dir=self.templates_dir_path,
overwrite=overwrite,
)
-
- # plugins
- copy_folder_from_package(
- package_name=self.package_name,
- folder_path_in_package=f"{PIPELEX_LIBRARIES_PATH}/plugins",
- target_dir=self.plugins_dir_path,
- overwrite=overwrite,
- )
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index f9cfeb094..ad8b5c887 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -1,11 +1,10 @@
import os
from pathlib import Path
-from typing import Any, ClassVar, Dict, List, Optional, Type
+from typing import ClassVar, List, Optional, Type
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.llm.llm_models.llm_deck import LLMDeck
from pipelex.config import get_config
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.concepts.concept import Concept
@@ -30,16 +29,11 @@ from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
from pipelex.tools.class_registry_utils import ClassRegistryUtils
from pipelex.tools.func_registry_utils import FuncRegistryUtils
from pipelex.tools.misc.file_utils import find_files_in_dir
-from pipelex.tools.misc.json_utils import deep_update
-from pipelex.tools.misc.toml_utils import TOMLValidationError, load_toml_from_path, validate_toml_file
+from pipelex.tools.misc.toml_utils import TOMLValidationError, validate_toml_file
from pipelex.tools.runtime_manager import runtime_manager
from pipelex.types import StrEnum
-class LLMDeckNotFoundError(LibraryError):
- pass
-
-
class LibraryComponent(StrEnum):
CONCEPT = "concept"
PIPE = "pipe"
@@ -73,34 +67,18 @@ class LibraryManager(LibraryManagerAbstract):
self.concept_library = concept_library
self.pipe_library = pipe_library
self.library_config = library_config
- self.llm_deck: Optional[LLMDeck] = None
@override
def validate_libraries(self):
log.debug("LibraryManager validating libraries")
- if self.llm_deck is None:
- raise LibraryError("LLM deck is not loaded")
-
- self.llm_deck.validate_llm_presets()
- LLMDeck.final_validate(deck=self.llm_deck)
self.concept_library.validate_with_libraries()
self.pipe_library.validate_with_libraries()
self.domain_library.validate_with_libraries()
- def _validate_plx_files(self):
- """Validate all PLX files used by the library manager for formatting issues."""
- log.debug("LibraryManager validating PLX file formatting")
-
- # Validation of LLM deck paths
- llm_deck_paths = self.library_config.get_llm_deck_paths()
- for llm_deck_path in llm_deck_paths:
- if os.path.exists(llm_deck_path):
- try:
- validate_toml_file(llm_deck_path)
- except TOMLValidationError as exc:
- log.error(f"PLX formatting issues in LLM deck file '{llm_deck_path}': {exc}")
- raise LibraryError(f"PLX validation failed for LLM deck file '{llm_deck_path}': {exc}") from exc
+ def _validate_toml_files(self):
+ """Validate all TOML files used by the library manager for formatting issues."""
+ log.debug("LibraryManager validating TOML file formatting")
# Validation of template paths
template_paths = self.library_config.get_templates_paths()
@@ -118,7 +96,6 @@ class LibraryManager(LibraryManagerAbstract):
@override
def teardown(self) -> None:
- self.llm_deck = None
self.pipe_library.teardown()
self.concept_library.teardown()
self.domain_library.teardown()
@@ -276,24 +253,3 @@ class LibraryManager(LibraryManagerAbstract):
pipes = self._load_pipes_from_blueprint(blueprint)
all_pipes.extend(pipes)
self.pipe_library.add_pipes(pipes=all_pipes)
-
- # TODO: move to LLMDeckManager
- def load_deck(self) -> LLMDeck:
- llm_deck_paths = self.library_config.get_llm_deck_paths()
- full_llm_deck_dict: Dict[str, Any] = {}
- if not llm_deck_paths:
- raise LLMDeckNotFoundError("No LLM deck paths found. Please run `pipelex init-libraries` to create it.")
-
- for llm_deck_path in llm_deck_paths:
- if not os.path.exists(llm_deck_path):
- raise LLMDeckNotFoundError(f"LLM deck path `{llm_deck_path}` not found. Please run `pipelex init-libraries` to create it.")
- try:
- llm_deck_dict = load_toml_from_path(path=llm_deck_path)
- log.debug(f"Loaded LLM deck from {llm_deck_path}")
- deep_update(full_llm_deck_dict, llm_deck_dict)
- except Exception as exc:
- log.error(f"Failed to load LLM deck file '{llm_deck_path}': {exc}")
- raise
-
- self.llm_deck = LLMDeck.model_validate(full_llm_deck_dict)
- return self.llm_deck
diff --git a/pipelex/libraries/llm_integrations/anthropic.toml b/pipelex/libraries/llm_integrations/anthropic.toml
deleted file mode 100644
index 079fbcc13..000000000
--- a/pipelex/libraries/llm_integrations/anthropic.toml
+++ /dev/null
@@ -1,75 +0,0 @@
-
-
-[claude-3.claude-3-haiku.latest]
-max_tokens = 4096
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 0.25, output = 1.25 }
-platform_llm_id = { anthropic = "claude-3-haiku-20240307" }
-
-[claude-3.claude-3-opus.latest]
-max_tokens = 4096
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 15.0, output = 75.0 }
-platform_llm_id = { anthropic = "claude-3-opus-20240229" }
-
-["claude-3.5".claude-3-5-sonnet.latest]
-max_tokens = 8192
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-3-5-sonnet-20240620", bedrock_anthropic = "us.anthropic.claude-3-5-sonnet-20240620-v1:0" }
-default_platform = "anthropic"
-
-["claude-3.5".claude-3-5-sonnet-v2.latest]
-max_tokens = 8192
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-3-5-sonnet-20241022", bedrock_anthropic = "anthropic.claude-3-5-sonnet-20241022-v2:0" }
-default_platform = "anthropic"
-
-["claude-3.7".claude-3-7-sonnet.latest]
-max_tokens = 8192
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-3-7-sonnet-20250219", bedrock_anthropic = "us.anthropic.claude-3-7-sonnet-20250219-v1:0" }
-default_platform = "anthropic"
-
-
-[claude-4.claude-4-sonnet.latest]
-max_tokens = 64000
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-sonnet-4-20250514", bedrock_anthropic = "us.anthropic.claude-sonnet-4-20250514-v1:0" }
-default_platform = "anthropic"
-
-
-[claude-4.claude-4-opus.latest]
-max_tokens = 32000
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-opus-4-20250514", bedrock_anthropic = "us.anthropic.claude-opus-4-20250514-v1:0" }
-default_platform = "anthropic"
-
-
-["claude-4.1".claude-4-1-opus.latest]
-max_tokens = 32000
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 100
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { anthropic = "claude-opus-4-1-20250805", bedrock_anthropic = "us.anthropic.claude-opus-4-1-20250805-v1:0" }
-default_platform = "anthropic"
-
diff --git a/pipelex/libraries/llm_integrations/bedrock.toml b/pipelex/libraries/llm_integrations/bedrock.toml
deleted file mode 100644
index bf4404370..000000000
--- a/pipelex/libraries/llm_integrations/bedrock.toml
+++ /dev/null
@@ -1,31 +0,0 @@
-
-
-[bedrock-mistral-large.bedrock-mistral-large.latest]
-max_tokens = 8192
-is_gen_object_supported = false
-cost_per_million_tokens_usd = { input = 4.0, output = 12.0 }
-platform_llm_id = { bedrock = "mistral.mistral-large-2407-v1:0" }
-
-
-[bedrock-anthropic-claude.bedrock-claude-3-7-sonnet.latest]
-max_tokens = 8192
-is_gen_object_supported = false
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { bedrock = "us.anthropic.claude-3-7-sonnet-20250219-v1:0" }
-
-
-[bedrock-meta-llama-3.bedrock-meta-llama-3-3-70b-instruct.latest]
-max_tokens = 8192
-is_gen_object_supported = false
-# TODO: find out the actual cost per million tokens for llama3 on bedrock
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { bedrock = "us.meta.llama3-3-70b-instruct-v1:0" }
-
-
-[bedrock-amazon-nova.bedrock-nova-pro.latest]
-max_tokens = 5120
-is_gen_object_supported = false
-# TODO: find out the actual cost per million tokens for nova on bedrock
-cost_per_million_tokens_usd = { input = 3.0, output = 15.0 }
-platform_llm_id = { bedrock = "us.amazon.nova-pro-v1:0" }
-
diff --git a/pipelex/libraries/llm_integrations/blackboxai.toml b/pipelex/libraries/llm_integrations/blackboxai.toml
deleted file mode 100644
index 87cf08cf2..000000000
--- a/pipelex/libraries/llm_integrations/blackboxai.toml
+++ /dev/null
@@ -1,153 +0,0 @@
-
-# OpenAI Models
-[custom-blackboxai."blackboxai/openai/gpt-4o-mini".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.60 }
-platform_llm_id = { custom_llm = "blackboxai/openai/gpt-4o-mini" }
-
-[custom-blackboxai."blackboxai/openai/gpt-4o".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.50, output = 10.00 }
-platform_llm_id = { custom_llm = "blackboxai/openai/gpt-4o" }
-
-[custom-blackboxai."blackboxai/openai/o1-mini".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 1.10, output = 4.40 }
-platform_llm_id = { custom_llm = "blackboxai/openai/o1-mini" }
-
-[custom-blackboxai."blackboxai/openai/o4-mini".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 1.10, output = 4.40 }
-platform_llm_id = { custom_llm = "blackboxai/openai/o4-mini" }
-
-[custom-blackboxai."blackboxai/openai/gpt-4.5-preview".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 75.00, output = 150.00 }
-platform_llm_id = { custom_llm = "blackboxai/openai/gpt-4.5-preview" }
-
-# Anthropic Models
-[custom-blackboxai."blackboxai/anthropic/claude-3.5-haiku".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.80, output = 4.00 }
-platform_llm_id = { custom_llm = "blackboxai/anthropic/claude-3.5-haiku" }
-
-[custom-blackboxai."blackboxai/anthropic/claude-3.5-sonnet".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 3.00, output = 15.00 }
-platform_llm_id = { custom_llm = "blackboxai/anthropic/claude-3.5-sonnet" }
-
-[custom-blackboxai."blackboxai/anthropic/claude-3.7-sonnet".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 3.00, output = 15.00 }
-platform_llm_id = { custom_llm = "blackboxai/anthropic/claude-3.7-sonnet" }
-
-[custom-blackboxai."blackboxai/anthropic/claude-opus-4".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 15.00, output = 75.00 }
-platform_llm_id = { custom_llm = "blackboxai/anthropic/claude-opus-4" }
-
-[custom-blackboxai."blackboxai/anthropic/claude-sonnet-4".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 3.00, output = 15.00 }
-platform_llm_id = { custom_llm = "blackboxai/anthropic/claude-sonnet-4" }
-
-# Google Models
-[custom-blackboxai."blackboxai/google/gemini-2.5-flash".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.30, output = 2.50 }
-platform_llm_id = { custom_llm = "blackboxai/google/gemini-2.5-flash" }
-
-[custom-blackboxai."blackboxai/google/gemini-2.5-pro".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 1.25, output = 10.00 }
-platform_llm_id = { custom_llm = "blackboxai/google/gemini-2.5-pro" }
-
-[custom-blackboxai."blackboxai/google/gemini-flash-1.5-8b".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.04, output = 0.15 }
-platform_llm_id = { custom_llm = "blackboxai/google/gemini-flash-1.5-8b" }
-
-# Free Models
-[custom-blackboxai."blackboxai/deepseek/deepseek-chat:free".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.00, output = 0.00 }
-platform_llm_id = { custom_llm = "blackboxai/deepseek/deepseek-chat:free" }
-
-[custom-blackboxai."blackboxai/deepseek/deepseek-r1:free".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.00, output = 0.00 }
-platform_llm_id = { custom_llm = "blackboxai/deepseek/deepseek-r1:free" }
-
-[custom-blackboxai."blackboxai/meta-llama/llama-3.3-70b-instruct:free".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.00, output = 0.00 }
-platform_llm_id = { custom_llm = "blackboxai/meta-llama/llama-3.3-70b-instruct:free" }
-
-# Mistral Models
-[custom-blackboxai."blackboxai/mistralai/mistral-large".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 2.00, output = 6.00 }
-platform_llm_id = { custom_llm = "blackboxai/mistralai/mistral-large" }
-
-[custom-blackboxai."blackboxai/mistralai/pixtral-large-2411".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.00, output = 6.00 }
-platform_llm_id = { custom_llm = "blackboxai/mistralai/pixtral-large-2411" }
-
-# Cost-Effective Models
-[custom-blackboxai."blackboxai/meta-llama/llama-3.3-70b-instruct".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.04, output = 0.12 }
-platform_llm_id = { custom_llm = "blackboxai/meta-llama/llama-3.3-70b-instruct" }
-
-[custom-blackboxai."blackboxai/qwen/qwen-2.5-72b-instruct".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.12, output = 0.39 }
-platform_llm_id = { custom_llm = "blackboxai/qwen/qwen-2.5-72b-instruct" }
-
-# Vision Models
-[custom-blackboxai."blackboxai/meta-llama/llama-3.2-11b-vision-instruct".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.05, output = 0.05 }
-platform_llm_id = { custom_llm = "blackboxai/meta-llama/llama-3.2-11b-vision-instruct" }
-
-[custom-blackboxai."blackboxai/qwen/qwen2.5-vl-72b-instruct".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.25, output = 0.75 }
-platform_llm_id = { custom_llm = "blackboxai/qwen/qwen2.5-vl-72b-instruct" }
-
-# Amazon Nova Models
-[custom-blackboxai."blackboxai/amazon/nova-micro-v1".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.04, output = 0.14 }
-platform_llm_id = { custom_llm = "blackboxai/amazon/nova-micro-v1" }
-
-[custom-blackboxai."blackboxai/amazon/nova-lite-v1".latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.06, output = 0.24 }
-platform_llm_id = { custom_llm = "blackboxai/amazon/nova-lite-v1" }
-
diff --git a/pipelex/libraries/llm_integrations/custom.toml b/pipelex/libraries/llm_integrations/custom.toml
deleted file mode 100644
index ecf6d747f..000000000
--- a/pipelex/libraries/llm_integrations/custom.toml
+++ /dev/null
@@ -1,30 +0,0 @@
-
-
-[custom-gemma-3."gemma3:4b".latest]
-is_gen_object_supported = false
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0, output = 0 }
-platform_llm_id = { custom_llm = "gemma3:4b" }
-
-[custom-llama-4."llama4:scout".latest]
-is_gen_object_supported = false
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0, output = 0 }
-platform_llm_id = { custom_llm = "llama4:scout" }
-
-["custom-mistral-small3.1"."mistral-small3.1".latest]
-is_gen_object_supported = false
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0, output = 0 }
-platform_llm_id = { custom_llm = "mistral-small3.1:24b" }
-
-["custom-qwen3"."qwen3:8b".latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0, output = 0 }
-platform_llm_id = { custom_llm = "qwen3:8b" }
-# TODO: support <think> tokens
-
diff --git a/pipelex/libraries/llm_integrations/mistral.toml b/pipelex/libraries/llm_integrations/mistral.toml
deleted file mode 100644
index d27135fc2..000000000
--- a/pipelex/libraries/llm_integrations/mistral.toml
+++ /dev/null
@@ -1,83 +0,0 @@
-
-
-[ministral.ministral-3b.latest]
-max_tokens = 131072
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.04, output = 0.04 }
-platform_llm_id = { mistral = "ministral-3b-latest" }
-
-[ministral.ministral-8b.latest]
-max_tokens = 131072
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.1, output = 0.1 }
-platform_llm_id = { mistral = "ministral-8b-latest" }
-
-[mistral-7b.mistral-7b."2312"]
-max_tokens = 32768
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.25, output = 0.25 }
-platform_llm_id = { mistral = "mistral-large-2402" }
-
-[mistral-8x7b.mistral-8x7b."2312"]
-max_tokens = 32768
-is_gen_object_supported = false
-cost_per_million_tokens_usd = { input = 0.7, output = 0.7 }
-platform_llm_id = { mistral = "open-mixtral-8x7b" }
-
-[mistral-codestral.mistral-codestral."2405"]
-max_tokens = 262144
-is_gen_object_supported = false
-cost_per_million_tokens_usd = { input = 1.0, output = 3.0 }
-platform_llm_id = { mistral = "codestral-2405" }
-
-[mistral-large.mistral-large."2402"]
-max_tokens = 32768
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 4.0, output = 12.0 }
-platform_llm_id = { mistral = "mistral-large-2402" }
-
-[mistral-large.mistral-large.latest]
-max_tokens = 131072
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 4.0, output = 12.0 }
-platform_llm_id = { mistral = "mistral-large-latest" }
-
-[mistral-small.mistral-small."2402"]
-max_tokens = 32768
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 1.0, output = 3.0 }
-platform_llm_id = { mistral = "mistral-small-2402" }
-
-[mistral-small.mistral-small.latest]
-max_tokens = 32768
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 1.0, output = 3.0 }
-platform_llm_id = { mistral = "mistral-small-latest" }
-
-[pixtral.pixtral-12b.latest]
-max_tokens = 131072
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.15 }
-platform_llm_id = { mistral = "pixtral-12b-latest" }
-
-[pixtral.pixtral-large.latest]
-max_tokens = 131072
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.0, output = 6.0 }
-platform_llm_id = { mistral = "pixtral-large-latest" }
-
-[mistral-medium.mistral-medium.latest]
-max_tokens = 128000
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.4, output = 2.0 }
-platform_llm_id = { mistral = "mistral-medium-latest" }
-
-
-[mistral-medium.mistral-medium."2508"]
-max_tokens = 128000
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.4, output = 2.0 }
-platform_llm_id = { mistral = "mistral-medium-2508" }
-
diff --git a/pipelex/libraries/llm_integrations/openai.toml b/pipelex/libraries/llm_integrations/openai.toml
deleted file mode 100644
index 895e8c3d4..000000000
--- a/pipelex/libraries/llm_integrations/openai.toml
+++ /dev/null
@@ -1,153 +0,0 @@
-["gpt-3.5"."gpt-3.5-turbo".latest]
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 0.5, output = 1.5 }
-platform_llm_id = { openai = "gpt-3.5-turbo-1106" }
-default_platform = "openai"
-
-[gpt-4.gpt-4.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 30.0, output = 60.0 }
-platform_llm_id = { openai = "gpt-4" }
-
-[gpt-4.gpt-4-turbo.0125-preview]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 10.0, output = 30.0 }
-platform_llm_id = { openai = "gpt-4-0125-preview" }
-
-[gpt-4.gpt-4-turbo.1106-preview]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 10.0, output = 30.0 }
-platform_llm_id = { openai = "gpt-4-1106-preview" }
-
-[gpt-4.gpt-4-turbo."2024-04-09"]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 10.0, output = 30.0 }
-platform_llm_id = { openai = "gpt-4-turbo-2024-04-09" }
-
-[gpt-4.gpt-4-turbo.latest]
-is_gen_object_supported = true
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 10.0, output = 30.0 }
-platform_llm_id = { openai = "gpt-4-turbo" }
-
-[gpt-4o.gpt-4o."2024-05-13"]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 5.0, output = 15.0 }
-platform_llm_id = { openai = "gpt-4o-2024-05-13" }
-
-[gpt-4.gpt-4o."2024-08-06"]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.5, output = 10.0 }
-platform_llm_id = { openai = "gpt-4o-2024-08-06" }
-
-[gpt-4.gpt-4o."2024-11-20"]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.5, output = 10.0 }
-platform_llm_id = { azure_openai = "gpt-4o-2024-11-20", openai = "gpt-4o-2024-11-20" }
-default_platform = "openai"
-
-[gpt-4o.gpt-4o.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2.5, output = 10.0 }
-platform_llm_id = { azure_openai = "gpt-4o-2024-11-20", openai = "gpt-4o" }
-default_platform = "openai"
-
-[gpt-4o.gpt-4o-mini."2024-07-18"]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.6 }
-platform_llm_id = { azure_openai = "gpt-4o-mini-2024-07-18", openai = "gpt-4o-mini-2024-07-18" }
-default_platform = "openai"
-
-[gpt-4o.gpt-4o-mini.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.6 }
-platform_llm_id = { azure_openai = "gpt-4o-mini-2024-07-18", openai = "gpt-4o-mini" }
-default_platform = "openai"
-
-["gpt-4.1"."gpt-4.1".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 2, output = 8 }
-platform_llm_id = { azure_openai = "gpt-4.1-2025-04-14", openai = "gpt-4.1" }
-default_platform = "openai"
-
-["gpt-4.1"."gpt-4.1-mini".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.4, output = 1.6 }
-platform_llm_id = { azure_openai = "gpt-4.1-mini-2025-04-14", openai = "gpt-4.1-mini" }
-default_platform = "openai"
-
-["gpt-4.1"."gpt-4.1-nano".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.1, output = 0.4 }
-platform_llm_id = { azure_openai = "gpt-4.1-nano-2025-04-14", openai = "gpt-4.1-nano" }
-default_platform = "openai"
-
-[o.o1-mini.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 3.0, output = 12.0 }
-platform_llm_id = { azure_openai = "o1-mini-2024-09-12", openai = "o1-mini" }
-
-[o.o1.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 15.0, output = 60.0 }
-platform_llm_id = { azure_openai = "o1-2024-12-17", openai = "o1" }
-
-[o.o3-mini.latest]
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 1.1, output = 4.4 }
-platform_llm_id = { azure_openai = "o3-mini-2025-01-31", openai = "o3-mini" }
-default_platform = "openai"
-
-[o.o3.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 10.0, output = 40.0 }
-platform_llm_id = { openai = "o3" }
-
-[o.o4-mini.latest]
-is_gen_object_supported = true
-cost_per_million_tokens_usd = { input = 1.1, output = 4.4 }
-platform_llm_id = { openai = "o4-mini" }
-
-[gpt-5.gpt-5.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 1.25, output = 10.0 }
-platform_llm_id = { openai = "gpt-5" }
-
-[gpt-5.gpt-5-mini.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.25, output = 2.0 }
-platform_llm_id = { azure_openai = "gpt-5-mini-2025-08-07", openai = "gpt-5-mini" }
-default_platform = "openai"
-
-[gpt-5.gpt-5-nano.latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.05, output = 0.4 }
-platform_llm_id = { azure_openai = "gpt-5-nano-2025-08-07", openai = "gpt-5-nano" }
-default_platform = "openai"
-
-[gpt-5-chat.gpt-5-chat.latest]
-is_gen_object_supported = false
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 1.25, output = 10.0 }
-platform_llm_id = { azure_openai = "gpt-5-chat-2025-08-07", openai = "gpt-5-chat-latest" }
-default_platform = "openai"
-
diff --git a/pipelex/libraries/llm_integrations/perplexity.toml b/pipelex/libraries/llm_integrations/perplexity.toml
deleted file mode 100644
index fb681f7bf..000000000
--- a/pipelex/libraries/llm_integrations/perplexity.toml
+++ /dev/null
@@ -1,26 +0,0 @@
-
-
-[perplexity-search.sonar-pro.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "sonar-pro" }
-
-[perplexity-search.sonar.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "sonar" }
-
-[perplexity-research.sonar-deep-research.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "sonar-deep-research" }
-
-[perplexity-reasoning.sonar-reasoning-pro.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "sonar-reasoning-pro" }
-
-[perplexity-reasoning.sonar-reasoning.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "sonar-reasoning" }
-
-[perplexity-deepseek.perplexity-deepseek-r1.latest]
-is_gen_object_supported = false
-platform_llm_id = { perplexity = "r1-1776" }
-
diff --git a/pipelex/libraries/llm_integrations/pipelex-inference.toml b/pipelex/libraries/llm_integrations/pipelex-inference.toml
deleted file mode 100644
index 16c910064..000000000
--- a/pipelex/libraries/llm_integrations/pipelex-inference.toml
+++ /dev/null
@@ -1,33 +0,0 @@
-
-# OpenAI Models
-[pipelex-inference."pipelex/gpt-4o-mini".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.60 }
-platform_llm_id = { custom_llm = "pipelex/gpt-4o-mini" }
-preferred_platform = "custom_llm"
-
-# Anthropic Models
-[pipelex-inference."pipelex/claude-3.7-sonnet".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.60 }
-platform_llm_id = { custom_llm = "pipelex/claude-3.7-sonnet" }
-preferred_platform = "custom_llm"
-
-# Google Models using VertexAI
-[pipelex-inference."pipelex/gemini-2.0-flash-vertex".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.60 }
-platform_llm_id = { custom_llm = "pipelex/gemini-2.0-flash-vertex" }
-preferred_platform = "custom_llm"
-
-# Google Models using Google API
-[pipelex-inference."pipelex/gemini-2.0-flash".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-cost_per_million_tokens_usd = { input = 0.15, output = 0.60 }
-platform_llm_id = { custom_llm = "gemini/gemini-2.0-flash" }
-preferred_platform = "custom_llm"
-
diff --git a/pipelex/libraries/llm_integrations/vertexai.toml b/pipelex/libraries/llm_integrations/vertexai.toml
deleted file mode 100644
index 8b6d3a7f5..000000000
--- a/pipelex/libraries/llm_integrations/vertexai.toml
+++ /dev/null
@@ -1,21 +0,0 @@
-[gemini."gemini-2.0-flash".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0.1, output = 0.4 }
-platform_llm_id = { vertexai = "google/gemini-2.0-flash" }
-
-[gemini."gemini-2.5-pro"."latest"]
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0.0, output = 0.0 }
-platform_llm_id = { vertexai = "google/gemini-2.5-pro" }
-
-[gemini."gemini-2.5-flash".latest]
-is_gen_object_supported = true
-is_vision_supported = true
-max_prompt_images = 3000
-cost_per_million_tokens_usd = { input = 0.15, output = 0.6 }
-platform_llm_id = { vertexai = "google/gemini-2.5-flash" }
-
diff --git a/pipelex/libraries/llm_integrations/xai.toml b/pipelex/libraries/llm_integrations/xai.toml
deleted file mode 100644
index 42c110edf..000000000
--- a/pipelex/libraries/llm_integrations/xai.toml
+++ /dev/null
@@ -1,27 +0,0 @@
-
-[grok-3.grok-3.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 3, output = 15 }
-platform_llm_id = { xai = "grok-3-latest", azure_openai = "grok-3" }
-preferred_platform = "azure_openai"
-
-[grok-3.grok-3-mini.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.3, output = 0.5 }
-platform_llm_id = { xai = "grok-3-mini-latest", azure_openai = "grok-3-mini" }
-preferred_platform = "azure_openai"
-
-[grok-3.grok-3-fast.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 5, output = 25 }
-platform_llm_id = { xai = "grok-3-fast-latest" }
-
-[grok-3.grok-3-mini-fast.latest]
-is_gen_object_supported = false
-is_vision_supported = false
-cost_per_million_tokens_usd = { input = 0.15, output = 4 }
-platform_llm_id = { xai = "grok-3-mini-fast-latest" }
-
diff --git a/pipelex/libraries/plugins/plugin_config.toml b/pipelex/libraries/plugins/plugin_config.toml
deleted file mode 100644
index 30cde64c3..000000000
--- a/pipelex/libraries/plugins/plugin_config.toml
+++ /dev/null
@@ -1,40 +0,0 @@
-
-####################################################################################################
-# Plugins config
-####################################################################################################
-
-[openai_config]
-image_output_compression = 100
-api_key_method = "env"
-
-[azure_openai_config]
-api_key_method = "env"
-
-# TODO: handle multiple azure openai accounts with different resource groups and account names for various llm model
-
-[perplexity_config]
-api_key_method = "env"
-
-[xai_config]
-api_key_method = "env"
-
-[vertexai_config]
-api_key_method = "env"
-
-[mistral_config]
-api_key_method = "env"
-
-[bedrock_config]
-client_method = "aioboto3"
-
-[anthropic_config]
-claude_4_reduced_tokens_limit = 8192 # use "unlimited" to enable the full 32/64K tokens Opus/Sonet but it raises streaming/timeout issues
-api_key_method = "env"
-
-[custom_endpoint_config]
-api_key_method = "env"
-
-[fal_config]
-flux_map_quality_to_steps = { "low" = 14, "medium" = 28, "high" = 56 }
-sdxl_lightning_map_quality_to_steps = { "low" = 2, "medium" = 4, "high" = 8 }
-
diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.py
index dadbafd8a..1432ab151 100644
--- a/pipelex/pipe_operators/llm/pipe_llm.py
+++ b/pipelex/pipe_operators/llm/pipe_llm.py
@@ -6,12 +6,12 @@ from typing_extensions import Self, override
from pipelex import log
from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
-from pipelex.cogt.llm.llm_models.llm_deck_check import check_llm_setting_with_deck
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingOrPresetId
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_prompt_spec import LLMPromptSpec
from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingOrPresetId
+from pipelex.cogt.models.model_deck_check import check_llm_setting_with_deck
from pipelex.config import StaticValidationReaction, get_config
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.concept_native import NativeConceptEnum
@@ -39,7 +39,7 @@ from pipelex.hub import (
get_class_registry,
get_concept_provider,
get_content_generator,
- get_llm_deck,
+ get_models_manager,
get_optional_pipe,
get_required_concept,
get_required_domain,
@@ -265,27 +265,29 @@ class PipeLLM(PipeOperator):
llm_for_text_choice = self.llm_choices.for_text
llm_for_object_choice = self.llm_choices.for_object
+ llm_deck = get_models_manager().get_llm_deck()
+
# Choice of main LLM for text first from this PipeLLM setting (self.llm_choices)
# or from the llm_choice_overrides or fallback on the llm_choice_defaults
llm_setting_or_preset_id_for_text: LLMSettingOrPresetId = (
- llm_for_text_choice or get_llm_deck().llm_choice_overrides.for_text or get_llm_deck().llm_choice_defaults.for_text
+ llm_for_text_choice or llm_deck.llm_choice_overrides.for_text or llm_deck.llm_choice_defaults.for_text
)
- llm_setting_main: LLMSetting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_setting_or_preset_id_for_text)
+ llm_setting_main: LLMSetting = llm_deck.get_llm_setting(llm_setting_or_preset_id=llm_setting_or_preset_id_for_text)
# Choice of main LLM for object from this PipeLLM setting (self.llm_choices)
# OR FROM THE llm_for_text_choice (if any)
# then fallback on the llm_choice_overrides or llm_choice_defaults
llm_setting_or_preset_id_for_object: LLMSettingOrPresetId = (
- llm_for_object_choice
- or llm_for_text_choice
- or get_llm_deck().llm_choice_overrides.for_object
- or get_llm_deck().llm_choice_defaults.for_object
+ llm_for_object_choice or llm_for_text_choice or llm_deck.llm_choice_overrides.for_object or llm_deck.llm_choice_defaults.for_object
)
- llm_setting_for_object: LLMSetting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_setting_or_preset_id_for_object)
+ llm_setting_for_object: LLMSetting = llm_deck.get_llm_setting(llm_setting_or_preset_id=llm_setting_or_preset_id_for_object)
- if not self.llm_prompt_spec.prompting_style and (llm_model := get_llm_deck().find_optional_llm_model(llm_handle=llm_setting_main.llm_handle)):
- llm_family = llm_model.llm_family
- prompting_target = llm_setting_main.prompting_target or llm_family.prompting_target
+ if (not self.llm_prompt_spec.prompting_style) and (
+ inference_model := llm_deck.get_optional_inference_model(llm_handle=llm_setting_main.llm_handle)
+ ):
+ # Note: the case where we don't get an inference model corresponds to the use of an external LLM Plugin
+ # TODO: improve this by making it possible to get the inference model for external LLM Plugins
+ prompting_target = llm_setting_main.prompting_target or inference_model.prompting_target
self.llm_prompt_spec.prompting_style = get_config().pipelex.prompting_config.get_prompting_style(
prompting_target=prompting_target,
)
diff --git a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
index 5ef3be93f..497adaa6b 100644
--- a/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
+++ b/pipelex/pipe_operators/llm/pipe_llm_blueprint.py
@@ -3,7 +3,7 @@ from typing import Literal, Optional
from pydantic import field_validator, model_validator
from typing_extensions import Self
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSettingOrPresetId
+from pipelex.cogt.llm.llm_setting import LLMSettingOrPresetId
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
from pipelex.exceptions import PipeDefinitionError
from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_lists
diff --git a/pipelex/pipe_operators/llm/pipe_llm_factory.py b/pipelex/pipe_operators/llm/pipe_llm_factory.py
index 456d11f57..c3658de28 100644
--- a/pipelex/pipe_operators/llm/pipe_llm_factory.py
+++ b/pipelex/pipe_operators/llm/pipe_llm_factory.py
@@ -2,8 +2,8 @@ from typing import List, Optional
from typing_extensions import override
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSettingChoices
from pipelex.cogt.llm.llm_prompt_spec import LLMPromptSpec
+from pipelex.cogt.llm.llm_setting import LLMSettingChoices
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.pipes.pipe_factory import PipeFactoryProtocol
from pipelex.core.pipes.pipe_input_spec_blueprint import InputRequirementBlueprint
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index ee74a1c9d..f17bc55e4 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -17,9 +17,10 @@ from pipelex.cogt.content_generation.content_generator import ContentGenerator
from pipelex.cogt.content_generation.content_generator_protocol import (
ContentGeneratorProtocol,
)
+from pipelex.cogt.exceptions import InferenceBackendCredentialsError, RoutingProfileLibraryNotFoundError
from pipelex.cogt.inference.inference_manager import InferenceManager
-from pipelex.cogt.llm.llm_models.llm_model import LATEST_VERSION_NAME
-from pipelex.cogt.llm.llm_models.llm_model_library import LLMModelLibrary
+from pipelex.cogt.models.model_manager import ModelManager
+from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
from pipelex.config import PipelexConfig, get_config
from pipelex.core.concepts.concept_library import ConceptLibrary
from pipelex.core.domains.domain_library import DomainLibrary
@@ -45,7 +46,7 @@ from pipelex.plugins.plugin_manager import PluginManager
from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocol
from pipelex.test_extras.registry_test_models import PipelexTestModels
-from pipelex.tools.config.models import ConfigRoot
+from pipelex.tools.config.config_root import ConfigRoot
from pipelex.tools.func_registry import func_registry
from pipelex.tools.runtime_manager import runtime_manager
from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
@@ -67,7 +68,7 @@ class Pipelex(metaclass=MetaSingleton):
config_cls: Optional[Type[ConfigRoot]] = None,
class_registry: Optional[ClassRegistryAbstract] = None,
template_provider: Optional[TemplateLibrary] = None,
- llm_model_provider: Optional[LLMModelLibrary] = None,
+ models_manager: Optional[ModelManagerAbstract] = None,
inference_manager: Optional[InferenceManager] = None,
pipeline_manager: Optional[PipelineManager] = None,
pipeline_tracker: Optional[PipelineTracker] = None,
@@ -105,8 +106,10 @@ class Pipelex(metaclass=MetaSingleton):
# cogt
self.plugin_manager = PluginManager()
self.pipelex_hub.set_plugin_manager(self.plugin_manager)
- self.llm_model_provider = llm_model_provider or LLMModelLibrary.make_empty(config_dir_path=config_dir_path)
- self.pipelex_hub.set_llm_models_provider(self.llm_model_provider)
+
+ self.models_manager: ModelManagerAbstract = models_manager or ModelManager()
+ self.pipelex_hub.set_models_manager(models_manager=self.models_manager)
+
self.inference_manager = inference_manager or InferenceManager()
self.pipelex_hub.set_inference_manager(self.inference_manager)
@@ -167,7 +170,22 @@ class Pipelex(metaclass=MetaSingleton):
self.pipelex_hub.set_secrets_provider(secrets_provider or EnvSecretsProvider())
self.pipelex_hub.set_storage_provider(storage_provider)
# cogt
- self.plugin_manager.setup(library_config=self.library_manager.library_config)
+ self.plugin_manager.setup()
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as routing_profile_library_exc:
+ raise PipelexSetupError(
+ "The routing library could not be found, please call `pipelex init config` to create it"
+ ) from routing_profile_library_exc
+ except InferenceBackendCredentialsError as credentials_exc:
+ if secrets_provider:
+ raise PipelexSetupError(
+ f"Missing credentials for inference backend: {credentials_exc}, check that it's available from your secrets provider."
+ ) from credentials_exc
+ else:
+ raise PipelexSetupError(
+ f"Missing credentials for inference backend: {credentials_exc}, add it to your environment variables or to your .env file."
+ ) from credentials_exc
self.pipelex_hub.set_content_generator(content_generator or ContentGenerator())
self.reporting_delegate.setup()
self.class_registry.register_classes(PipelexRegistryModels.get_all_models())
@@ -187,16 +205,8 @@ class Pipelex(metaclass=MetaSingleton):
def setup_libraries(self):
try:
self.template_provider.setup()
- self.llm_model_provider.setup()
self.library_manager.setup()
- llm_deck = self.library_manager.load_deck()
- for llm_model in self.llm_model_provider.get_all_llm_models():
- if llm_model.version == LATEST_VERSION_NAME:
- llm_deck.add_llm_name_as_handle_with_defaults(
- llm_name=llm_model.llm_name,
- )
self.library_manager.load_libraries()
- self.pipelex_hub.set_llm_deck_provider(llm_deck_provider=llm_deck)
except ValidationError as exc:
error_msg = format_pydantic_validation_error(exc)
raise PipelexSetupError(f"Could not setup libraries because of: {error_msg}") from exc
@@ -221,7 +231,6 @@ class Pipelex(metaclass=MetaSingleton):
# cogt
self.inference_manager.teardown()
self.reporting_delegate.teardown()
- self.llm_model_provider.teardown()
self.plugin_manager.teardown()
# tools
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index 5d686394c..0256c600a 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -76,6 +76,9 @@ api_key_method = "env"
[cogt]
+[cogt.inference_config]
+inference_config_path = ".pipelex/inference"
+
[cogt.inference_manager_config]
is_auto_setup_preset_llm = true
is_auto_setup_preset_imgg = true
@@ -88,20 +91,16 @@ default_max_images = 100
is_openai_structured_output_enabled = false
[cogt.llm_config.llm_job_config]
-max_retries = 3
+max_retries = 1
is_streaming_enabled = false
-[cogt.llm_config.preferred_platforms]
-# These overrride the defaults set for any llm handle
-# "gpt-4o-mini" = "openai"
-
####################################################################################################
# Image generation config
####################################################################################################
[cogt.imgg_config]
imgg_handles = ["fal-ai/flux-pro", "fal-ai/fast-lightning-sdxl"]
-default_imgg_handle = "fal-ai/flux-pro/v1.1-ultra" # use "fal-ai/fast-lightning-sdxl" for SDXL Lightning cheap and fast
+default_imgg_handle = "fal-ai/flux-pro/v1.1-ultra" # use "fal-ai/fast-lightning-sdxl" for SDXL Lightning cheap and fast
[cogt.imgg_config.imgg_job_config]
is_sync_mode = false
@@ -110,7 +109,7 @@ is_sync_mode = false
# quality and nb_steps cannot be set at the same time
# nb_steps = 1 # 28 is a good default for Flux, possible values are [1,2,4,8] for SDXL Lightning
quality = "low"
-aspect_ratio = "square" # "square", "landscape_4_3", "landscape_16_9", "landscape_21_9", "portrait_3_4", "portrait_9_16", "portrait_9_21"
+aspect_ratio = "square" # "square", "landscape_4_3", "landscape_16_9", "landscape_21_9", "portrait_3_4", "portrait_9_16", "portrait_9_21"
guidance_scale = 3.5
background = "auto"
is_moderated = true
@@ -119,6 +118,10 @@ is_raw = false
output_format = "jpg"
seed = "auto"
+[cogt.imgg_config.fal_config]
+flux_map_quality_to_steps = { "low" = 14, "medium" = 28, "high" = 56 }
+sdxl_lightning_map_quality_to_steps = { "low" = 2, "medium" = 4, "high" = 8 }
+
####################################################################################################
# OCR config
####################################################################################################
@@ -142,7 +145,7 @@ mistral = { tag_style = "square_brackets" }
gemini = { tag_style = "xml" }
[pipelex.structure_config]
-is_default_text_then_structure = false # turn this to true to get better results: generates text before structuring
+is_default_text_then_structure = false # turn this to true to get better results: generates text before structuring
####################################################################################################
@@ -178,7 +181,7 @@ is_include_text_preview = false
is_include_interactivity = false
nb_items_limit = "unlimited"
theme = "base"
-layout = "dagre" # "elk", "dagre", "fixed"
+layout = "dagre" # "elk", "dagre", "fixed"
sub_graph_colors = ["#e6f5ff", "#fff5f7", "#f0fff0"]
wrapping_width = "auto"
@@ -205,9 +208,13 @@ apply_to_jinja2_rendering = false
text_gen_truncate_length = 256
nb_list_items = 3
nb_ocr_pages = 4
-allowed_to_fail_pipes = ["infinite_loop_1", "build_blueprint", "structure_blueprint"]
+allowed_to_fail_pipes = [
+ "infinite_loop_1",
+ "build_blueprint",
+ "structure_blueprint",
+]
image_urls = [
"https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg",
"https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_2.png",
- ]
+]
diff --git a/pipelex/pipeline/track/tracker_config.py b/pipelex/pipeline/track/tracker_config.py
index 99d8d2160..e74a14129 100644
--- a/pipelex/pipeline/track/tracker_config.py
+++ b/pipelex/pipeline/track/tracker_config.py
@@ -1,6 +1,6 @@
from typing import List, Literal, Optional, Union
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
class TrackerConfig(ConfigModel):
diff --git a/pipelex/plugins/anthropic/anthropic_config.py b/pipelex/plugins/anthropic/anthropic_config.py
deleted file mode 100644
index 6c45fa1f7..000000000
--- a/pipelex/plugins/anthropic/anthropic_config.py
+++ /dev/null
@@ -1,54 +0,0 @@
-from typing import Literal, Optional, Union
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.cogt.exceptions import CogtError
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class AnthropicCredentialsError(CogtError):
- pass
-
-
-class AnthropicKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-ANTHROPIC_API_KEY_VAR_NAME = "ANTHROPIC_API_KEY"
-
-
-class AnthropicConfig(ConfigModel):
- claude_4_reduced_tokens_limit: Union[int, Literal["unlimited"]] = Field(default="unlimited")
- api_key_method: AnthropicKeyMethod = Field(strict=False)
-
- @property
- def claude_4_tokens_limit(self) -> Optional[int]:
- if self.claude_4_reduced_tokens_limit == "unlimited":
- return None
- return self.claude_4_reduced_tokens_limit
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> str:
- return self.get_api_key(secrets_provider=secrets_provider)
-
- def get_api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- match self.api_key_method:
- case AnthropicKeyMethod.ENV:
- log.verbose("Getting Anthropic API key from environment.")
- try:
- key_from_env = get_required_env(ANTHROPIC_API_KEY_VAR_NAME)
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise AnthropicCredentialsError(f"Error getting Anthropic API key from environment: {exc}") from exc
- case AnthropicKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting Anthropic API key from secrets provider.")
- try:
- key_from_service_provider = secrets_provider.get_secret(secret_id=ANTHROPIC_API_KEY_VAR_NAME)
- except SecretNotFoundError as exc:
- raise AnthropicCredentialsError("Error getting Anthropic API key from secrets provider.") from exc
- return key_from_service_provider
diff --git a/pipelex/plugins/anthropic/anthropic_exceptions.py b/pipelex/plugins/anthropic/anthropic_exceptions.py
new file mode 100644
index 000000000..4f1622e95
--- /dev/null
+++ b/pipelex/plugins/anthropic/anthropic_exceptions.py
@@ -0,0 +1,13 @@
+from pipelex.cogt.exceptions import CogtError
+
+
+class AnthropicModelListingError(CogtError):
+ pass
+
+
+class AnthropicSDKUnsupportedError(CogtError):
+ pass
+
+
+class AnthropicWorkerConfigurationError(CogtError):
+ pass
diff --git a/pipelex/plugins/anthropic/anthropic_factory.py b/pipelex/plugins/anthropic/anthropic_factory.py
index bcabb7c7b..bad120729 100644
--- a/pipelex/plugins/anthropic/anthropic_factory.py
+++ b/pipelex/plugins/anthropic/anthropic_factory.py
@@ -23,30 +23,42 @@ from pipelex.cogt.image.prompt_image import (
)
from pipelex.cogt.image.prompt_image_factory import PromptImageFactory
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.config import get_config
-from pipelex.hub import get_plugin_manager, get_secrets_provider
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.tools.misc.base_64_utils import load_binary_as_base64_async
from pipelex.tools.misc.filetype_utils import detect_file_type_from_base64
+from pipelex.types import StrEnum
class AnthropicFactoryError(CogtError):
pass
+class AnthropicSdkVariant(StrEnum):
+ ANTHROPIC = "anthropic"
+ BEDROCK_ANTHROPIC = "bedrock_anthropic"
+
+
class AnthropicFactory:
@staticmethod
def make_anthropic_client(
- llm_platform: LLMPlatform,
+ plugin: Plugin,
+ backend: InferenceBackend,
) -> Union[AsyncAnthropic, AsyncAnthropicBedrock]:
- # TODO: also support Anthropic with VertexAI
- match llm_platform:
- case LLMPlatform.ANTHROPIC:
- anthropic_config = get_plugin_manager().plugin_configs.anthropic_config
- api_key = anthropic_config.get_api_key(secrets_provider=get_secrets_provider())
- return AsyncAnthropic(api_key=api_key)
- case LLMPlatform.BEDROCK_ANTHROPIC:
+ try:
+ sdk_variant = AnthropicSdkVariant(plugin.sdk)
+ except ValueError:
+ raise AnthropicFactoryError(f"Plugin '{plugin}' is not supported by AnthropicFactory")
+
+ match sdk_variant:
+ case AnthropicSdkVariant.ANTHROPIC:
+ return AsyncAnthropic(
+ api_key=backend.api_key,
+ base_url=backend.endpoint,
+ )
+ case AnthropicSdkVariant.BEDROCK_ANTHROPIC:
aws_config = get_config().pipelex.aws_config
aws_access_key_id, aws_secret_access_key, aws_region = aws_config.get_aws_access_keys()
return AsyncAnthropicBedrock(
@@ -54,8 +66,6 @@ class AnthropicFactory:
aws_access_key=aws_access_key_id,
aws_region=aws_region,
)
- case _:
- raise AnthropicFactoryError(f"Unsupported LLM platform for Anthropic sdk: '{llm_platform}'")
@classmethod
async def make_user_message(
diff --git a/pipelex/plugins/anthropic/anthropic_llm_worker.py b/pipelex/plugins/anthropic/anthropic_llm_worker.py
index 97aa88dd1..5db6beccf 100644
--- a/pipelex/plugins/anthropic/anthropic_llm_worker.py
+++ b/pipelex/plugins/anthropic/anthropic_llm_worker.py
@@ -1,49 +1,55 @@
-from typing import Any, Optional, Type
+from typing import Any, Dict, Optional, Type
import instructor
from anthropic import NOT_GIVEN, AsyncAnthropic, AsyncAnthropicBedrock
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCompletionError, LLMEngineParameterError, SdkTypeError
+from pipelex.cogt.exceptions import LLMCompletionError, SdkTypeError
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
-from pipelex.hub import get_plugin_manager
-from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicWorkerConfigurationError
+from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory, AnthropicSdkVariant
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.types import StrEnum
+
+
+class AnthropicExtraField(StrEnum):
+ CLAUDE_4_TOKENS_LIMIT = "claude_4_tokens_limit"
class AnthropicLLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
sdk_instance: Any,
- llm_engine: LLMEngine,
+ extra_config: Dict[str, Any],
+ inference_model: InferenceModelSpec,
structure_method: Optional[StructureMethod] = None,
reporting_delegate: Optional[ReportingProtocol] = None,
):
LLMWorkerInternalAbstract.__init__(
self,
- llm_engine=llm_engine,
+ inference_model=inference_model,
structure_method=structure_method,
reporting_delegate=reporting_delegate,
)
- self.default_max_tokens: int
- if default_max_tokens := llm_engine.llm_model.max_tokens:
- self.default_max_tokens = default_max_tokens
+ self.extra_config: Dict[str, Any] = extra_config
+ self.default_max_tokens: int = 0
+ if inference_model.max_tokens:
+ self.default_max_tokens = inference_model.max_tokens
else:
- raise LLMEngineParameterError(
- f"No max_tokens provided for llm model '{self.llm_engine.llm_model.desc}', but it is required for Anthropic"
+ raise AnthropicWorkerConfigurationError(
+ f"No max_tokens provided for llm model '{self.inference_model.desc}', but it is required for Anthropic"
)
# Verify if the sdk_instance is compatible with the current LLM platform
if isinstance(sdk_instance, (AsyncAnthropic, AsyncAnthropicBedrock)):
- if llm_engine.llm_platform == LLMPlatform.ANTHROPIC and not (isinstance(sdk_instance, AsyncAnthropic)):
+ if inference_model.sdk == AnthropicSdkVariant.ANTHROPIC and not (isinstance(sdk_instance, AsyncAnthropic)):
raise SdkTypeError(f"Provided sdk_instance does not match LLMEngine platform:{sdk_instance}")
- elif llm_engine.llm_platform == LLMPlatform.BEDROCK_ANTHROPIC and not (isinstance(sdk_instance, AsyncAnthropicBedrock)):
+ elif inference_model.sdk == AnthropicSdkVariant.BEDROCK_ANTHROPIC and not (isinstance(sdk_instance, AsyncAnthropicBedrock)):
raise SdkTypeError(f"Provided sdk_instance does not match LLMEngine platform:{sdk_instance}")
else:
raise SdkTypeError(f"Provided sdk_instance does not match LLMEngine platform:{sdk_instance}")
@@ -63,10 +69,12 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
# TODO: implement streaming behind the scenes to avoid timeout/streaming errors with Claude 4 and high tokens
def _adapt_max_tokens(self, max_tokens: Optional[int]) -> int:
max_tokens = max_tokens or self.default_max_tokens
- if claude_4_tokens_limit := get_plugin_manager().plugin_configs.anthropic_config.claude_4_tokens_limit:
- if max_tokens > claude_4_tokens_limit:
- max_tokens = claude_4_tokens_limit
- log.warning(f"Max tokens is greater than the claude 4 reduced tokens limit, reducing to {max_tokens}")
+
+ if (claude_4_tokens_limit := self.extra_config.get(AnthropicExtraField.CLAUDE_4_TOKENS_LIMIT)) and max_tokens > claude_4_tokens_limit:
+ max_tokens = claude_4_tokens_limit
+ log.warning(f"Max tokens is greater than the claude 4 reduced tokens limit, reducing to {max_tokens}")
+ if not max_tokens:
+ raise AnthropicWorkerConfigurationError(f"Max tokens is None for model {self.inference_model.desc}")
return max_tokens
@override
@@ -79,19 +87,19 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
response = await self.anthropic_async_client.messages.create(
messages=[message],
system=llm_job.llm_prompt.system_text or NOT_GIVEN,
- model=self.llm_engine.llm_id,
+ model=self.inference_model.model_id,
temperature=llm_job.job_params.temperature,
max_tokens=max_tokens,
)
single_content_block = response.content[0]
if single_content_block.type != "text":
- raise LLMCompletionError(f"Unexpected content block type: {single_content_block.type}\nmodel: {self.llm_engine.llm_model.desc}")
+ raise LLMCompletionError(f"Unexpected content block type: {single_content_block.type}\nmodel: {self.inference_model.desc}")
full_reply_content = single_content_block.text
single_content_block = response.content[0]
if single_content_block.type != "text":
- raise LLMCompletionError(f"Unexpected content block type: {single_content_block.type}\nmodel: {self.llm_engine.llm_model.desc}")
+ raise LLMCompletionError(f"Unexpected content block type: {single_content_block.type}\nmodel: {self.inference_model.desc}")
full_reply_content = single_content_block.text
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
@@ -111,7 +119,7 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
messages=messages,
response_model=schema,
max_retries=llm_job.job_config.max_retries,
- model=self.llm_engine.llm_id,
+ model=self.inference_model.model_id,
temperature=llm_job.job_params.temperature,
max_tokens=max_tokens,
)
diff --git a/pipelex/plugins/anthropic/anthropic_llms.py b/pipelex/plugins/anthropic/anthropic_llms.py
index 92b722d48..7c9436e5f 100644
--- a/pipelex/plugins/anthropic/anthropic_llms.py
+++ b/pipelex/plugins/anthropic/anthropic_llms.py
@@ -3,27 +3,28 @@ from typing import List
from anthropic import AsyncAnthropic
from anthropic.types import ModelInfo
-from pipelex.cogt.exceptions import LLMModelProviderError, LLMSDKError
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicModelListingError, AnthropicSDKUnsupportedError
from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory
+from pipelex.plugins.plugin_sdk_registry import Plugin
-async def anthropic_list_anthropic_models(llm_platform: LLMPlatform) -> List[ModelInfo]:
+async def anthropic_list_anthropic_models(plugin: Plugin, backend: InferenceBackend) -> List[ModelInfo]:
"""
List available Anthropic models.
Returns:
List[ModelInfo]: A list of Anthropic model information objects
"""
- anthropic_client = AnthropicFactory.make_anthropic_client(llm_platform=llm_platform)
+ anthropic_client = AnthropicFactory.make_anthropic_client(plugin=plugin, backend=backend)
if not hasattr(anthropic_client, "models"):
- raise LLMSDKError(f"{type(anthropic_client).__name__} does not support listing models")
+ raise AnthropicSDKUnsupportedError(f"{type(anthropic_client).__name__} does not support listing models")
if not isinstance(anthropic_client, AsyncAnthropic):
- raise LLMSDKError("We only support the standard Anthropic client for listing models")
+ raise AnthropicSDKUnsupportedError("We only support the standard Anthropic client for listing models")
models_response = await anthropic_client.models.list()
if not models_response:
- raise LLMModelProviderError("No models found")
+ raise AnthropicModelListingError("No models found")
models_list = models_response.data
if not models_list:
- raise LLMModelProviderError("No models found")
+ raise AnthropicModelListingError("No models found")
return sorted(models_list, key=lambda model: model.created_at)
diff --git a/pipelex/plugins/bedrock/bedrock_client_aioboto3.py b/pipelex/plugins/bedrock/bedrock_client_aioboto3.py
index 2c46a3173..7d0dbe647 100644
--- a/pipelex/plugins/bedrock/bedrock_client_aioboto3.py
+++ b/pipelex/plugins/bedrock/bedrock_client_aioboto3.py
@@ -5,7 +5,7 @@ from types_aiobotocore_bedrock_runtime.type_defs import ConverseResponseTypeDef
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtocol
from pipelex.plugins.bedrock.bedrock_message import BedrockMessageDictList
diff --git a/pipelex/plugins/bedrock/bedrock_client_boto3.py b/pipelex/plugins/bedrock/bedrock_client_boto3.py
index 4a45edee6..1a7183238 100644
--- a/pipelex/plugins/bedrock/bedrock_client_boto3.py
+++ b/pipelex/plugins/bedrock/bedrock_client_boto3.py
@@ -5,7 +5,7 @@ import boto3
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtocol
from pipelex.plugins.bedrock.bedrock_message import BedrockMessageDictList
diff --git a/pipelex/plugins/bedrock/bedrock_client_protocol.py b/pipelex/plugins/bedrock/bedrock_client_protocol.py
index f025a4c4d..c3d7bd447 100644
--- a/pipelex/plugins/bedrock/bedrock_client_protocol.py
+++ b/pipelex/plugins/bedrock/bedrock_client_protocol.py
@@ -1,6 +1,6 @@
from typing import Optional, Protocol, Tuple, runtime_checkable
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict
from pipelex.plugins.bedrock.bedrock_message import BedrockMessageDictList
diff --git a/pipelex/plugins/bedrock/bedrock_config.py b/pipelex/plugins/bedrock/bedrock_config.py
deleted file mode 100644
index f56377b3e..000000000
--- a/pipelex/plugins/bedrock/bedrock_config.py
+++ /dev/null
@@ -1,41 +0,0 @@
-from pydantic import Field
-
-from pipelex import log
-from pipelex.cogt.exceptions import CogtError
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class BedrockCredentialsError(CogtError):
- pass
-
-
-class BedrockClientMethod(StrEnum):
- BOTO3 = "boto3"
- AIBOTO3 = "aioboto3"
-
-
-AWS_REGION_VAR_NAME = "AWS_REGION"
-
-
-class BedrockConfig(ConfigModel):
- client_method: BedrockClientMethod = Field(strict=False)
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Configure and return AWS region."""
- return self._get_region(secrets_provider=secrets_provider)
-
- def _get_region(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get AWS region from environment or secrets provider."""
- log.debug("Getting AWS region.")
- try:
- return get_required_env(AWS_REGION_VAR_NAME)
- except EnvVarNotFoundError:
- log.verbose("AWS region not found in environment, trying secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=AWS_REGION_VAR_NAME)
- except SecretNotFoundError as exc:
- raise BedrockCredentialsError("Error getting AWS region from secrets provider.") from exc
diff --git a/pipelex/plugins/bedrock/bedrock_factory.py b/pipelex/plugins/bedrock/bedrock_factory.py
index 513a2c458..3bd56c354 100644
--- a/pipelex/plugins/bedrock/bedrock_factory.py
+++ b/pipelex/plugins/bedrock/bedrock_factory.py
@@ -1,11 +1,25 @@
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, PromptImageFormatError
+from pipelex.cogt.exceptions import CogtError, LLMCapabilityError, PromptImageFormatError
from pipelex.cogt.image.prompt_image import PromptImageBytes
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.hub import get_plugin_manager, get_secrets_provider
+from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtocol
-from pipelex.plugins.bedrock.bedrock_config import BedrockClientMethod
from pipelex.plugins.bedrock.bedrock_message import BedrockContentItem, BedrockImage, BedrockMessage, BedrockSource, ImageFormat
+from pipelex.plugins.plugin_sdk_registry import Plugin
+from pipelex.types import StrEnum
+
+
+class BedrockFactoryError(CogtError):
+ pass
+
+
+class BedrockSdkVariant(StrEnum):
+ BOTO3 = "bedrock_boto3"
+ AIBOTO3 = "bedrock_aioboto3"
+
+
+class BedrockExtraField(StrEnum):
+ AWS_REGION = "aws_region"
class BedrockFactory:
@@ -14,21 +28,31 @@ class BedrockFactory:
#########################################################
@classmethod
- def make_bedrock_client(cls) -> BedrockClientProtocol:
- bedrock_config = get_plugin_manager().plugin_configs.bedrock_config
- aws_region = bedrock_config.configure(secrets_provider=get_secrets_provider())
- client_method = bedrock_config.client_method
+ def make_bedrock_client(
+ cls,
+ plugin: Plugin,
+ backend: InferenceBackend,
+ ) -> BedrockClientProtocol:
+ try:
+ sdk_variant = BedrockSdkVariant(plugin.sdk)
+ except ValueError:
+ raise BedrockFactoryError(f"Plugin '{plugin}' is not supported by BedrockFactory")
+
bedrock_async_client: BedrockClientProtocol
- log.verbose(f"Using '{client_method}' for BedrockClient")
- match client_method:
- case BedrockClientMethod.AIBOTO3:
+ log.verbose(f"Using '{sdk_variant}' for BedrockClient")
+ match sdk_variant:
+ case BedrockSdkVariant.AIBOTO3:
from pipelex.plugins.bedrock.bedrock_client_aioboto3 import BedrockClientAioboto3
- bedrock_async_client = BedrockClientAioboto3(aws_region=aws_region)
- case BedrockClientMethod.BOTO3:
+ bedrock_async_client = BedrockClientAioboto3(
+ aws_region=backend.extra_config[BedrockExtraField.AWS_REGION],
+ )
+ case BedrockSdkVariant.BOTO3:
from pipelex.plugins.bedrock.bedrock_client_boto3 import BedrockClientBoto3
- bedrock_async_client = BedrockClientBoto3(aws_region=aws_region)
+ bedrock_async_client = BedrockClientBoto3(
+ aws_region=backend.extra_config[BedrockExtraField.AWS_REGION],
+ )
return bedrock_async_client
diff --git a/pipelex/plugins/bedrock/bedrock_llm_worker.py b/pipelex/plugins/bedrock/bedrock_llm_worker.py
index 87e479573..da02b300b 100644
--- a/pipelex/plugins/bedrock/bedrock_llm_worker.py
+++ b/pipelex/plugins/bedrock/bedrock_llm_worker.py
@@ -3,28 +3,32 @@ from typing import Any, Optional, Type
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, LLMEngineParameterError, SdkTypeError
+from pipelex.cogt.exceptions import CogtError, LLMCapabilityError, SdkTypeError
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtocol
from pipelex.plugins.bedrock.bedrock_factory import BedrockFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+class BedrockWorkerConfigurationError(CogtError):
+ pass
+
+
class BedrockLLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
sdk_instance: Any,
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
structure_method: Optional[StructureMethod] = None,
reporting_delegate: Optional[ReportingProtocol] = None,
):
LLMWorkerInternalAbstract.__init__(
self,
- llm_engine=llm_engine,
+ inference_model=inference_model,
structure_method=structure_method,
reporting_delegate=reporting_delegate,
)
@@ -34,10 +38,12 @@ class BedrockLLMWorker(LLMWorkerInternalAbstract):
f"Provided sdk_instance for {self.__class__.__name__} is not of type BedrockClientProtocol: it's a '{type(sdk_instance)}'"
)
- if default_max_tokens := llm_engine.llm_model.max_tokens:
+ if default_max_tokens := inference_model.max_tokens:
self.default_max_tokens = default_max_tokens
else:
- raise LLMEngineParameterError(f"No max_tokens provided for llm model '{self.llm_engine.llm_model.desc}', but it is required for Bedrock")
+ raise BedrockWorkerConfigurationError(
+ f"No max_tokens provided for llm model '{self.inference_model.desc}', but it is required for Bedrock"
+ )
self.bedrock_client_for_text = sdk_instance
@override
@@ -47,12 +53,12 @@ class BedrockLLMWorker(LLMWorkerInternalAbstract):
) -> str:
message = BedrockFactory.make_simple_message(llm_job=llm_job)
- log.debug(self.llm_engine.llm_id)
+ log.debug(self.inference_model.model_id)
bedrock_response_text, nb_tokens_by_category = await self.bedrock_client_for_text.chat(
messages=message.to_dict_list(),
system_text=llm_job.llm_prompt.system_text,
- model=self.llm_engine.llm_id,
+ model=self.inference_model.model_id,
temperature=llm_job.job_params.temperature,
max_tokens=llm_job.job_params.max_tokens or self.default_max_tokens,
)
diff --git a/pipelex/plugins/fal/fal_config.py b/pipelex/plugins/fal/fal_config.py
index 6a5a1d01a..93e4cb9e9 100644
--- a/pipelex/plugins/fal/fal_config.py
+++ b/pipelex/plugins/fal/fal_config.py
@@ -3,7 +3,7 @@ from typing import Dict
from pydantic import field_validator
from pipelex.cogt.imgg.imgg_job_components import Quality
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.tools.exceptions import ConfigValidationError
diff --git a/pipelex/plugins/fal/fal_factory.py b/pipelex/plugins/fal/fal_factory.py
index 88d47dc11..4d4780466 100644
--- a/pipelex/plugins/fal/fal_factory.py
+++ b/pipelex/plugins/fal/fal_factory.py
@@ -7,7 +7,7 @@ from pipelex.cogt.imgg.imgg_engine import ImggEngine
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_job import ImggJob
from pipelex.cogt.imgg.imgg_job_components import AspectRatio, OutputFormat, Quality
-from pipelex.hub import get_plugin_manager
+from pipelex.config import get_config
class FalFactory:
@@ -17,12 +17,12 @@ class FalFactory:
@classmethod
def make_nb_steps_from_quality_for_flux_pro(cls, quality: Quality) -> int:
- flux_map_quality_to_steps = get_plugin_manager().plugin_configs.fal_config.flux_map_quality_to_steps
+ flux_map_quality_to_steps = get_config().cogt.imgg_config.fal_config.flux_map_quality_to_steps
return flux_map_quality_to_steps[quality.value]
@classmethod
def make_nb_steps_from_quality_for_sdxl_lightning(cls, quality: Quality) -> int:
- sdxl_lightning_map_quality_to_steps = get_plugin_manager().plugin_configs.fal_config.sdxl_lightning_map_quality_to_steps
+ sdxl_lightning_map_quality_to_steps = get_config().cogt.imgg_config.fal_config.sdxl_lightning_map_quality_to_steps
return sdxl_lightning_map_quality_to_steps[quality.value]
@classmethod
diff --git a/pipelex/plugins/mistral/mistral_config.py b/pipelex/plugins/mistral/mistral_config.py
deleted file mode 100644
index dff917702..000000000
--- a/pipelex/plugins/mistral/mistral_config.py
+++ /dev/null
@@ -1,39 +0,0 @@
-from pydantic import Field
-
-from pipelex import log
-from pipelex.cogt.exceptions import CogtError
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class MistralCredentialsError(CogtError):
- pass
-
-
-class MistralKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-class MistralConfig(ConfigModel):
- api_key_method: MistralKeyMethod = Field(strict=False)
-
- def api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- match self.api_key_method:
- case MistralKeyMethod.ENV:
- log.debug("Using Mistral API key from environment.")
- try:
- key_from_env = get_required_env("MISTRAL_API_KEY")
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise MistralCredentialsError(f"Error getting Mistral API key from environment: {exc}") from exc
- case MistralKeyMethod.SECRET_PROVIDER:
- log.verbose("Using Mistral API key from secrets provider.")
- try:
- key_from_secrets_provider = secrets_provider.get_secret(secret_id="MISTRAL_API_KEY")
- return key_from_secrets_provider
- except SecretNotFoundError as exc:
- raise MistralCredentialsError("Error getting Mistral API key from secrets provider.") from exc
diff --git a/pipelex/plugins/mistral/mistral_exceptions.py b/pipelex/plugins/mistral/mistral_exceptions.py
new file mode 100644
index 000000000..1318bb466
--- /dev/null
+++ b/pipelex/plugins/mistral/mistral_exceptions.py
@@ -0,0 +1,9 @@
+from pipelex.cogt.exceptions import CogtError
+
+
+class MistralModelListingError(CogtError):
+ pass
+
+
+class MistralWorkerConfigurationError(CogtError):
+ pass
diff --git a/pipelex/plugins/mistral/mistral_factory.py b/pipelex/plugins/mistral/mistral_factory.py
index 2d00aa6b7..bed02e2b4 100644
--- a/pipelex/plugins/mistral/mistral_factory.py
+++ b/pipelex/plugins/mistral/mistral_factory.py
@@ -22,9 +22,9 @@ from openai.types.chat import (
from pipelex.cogt.exceptions import PromptImageFormatError
from pipelex.cogt.image.prompt_image import PromptImage, PromptImageBytes, PromptImagePath, PromptImageUrl
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.ocr.ocr_output import ExtractedImageFromPage, OcrOutput, Page
-from pipelex.hub import get_plugin_manager, get_secrets_provider
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.plugins.openai.openai_factory import OpenAIFactory
from pipelex.tools.misc.base_64_utils import encode_to_base64, load_binary_as_base64
@@ -35,10 +35,11 @@ class MistralFactory:
#########################################################
@classmethod
- def make_mistral_client(cls) -> Mistral:
- return Mistral(
- api_key=get_plugin_manager().plugin_configs.mistral_config.api_key(secrets_provider=get_secrets_provider()),
- )
+ def make_mistral_client(
+ cls,
+ backend: InferenceBackend,
+ ) -> Mistral:
+ return Mistral(api_key=backend.api_key)
#########################################################
# Message
diff --git a/pipelex/plugins/mistral/mistral_llm_worker.py b/pipelex/plugins/mistral/mistral_llm_worker.py
index 5cbb1965d..f0de7ad25 100644
--- a/pipelex/plugins/mistral/mistral_llm_worker.py
+++ b/pipelex/plugins/mistral/mistral_llm_worker.py
@@ -6,11 +6,12 @@ from mistralai.models import ChatCompletionResponse
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCompletionError, LLMEngineParameterError, SdkTypeError
+from pipelex.cogt.exceptions import LLMCompletionError, SdkTypeError
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.plugins.mistral.mistral_exceptions import MistralWorkerConfigurationError
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -20,13 +21,13 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
sdk_instance: Any,
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
structure_method: Optional[StructureMethod] = None,
reporting_delegate: Optional[ReportingProtocol] = None,
):
LLMWorkerInternalAbstract.__init__(
self,
- llm_engine=llm_engine,
+ inference_model=inference_model,
structure_method=structure_method,
reporting_delegate=reporting_delegate,
)
@@ -34,10 +35,12 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
if not isinstance(sdk_instance, Mistral):
raise SdkTypeError(f"Provided LLM sdk_instance for {self.__class__.__name__} is not of type Mistral: it's a '{type(sdk_instance)}'")
- if default_max_tokens := llm_engine.llm_model.max_tokens:
+ if default_max_tokens := inference_model.max_tokens:
self.default_max_tokens = default_max_tokens
else:
- raise LLMEngineParameterError(f"No max_tokens provided for llm model '{self.llm_engine.llm_model.desc}', but it is required for Mistral")
+ raise MistralWorkerConfigurationError(
+ f"No max_tokens provided for llm model '{self.inference_model.desc}', but it is required for Mistral"
+ )
self.mistral_client_for_text: Mistral = sdk_instance
if structure_method:
@@ -55,7 +58,7 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
messages = MistralFactory.make_simple_messages(llm_job=llm_job)
response: Optional[ChatCompletionResponse] = await self.mistral_client_for_text.chat.complete_async(
messages=messages,
- model=self.llm_engine.llm_id,
+ model=self.inference_model.model_id,
temperature=llm_job.job_params.temperature,
max_tokens=llm_job.job_params.max_tokens or self.default_max_tokens,
)
@@ -81,7 +84,7 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
response_model=schema,
messages=MistralFactory.make_simple_messages_openai_typed(llm_job=llm_job),
- model=self.llm_engine.llm_id,
+ model=self.inference_model.model_id,
temperature=llm_job.job_params.temperature,
max_tokens=llm_job.job_params.max_tokens or self.default_max_tokens,
)
diff --git a/pipelex/plugins/mistral/mistral_llms.py b/pipelex/plugins/mistral/mistral_llms.py
index 3cca53791..c9a22d23f 100644
--- a/pipelex/plugins/mistral/mistral_llms.py
+++ b/pipelex/plugins/mistral/mistral_llms.py
@@ -2,16 +2,18 @@ from typing import List
from mistralai.models import Data
-from pipelex.cogt.exceptions import LLMModelProviderError
+from pipelex.hub import get_models_manager
+from pipelex.plugins.mistral.mistral_exceptions import MistralModelListingError
from pipelex.plugins.mistral.mistral_factory import MistralFactory
def list_mistral_models() -> List[Data]:
- mistral_client = MistralFactory.make_mistral_client()
+ backend = get_models_manager().get_required_inference_backend("mistral")
+ mistral_client = MistralFactory.make_mistral_client(backend=backend)
models_list_response = mistral_client.models.list()
if not models_list_response:
- raise LLMModelProviderError("No models found")
+ raise MistralModelListingError("No models found")
models_list = models_list_response.data
if not models_list:
- raise LLMModelProviderError("No models found")
+ raise MistralModelListingError("No models found")
return sorted(models_list, key=lambda model: model.id)
diff --git a/pipelex/plugins/openai/azure_openai_config.py b/pipelex/plugins/openai/azure_openai_config.py
deleted file mode 100644
index 130993e02..000000000
--- a/pipelex/plugins/openai/azure_openai_config.py
+++ /dev/null
@@ -1,87 +0,0 @@
-from typing import Tuple
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.cogt.exceptions import CogtError
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class AzureOpenAICredentialsError(CogtError):
- pass
-
-
-class AzureOpenAIKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-AZURE_OPENAI_API_KEY_VAR_NAME = "AZURE_OPENAI_API_KEY"
-AZURE_OPENAI_API_ENDPOINT_VAR_NAME = "AZURE_OPENAI_API_ENDPOINT"
-AZURE_OPENAI_API_VERSION_VAR_NAME = "AZURE_OPENAI_API_VERSION"
-
-
-class AzureOpenAIConfig(ConfigModel):
- api_key_method: AzureOpenAIKeyMethod = Field(strict=False)
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> Tuple[str, str, str]:
- """Configure and return endpoint, version, and API key."""
- api_endpoint = self._get_api_endpoint(secrets_provider=secrets_provider)
- api_version = self._get_api_version(secrets_provider=secrets_provider)
- api_key = self._get_api_key(secrets_provider=secrets_provider)
- return api_endpoint, api_version, api_key
-
- def _get_api_endpoint(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API endpoint from environment or secrets provider."""
- match self.api_key_method:
- case AzureOpenAIKeyMethod.ENV:
- log.debug("Getting Azure OpenAI API endpoint from environment.")
- try:
- return get_required_env(AZURE_OPENAI_API_ENDPOINT_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise AzureOpenAICredentialsError(f"Error getting Azure OpenAI API endpoint from environment: {exc}") from exc
- case AzureOpenAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting Azure OpenAI API endpoint from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=AZURE_OPENAI_API_ENDPOINT_VAR_NAME)
- except SecretNotFoundError as exc:
- raise AzureOpenAICredentialsError("Error getting Azure OpenAI API endpoint from secrets provider.") from exc
-
- def _get_api_version(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API version from environment or secrets provider."""
- match self.api_key_method:
- case AzureOpenAIKeyMethod.ENV:
- log.debug("Getting Azure OpenAI API version from environment.")
- try:
- return get_required_env(AZURE_OPENAI_API_VERSION_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise AzureOpenAICredentialsError(f"Error getting Azure OpenAI API version from environment: {exc}") from exc
- case AzureOpenAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting Azure OpenAI API version from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=AZURE_OPENAI_API_VERSION_VAR_NAME)
- except SecretNotFoundError as exc:
- raise AzureOpenAICredentialsError("Error getting Azure OpenAI API version from secrets provider.") from exc
-
- def _get_api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API key from environment or secrets provider."""
- match self.api_key_method:
- case AzureOpenAIKeyMethod.ENV:
- log.debug("Using Azure OpenAI API key from environment.")
- try:
- key_from_env = get_required_env(AZURE_OPENAI_API_KEY_VAR_NAME)
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise AzureOpenAICredentialsError(f"Error getting Azure OpenAI API key from environment: {exc}") from exc
- case AzureOpenAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Using Azure OpenAI API key from secrets provider.")
- # TODO: make it automatically select the right key for the right azure resource
- try:
- key_from_secrets_provider = secrets_provider.get_secret(secret_id=AZURE_OPENAI_API_KEY_VAR_NAME)
- except SecretNotFoundError as exc:
- raise AzureOpenAICredentialsError("Error getting Azure OpenAI API key from secrets provider.") from exc
- return key_from_secrets_provider
diff --git a/pipelex/plugins/openai/custom_endpoint_config.py b/pipelex/plugins/openai/custom_endpoint_config.py
deleted file mode 100644
index 2564ee6fa..000000000
--- a/pipelex/plugins/openai/custom_endpoint_config.py
+++ /dev/null
@@ -1,74 +0,0 @@
-from typing import Optional, Tuple
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.exceptions import CredentialsError
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class CustomEndpointCredentialsError(CredentialsError):
- pass
-
-
-class OpenAIKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-CUSTOM_ENDPOINT_API_KEY_VAR_NAME = "CUSTOM_ENDPOINT_API_KEY"
-CUSTOM_ENDPOINT_BASE_URL_VAR_NAME = "CUSTOM_ENDPOINT_BASE_URL"
-
-
-class CustomEndpointConfig(ConfigModel):
- """Configuration for custom OpenAI-compatible endpoints (e.g., Ollama, LM Studio, etc.)"""
-
- api_key_method: OpenAIKeyMethod = Field(strict=False)
-
- def configure(self, secrets_provider: Optional[SecretsProviderAbstract] = None) -> Tuple[str, Optional[str]]:
- """Configure and return base URL and optional API key."""
- base_url = self._get_base_url(secrets_provider=secrets_provider)
- api_key = self._get_api_key(secrets_provider=secrets_provider)
- return base_url, api_key
-
- def _get_base_url(self, secrets_provider: Optional[SecretsProviderAbstract]) -> str:
- """Get base URL from environment or secrets provider."""
- match self.api_key_method:
- case OpenAIKeyMethod.ENV:
- log.debug("Getting Custom OpenAI base URL from environment.")
- try:
- return get_required_env(CUSTOM_ENDPOINT_BASE_URL_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise CustomEndpointCredentialsError(f"Error getting Custom OpenAI base URL from environment: {exc}") from exc
- case OpenAIKeyMethod.SECRET_PROVIDER:
- if not secrets_provider:
- raise CustomEndpointCredentialsError("Secrets provider is required when using SECRET_PROVIDER method")
- log.verbose("Getting Custom OpenAI base URL from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=CUSTOM_ENDPOINT_BASE_URL_VAR_NAME)
- except SecretNotFoundError as exc:
- raise CustomEndpointCredentialsError("Error getting Custom OpenAI base URL from secrets provider.") from exc
-
- def _get_api_key(self, secrets_provider: Optional[SecretsProviderAbstract] = None) -> Optional[str]:
- """Get API key if configured, otherwise return None for services that don't require authentication"""
- match self.api_key_method:
- case OpenAIKeyMethod.ENV:
- log.debug("Using Custom OpenAI API key from environment.")
- try:
- return get_required_env(CUSTOM_ENDPOINT_API_KEY_VAR_NAME)
- except EnvVarNotFoundError:
- log.debug("No Custom OpenAI API key found in environment - using None.")
- return None
- case OpenAIKeyMethod.SECRET_PROVIDER:
- if not secrets_provider:
- raise CustomEndpointCredentialsError("Secrets provider is required when using SECRET_PROVIDER method")
- log.verbose("Using Custom OpenAI API key from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=CUSTOM_ENDPOINT_API_KEY_VAR_NAME)
- except SecretNotFoundError:
- log.debug("No Custom OpenAI API key found in secrets provider - using None.")
- return None
diff --git a/pipelex/plugins/openai/openai_config.py b/pipelex/plugins/openai/openai_config.py
deleted file mode 100644
index c79804502..000000000
--- a/pipelex/plugins/openai/openai_config.py
+++ /dev/null
@@ -1,43 +0,0 @@
-from pydantic import Field
-
-from pipelex import log
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.exceptions import CredentialsError
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class OpenAICredentialsError(CredentialsError):
- pass
-
-
-class OpenAIKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-OPENAI_API_KEY_VAR_NAME = "OPENAI_API_KEY"
-
-
-class OpenAIConfig(ConfigModel):
- image_output_compression: int = Field(ge=1, le=100)
- api_key_method: OpenAIKeyMethod = Field(strict=False)
-
- def get_api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- match self.api_key_method:
- case OpenAIKeyMethod.ENV:
- log.debug("Using OpenAI API key from environment.")
- try:
- key_from_env = get_required_env(OPENAI_API_KEY_VAR_NAME)
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise OpenAICredentialsError(f"Error getting OpenAI API key from environment: {exc}") from exc
- case OpenAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Using OpenAI API key from secrets provider.")
- try:
- key_from_service_provider = secrets_provider.get_secret(secret_id=OPENAI_API_KEY_VAR_NAME)
- except SecretNotFoundError as exc:
- raise OpenAICredentialsError("Error getting OpenAI API key from secrets provider.") from exc
- return key_from_service_provider
diff --git a/pipelex/plugins/openai/openai_factory.py b/pipelex/plugins/openai/openai_factory.py
index 844e5e0ff..e46a6b9d9 100644
--- a/pipelex/plugins/openai/openai_factory.py
+++ b/pipelex/plugins/openai/openai_factory.py
@@ -1,4 +1,4 @@
-from typing import Dict, List, Optional
+from typing import Dict, List
import openai
from openai.types.chat import (
@@ -13,74 +13,60 @@ from openai.types.chat.chat_completion_content_part_image_param import ImageURL
from openai.types.completion_usage import CompletionUsage
from pipelex import log
-from pipelex.cogt.exceptions import LLMEngineParameterError, LLMPromptParameterError
+from pipelex.cogt.exceptions import CogtError, LLMPromptParameterError
from pipelex.cogt.image.prompt_image import PromptImage, PromptImageBytes, PromptImagePath, PromptImageUrl
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.hub import get_plugin_manager, get_secrets_provider
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.tools.misc.base_64_utils import load_binary_as_base64
+from pipelex.types import StrEnum
+
+
+class OpenAIFactoryError(CogtError):
+ pass
+
+
+class OpenAISdkVariant(StrEnum):
+ AZURE_OPENAI = "azure_openai"
+ OPENAI = "openai"
+
+
+class AzureExtraField(StrEnum):
+ API_VERSION = "api_version"
class OpenAIFactory:
@classmethod
- def make_openai_client(cls, llm_platform: LLMPlatform) -> openai.AsyncClient:
- the_client: openai.AsyncOpenAI
- api_key: Optional[str] = None
- match llm_platform:
- case LLMPlatform.AZURE_OPENAI:
- azure_openai_config = get_plugin_manager().plugin_configs.azure_openai_config
- endpoint, api_version, api_key = azure_openai_config.configure(secrets_provider=get_secrets_provider())
+ def make_openai_client(
+ cls,
+ plugin: Plugin,
+ backend: InferenceBackend,
+ ) -> openai.AsyncClient:
+ try:
+ sdk_variant = OpenAISdkVariant(plugin.sdk)
+ except ValueError:
+ raise OpenAIFactoryError(f"Plugin '{plugin}' is not supported by OpenAIFactory")
- log.verbose(f"Making AsyncAzureOpenAI client with endpoint: {endpoint}, api_version: {api_version}")
+ the_client: openai.AsyncOpenAI
+ match sdk_variant:
+ case OpenAISdkVariant.AZURE_OPENAI:
+ log.debug(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")
+ if backend.endpoint is None:
+ raise OpenAIFactoryError("Azure OpenAI endpoint is not set")
the_client = openai.AsyncAzureOpenAI(
- azure_endpoint=endpoint,
- api_key=api_key,
- api_version=api_version,
- )
- case LLMPlatform.PERPLEXITY:
- perplexity_config = get_plugin_manager().plugin_configs.perplexity_config
- endpoint, api_key = perplexity_config.configure(secrets_provider=get_secrets_provider())
-
- log.verbose(f"Making perplexity AsyncOpenAI client with endpoint: {endpoint}")
- the_client = openai.AsyncOpenAI(
- api_key=api_key,
- base_url=endpoint,
- )
- case LLMPlatform.OPENAI:
- openai_config = get_plugin_manager().plugin_configs.openai_config
- api_key = openai_config.get_api_key(secrets_provider=get_secrets_provider())
- the_client = openai.AsyncOpenAI(api_key=api_key)
- case LLMPlatform.VERTEXAI:
- vertexai_config = get_plugin_manager().plugin_configs.vertexai_config
- endpoint, api_key = vertexai_config.configure(secrets_provider=get_secrets_provider())
-
- log.verbose(f"Making vertex AsyncOpenAI client with endpoint: {endpoint}")
- the_client = openai.AsyncOpenAI(
- api_key=api_key,
- base_url=endpoint,
- )
- case LLMPlatform.XAI:
- xai_config = get_plugin_manager().plugin_configs.xai_config
- endpoint, api_key = xai_config.configure(secrets_provider=get_secrets_provider())
-
- log.verbose(f"Making Xai AsyncOpenAI client with endpoint: {endpoint}")
- the_client = openai.AsyncOpenAI(
- api_key=api_key,
- base_url=endpoint,
+ azure_endpoint=backend.endpoint,
+ api_key=backend.api_key,
+ api_version=backend.get_extra_config(AzureExtraField.API_VERSION),
)
- case LLMPlatform.CUSTOM_LLM:
- custom_endpoint_config = get_plugin_manager().plugin_configs.custom_endpoint_config
- base_url, api_key = custom_endpoint_config.configure(secrets_provider=get_secrets_provider())
- log.verbose(f"Making custom AsyncOpenAI client with base_url: {base_url}")
+ case OpenAISdkVariant.OPENAI:
+ log.debug(f"Making AsyncOpenAI client with endpoint: {backend.endpoint}")
the_client = openai.AsyncOpenAI(
- api_key=api_key,
- base_url=base_url,
+ api_key=backend.api_key,
+ base_url=backend.endpoint,
)
- case LLMPlatform.ANTHROPIC | LLMPlatform.BEDROCK | LLMPlatform.BEDROCK_ANTHROPIC | LLMPlatform.MISTRAL:
- raise LLMEngineParameterError(f"Platform '{llm_platform}' is not supported by this factory '{cls.__name__}'")
return the_client
@@ -88,7 +74,7 @@ class OpenAIFactory:
def make_simple_messages(
cls,
llm_job: LLMJob,
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
) -> List[ChatCompletionMessageParam]:
"""
Makes a list of messages with a system message (if provided) and followed by a user message.
@@ -96,7 +82,7 @@ class OpenAIFactory:
llm_prompt = llm_job.llm_prompt
messages: List[ChatCompletionMessageParam] = []
user_contents: List[ChatCompletionContentPartParam] = []
- if llm_engine.llm_model.is_system_prompt_supported and (system_content := llm_prompt.system_text):
+ if inference_model.is_system_prompt_supported and (system_content := llm_prompt.system_text):
messages.append(ChatCompletionSystemMessageParam(role="system", content=system_content))
# TODO: confirm that we can prompt without user_contents, for instance if we have only images,
# otherwise consider using a default user_content
diff --git a/pipelex/plugins/openai/openai_imgg_factory.py b/pipelex/plugins/openai/openai_imgg_factory.py
index 9377cc256..0a1c2cc1d 100644
--- a/pipelex/plugins/openai/openai_imgg_factory.py
+++ b/pipelex/plugins/openai/openai_imgg_factory.py
@@ -53,3 +53,8 @@ class OpenAIImggFactory:
def background_for_gpt_image_1(cls, background: Background) -> GptImage1BackgroundType:
"""This method only converts the Background string value as a Literal, as expected by the OpenAI API"""
return background.value
+
+ @classmethod
+ def output_compression_for_gpt_image_1(cls) -> int:
+ """This method only converts the OutputCompression int value as a Literal, as expected by the OpenAI API"""
+ return 100
diff --git a/pipelex/plugins/openai/openai_imgg_worker.py b/pipelex/plugins/openai/openai_imgg_worker.py
index 6ac84f67f..83553eba8 100644
--- a/pipelex/plugins/openai/openai_imgg_worker.py
+++ b/pipelex/plugins/openai/openai_imgg_worker.py
@@ -11,7 +11,6 @@ from pipelex.cogt.imgg.imgg_engine import ImggEngine
from pipelex.cogt.imgg.imgg_job import ImggJob
from pipelex.cogt.imgg.imgg_job_components import Quality
from pipelex.cogt.imgg.imgg_worker_abstract import ImggWorkerAbstract
-from pipelex.hub import get_plugin_manager
from pipelex.plugins.openai.openai_imgg_factory import OpenAIImggFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file
@@ -54,7 +53,7 @@ class OpenAIImggWorker(ImggWorkerAbstract):
moderation = OpenAIImggFactory.moderation_for_gpt_image_1(is_moderated=imgg_job.job_params.is_moderated)
background = OpenAIImggFactory.background_for_gpt_image_1(background=imgg_job.job_params.background)
quality = OpenAIImggFactory.quality_for_gpt_image_1(quality=imgg_job.job_params.quality or Quality.LOW)
- output_compression = get_plugin_manager().plugin_configs.openai_config.image_output_compression
+ output_compression = OpenAIImggFactory.output_compression_for_gpt_image_1()
result = await self.openai_client.images.generate(
prompt=imgg_job.imgg_prompt.positive_text,
model=self.imgg_engine.imgg_model_name,
diff --git a/pipelex/plugins/openai/openai_llm_worker.py b/pipelex/plugins/openai/openai_llm_worker.py
index ef82a408d..b5c021c71 100644
--- a/pipelex/plugins/openai/openai_llm_worker.py
+++ b/pipelex/plugins/openai/openai_llm_worker.py
@@ -7,12 +7,12 @@ from openai.types.chat import ChatCompletionMessage
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCompletionError, LLMEngineParameterError, LLMModelNotFoundError, SdkTypeError
+from pipelex.cogt.exceptions import LLMCompletionError, LLMModelNotFoundError, SdkTypeError
from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_family import LLMFamily
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.openai.openai_factory import OpenAIFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -22,13 +22,13 @@ class OpenAILLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
sdk_instance: Any,
- llm_engine: LLMEngine,
+ inference_model: InferenceModelSpec,
structure_method: Optional[StructureMethod],
reporting_delegate: Optional[ReportingProtocol] = None,
):
LLMWorkerInternalAbstract.__init__(
self,
- llm_engine=llm_engine,
+ inference_model=inference_model,
structure_method=structure_method,
reporting_delegate=reporting_delegate,
)
@@ -55,94 +55,37 @@ class OpenAILLMWorker(LLMWorkerInternalAbstract):
) -> str:
messages = OpenAIFactory.make_simple_messages(
llm_job=llm_job,
- llm_engine=self.llm_engine,
+ inference_model=self.inference_model,
)
try:
- match self.llm_engine.llm_model.llm_family:
- case LLMFamily.O_SERIES | LLMFamily.GPT_5:
- # for o1 models, we must use temperature=1, and tokens limit is named max_completion_tokens
- response = await self.openai_client_for_text.chat.completions.create(
- model=self.llm_engine.llm_id,
- temperature=1,
- max_completion_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- )
- case LLMFamily.GEMINI:
- # for gemini models, we multiply the temperature by 2 because the range is 0-2
- response = await self.openai_client_for_text.chat.completions.create(
- model=self.llm_engine.llm_id,
- temperature=llm_job.job_params.temperature * 2,
- max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- )
- case (
- LLMFamily.GPT_4
- | LLMFamily.GPT_3_5
- | LLMFamily.GPT_3
- | LLMFamily.GPT_4_5
- | LLMFamily.GPT_4_1
- | LLMFamily.GPT_4O
- | LLMFamily.GPT_5_CHAT
- | LLMFamily.CUSTOM_LLAMA_4
- | LLMFamily.CUSTOM_GEMMA_3
- | LLMFamily.CUSTOM_MISTRAL_SMALL_3_1
- | LLMFamily.CUSTOM_QWEN_3
- | LLMFamily.CUSTOM_BLACKBOXAI
- | LLMFamily.PERPLEXITY_SEARCH
- | LLMFamily.PERPLEXITY_RESEARCH
- | LLMFamily.PERPLEXITY_REASONING
- | LLMFamily.PERPLEXITY_DEEPSEEK
- | LLMFamily.GROK_3
- | LLMFamily.PIPELEX_INFERENCE
- ):
- response = await self.openai_client_for_text.chat.completions.create(
- model=self.llm_engine.llm_id,
- temperature=llm_job.job_params.temperature,
- max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- )
- case (
- LLMFamily.CLAUDE_3
- | LLMFamily.CLAUDE_3_5
- | LLMFamily.CLAUDE_3_7
- | LLMFamily.CLAUDE_4
- | LLMFamily.CLAUDE_4_1
- | LLMFamily.MISTRAL_7B
- | LLMFamily.MISTRAL_8X7B
- | LLMFamily.MISTRAL_LARGE
- | LLMFamily.MISTRAL_SMALL
- | LLMFamily.MISTRAL_MEDIUM
- | LLMFamily.MISTRAL_CODESTRAL
- | LLMFamily.MINISTRAL
- | LLMFamily.PIXTRAL
- | LLMFamily.LLAMA_3
- | LLMFamily.LLAMA_3_1
- | LLMFamily.BEDROCK_MISTRAL_LARGE
- | LLMFamily.BEDROCK_ANTHROPIC_CLAUDE
- | LLMFamily.BEDROCK_META_LLAMA_3
- | LLMFamily.BEDROCK_AMAZON_NOVA
- ):
- raise LLMEngineParameterError(f"LLM family {self.llm_engine.llm_model.llm_family} is not supported by OpenAILLMWorker")
+ temperature = llm_job.job_params.temperature
+ if ModelConstraints.TEMPERATURE_MUST_BE_MULTIPLIED_BY_2 in self.inference_model.constraints:
+ temperature *= 2
+ if ModelConstraints.TEMPERATURE_MUST_BE_1 in self.inference_model.constraints and temperature != 1:
+ log.warning(f"OpenAI model {self.inference_model.desc} used with a temperature of {temperature}, but it must be 1 for this model")
+ temperature = 1
+ response = await self.openai_client_for_text.chat.completions.create(
+ model=self.inference_model.model_id,
+ temperature=temperature,
+ max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
+ seed=llm_job.job_params.seed,
+ messages=messages,
+ )
except NotFoundError as not_found_error:
# TODO: record llm config so it can be displayed here
raise LLMModelNotFoundError(
- f"OpenAI model or deployment not found:\n{self.llm_engine.desc}\nmodel: {self.llm_engine.llm_model.desc}\n{not_found_error}"
+ f"OpenAI model or deployment not found:\n{self.inference_model.desc}\nmodel: {self.inference_model.desc}\n{not_found_error}"
) from not_found_error
except APIConnectionError as api_connection_error:
raise LLMCompletionError(f"OpenAI API connection error: {api_connection_error}") from api_connection_error
except BadRequestError as bad_request_error:
- raise LLMCompletionError(
- f"OpenAI bad request error with model: {self.llm_engine.llm_model.desc}:\n{bad_request_error}"
- ) from bad_request_error
+ raise LLMCompletionError(f"OpenAI bad request error with model: {self.inference_model.desc}:\n{bad_request_error}") from bad_request_error
openai_message: ChatCompletionMessage = response.choices[0].message
response_text = openai_message.content
if response_text is None:
- raise LLMCompletionError(f"OpenAI response message content is None: {response}\nmodel: {self.llm_engine.llm_model.desc}")
+ raise LLMCompletionError(f"OpenAI response message content is None: {response}\nmodel: {self.inference_model.desc}")
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
llm_tokens_usage.nb_tokens_by_category = OpenAIFactory.make_nb_tokens_by_category(usage=usage)
@@ -156,89 +99,28 @@ class OpenAILLMWorker(LLMWorkerInternalAbstract):
) -> BaseModelTypeVar:
messages = OpenAIFactory.make_simple_messages(
llm_job=llm_job,
- llm_engine=self.llm_engine,
+ inference_model=self.inference_model,
)
try:
- match self.llm_engine.llm_model.llm_family:
- case LLMFamily.O_SERIES | LLMFamily.GPT_5:
- # for o1 models, we must use temperature=1, and tokens limit is named max_completion_tokens
- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
- model=self.llm_engine.llm_id,
- temperature=1,
- max_completion_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- response_model=schema,
- max_retries=llm_job.job_config.max_retries,
- )
- case LLMFamily.GEMINI:
- # for gemini models, we multiply the temperature by 2 because the range is 0-2
- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
- model=self.llm_engine.llm_id,
- temperature=llm_job.job_params.temperature * 2,
- max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- response_model=schema,
- max_retries=llm_job.job_config.max_retries,
- )
- case (
- LLMFamily.GPT_4
- | LLMFamily.GPT_3_5
- | LLMFamily.GPT_3
- | LLMFamily.GPT_4_5
- | LLMFamily.GPT_4_1
- | LLMFamily.GPT_4O
- | LLMFamily.GPT_5_CHAT
- | LLMFamily.CUSTOM_LLAMA_4
- | LLMFamily.CUSTOM_GEMMA_3
- | LLMFamily.CUSTOM_MISTRAL_SMALL_3_1
- | LLMFamily.CUSTOM_QWEN_3
- | LLMFamily.CUSTOM_BLACKBOXAI
- | LLMFamily.PERPLEXITY_SEARCH
- | LLMFamily.PERPLEXITY_RESEARCH
- | LLMFamily.PERPLEXITY_REASONING
- | LLMFamily.PERPLEXITY_DEEPSEEK
- | LLMFamily.GROK_3
- | LLMFamily.PIPELEX_INFERENCE
- ):
- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
- model=self.llm_engine.llm_id,
- temperature=llm_job.job_params.temperature,
- max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
- seed=llm_job.job_params.seed,
- messages=messages,
- response_model=schema,
- max_retries=llm_job.job_config.max_retries,
- )
- case (
- LLMFamily.CLAUDE_3
- | LLMFamily.CLAUDE_3_5
- | LLMFamily.CLAUDE_3_7
- | LLMFamily.CLAUDE_4
- | LLMFamily.CLAUDE_4_1
- | LLMFamily.MISTRAL_7B
- | LLMFamily.MISTRAL_8X7B
- | LLMFamily.MISTRAL_LARGE
- | LLMFamily.MISTRAL_SMALL
- | LLMFamily.MISTRAL_MEDIUM
- | LLMFamily.MISTRAL_CODESTRAL
- | LLMFamily.MINISTRAL
- | LLMFamily.PIXTRAL
- | LLMFamily.LLAMA_3
- | LLMFamily.LLAMA_3_1
- | LLMFamily.BEDROCK_MISTRAL_LARGE
- | LLMFamily.BEDROCK_ANTHROPIC_CLAUDE
- | LLMFamily.BEDROCK_META_LLAMA_3
- | LLMFamily.BEDROCK_AMAZON_NOVA
- ):
- raise LLMEngineParameterError(f"LLM family {self.llm_engine.llm_model.llm_family} is not supported by OpenAILLMWorker")
+ temperature = llm_job.job_params.temperature
+ if ModelConstraints.TEMPERATURE_MUST_BE_MULTIPLIED_BY_2 in self.inference_model.constraints:
+ temperature *= 2
+ if ModelConstraints.TEMPERATURE_MUST_BE_1 in self.inference_model.constraints and temperature != 1:
+ log.warning(f"OpenAI model {self.inference_model.desc} used with a temperature of {temperature}, but it must be 1 for this model")
+ temperature = 1
+ result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
+ model=self.inference_model.model_id,
+ temperature=temperature,
+ max_tokens=llm_job.job_params.max_tokens or NOT_GIVEN,
+ seed=llm_job.job_params.seed,
+ messages=messages,
+ response_model=schema,
+ max_retries=llm_job.job_config.max_retries,
+ )
except NotFoundError as exc:
- raise LLMCompletionError(f"OpenAI model or deployment '{self.llm_engine.llm_id}' not found: {exc}") from exc
+ raise LLMCompletionError(f"OpenAI model or deployment '{self.inference_model.model_id}' not found: {exc}") from exc
except BadRequestError as bad_request_error:
- raise LLMCompletionError(
- f"OpenAI bad request error with model: {self.llm_engine.llm_model.desc}:\n{bad_request_error}"
- ) from bad_request_error
+ raise LLMCompletionError(f"OpenAI bad request error with model: {self.inference_model.desc}:\n{bad_request_error}") from bad_request_error
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := completion.usage):
llm_tokens_usage.nb_tokens_by_category = OpenAIFactory.make_nb_tokens_by_category(usage=usage)
diff --git a/pipelex/plugins/openai/openai_llms.py b/pipelex/plugins/openai/openai_llms.py
index 2fdfa4dc2..61f5a13ca 100644
--- a/pipelex/plugins/openai/openai_llms.py
+++ b/pipelex/plugins/openai/openai_llms.py
@@ -2,18 +2,19 @@ from typing import List
from openai.types import Model
-from pipelex.cogt.exceptions import LLMSDKError
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.plugins.openai.openai_factory import OpenAIFactory
+from pipelex.plugins.plugin_sdk_registry import Plugin
-async def openai_list_available_models(llm_platform: LLMPlatform) -> List[Model]:
- openai_client_async = OpenAIFactory.make_openai_client(llm_platform=llm_platform)
- match llm_platform:
- case LLMPlatform.VERTEXAI | LLMPlatform.PERPLEXITY | LLMPlatform.CUSTOM_LLM:
- raise LLMSDKError(f"Platform '{llm_platform}' does not support listing models with OpenAI SDK")
- case _:
- pass
+async def openai_list_available_models(
+ plugin: Plugin,
+ backend: InferenceBackend,
+) -> List[Model]:
+ openai_client_async = OpenAIFactory.make_openai_client(
+ plugin=plugin,
+ backend=backend,
+ )
models = await openai_client_async.models.list()
data = models.data
diff --git a/pipelex/plugins/openai/perplexity_config.py b/pipelex/plugins/openai/perplexity_config.py
deleted file mode 100644
index 3fd06d0e0..000000000
--- a/pipelex/plugins/openai/perplexity_config.py
+++ /dev/null
@@ -1,68 +0,0 @@
-from typing import Tuple
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.exceptions import CredentialsError
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class PerplexityCredentialsError(CredentialsError):
- pass
-
-
-class PerplexityKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-PERPLEXITY_API_KEY_VAR_NAME = "PERPLEXITY_API_KEY"
-PERPLEXITY_API_ENDPOINT_VAR_NAME = "PERPLEXITY_API_ENDPOINT"
-
-
-class PerplexityConfig(ConfigModel):
- api_key_method: PerplexityKeyMethod = Field(strict=False)
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> Tuple[str, str]:
- """Configure and return endpoint and API key."""
- api_endpoint = self._get_api_endpoint(secrets_provider=secrets_provider)
- api_key = self._get_api_key(secrets_provider=secrets_provider)
- return api_endpoint, api_key
-
- def _get_api_endpoint(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API endpoint from environment or secrets provider."""
- match self.api_key_method:
- case PerplexityKeyMethod.ENV:
- log.debug("Getting Perplexity API endpoint from environment.")
- try:
- return get_required_env(PERPLEXITY_API_ENDPOINT_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise PerplexityCredentialsError(f"Error getting Perplexity API endpoint from environment: {exc}") from exc
- case PerplexityKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting Perplexity API endpoint from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=PERPLEXITY_API_ENDPOINT_VAR_NAME)
- except SecretNotFoundError as exc:
- raise PerplexityCredentialsError("Error getting Perplexity API endpoint from secrets provider.") from exc
-
- def _get_api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API key from environment or secrets provider."""
- match self.api_key_method:
- case PerplexityKeyMethod.ENV:
- log.debug("Using Perplexity API key from environment.")
- try:
- key_from_env = get_required_env(PERPLEXITY_API_KEY_VAR_NAME)
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise PerplexityCredentialsError(f"Error getting Perplexity API key from environment: {exc}") from exc
- case PerplexityKeyMethod.SECRET_PROVIDER:
- log.verbose("Using Perplexity API key from secrets provider.")
- try:
- key_from_secrets_provider = secrets_provider.get_secret(secret_id=PERPLEXITY_API_KEY_VAR_NAME)
- except SecretNotFoundError as exc:
- raise PerplexityCredentialsError("Error getting Perplexity API key from secrets provider.") from exc
- return key_from_secrets_provider
diff --git a/pipelex/plugins/openai/vertexai_config.py b/pipelex/plugins/openai/vertexai_config.py
deleted file mode 100644
index 92b4e5bce..000000000
--- a/pipelex/plugins/openai/vertexai_config.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from typing import Any, Dict, Tuple
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.cogt.exceptions import MissingDependencyError
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.exceptions import CredentialsError
-from pipelex.tools.misc.json_utils import load_json_dict_from_path
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class VertexAICredentialsError(CredentialsError):
- pass
-
-
-class VertexAIKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-GCP_PROJECT_ID_VAR_NAME = "GCP_PROJECT_ID"
-GCP_REGION_VAR_NAME = "GCP_REGION"
-GCP_CREDENTIALS_FILE_PATH_VAR_NAME = "GCP_CREDENTIALS_FILE_PATH"
-
-
-class VertexAIConfig(ConfigModel):
- api_key_method: VertexAIKeyMethod = Field(strict=False)
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> Tuple[str, str]:
- """Configure and return project ID, region, and API key."""
- project_id, region = self._get_project_settings(secrets_provider=secrets_provider)
- api_key = self._get_credentials()
- endpoint = f"https://{region}-aiplatform.googleapis.com/v1beta1/projects/{project_id}/locations/{region}/endpoints/openapi"
- return endpoint, api_key
-
- def _get_project_settings(self, secrets_provider: SecretsProviderAbstract) -> Tuple[str, str]:
- """Get project ID and region from environment or secrets provider."""
- match self.api_key_method:
- case VertexAIKeyMethod.ENV:
- log.debug("Getting GCP project settings from environment.")
- try:
- project_id = get_required_env(GCP_PROJECT_ID_VAR_NAME)
- region = get_required_env(GCP_REGION_VAR_NAME)
- return project_id, region
- except EnvVarNotFoundError as exc:
- raise VertexAICredentialsError(f"Error getting GCP project settings from environment: {exc}") from exc
- case VertexAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting GCP project settings from secrets provider.")
- try:
- project_id = secrets_provider.get_secret(secret_id=GCP_PROJECT_ID_VAR_NAME)
- region = secrets_provider.get_secret(secret_id=GCP_REGION_VAR_NAME)
- return project_id, region
- except SecretNotFoundError as exc:
- raise VertexAICredentialsError("Error getting GCP project settings from secrets provider.") from exc
-
- def _get_credentials(self) -> str:
- """Get GCP credentials and return access token."""
- try:
- from google.auth.transport.requests import Request
- from google.oauth2.service_account import Credentials
- except ImportError as exc:
- raise MissingDependencyError(
- "google-auth-oauthlib",
- "google",
- (
- "The google-auth-oauthlib SDK is required to use Google connection. "
- "You can install it with 'pip install pipelex[google]', or use this model via another provider "
- "(such as Azure OpenAI, OpenAI, anthropic or bedrock)."
- ),
- ) from exc
-
- match self.api_key_method:
- case VertexAIKeyMethod.ENV:
- log.debug("Getting GCP credentials from environment.")
- try:
- gcp_credentials_file_path = get_required_env(GCP_CREDENTIALS_FILE_PATH_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise VertexAICredentialsError(f"{GCP_CREDENTIALS_FILE_PATH_VAR_NAME} environment variable not found") from exc
- case VertexAIKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting GCP credentials from secrets provider.")
- try:
- gcp_credentials_file_path = get_required_env(GCP_CREDENTIALS_FILE_PATH_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise VertexAICredentialsError(f"{GCP_CREDENTIALS_FILE_PATH_VAR_NAME} environment variable not found") from exc
-
- try:
- credentials_dict: Dict[str, Any] = load_json_dict_from_path(path=gcp_credentials_file_path)
- except FileNotFoundError as exc:
- raise VertexAICredentialsError(f"File not found: {gcp_credentials_file_path}") from exc
-
- credentials = Credentials.from_service_account_info( # pyright: ignore[reportUnknownMemberType]
- credentials_dict, scopes=["https://www.googleapis.com/auth/cloud-platform"]
- )
- auth_req = Request()
- credentials.refresh(auth_req) # pyright: ignore[reportUnknownMemberType]
- if not isinstance(credentials.token, str): # pyright: ignore[reportUnknownMemberType]
- raise VertexAICredentialsError("Token is not a string")
- return credentials.token
diff --git a/pipelex/plugins/openai/vertexai_factory.py b/pipelex/plugins/openai/vertexai_factory.py
new file mode 100644
index 000000000..d721ffc08
--- /dev/null
+++ b/pipelex/plugins/openai/vertexai_factory.py
@@ -0,0 +1,82 @@
+from typing import Any, Dict, Tuple
+
+from pipelex.cogt.exceptions import CogtError, MissingDependencyError
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.exceptions import CredentialsError
+from pipelex.tools.misc.json_utils import load_json_dict_from_path
+from pipelex.types import StrEnum
+
+
+class VertexAIConfigError(CogtError):
+ pass
+
+
+class VertexAICredentialsError(CredentialsError):
+ pass
+
+
+class VertexAIExtraField(StrEnum):
+ GCP_PROJECT_ID = "gcp_project_id"
+ GCP_LOCATION = "gcp_location"
+ GCP_CREDENTIALS_FILE_PATH = "gcp_credentials_file_path"
+
+
+class VertexAIFactory(ConfigModel):
+ @classmethod
+ def make_endpoint_and_api_key(
+ cls,
+ extra_config: Dict[str, Any],
+ ) -> Tuple[str, str]:
+ """Configure and return endpoint, and API key."""
+ gcp_project_id = extra_config.get(VertexAIExtraField.GCP_PROJECT_ID)
+ if not gcp_project_id:
+ raise VertexAIConfigError("GCP project ID is not set in VertexAI config")
+
+ gcp_location = extra_config.get(VertexAIExtraField.GCP_LOCATION)
+ if not gcp_location:
+ raise VertexAIConfigError("GCP location is not set in VertexAI config")
+
+ endpoint = cls._make_endpoint(gcp_project_id=gcp_project_id, gcp_location=gcp_location)
+
+ gcp_credentials_file_path = extra_config.get(VertexAIExtraField.GCP_CREDENTIALS_FILE_PATH)
+ if not gcp_credentials_file_path:
+ raise VertexAIConfigError("GCP credentials file path is not set in VertexAI config")
+
+ api_key = cls._make_api_key(gcp_credentials_file_path=gcp_credentials_file_path)
+
+ return endpoint, api_key
+
+ @classmethod
+ def _make_endpoint(cls, gcp_project_id: str, gcp_location: str) -> str:
+ return f"https://{gcp_location}-aiplatform.googleapis.com/v1beta1/projects/{gcp_project_id}/locations/{gcp_location}/endpoints/openapi"
+
+ @classmethod
+ def _make_api_key(cls, gcp_credentials_file_path: str) -> str:
+ """Get GCP credentials and return access token."""
+ try:
+ from google.auth.transport.requests import Request
+ from google.oauth2.service_account import Credentials
+ except ImportError as exc:
+ raise MissingDependencyError(
+ "google-auth-oauthlib",
+ "google",
+ (
+ "The google-auth-oauthlib SDK is required to use Google connection. "
+ "You can install it with 'pip install pipelex[google]', or use this model via another provider "
+ "(such as Azure OpenAI, OpenAI, anthropic or bedrock)."
+ ),
+ ) from exc
+
+ try:
+ credentials_dict: Dict[str, Any] = load_json_dict_from_path(path=gcp_credentials_file_path)
+ except FileNotFoundError as exc:
+ raise VertexAICredentialsError(f"File not found: {gcp_credentials_file_path}") from exc
+
+ credentials = Credentials.from_service_account_info( # pyright: ignore[reportUnknownMemberType]
+ credentials_dict, scopes=["https://www.googleapis.com/auth/cloud-platform"]
+ )
+ auth_req = Request()
+ credentials.refresh(auth_req) # pyright: ignore[reportUnknownMemberType]
+ if not isinstance(credentials.token, str): # pyright: ignore[reportUnknownMemberType]
+ raise VertexAICredentialsError("Token is not a string")
+ return credentials.token
diff --git a/pipelex/plugins/openai/xai_config.py b/pipelex/plugins/openai/xai_config.py
deleted file mode 100644
index 1719e8d9e..000000000
--- a/pipelex/plugins/openai/xai_config.py
+++ /dev/null
@@ -1,68 +0,0 @@
-from typing import Tuple
-
-from pydantic import Field
-
-from pipelex import log
-from pipelex.tools.config.models import ConfigModel
-from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
-from pipelex.tools.exceptions import CredentialsError
-from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
-from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
-from pipelex.types import StrEnum
-
-
-class XaiCredentialsError(CredentialsError):
- pass
-
-
-class XaiKeyMethod(StrEnum):
- SECRET_PROVIDER = "secret_provider"
- ENV = "env"
-
-
-XAI_API_KEY_VAR_NAME = "XAI_API_KEY"
-XAI_API_ENDPOINT_VAR_NAME = "XAI_API_ENDPOINT"
-
-
-class XaiConfig(ConfigModel):
- api_key_method: XaiKeyMethod = Field(strict=False)
-
- def configure(self, secrets_provider: SecretsProviderAbstract) -> Tuple[str, str]:
- """Configure and return endpoint and API key."""
- api_endpoint = self._get_api_endpoint(secrets_provider=secrets_provider)
- api_key = self._get_api_key(secrets_provider=secrets_provider)
- return api_endpoint, api_key
-
- def _get_api_endpoint(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API endpoint from environment or secrets provider."""
- match self.api_key_method:
- case XaiKeyMethod.ENV:
- log.debug("Getting Xai API endpoint from environment.")
- try:
- return get_required_env(XAI_API_ENDPOINT_VAR_NAME)
- except EnvVarNotFoundError as exc:
- raise XaiCredentialsError(f"Error getting Xai API endpoint from environment: {exc}") from exc
- case XaiKeyMethod.SECRET_PROVIDER:
- log.verbose("Getting Xai API endpoint from secrets provider.")
- try:
- return secrets_provider.get_secret(secret_id=XAI_API_ENDPOINT_VAR_NAME)
- except SecretNotFoundError as exc:
- raise XaiCredentialsError("Error getting Xai API endpoint from secrets provider.") from exc
-
- def _get_api_key(self, secrets_provider: SecretsProviderAbstract) -> str:
- """Get API key from environment or secrets provider."""
- match self.api_key_method:
- case XaiKeyMethod.ENV:
- log.debug("Using Xai API key from environment.")
- try:
- key_from_env = get_required_env(XAI_API_KEY_VAR_NAME)
- return key_from_env
- except EnvVarNotFoundError as exc:
- raise XaiCredentialsError(f"Error getting Xai API key from environment: {exc}") from exc
- case XaiKeyMethod.SECRET_PROVIDER:
- log.verbose("Using Xai API key from secrets provider.")
- try:
- key_from_secrets_provider = secrets_provider.get_secret(secret_id=XAI_API_KEY_VAR_NAME)
- except SecretNotFoundError as exc:
- raise XaiCredentialsError("Error getting Xai API key from secrets provider.") from exc
- return key_from_secrets_provider
diff --git a/pipelex/plugins/plugin_manager.py b/pipelex/plugins/plugin_manager.py
index 90b0adfdf..0f6fd8946 100644
--- a/pipelex/plugins/plugin_manager.py
+++ b/pipelex/plugins/plugin_manager.py
@@ -1,27 +1,12 @@
-from typing import Optional
-
-from pipelex.libraries.library_config import LibraryConfig
from pipelex.plugins.plugin_sdk_registry import PluginSdkRegistry
-from pipelex.plugins.plugins_config import PluginConfig
-from pipelex.tools.misc.toml_utils import load_toml_from_path
class PluginManager:
def __init__(self):
- self._plugin_configs: Optional[PluginConfig] = None
self.plugin_sdk_registry = PluginSdkRegistry()
- @property
- def plugin_configs(self) -> PluginConfig:
- if self._plugin_configs is None:
- raise RuntimeError("Plugin configs not loaded")
- return self._plugin_configs
-
- def setup(self, library_config: LibraryConfig):
- plugin_config_path = library_config.get_default_plugin_config_path()
- plugin_config_dict = load_toml_from_path(path=plugin_config_path)
- self._plugin_configs = PluginConfig.model_validate(plugin_config_dict)
+ def setup(self):
+ pass
def teardown(self):
- self._plugin_configs = None
self.plugin_sdk_registry.teardown()
diff --git a/pipelex/plugins/plugin_sdk_registry.py b/pipelex/plugins/plugin_sdk_registry.py
index 6b350d4cb..c8643f208 100644
--- a/pipelex/plugins/plugin_sdk_registry.py
+++ b/pipelex/plugins/plugin_sdk_registry.py
@@ -1,66 +1,42 @@
from typing import Any, Dict, Optional
-from pydantic import Field, RootModel
+from pydantic import BaseModel, Field, RootModel
from pipelex.cogt.imgg.imgg_platform import ImggPlatform
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.ocr.ocr_platform import OcrPlatform
-from pipelex.types import StrEnum
-class PluginSdkHandle(StrEnum):
- OPENAI_SDK = "openai_sdk"
- AZURE_OPENAI_SDK = "azure_openai_sdk"
- ANTHROPIC_SDK = "anthropic_sdk"
- BEDROCK_ANTHROPIC_SDK = "bedrock_anthropic_sdk"
- MISTRAL_SDK = "mistral_sdk"
- BASIC_SDK = "basic_sdk"
- BEDROCK_SDK = "bedrock_sdk"
- PERPLEXITY_OPENAI_SDK = "perplexity_openai_sdk"
- VERTEXAI_OPENAI_SDK = "vertexai_openai_sdk"
- XAI_OPENAI_SDK = "xai_openai_sdk"
- CUSTOM_LLM_OPENAI_SDK = "custom_llm_openai_sdk"
- FAL_SDK = "fal_sdk"
+class Plugin(BaseModel):
+ sdk: str
+ backend: str
- @staticmethod
- def get_for_llm_platform(llm_platform: LLMPlatform) -> "PluginSdkHandle":
- match llm_platform:
- case LLMPlatform.OPENAI:
- return PluginSdkHandle.OPENAI_SDK
- case LLMPlatform.AZURE_OPENAI:
- return PluginSdkHandle.AZURE_OPENAI_SDK
- case LLMPlatform.ANTHROPIC:
- return PluginSdkHandle.ANTHROPIC_SDK
- case LLMPlatform.MISTRAL:
- return PluginSdkHandle.MISTRAL_SDK
- case LLMPlatform.BEDROCK:
- return PluginSdkHandle.BEDROCK_SDK
- case LLMPlatform.BEDROCK_ANTHROPIC:
- return PluginSdkHandle.BEDROCK_ANTHROPIC_SDK
- case LLMPlatform.PERPLEXITY:
- return PluginSdkHandle.PERPLEXITY_OPENAI_SDK
- case LLMPlatform.VERTEXAI:
- return PluginSdkHandle.VERTEXAI_OPENAI_SDK
- case LLMPlatform.XAI:
- return PluginSdkHandle.XAI_OPENAI_SDK
- case LLMPlatform.CUSTOM_LLM:
- return PluginSdkHandle.CUSTOM_LLM_OPENAI_SDK
+ @property
+ def sdk_handle(self) -> str:
+ return f"{self.sdk}@{self.backend}"
- @staticmethod
- def get_for_ocr_engine(ocr_platform: OcrPlatform) -> "PluginSdkHandle":
+ @classmethod
+ def make_for_ocr_engine(cls, ocr_platform: OcrPlatform) -> "Plugin":
match ocr_platform:
case OcrPlatform.MISTRAL:
- return PluginSdkHandle.MISTRAL_SDK
+ return Plugin(sdk="mistral", backend="mistral")
case OcrPlatform.BASIC:
- return PluginSdkHandle.BASIC_SDK
+ return Plugin(sdk="basic", backend="none")
- @staticmethod
- def get_for_imgg_engine(imgg_platform: ImggPlatform) -> "PluginSdkHandle":
+ @classmethod
+ def make_for_imgg_engine(cls, imgg_platform: ImggPlatform) -> "Plugin":
match imgg_platform:
case ImggPlatform.FAL_AI:
- return PluginSdkHandle.FAL_SDK
+ return Plugin(sdk="fal", backend="fal")
case ImggPlatform.OPENAI:
- return PluginSdkHandle.OPENAI_SDK
+ return Plugin(sdk="openai", backend="openai")
+
+ @classmethod
+ def make_for_inference_model(cls, inference_model: InferenceModelSpec) -> "Plugin":
+ return Plugin(
+ sdk=inference_model.sdk,
+ backend=inference_model.backend_name,
+ )
PluginSdkRegistryRoot = Dict[str, Any]
@@ -70,28 +46,14 @@ class PluginSdkRegistry(RootModel[PluginSdkRegistryRoot]):
root: PluginSdkRegistryRoot = Field(default_factory=dict)
def teardown(self):
- for llm_sdk_instance in self.root.values():
- if hasattr(llm_sdk_instance, "teardown"):
- llm_sdk_instance.teardown()
+ for sdk_instance in self.root.values():
+ if hasattr(sdk_instance, "teardown"):
+ sdk_instance.teardown()
self.root = {}
- def get_llm_sdk_instance(self, llm_sdk_handle: PluginSdkHandle) -> Optional[Any]:
- return self.root.get(llm_sdk_handle)
-
- def set_llm_sdk_instance(self, llm_sdk_handle: PluginSdkHandle, llm_sdk_instance: Any) -> Any:
- self.root[llm_sdk_handle] = llm_sdk_instance
- return llm_sdk_instance
-
- def get_ocr_sdk_instance(self, ocr_sdk_handle: PluginSdkHandle) -> Optional[Any]:
- return self.root.get(ocr_sdk_handle)
-
- def set_ocr_sdk_instance(self, ocr_sdk_handle: PluginSdkHandle, ocr_sdk_instance: Any) -> Any:
- self.root[ocr_sdk_handle] = ocr_sdk_instance
- return ocr_sdk_instance
-
- def get_imgg_sdk_instance(self, imgg_sdk_handle: PluginSdkHandle) -> Optional[Any]:
- return self.root.get(imgg_sdk_handle)
+ def get_sdk_instance(self, plugin: Plugin) -> Optional[Any]:
+ return self.root.get(plugin.sdk_handle)
- def set_imgg_sdk_instance(self, imgg_sdk_handle: PluginSdkHandle, imgg_sdk_instance: Any) -> Any:
- self.root[imgg_sdk_handle] = imgg_sdk_instance
- return imgg_sdk_instance
+ def set_sdk_instance(self, plugin: Plugin, sdk_instance: Any) -> Any:
+ self.root[plugin.sdk_handle] = sdk_instance
+ return sdk_instance
diff --git a/pipelex/plugins/plugins_config.py b/pipelex/plugins/plugins_config.py
deleted file mode 100644
index cd11b970f..000000000
--- a/pipelex/plugins/plugins_config.py
+++ /dev/null
@@ -1,24 +0,0 @@
-from pipelex.plugins.anthropic.anthropic_config import AnthropicConfig
-from pipelex.plugins.bedrock.bedrock_config import BedrockConfig
-from pipelex.plugins.fal.fal_config import FalConfig
-from pipelex.plugins.mistral.mistral_config import MistralConfig
-from pipelex.plugins.openai.azure_openai_config import AzureOpenAIConfig
-from pipelex.plugins.openai.custom_endpoint_config import CustomEndpointConfig
-from pipelex.plugins.openai.openai_config import OpenAIConfig
-from pipelex.plugins.openai.perplexity_config import PerplexityConfig
-from pipelex.plugins.openai.vertexai_config import VertexAIConfig
-from pipelex.plugins.openai.xai_config import XaiConfig
-from pipelex.tools.config.models import ConfigModel
-
-
-class PluginConfig(ConfigModel):
- anthropic_config: AnthropicConfig
- azure_openai_config: AzureOpenAIConfig
- bedrock_config: BedrockConfig
- vertexai_config: VertexAIConfig
- mistral_config: MistralConfig
- openai_config: OpenAIConfig
- perplexity_config: PerplexityConfig
- xai_config: XaiConfig
- custom_endpoint_config: CustomEndpointConfig
- fal_config: FalConfig
diff --git a/pipelex/plugins/specific_llm/specific_llm_config.py b/pipelex/plugins/specific_llm/specific_llm_config.py
deleted file mode 100644
index 3eff2f16a..000000000
--- a/pipelex/plugins/specific_llm/specific_llm_config.py
+++ /dev/null
@@ -1,7 +0,0 @@
-from typing import Dict
-
-from pipelex.tools.config.models import ConfigModel
-
-
-class SpecificLLMConfig(ConfigModel):
- llm_worker_classes: Dict[str, str]
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index cf6370856..85c063824 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -5,10 +5,10 @@ from typing_extensions import override
from pipelex import log
from pipelex.cogt.exceptions import ReportingManagerError
-from pipelex.cogt.inference.cost_registry import CostRegistry
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokensUsage
+from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.config import ReportingConfig
from pipelex.pipeline.pipeline_models import SpecialPipelineId
from pipelex.reporting.reporting_protocol import ReportingProtocol
diff --git a/pipelex/tools/aws/aws_config.py b/pipelex/tools/aws/aws_config.py
index 1dbda8abb..b3aa13c63 100644
--- a/pipelex/tools/aws/aws_config.py
+++ b/pipelex/tools/aws/aws_config.py
@@ -4,7 +4,7 @@ from pydantic import Field
from pipelex import log
from pipelex.hub import get_secret
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.tools.environment import EnvVarNotFoundError, get_required_env
from pipelex.tools.exceptions import CredentialsError
from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
diff --git a/pipelex/tools/config/models.py b/pipelex/tools/config/config_model.py
similarity index 55%
rename from pipelex/tools/config/models.py
rename to pipelex/tools/config/config_model.py
index 725cadac7..64a3f7e0d 100644
--- a/pipelex/tools/config/models.py
+++ b/pipelex/tools/config/config_model.py
@@ -1,21 +1,10 @@
-from typing import Any, Dict, Optional, Type, TypeVar, Union
+from typing import Dict, List, Optional, Type, TypeVar, Union
-from pydantic import BaseModel, ConfigDict, ValidationError
+from pydantic import BaseModel, ConfigDict
-from pipelex.tools.exceptions import ConfigModelError, ConfigValidationError
-from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+from pipelex.tools.exceptions import ConfigModelError
from pipelex.types import StrEnum
-CONFIG_BASE_OVERRIDES_BEFORE_ENV = ["local"]
-CONFIG_BASE_OVERRIDES_AFTER_ENV = ["super"]
-
-
-class SecretMethod(StrEnum):
- NONE = "none"
- ENV_VAR = "env_var"
- SECRET_PROVIDER = "secret_provider"
-
-
StrEnumType = TypeVar("StrEnumType", bound=StrEnum)
@@ -49,31 +38,26 @@ class ConfigModel(BaseModel):
else:
raise ConfigModelError("Either key_enum_cls or value_enum_cls must be provided.")
-
-class ConfigRoot(ConfigModel):
- """
- Main configuration class for the project.
-
- Attributes:
- project_name (str): Name of the current project.
- """
-
- project_name: Optional[str] = None
-
- def __init__(self, **kwargs: Any):
+ @staticmethod
+ def transform_dict_of_floats_str_to_enum(
+ input_dict: Dict[str, float],
+ key_enum_cls: Type[StrEnumType],
+ ) -> Dict[StrEnumType, float]:
"""
- Initialize the Config instance.
+ Transforms a dictionary with str keys and float values into a dictionary with enum keys and float values.
Args:
- **kwargs: Keyword arguments for configuration.
+ input_dict: Dictionary with string values to be transformed.
+ key_enum_cls: The StrEnum class to convert the keys to
- Raises:
- ConfigValidationError: If the provided data is invalid.
+ Returns:
+ A dictionary where the keys are converted to the given StrEnum type.
"""
+ return {key_enum_cls(key): value for key, value in input_dict.items()}
- try:
- super().__init__(**kwargs)
- except ValidationError as exc:
- validation_error_msg = format_pydantic_validation_error(exc)
- error_msg = f"Could not create config of type {type(self)} with provided data: {validation_error_msg}"
- raise ConfigValidationError(message=error_msg) from exc
+ @staticmethod
+ def transform_list_of_str_to_enum(
+ input_list: List[str],
+ enum_cls: Type[StrEnumType],
+ ) -> List[StrEnumType]:
+ return [enum_cls(item) for item in input_list]
diff --git a/pipelex/tools/config/config_root.py b/pipelex/tools/config/config_root.py
new file mode 100644
index 000000000..fcf42c180
--- /dev/null
+++ b/pipelex/tools/config/config_root.py
@@ -0,0 +1,46 @@
+from typing import Any, Optional
+
+from pydantic import ValidationError
+
+from pipelex.tools.config.config_model import ConfigModel
+from pipelex.tools.exceptions import ConfigValidationError
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+from pipelex.types import StrEnum
+
+CONFIG_BASE_OVERRIDES_BEFORE_ENV = ["local"]
+CONFIG_BASE_OVERRIDES_AFTER_ENV = ["super"]
+
+
+class SecretMethod(StrEnum):
+ NONE = "none"
+ ENV_VAR = "env_var"
+ SECRET_PROVIDER = "secret_provider"
+
+
+class ConfigRoot(ConfigModel):
+ """
+ Main configuration class for the project.
+
+ Attributes:
+ project_name (str): Name of the current project.
+ """
+
+ project_name: Optional[str] = None
+
+ def __init__(self, **kwargs: Any):
+ """
+ Initialize the Config instance.
+
+ Args:
+ **kwargs: Keyword arguments for configuration.
+
+ Raises:
+ ConfigValidationError: If the provided data is invalid.
+ """
+
+ try:
+ super().__init__(**kwargs)
+ except ValidationError as exc:
+ validation_error_msg = format_pydantic_validation_error(exc)
+ error_msg = f"Could not create config of type {type(self)} with provided data: {validation_error_msg}"
+ raise ConfigValidationError(message=error_msg) from exc
diff --git a/pipelex/tools/config/manager.py b/pipelex/tools/config/manager.py
index 454ff0bc9..0e57889ff 100644
--- a/pipelex/tools/config/manager.py
+++ b/pipelex/tools/config/manager.py
@@ -5,7 +5,7 @@ from typing import Any, Dict, Optional
import toml
-from pipelex.tools.config.models import (
+from pipelex.tools.config.config_root import (
CONFIG_BASE_OVERRIDES_AFTER_ENV,
CONFIG_BASE_OVERRIDES_BEFORE_ENV,
)
@@ -13,7 +13,10 @@ from pipelex.tools.misc.json_utils import deep_update
from pipelex.tools.misc.toml_utils import failable_load_toml_from_path
from pipelex.tools.runtime_manager import runtime_manager
+CONFIG_DIR_NAME = ".pipelex"
CONFIG_NAME = "pipelex.toml"
+CONFIG_TEMPLATE_SUBPATH = "config_template"
+INFERENCE_CONFIG_SUBPATH = "inference"
class ConfigException(Exception):
@@ -35,21 +38,35 @@ class ConfigManager:
current_dir = os.path.dirname(os.path.abspath(__file__))
return os.path.dirname(os.path.dirname(current_dir))
+ @property
+ def pipelex_root_config_path(self) -> str:
+ return os.path.join(self.pipelex_root_dir, CONFIG_NAME)
+
@property
def local_root_dir(self) -> str:
"""Get the root directory of the project using pipelex.
This is the directory from where the command is being run."""
return os.path.abspath(os.getcwd())
+ @property
+ def pipelex_config_dir(self) -> str:
+ return os.path.join(self.local_root_dir, CONFIG_DIR_NAME)
+
+ @property
+ def pipelex_specific_config_file_path(self) -> str:
+ return os.path.join(self.pipelex_config_dir, CONFIG_NAME)
+
def get_pipelex_config(self) -> Dict[str, Any]:
"""Get the pipelex configuration from pipelex.toml.
Returns:
Dict[str, Any]: The configuration dictionary from pipelex.toml
"""
- config_path = os.path.join(self.pipelex_root_dir, CONFIG_NAME)
+ config_path = self.pipelex_root_config_path
config = failable_load_toml_from_path(config_path)
- return config or {}
+ if not config:
+ raise ConfigException(f"Pipelex root config could not be found at {self.pipelex_specific_config_file_path}.")
+ return config
def get_local_config(self) -> Dict[str, Any]:
"""Get the local pipelex configuration from pipelex.toml in the project root.
diff --git a/pipelex/tools/environment.py b/pipelex/tools/environment.py
index 4fb965291..99ff29270 100644
--- a/pipelex/tools/environment.py
+++ b/pipelex/tools/environment.py
@@ -15,7 +15,7 @@ class EnvVarNotFoundError(ToolException):
def get_required_env(key: str) -> str:
value = os.getenv(key)
if not value:
- raise EnvVarNotFoundError(f"Missing '{key} 'in environment.")
+ raise EnvVarNotFoundError(f"Environment variable '{key}' is required but not set")
return value
diff --git a/pipelex/tools/log/log_config.py b/pipelex/tools/log/log_config.py
index 2d180bc73..1a49e6295 100644
--- a/pipelex/tools/log/log_config.py
+++ b/pipelex/tools/log/log_config.py
@@ -5,7 +5,7 @@ from pydantic import Field, field_validator
from rich.highlighter import Highlighter, JSONHighlighter, ReprHighlighter
from rich.logging import RichHandler
-from pipelex.tools.config.models import ConfigModel
+from pipelex.tools.config.config_model import ConfigModel
from pipelex.tools.log.log_levels import LogLevel
from pipelex.types import StrEnum
diff --git a/pipelex/tools/misc/dict_utils.py b/pipelex/tools/misc/dict_utils.py
index ac48cdb45..1d54df1e7 100644
--- a/pipelex/tools/misc/dict_utils.py
+++ b/pipelex/tools/misc/dict_utils.py
@@ -1,6 +1,6 @@
"""Dictionary utility functions for manipulating dictionary order and structure."""
-from typing import Dict, TypeVar
+from typing import Any, Callable, Dict, List, TypeVar, cast
K = TypeVar("K")
V = TypeVar("V")
@@ -41,3 +41,54 @@ def insert_before(dictionary: Dict[K, V], target_key: K, new_key: K, new_value:
result[new_key] = new_value
return result
+
+
+def apply_to_strings_recursive(data: Any, transform_func: Callable[[str], str]) -> Dict[str, Any]:
+ """
+ Recursively traverse a data structure and apply a transformation function to all string values.
+
+ This function walks through dictionaries, lists, and other nested structures,
+ applying the provided transformation function only to string values while
+ preserving the original structure.
+
+ Args:
+ data: The data structure to traverse (dict, list, or any value)
+ transform_func: Function to apply to each string value found
+
+ Returns:
+ A new data structure with the same shape but with transformed strings
+
+ Example:
+ >>> data = {'a': 'hello ${USER}', 'b': [1, 'world ${HOME}'], 'c': {'d': 'test ${PATH}'}}
+ >>> result = apply_to_strings_recursive(data, lambda s: s.replace('${USER}', 'john'))
+ >>> # Returns: {'a': 'hello john', 'b': [1, 'world ${HOME}'], 'c': {'d': 'test ${PATH}'}}
+ """
+ result: Dict[str, Any] = {}
+ for key, value in data.items():
+ if isinstance(value, dict):
+ result[key] = apply_to_strings_recursive(value, transform_func)
+ elif isinstance(value, list):
+ list_value: List[Any] = cast(List[Any], value)
+ result[key] = apply_to_strings_in_list(list_value, transform_func)
+ elif isinstance(value, str):
+ result[key] = transform_func(value)
+ else:
+ # For all other types (int, float, bool, None, etc.), return as-is
+ result[key] = value
+ return result
+
+
+def apply_to_strings_in_list(data: List[Any], transform_func: Callable[[str], str]) -> List[Any]:
+ """Helper function to apply string transformation to items in a list."""
+ result: List[Any] = []
+ for item in data:
+ if isinstance(item, dict):
+ result.append(apply_to_strings_recursive(item, transform_func))
+ elif isinstance(item, list):
+ list_item: List[Any] = cast(List[Any], item)
+ result.append(apply_to_strings_in_list(list_item, transform_func))
+ elif isinstance(item, str):
+ result.append(transform_func(item))
+ else:
+ result.append(item)
+ return result
diff --git a/pipelex/tools/misc/string_utils.py b/pipelex/tools/misc/string_utils.py
index 940705b24..6237f1dfe 100644
--- a/pipelex/tools/misc/string_utils.py
+++ b/pipelex/tools/misc/string_utils.py
@@ -204,3 +204,48 @@ def is_snake_case(word: str) -> bool:
def is_pascal_case(word: str) -> bool:
return re.match(r"^[A-Z][a-zA-Z0-9]*$", word) is not None
+
+
+def matches_wildcard_pattern(text: str, pattern: str) -> bool:
+ """Check if a text matches a wildcard pattern.
+
+ Supports wildcards (*) at the beginning, end, or both.
+ A single asterisk (*) matches any text.
+
+ Args:
+ text: The text to check against the pattern
+ pattern: The pattern to match against, supporting wildcards (*)
+
+ Returns:
+ True if the text matches the pattern
+
+ Examples:
+ >>> matches_wildcard_pattern("claude-3-sonnet", "claude-*")
+ True
+ >>> matches_wildcard_pattern("gpt-4o-mini", "*mini")
+ True
+ >>> matches_wildcard_pattern("mistral-large", "*large*")
+ True
+ >>> matches_wildcard_pattern("any-model", "*")
+ True
+ >>> matches_wildcard_pattern("exact-match", "exact-match")
+ True
+ """
+ if pattern == "*":
+ return True
+
+ if pattern.startswith("*") and pattern.endswith("*"):
+ # Pattern like "*sonnet*"
+ middle = pattern[1:-1]
+ return middle in text
+ elif pattern.startswith("*"):
+ # Pattern like "*sonnet"
+ suffix = pattern[1:]
+ return text.endswith(suffix)
+ elif pattern.endswith("*"):
+ # Pattern like "claude-*"
+ prefix = pattern[:-1]
+ return text.startswith(prefix)
+ else:
+ # Exact match
+ return text == pattern
diff --git a/pipelex/tools/misc/toml_utils.py b/pipelex/tools/misc/toml_utils.py
index 4545f596a..3b83621cc 100644
--- a/pipelex/tools/misc/toml_utils.py
+++ b/pipelex/tools/misc/toml_utils.py
@@ -113,7 +113,9 @@ def load_toml_from_path(path: str) -> Dict[str, Any]:
with open(path, "w", encoding="utf-8") as file:
file.write(cleaned_content)
+ # Parse TOML first
dict_from_toml = toml.loads(cleaned_content)
+
return dict_from_toml
except toml.TomlDecodeError as exc:
raise toml.TomlDecodeError(f"TOML parsing error in file '{path}': {exc}", exc.doc, exc.pos) from exc
@@ -125,7 +127,7 @@ def failable_load_toml_from_path(path: str) -> Optional[Dict[str, Any]]:
return None
try:
return load_toml_from_path(path)
- except toml.TomlDecodeError as exc:
+ except (toml.TomlDecodeError, TOMLValidationError) as exc:
print(f"Failed to parse TOML file '{path}': {exc}")
return None
diff --git a/pipelex/tools/secrets/secrets_utils.py b/pipelex/tools/secrets/secrets_utils.py
new file mode 100644
index 000000000..5807c77de
--- /dev/null
+++ b/pipelex/tools/secrets/secrets_utils.py
@@ -0,0 +1,125 @@
+import re
+
+from pipelex.hub import get_secrets_provider
+from pipelex.tools.environment import EnvVarNotFoundError, get_optional_env, get_required_env
+from pipelex.tools.exceptions import ToolException
+from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+from pipelex.types import StrEnum
+
+
+class VarNotFoundError(ToolException):
+ pass
+
+
+class UnknownVarPrefixError(ToolException):
+ """Raised when an unknown variable prefix is used in variable substitution."""
+
+ pass
+
+
+class VarPrefix(StrEnum):
+ """Variable prefix types for variable substitution."""
+
+ ENV = "env"
+ SECRET = "secret"
+
+
+def substitute_vars(content: str) -> str:
+ """Substitute variable placeholders with values from environment variables or secrets.
+
+ Supports the following placeholder formats:
+ - ${VAR_NAME} -> use secrets provider by default
+ - ${env:ENV_VAR_NAME} -> force use environment variable
+ - ${secret:SECRET_NAME} -> force use secrets provider
+ - ${env:ENV_VAR_NAME|secret:SECRET_NAME} -> try env first, then secret as fallback
+
+ Args:
+ content: Text content with variable placeholders
+
+ Returns:
+ Content with variables substituted
+
+ Raises:
+ VarNotFoundError: If required variable is missing from all specified sources
+ """
+
+ def replace_var(match: re.Match[str]) -> str:
+ var_spec = match.group(1)
+
+ # Check if it's a fallback pattern (contains |)
+ if "|" in var_spec:
+ return _handle_fallback_pattern(var_spec)
+
+ # Check if it has a prefix (env: or secret:)
+ if ":" in var_spec:
+ prefix_str, var_name = var_spec.split(":", 1)
+ prefix_str = prefix_str.strip()
+
+ try:
+ prefix = VarPrefix(prefix_str)
+ except ValueError:
+ raise UnknownVarPrefixError(f"Unknown variable prefix: '{prefix_str}'")
+
+ match prefix:
+ case VarPrefix.ENV:
+ return _get_env_var(var_name)
+ case VarPrefix.SECRET:
+ return _get_secret(var_name)
+ else:
+ # Default behavior: use secrets provider
+ return _get_secret(var_spec)
+
+ # Pattern matches ${VAR_NAME} or ${prefix:VAR_NAME} or ${env:VAR|secret:VAR}
+ # Restrict to not match across newlines, quotes, or nested braces
+ pattern = r"\$\{([^}\n\"'$]+)\}"
+ return re.sub(pattern, replace_var, content)
+
+
+def _handle_fallback_pattern(var_spec: str) -> str:
+ """Handle fallback pattern like 'env:VAR|secret:VAR'."""
+ parts = [part.strip() for part in var_spec.split("|")]
+
+ for part in parts:
+ if ":" in part:
+ prefix_str, var_name = part.split(":", 1)
+ prefix_str = prefix_str.strip()
+
+ try:
+ prefix = VarPrefix(prefix_str)
+ except ValueError:
+ raise UnknownVarPrefixError(f"Unknown variable prefix: '{prefix_str}'")
+
+ match prefix:
+ case VarPrefix.ENV:
+ value = get_optional_env(var_name)
+ if value is not None:
+ return value
+ case VarPrefix.SECRET:
+ try:
+ return get_secrets_provider().get_secret(secret_id=var_name)
+ except SecretNotFoundError:
+ continue # Try next option
+ else:
+ # No prefix, try as secret
+ try:
+ return get_secrets_provider().get_secret(secret_id=part)
+ except SecretNotFoundError:
+ continue # Try next option
+
+ raise VarNotFoundError(f"Variable not found in any source: {var_spec}")
+
+
+def _get_env_var(var_name: str) -> str:
+ """Get environment variable, raising VarNotFoundError if not found."""
+ try:
+ return get_required_env(var_name)
+ except EnvVarNotFoundError as exc:
+ raise VarNotFoundError(f"Could not get variable '{var_name}': {str(exc)}") from exc
+
+
+def _get_secret(secret_name: str) -> str:
+ """Get secret, raising VarNotFoundError if not found."""
+ try:
+ return get_secrets_provider().get_secret(secret_id=secret_name)
+ except SecretNotFoundError as exc:
+ raise VarNotFoundError(f"Could not get variable '{secret_name}': {str(exc)}") from exc
diff --git a/pyproject.toml b/pyproject.toml
index be22dfa36..933d7480e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "pipelex"
-version = "0.9.6"
+version = "0.10.0"
description = "Pipelex is an open-source dev tool based on a simple declarative language that lets you define replicable, structured, composable LLM pipelines."
authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]
maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]
diff --git a/tests/conftest.py b/tests/conftest.py
index 1e93d46e4..3ab87df43 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,3 +1,6 @@
+import atexit
+import os
+
import pytest
from rich import print
from rich.console import Console
@@ -18,6 +21,98 @@ pytest_plugins = [
TEST_OUTPUTS_DIR = "temp/test_outputs"
+def _setup_ci_env_vars():
+ """Set placeholder environment variables when running in CI to prevent import failures.
+
+ These placeholders allow the code to import successfully, while actual inference tests
+ remain skipped via pytest markers.
+ """
+ # Check if we're running in CI (GitHub Actions or generic CI environment)
+ if os.getenv("CI") or os.getenv("GITHUB_ACTIONS"):
+ print("[yellow]CI environment detected - setting placeholder API keys[/yellow]")
+
+ # Define placeholder values for all inference-related env vars
+ ci_placeholders = {
+ "PIPELEX_API_TOKEN": "ci-placeholder-token",
+ "PIPELEX_API_BASE_URL": "https://app.pipelex.ai/api/v1",
+ "PIPELEX_INFERENCE_API_KEY": "ci-placeholder-inference-key",
+ "OPENAI_API_KEY": "sk-ci-placeholder-key",
+ "AWS_ACCESS_KEY_ID": "ci-placeholder-aws-key",
+ "AWS_SECRET_ACCESS_KEY": "ci-placeholder-aws-secret",
+ "AWS_REGION": "us-east-1",
+ "AZURE_OPENAI_API_ENDPOINT": "https://ci-placeholder.openai.azure.com",
+ "AZURE_OPENAI_API_VERSION": "2025-04-01-preview",
+ "AZURE_OPENAI_API_KEY": "ci-placeholder-azure-key",
+ "AZURE_API_BASE": "https://ci-placeholder.openai.azure.com",
+ "AZURE_API_KEY": "ci-placeholder-azure-api-key",
+ "AZURE_API_VERSION": "2025-04-01-preview",
+ "ANTHROPIC_API_KEY": "sk-ant-ci-placeholder-key",
+ "MISTRAL_API_KEY": "ci-placeholder-mistral-key",
+ "PERPLEXITY_API_KEY": "ci-placeholder-perplexity-key",
+ "PERPLEXITY_API_ENDPOINT": "https://api.perplexity.ai",
+ "FAL_API_KEY": "ci-placeholder-fal-key",
+ "GCP_PROJECT_ID": "ci-placeholder-project",
+ "GCP_REGION": "us-central1",
+ "GCP_LOCATION": "us-central1",
+ "GCP_CREDENTIALS_FILE_PATH": "/tmp/ci-placeholder-gcp-credentials.json",
+ "XAI_API_KEY": "ci-placeholder-xai-key",
+ "XAI_API_ENDPOINT": "https://api.x.ai/v1/",
+ "BLACKBOX_API_KEY": "ci-placeholder-blackbox-key",
+ "CUSTOM_ENDPOINT_BASE_URL": "http://localhost:11434/v1/",
+ "CUSTOM_ENDPOINT_API_KEY": "ci-placeholder-custom-key",
+ }
+
+ # Set placeholders, overriding any existing values in CI
+ # This ensures tests won't accidentally use real API keys in CI
+ for key, value in ci_placeholders.items():
+ os.environ[key] = value
+
+ # Temporarily modify backends.toml to disable problematic backends in CI
+ backends_path = ".pipelex/inference/backends.toml"
+ backup_path = ".pipelex/inference/backends.toml.backup"
+
+ # Create backup if it doesn't exist
+ if not os.path.exists(backup_path):
+ with open(backends_path, "r") as f:
+ original_content = f.read()
+ with open(backup_path, "w") as f:
+ f.write(original_content)
+
+ # Modify the original file to disable vertexai
+ with open(backends_path, "r") as f:
+ content = f.read()
+
+ # Disable vertexai backend for CI
+ content = content.replace("[vertexai]\nenabled = true", "[vertexai]\nenabled = false")
+
+ with open(backends_path, "w") as f:
+ f.write(content)
+
+ print(f"[yellow]Set {len(ci_placeholders)} placeholder environment variables[/yellow]")
+
+
+def _restore_ci_backends_file():
+ """Restore original backends.toml file if we're in CI and a backup exists."""
+ if os.getenv("CI") or os.getenv("GITHUB_ACTIONS"):
+ backends_path = ".pipelex/inference/backends.toml"
+ backup_path = ".pipelex/inference/backends.toml.backup"
+
+ if os.path.exists(backup_path):
+ with open(backup_path, "r") as f:
+ original_content = f.read()
+ with open(backends_path, "w") as f:
+ f.write(original_content)
+ os.remove(backup_path)
+ print("[yellow]Restored original backends.toml file[/yellow]")
+
+
+# Set up CI environment variables before any imports that might need them
+_setup_ci_env_vars()
+
+# Register cleanup function to restore backends file
+atexit.register(_restore_ci_backends_file)
+
+
@pytest.fixture(scope="module", autouse=True)
def reset_pipelex_config_fixture():
# Code to run before each test
diff --git a/tests/data/tools_data/problematic_test_cases.plx b/tests/data/tools_data/problematic_test_cases.toml
similarity index 100%
rename from tests/data/tools_data/problematic_test_cases.plx
rename to tests/data/tools_data/problematic_test_cases.toml
diff --git a/tests/integration/pipelex/cogt/test_content_generator.py b/tests/integration/pipelex/cogt/test_content_generator.py
index b0614b2b3..1fcb3c305 100644
--- a/tests/integration/pipelex/cogt/test_content_generator.py
+++ b/tests/integration/pipelex/cogt/test_content_generator.py
@@ -8,13 +8,13 @@ from pipelex.cogt.exceptions import LLMHandleNotFoundError
from pipelex.cogt.image.generated_image import GeneratedImage
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.imgg.imgg_prompt import ImggPrompt
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.ocr.ocr_handle import OcrHandle
from pipelex.cogt.ocr.ocr_input import OcrInput
from pipelex.cogt.ocr.ocr_job_components import OcrJobConfig, OcrJobParams
from pipelex.cogt.ocr.ocr_output import OcrOutput
-from pipelex.hub import get_content_generator, get_llm_deck
+from pipelex.hub import get_content_generator, get_models_manager
from pipelex.pipeline.job_metadata import JobMetadata
from tests.cases import ImageTestCases
from tests.integration.pipelex.cogt.test_data import Employee
@@ -61,7 +61,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_llm_text_only(self, request: FixtureRequest):
- llm_setting_main = get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_text")
+ llm_setting_main = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_text")
text: str = await get_content_generator().make_llm_text(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
@@ -75,7 +75,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_object_direct(self, request: FixtureRequest):
- llm_setting_for_object = get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
+ llm_setting_for_object = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
person_direct: Employee = await get_content_generator().make_object_direct(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
@@ -90,7 +90,7 @@ class TestContentGenerator:
@pytest.mark.llm
@pytest.mark.inference
async def test_make_object_list_direct(self, request: FixtureRequest):
- llm_setting_for_object = get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
+ llm_setting_for_object = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_object")
person_list_direct: List[Employee] = await get_content_generator().make_object_list_direct(
job_metadata=JobMetadata(job_name=request.node.originalname), # type: ignore
diff --git a/tests/integration/pipelex/cogt/test_image_gen.py b/tests/integration/pipelex/cogt/test_image_gen.py
index 21c4fdb8d..c7f408cf1 100644
--- a/tests/integration/pipelex/cogt/test_image_gen.py
+++ b/tests/integration/pipelex/cogt/test_image_gen.py
@@ -19,4 +19,4 @@ class TestImageGeneration:
generated_image = await imgg_worker_async.gen_image(
imgg_job=imgg_job,
)
- pretty_print(generated_image, title=f"Generated Image, topic={topic}")
+ pretty_print(generated_image, title=f"Generated Image, topic={topic}, imgg_handle={imgg_handle}")
diff --git a/tests/integration/pipelex/cogt/test_llm_engines.py b/tests/integration/pipelex/cogt/test_llm_engines.py
deleted file mode 100644
index 73e70f708..000000000
--- a/tests/integration/pipelex/cogt/test_llm_engines.py
+++ /dev/null
@@ -1,147 +0,0 @@
-import pytest
-
-from pipelex import log, pretty_print
-from pipelex.cogt.llm.llm_job import LLMJob
-from pipelex.cogt.llm.llm_job_components import LLMJobParams
-from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_engine_factory import LLMEngineFactory
-from pipelex.cogt.llm.llm_models.llm_family import LLMCreator, LLMFamily
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
-from pipelex.cogt.llm.llm_worker_factory import LLMWorkerFactory
-from pipelex.hub import get_inference_manager, get_llm_deck, get_llm_models_provider, get_report_delegate
-from tests.integration.pipelex.cogt.test_data import LLMTestConstants, Person
-
-
[email protected](loop_scope="class")
-class TestLLMEngines:
- async def run_inference(self, llm_worker: LLMWorkerAbstract, llm_job: LLMJob):
- generated_text = await llm_worker.gen_text(llm_job=llm_job)
- assert generated_text
- pretty_print(generated_text)
- if llm_worker.is_gen_object_supported:
- generated_object = await llm_worker.gen_object(llm_job=llm_job, schema=Person)
- assert generated_object
- pretty_print(generated_object)
- else:
- log.info(f"No object generation supported for this worker: '{llm_worker.desc}'")
-
- async def test_one_llm_engine_by_llm_handle(self, llm_job_params: LLMJobParams, llm_handle: str):
- log.info(f"Testing {llm_handle}")
- inference_manager = get_inference_manager()
- llm_worker = inference_manager.get_llm_worker(llm_handle=llm_handle)
- log.info(f"LLM Worker: {llm_worker.desc}")
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
-
- async def test_one_llm_engine_by_llm_id_and_platform(self, llm_job_params: LLMJobParams, llm_id: str, llm_platform: LLMPlatform):
- log.info(f"Testing {llm_id} on {llm_platform}")
- llm_models_provider = get_llm_models_provider()
- llm_models = llm_models_provider.get_all_llm_models()
- count = 0
- for llm_model in llm_models:
- platform_llm_id = llm_model.platform_llm_id.get(llm_platform)
- if llm_id != platform_llm_id:
- continue
-
- llm_engine = LLMEngine(
- llm_platform=llm_platform,
- llm_model=llm_model,
- )
- llm_worker = LLMWorkerFactory.make_llm_worker(
- llm_engine=llm_engine,
- reporting_delegate=get_report_delegate(),
- )
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- log.info(f"Running inference for {llm_model.name_and_version_and_platform}")
- await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
- count += 1
- if count > 0:
- log.info(f"Tested {count} llm_engines for llm_id '{llm_id}' on platform '{llm_platform}'")
- else:
- pytest.fail(f"No llm_engines found for llm_id '{llm_id}' on platform '{llm_platform}'")
-
- async def test_llm_engines_from_one_family(self, llm_job_params: LLMJobParams, llm_family: LLMFamily):
- llm_handle_to_llm_engine_blueprint = get_llm_deck().llm_handles
- count = 0
- for _, llm_engine_blueprint in llm_handle_to_llm_engine_blueprint.items():
- llm_engine = LLMEngineFactory.make_llm_engine(llm_engine_blueprint=llm_engine_blueprint)
- if llm_engine.llm_model.llm_family != llm_family:
- continue
- llm_worker = LLMWorkerFactory.make_llm_worker(
- llm_engine=llm_engine,
- reporting_delegate=get_report_delegate(),
- )
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
- count += 1
-
- log.info(f"Tested {count} llm_engines for family {llm_family}")
-
- async def test_llm_engines_from_one_creator(self, llm_job_params: LLMJobParams, llm_creator: LLMCreator):
- llm_handle_to_llm_engine_blueprint = get_llm_deck().llm_handles
- for _, llm_engine_blueprint in llm_handle_to_llm_engine_blueprint.items():
- llm_engine = LLMEngineFactory.make_llm_engine(llm_engine_blueprint=llm_engine_blueprint)
- if llm_engine.llm_model.llm_family.creator != llm_creator:
- continue
- llm_worker = LLMWorkerFactory.make_llm_worker(
- llm_engine=llm_engine,
- reporting_delegate=get_report_delegate(),
- )
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
-
- async def test_llm_engines_from_one_platform(self, llm_job_params: LLMJobParams, llm_platform: LLMPlatform):
- llm_models_provider = get_llm_models_provider()
- llm_models = llm_models_provider.get_all_llm_models()
- for llm_model in llm_models:
- if llm_platform not in llm_model.enabled_platforms:
- continue
- llm_engine = LLMEngine(
- llm_platform=llm_platform,
- llm_model=llm_model,
- )
- llm_worker = LLMWorkerFactory.make_llm_worker(
- llm_engine=llm_engine,
- reporting_delegate=get_report_delegate(),
- )
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
-
- async def test_llm_handle_to_llm_engine_default(self, llm_job_params: LLMJobParams):
- inference_manager = get_inference_manager()
- llm_handle_to_llm_engine_blueprint = get_llm_deck().llm_handles
- for llm_handle in llm_handle_to_llm_engine_blueprint.keys():
- llm_worker = inference_manager.get_llm_worker(llm_handle=llm_handle)
- assert llm_worker
- llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
- system_text=None,
- user_text=LLMTestConstants.USER_TEXT_SHORT,
- llm_job_params=llm_job_params,
- )
- assert llm_job
- # ENABLE THIS TO TEST THE INFERENCE with all base models
- # await self.run_inference(llm_worker=llm_worker, llm_job=llm_job)
diff --git a/tests/integration/pipelex/cogt/test_llm_gen_object.py b/tests/integration/pipelex/cogt/test_llm_gen_object.py
index 05005e319..ce6f29c8e 100644
--- a/tests/integration/pipelex/cogt/test_llm_gen_object.py
+++ b/tests/integration/pipelex/cogt/test_llm_gen_object.py
@@ -7,12 +7,12 @@ from pydantic import BaseModel
from pipelex import pretty_print
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.hub import get_llm_deck, get_llm_worker
+from pipelex.hub import get_llm_worker, get_models_manager
from tests.integration.pipelex.cogt.test_data import LLMTestCases
def get_async_worker_and_job(llm_preset_id: str, user_text: str):
- llm_setting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(user_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
diff --git a/tests/integration/pipelex/cogt/test_llm_gen_text.py b/tests/integration/pipelex/cogt/test_llm_gen_text.py
index fc42fd90f..2dfa3fa1a 100644
--- a/tests/integration/pipelex/cogt/test_llm_gen_text.py
+++ b/tests/integration/pipelex/cogt/test_llm_gen_text.py
@@ -7,15 +7,15 @@ from pipelex import log, pretty_print
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.cogt.llm.llm_models.llm_family import LLMFamily
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
-from pipelex.hub import get_llm_deck, get_llm_worker, get_report_delegate
+from pipelex.cogt.model_backends.model_constraints import ModelConstraints
+from pipelex.hub import get_llm_worker, get_models_manager, get_report_delegate
from tests.integration.pipelex.cogt.test_data import LLMTestCases
def get_worker_and_job(llm_preset_id: str, user_text: str) -> Tuple[LLMWorkerAbstract, LLMJob]:
- llm_setting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(user_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
@@ -32,7 +32,7 @@ def get_worker_and_job(llm_preset_id: str, user_text: str) -> Tuple[LLMWorkerAbs
@pytest.mark.asyncio(loop_scope="class")
class TestLLMGenText:
@pytest.mark.parametrize("topic, prompt_text", LLMTestCases.SINGLE_TEXT)
- async def test_gen_text_async_using_handle(self, llm_job_params: LLMJobParams, llm_handle: str, topic: str, prompt_text: str):
+ async def test_gen_text_using_handle(self, llm_job_params: LLMJobParams, llm_handle: str, topic: str, prompt_text: str):
pretty_print(prompt_text, title=topic)
llm_worker = get_llm_worker(llm_handle=llm_handle)
llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
@@ -45,14 +45,14 @@ class TestLLMGenText:
get_report_delegate().generate_report()
@pytest.mark.parametrize("topic, prompt_text", LLMTestCases.SINGLE_TEXT)
- async def test_gen_text_async_using_llm_preset(self, llm_preset_id: str, topic: str, prompt_text: str):
+ async def test_gen_text_using_llm_preset(self, llm_preset_id: str, topic: str, prompt_text: str):
llm_worker, llm_job = get_worker_and_job(llm_preset_id=llm_preset_id, user_text=prompt_text)
generated_text = await llm_worker.gen_text(llm_job=llm_job)
assert generated_text
pretty_print(generated_text)
@pytest.mark.parametrize("topic, prompt_text", LLMTestCases.SINGLE_TEXT)
- async def test_gen_text_async_multiple_using_llm_preset(self, llm_preset_id: str, topic: str, prompt_text: str):
+ async def test_gen_text_multiple_using_llm_preset(self, llm_preset_id: str, topic: str, prompt_text: str):
llm_worker, llm_job = get_worker_and_job(llm_preset_id=llm_preset_id, user_text=prompt_text)
job_params_base = llm_job.job_params
max_tokens = 30
@@ -63,12 +63,15 @@ class TestLLMGenText:
temperature += 0.2
if temperature > 1:
break
- if isinstance(llm_worker, LLMWorkerInternalAbstract) and llm_worker.llm_engine.llm_model.llm_family == LLMFamily.O_SERIES:
- log.warning("LLMFamily O_series, forcing temprature to 1, setting minimum tokens to avoid empty output")
- completion_max_tokens = max(max_tokens, 2000)
- llm_job.job_params = job_params_base.model_copy(update={"max_tokens": completion_max_tokens, "temperature": 1})
- else:
- llm_job.job_params = job_params_base.model_copy(update={"max_tokens": max_tokens, "temperature": temperature})
+ llm_job.job_params = job_params_base.model_copy(update={"max_tokens": max_tokens, "temperature": temperature})
+ if isinstance(llm_worker, LLMWorkerInternalAbstract):
+ if ModelConstraints.TEMPERATURE_MUST_BE_1 in llm_worker.inference_model.constraints:
+ log.warning("ModelConstraints TEMPERATURE_MUST_BE_1, forcing temprature to 1, setting minimum tokens to avoid empty output")
+ llm_job.job_params = job_params_base.model_copy(update={"temperature": 1})
+ if ModelConstraints.MAX_TOKENS_MUST_BE_HIGH_ENOUGH in llm_worker.inference_model.constraints:
+ log.warning("ModelConstraints MAX_TOKENS_MUST_BE_HIGH_ENOUGH, forcing max tokens to at least2000")
+ completion_max_tokens = max(max_tokens, 2000)
+ llm_job.job_params = job_params_base.model_copy(update={"max_tokens": completion_max_tokens})
task: asyncio.Task[str] = asyncio.create_task(llm_worker.gen_text(llm_job=llm_job))
tasks.append(task)
diff --git a/tests/integration/pipelex/cogt/test_llm_inference.py b/tests/integration/pipelex/cogt/test_llm_inference.py
new file mode 100644
index 000000000..77cf58ff3
--- /dev/null
+++ b/tests/integration/pipelex/cogt/test_llm_inference.py
@@ -0,0 +1,60 @@
+import pytest
+
+from pipelex import log, pretty_print
+from pipelex.cogt.exceptions import LLMCapabilityError, PromptImageFormatError
+from pipelex.cogt.image.prompt_image import PromptImagePath
+from pipelex.cogt.llm.llm_job_components import LLMJobParams
+from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
+from pipelex.hub import get_inference_manager
+from tests.integration.pipelex.cogt.test_data import LLMTestConstants, LLMVisionTestCases, Person
+
+
[email protected](loop_scope="class")
+class TestLLMInference:
+ async def test_simple_gen_text_from_text(self, llm_job_params: LLMJobParams, llm_handle: str):
+ log.info(f"Testing llm_handle '{llm_handle}'")
+ llm_worker = get_inference_manager().get_llm_worker(llm_handle=llm_handle)
+ log.info(f"Using llm_worker: {llm_worker.desc}")
+ llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
+ system_text=None,
+ user_text=LLMTestConstants.USER_TEXT_SHORT,
+ llm_job_params=llm_job_params,
+ )
+ generated_text = await llm_worker.gen_text(llm_job=llm_job)
+ assert generated_text
+ pretty_print(generated_text)
+
+ async def test_simple_gen_object_from_text(self, llm_job_params: LLMJobParams, llm_handle: str):
+ log.info(f"Testing llm_handle '{llm_handle}'")
+ llm_worker = get_inference_manager().get_llm_worker(llm_handle=llm_handle)
+ log.info(f"Using llm_worker: {llm_worker.desc}")
+ llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
+ system_text=None,
+ user_text=LLMTestConstants.USER_TEXT_SHORT,
+ llm_job_params=llm_job_params,
+ )
+ if not llm_worker.is_gen_object_supported:
+ pytest.skip(f"No object generation supported for this worker: '{llm_worker.desc}'")
+ generated_object = await llm_worker.gen_object(llm_job=llm_job, schema=Person)
+ assert generated_object
+ pretty_print(generated_object)
+
+ @pytest.mark.parametrize("image_path", [LLMVisionTestCases.PATH_IMG_PNG_1])
+ async def test_gen_text_from_image(self, llm_job_params: LLMJobParams, llm_handle: str, image_path: str):
+ prompt_image = PromptImagePath(file_path=image_path)
+ llm_worker = get_inference_manager().get_llm_worker(llm_handle=llm_handle)
+ llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
+ user_text=LLMVisionTestCases.VISION_USER_TEXT_2,
+ user_images=[prompt_image],
+ llm_job_params=llm_job_params,
+ )
+ try:
+ generated_text = await llm_worker.gen_text(llm_job=llm_job)
+ assert generated_text
+ pretty_print(generated_text, title=f"Vision of {image_path}")
+ except LLMCapabilityError as exc:
+ pytest.skip(f"Vision not supported for this LLM: {llm_handle} because {exc}")
+ except PromptImageFormatError as exc:
+ pytest.skip(f"Prompt Image format not supported for this LLM: {llm_handle} because {exc}")
diff --git a/tests/integration/pipelex/cogt/test_llm_models_library.py b/tests/integration/pipelex/cogt/test_llm_models_library.py
deleted file mode 100644
index d1d6e7b46..000000000
--- a/tests/integration/pipelex/cogt/test_llm_models_library.py
+++ /dev/null
@@ -1,138 +0,0 @@
-from typing import List, Optional
-
-import pytest
-from rich import box
-from rich.console import Console, Group, RenderableType
-from rich.padding import Padding
-from rich.table import Table
-from rich.text import Text
-
-from pipelex import log, pretty_print
-from pipelex.cogt.llm.llm_models.llm_engine import LLMEngine
-from pipelex.cogt.llm.llm_models.llm_engine_blueprint import LLMEngineBlueprint
-from pipelex.cogt.llm.llm_models.llm_engine_factory import LLMEngineFactory
-from pipelex.cogt.llm.llm_models.llm_model import LLMModel
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform, LLMPlatformChoice
-from pipelex.hub import get_llm_models_provider
-
-
-def print_models(models: List[LLMModel], title: Optional[str] = None):
- console = Console()
- table = Table(
- title=title,
- show_header=True,
- show_lines=True,
- header_style="bold cyan",
- box=box.SQUARE_DOUBLE_HEAD,
- )
- table.add_column("Creator")
- table.add_column("Family")
- table.add_column("Name")
- table.add_column("Version")
- table.add_column("Default platform")
- table.add_column("Platform ids")
-
- sorted_models = sorted(models, key=lambda m: (m.llm_family, m.llm_name, m.version))
- for model in sorted_models:
- platform_elemnts: List[RenderableType] = []
- for platform, llm_id in model.platform_llm_id.items():
- text = Text.assemble((platform, platform.console_color), f" - {llm_id}")
- if platform != list(model.platform_llm_id.keys())[-1]:
- ready = Padding(text, (0, 0, 1, 0))
- else:
- ready = Padding(text, (0, 0, 0, 0))
- platform_elemnts.append(ready)
- platforms = Group(*platform_elemnts)
-
- table.add_row(
- Text(model.llm_family.creator, model.llm_family.creator.console_color),
- model.llm_family,
- model.llm_name,
- model.version,
- model.default_platform,
- platforms,
- )
- console.print("\n")
- console.print(table)
-
-
-class TestLLMModelsDB:
- def test_llm_models_loading(self, pytestconfig: pytest.Config):
- all_llm_models = get_llm_models_provider().get_all_llm_models()
- if pytestconfig.get_verbosity() >= 2:
- print_models(models=all_llm_models, title="All available LLM Models (unit testing)")
- assert len(all_llm_models) > 0, "No LLM models found"
- assert all(isinstance(model, LLMModel) for model in all_llm_models), "Invalid model type found"
- for llm_model in all_llm_models:
- log.debug(f"Checking model: {llm_model.name_and_version}")
- assert llm_model.llm_name, "Missing llm_name"
- assert llm_model.version, "Missing version"
- assert llm_model.llm_family, "Missing llm_family"
- assert llm_model.default_platform, "Missing default_platform"
- assert llm_model.platform_llm_id, "Missing platform_llm_id"
-
- @pytest.mark.parametrize(
- "llm_name, llm_version, llm_platform_choice",
- [
- # ("mistral-large", "latest", "default"),
- # ("mistral-large", "latest", LLMPlatform.BEDROCK_MISTRAL),
- # ("mistral-large", "latest", LLMPlatform.MISTRAL),
- # ("gpt-4o", "2024-08-06", "default"),
- ("gpt-4o", "2024-08-06", LLMPlatform.OPENAI),
- # ("gpt-4.1-mini", "latest", LLMPlatform.OPENAI),
- # ("gpt-4o-mini", "latest", LLMPlatform.ANTHROPIC),
- # ("gpt-4o-mini", "latest", LLMPlatform.BEDROCK_ANTHROPIC),
- # ("gpt-4o-mini", "latest", LLMPlatform.AZURE_OPENAI),
- # ("gpt-4o-mini", "latest", LLMPlatform.MISTRAL),
- ],
- )
- def test_get_llm_model(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ):
- model = get_llm_models_provider().get_llm_model(llm_name=llm_name, llm_version=llm_version, llm_platform_choice=llm_platform_choice)
- pretty_print(model)
- assert isinstance(model, LLMModel)
- assert model.llm_name == llm_name
- if llm_version != "latest":
- assert model.version == llm_version
- if llm_platform_choice == "default":
- assert model.default_platform in model.platform_llm_id
- else:
- llm_platform = llm_platform_choice
- assert llm_platform in model.platform_llm_id
-
- @pytest.mark.parametrize(
- "llm_name, llm_version, llm_platform_choice",
- [
- # ("mistral-large", "latest", "default"),
- # ("mistral-large", "latest", LLMPlatform.BEDROCK_MISTRAL),
- # ("mistral-large", "latest", LLMPlatform.MISTRAL),
- # ("gpt-4o", "2024-08-06", "default"),
- ("gpt-4o", "2024-08-06", LLMPlatform.OPENAI),
- # ("gpt-4o-mini", "latest", LLMPlatform.ANTHROPIC),
- # ("gpt-4o-mini", "latest", LLMPlatform.BEDROCK_ANTHROPIC),
- # ("gpt-4o-mini", "latest", LLMPlatform.AZURE_OPENAI),
- # ("gpt-4o-mini", "latest", LLMPlatform.MISTRAL),
- ],
- )
- def test_make_llm_engine(
- self,
- llm_name: str,
- llm_version: str,
- llm_platform_choice: LLMPlatformChoice,
- ):
- engine_card: LLMEngineBlueprint = LLMEngineBlueprint(
- llm_name=llm_name,
- llm_version=llm_version,
- llm_platform_choice=llm_platform_choice,
- )
- engine = LLMEngineFactory.make_llm_engine(engine_card)
- pretty_print(engine)
-
- assert engine is not None
- assert isinstance(engine, LLMEngine)
- assert engine.llm_model.llm_name == llm_name
- assert engine.llm_model.version == llm_version
diff --git a/tests/integration/pipelex/cogt/test_llm_report.py b/tests/integration/pipelex/cogt/test_llm_report.py
index d3a9af4ca..04d2d62ae 100644
--- a/tests/integration/pipelex/cogt/test_llm_report.py
+++ b/tests/integration/pipelex/cogt/test_llm_report.py
@@ -5,7 +5,7 @@ import pytest
from pipelex import pretty_print
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.hub import get_llm_deck, get_llm_worker, get_report_delegate
+from pipelex.hub import get_llm_worker, get_models_manager, get_report_delegate
from tests.integration.pipelex.cogt.test_data import LLMTestCases
@@ -27,7 +27,7 @@ class TestLLMReport:
get_report_delegate().generate_report()
def _get_async_worker_and_job(self, llm_preset_id: str, prompt_text: str):
- llm_setting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
+ llm_setting = get_models_manager().get_llm_deck().get_llm_setting(llm_setting_or_preset_id=llm_preset_id)
pretty_print(llm_setting, title=llm_preset_id)
pretty_print(prompt_text)
llm_worker = get_llm_worker(llm_handle=llm_setting.llm_handle)
diff --git a/tests/integration/pipelex/cogt/test_llm_vision.py b/tests/integration/pipelex/cogt/test_llm_vision.py
index 8b4a43c27..145adcd6b 100644
--- a/tests/integration/pipelex/cogt/test_llm_vision.py
+++ b/tests/integration/pipelex/cogt/test_llm_vision.py
@@ -21,7 +21,7 @@ class TestLLMVision:
async def test_gen_text_from_vision_by_url(self, llm_handle_for_vision: str, topic: str, image_uri: str):
prompt_image = PromptImageFactory.make_prompt_image_from_uri(uri=image_uri)
llm_worker = get_llm_worker(llm_handle=llm_handle_for_vision)
- log.info(f"LLM Worker: {llm_worker.desc}")
+ log.info(f"Using llm_worker: {llm_worker.desc}")
llm_job = LLMJobFactory.make_llm_job_from_prompt_contents(
user_text=LLMVisionTestCases.VISION_USER_TEXT_2,
user_images=[prompt_image],
diff --git a/tests/integration/pipelex/cogt/test_setup_inference_workers.py b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
index 27321c84f..92fa56e71 100644
--- a/tests/integration/pipelex/cogt/test_setup_inference_workers.py
+++ b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
@@ -1,11 +1,21 @@
import pytest
-from pipelex.hub import get_inference_manager
+from pipelex import log
+from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
+from pipelex.hub import get_inference_manager, get_models_manager
@pytest.mark.gha_disabled
@pytest.mark.codex_disabled
class TestSetupInferenceWorkers:
def test_setup_inference_manager(self):
- get_inference_manager().setup_llm_workers()
+ inference_manager = get_inference_manager()
+ llm_handle_to_inference_model = get_models_manager().get_llm_deck().inference_models
+ log.verbose(f"{len(llm_handle_to_inference_model)} LLM engine_cards found")
+ for llm_handle, inference_model in llm_handle_to_inference_model.items():
+ llm_worker = inference_manager.get_llm_worker(llm_handle=llm_handle)
+ if isinstance(llm_worker, LLMWorkerInternalAbstract):
+ assert inference_model == llm_worker.inference_model
+ log.debug("Done setting up LLM Workers (async)")
+
get_inference_manager().setup_imgg_workers()
diff --git a/tests/integration/pipelex/conftest.py b/tests/integration/pipelex/conftest.py
index 8f93653e2..62c76226e 100644
--- a/tests/integration/pipelex/conftest.py
+++ b/tests/integration/pipelex/conftest.py
@@ -1,16 +1,17 @@
+from typing import Optional
+
import pytest
from pipelex.cogt.imgg.imgg_handle import ImggHandle
from pipelex.cogt.llm.llm_job_components import LLMJobParams
-from pipelex.cogt.llm.llm_models.llm_family import LLMCreator, LLMFamily
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
from pipelex.cogt.ocr.ocr_handle import OcrHandle
+from pipelex.plugins.plugin_sdk_registry import Plugin
@pytest.fixture(
params=[
- "llm_for_testing_gen_text",
- "llm_for_testing_gen_object",
+ # "llm_for_testing_gen_text",
+ # "llm_for_testing_gen_object",
"llm_for_creative_writing",
]
)
@@ -38,6 +39,8 @@ def llm_preset_id(request: pytest.FixtureRequest) -> str:
# "mistral-large",
# "ministral-3b",
# "ministral-8b",
+ "mistral-medium",
+ "mistral-medium-2508",
# "pixtral-12b",
# "pixtral-large",
# "gemini-1-5-pro",
@@ -54,7 +57,23 @@ def llm_preset_id(request: pytest.FixtureRequest) -> str:
# "pipelex/gpt-4o-mini",
# "pipelex/claude-3.7-sonnet",
# "pipelex/gemini-2.0-flash-vertex",
- "pipelex/gemini-2.0-flash",
+ # "pipelex/gemini-2.0-flash",
+ # "best-claude",
+ # "llm_to_engineer",
+ # "gpt-5-nano",
+ # "gpt-4o-mini",
+ # "gpt-5-mini",
+ # "gpt-5-chat",
+ # "claude-4-sonnet",
+ # "claude-4.1-opus",
+ # "claude-3.5-sonnet",
+ # "claude-3.5-sonnet-v2"
+ # "claude-3.7-sonnet",
+ # "gemini-2.5-flash-lite",
+ # "gemini-2.5-flash",
+ # "gemini-2.5-pro",
+ # "grok-3",
+ # "grok-3-mini",
]
)
def llm_handle(request: pytest.FixtureRequest) -> str:
@@ -67,24 +86,22 @@ def llm_handle(request: pytest.FixtureRequest) -> str:
# "o1",
# "o3-mini",
# "gpt-4o",
- "gpt-4o-mini",
+ # "gpt-4o-mini",
# "gpt-5-mini",
# "gpt-5-nano",
# "gpt-5-chat",
# "gpt-4-5-preview",
# "claude-3-haiku",
- # "claude-3-5-sonnet",
- # "claude-3-7-sonnet",
- # "claude-4-1-opus",
+ # "claude-3.5-sonnet",
+ # "claude-3.7-sonnet",
+ # "claude-4.1-opus",
# "pixtral-12b",
# "pixtral-large",
- # "gemini-1-5-pro",
- # "gemini-1-5-flash",
- # "gemini-2-flash",
- # "gemini-2-pro",
- # "gemini-2-5-pro",
- # "gemini-2-5-flash",
+ # "gemini-2.5-pro",
+ # "gemini-2.5-flash",
# "mistral-small3.1",
+ "mistral-medium",
+ "mistral-medium-2508",
# "qwen3:8b",
]
)
@@ -95,70 +112,40 @@ def llm_handle_for_vision(request: pytest.FixtureRequest) -> str:
@pytest.fixture(
params=[
- # LLMFamily.GPT_4,
- # LLMFamily.GPT_4O,
- LLMFamily.GPT_5,
- # LLMFamily.GPT_4_5,
- # LLMFamily.GPT_4_1,
- # LLMFamily.O_SERIES,
- # LLMFamily.CLAUDE_3_7,
- # LLMFamily.CLAUDE_4,
- # LLMFamily.CLAUDE_4_1,
- # LLMFamily.PERPLEXITY_SEARCH,
- # LLMFamily.PERPLEXITY_REASONING,
- # LLMFamily.PERPLEXITY_RESEARCH,
- # LLMFamily.PERPLEXITY_DEEPSEEK,
- # LLMFamily.GEMINI,
- # LLMFamily.GEMMA,
- # LLMFamily.CUSTOM_BLACKBOXAI,
+ Plugin(sdk="openai", backend="openai"),
+ Plugin(sdk="azure_openai", backend="azure_openai"),
]
)
-def llm_family(request: pytest.FixtureRequest) -> LLMFamily:
- assert isinstance(request.param, LLMFamily)
+def plugin_for_openai(request: pytest.FixtureRequest) -> Plugin:
+ assert isinstance(request.param, Plugin)
return request.param
@pytest.fixture(
params=[
- # LLMCreator.ALIBABA,
- # LLMCreator.AMAZON,
- # LLMCreator.ANTHROPIC,
- # LLMCreator.DEEPSEEK,
- # LLMCreator.GOOGLE,
- LLMCreator.OPENAI,
- # LLMCreator.META,
- # LLMCreator.MISTRAL,
- # LLMCreator.PERPLEXITY,
+ Plugin(sdk="anthropic", backend="anthropic"),
+ Plugin(sdk="bedrock_anthropic", backend="bedrock_anthropic"),
]
)
-def llm_creator(request: pytest.FixtureRequest) -> LLMCreator:
- assert isinstance(request.param, LLMCreator)
+def plugin_for_anthropic(request: pytest.FixtureRequest) -> Plugin:
+ assert isinstance(request.param, Plugin)
return request.param
-# TODO: build llm_id/platform combos dynalically from config data
@pytest.fixture(
params=[
- # LLMPlatform.ANTHROPIC,
- # LLMPlatform.AZURE_OPENAI,
- # LLMPlatform.BEDROCK,
- # LLMPlatform.BEDROCK_ANTHROPIC,
- # LLMPlatform.MISTRAL,
- LLMPlatform.OPENAI,
- # LLMPlatform.PERPLEXITY,
- # LLMPlatform.VERTEXAI,
- # LLMPlatform.XAI,
- # LLMPlatform.CUSTOM_LLM,
+ # None,
+ "https://inference.pipelex.com/v1",
]
)
-def llm_platform(request: pytest.FixtureRequest) -> LLMPlatform:
- assert isinstance(request.param, LLMPlatform)
+def openai_endpoint(request: pytest.FixtureRequest) -> Optional[str]:
+ assert isinstance(request.param, str) or request.param is None
return request.param
@pytest.fixture(
params=[
- # "gpt-5-mini-2025-08-07",
+ "gpt-5-mini-2025-08-07",
# "gpt-5-nano-2025-08-07",
# "gpt-5-chat-2025-08-07",
# "gpt-5-mini",
@@ -235,7 +222,7 @@ def imgg_handle(request: pytest.FixtureRequest) -> ImggHandle:
@pytest.fixture(
params=[
OcrHandle.BASIC_OCR,
- # OcrHandle.MISTRAL_OCR,
+ OcrHandle.MISTRAL_OCR,
]
)
def ocr_handle(request: pytest.FixtureRequest) -> OcrHandle:
diff --git a/tests/integration/pipelex/conftest.py.orig b/tests/integration/pipelex/conftest.py.orig
new file mode 100644
index 000000000..c0a4a4a55
--- /dev/null
+++ b/tests/integration/pipelex/conftest.py.orig
@@ -0,0 +1,235 @@
+from typing import Optional
+
+import pytest
+
+from pipelex.cogt.imgg.imgg_handle import ImggHandle
+from pipelex.cogt.llm.llm_job_components import LLMJobParams
+<<<<<<< HEAD
+from pipelex.plugins.plugin_sdk_registry import Plugin
+=======
+from pipelex.cogt.llm.llm_models.llm_family import LLMCreator, LLMFamily
+from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.cogt.ocr.ocr_handle import OcrHandle
+>>>>>>> dev
+
+
+ params=[
+ # "llm_for_testing_gen_text",
+ # "llm_for_testing_gen_object",
+ "llm_for_creative_writing",
+ ]
+)
+def llm_preset_id(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
+
+
+# TODO: make it efficient to also test multiple platforms like openai/azure and mistral/anthropic/bedrock
+ params=[
+ # "o1",
+ # "gpt-4o",
+ # "gpt-4o-mini",
+ # "gpt-4-5-preview",
+ # "o1-mini",
+ # "o3-mini",
+ # "gpt-5-mini", # TODO: fix this
+ # "gpt-5-nano", # TODO: fix this
+ # "best-gpt",
+ # "claude-3-haiku",
+ # "claude-3-5-sonnet",
+ # "claude-3-7-sonnet",
+ # "best-claude",
+ # "mistral-large",
+ # "ministral-3b",
+ # "ministral-8b",
+ "mistral-medium",
+ "mistral-medium-2508",
+ # "pixtral-12b",
+ # "pixtral-large",
+ # "gemini-1-5-pro",
+ # "gemini-1-5-flash",
+ # "gemini-2-flash",
+ # "gemini-2-pro",
+ # "gemini-2-5-flash",
+ # "gemini-2-5-pro",
+ # "bedrock-mistral-large",
+ # "bedrock-claude-3-7-sonnet",
+ # "bedrock-meta-llama-3-3-70b-instruct",
+ # "bedrock-nova-pro",
+ # "sonar",
+ # "pipelex/gpt-4o-mini",
+ # "pipelex/claude-3.7-sonnet",
+ # "pipelex/gemini-2.0-flash-vertex",
+ # "pipelex/gemini-2.0-flash",
+ # "best-claude",
+ # "llm_to_engineer",
+ # "gpt-5-nano",
+ # "gpt-4o-mini",
+ # "gpt-5-mini",
+ # "gpt-5-chat",
+ # "claude-4-sonnet",
+ # "claude-4.1-opus",
+ # "claude-3.5-sonnet",
+ # "claude-3.5-sonnet-v2"
+ # "claude-3.7-sonnet",
+ # "gemini-2.5-flash-lite",
+ # "gemini-2.5-flash",
+ # "gemini-2.5-pro",
+ # "grok-3",
+ # "grok-3-mini",
+ ]
+)
+def llm_handle(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
+
+
+ params=[
+ # "o1",
+ # "o3-mini",
+ # "gpt-4o",
+ # "gpt-4o-mini",
+ # "gpt-5-mini",
+ # "gpt-5-nano",
+ # "gpt-5-chat",
+ # "gpt-4-5-preview",
+ # "claude-3-haiku",
+ # "claude-3.5-sonnet",
+ # "claude-3.7-sonnet",
+ # "claude-4.1-opus",
+ # "pixtral-12b",
+ # "pixtral-large",
+ # "gemini-2.5-pro",
+ # "gemini-2.5-flash",
+ # "mistral-small3.1",
+ "mistral-medium",
+ "mistral-medium-2508",
+ # "qwen3:8b",
+ ]
+)
+def llm_handle_for_vision(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
+
+
+ params=[
+ Plugin(sdk="openai", backend="openai"),
+ Plugin(sdk="azure_openai", backend="azure_openai"),
+ ]
+)
+def plugin_for_openai(request: pytest.FixtureRequest) -> Plugin:
+ assert isinstance(request.param, Plugin)
+ return request.param
+
+
+ params=[
+ Plugin(sdk="anthropic", backend="anthropic"),
+ Plugin(sdk="bedrock_anthropic", backend="bedrock_anthropic"),
+ ]
+)
+def plugin_for_anthropic(request: pytest.FixtureRequest) -> Plugin:
+ assert isinstance(request.param, Plugin)
+ return request.param
+
+
+ params=[
+ # None,
+ "https://inference.pipelex.com/v1",
+ ]
+)
+def openai_endpoint(request: pytest.FixtureRequest) -> Optional[str]:
+ assert isinstance(request.param, str) or request.param is None
+ return request.param
+
+
+ params=[
+ "gpt-5-mini-2025-08-07",
+ # "gpt-5-nano-2025-08-07",
+ # "gpt-5-chat-2025-08-07",
+ # "gpt-5-mini",
+ # "gpt-5-nano",
+ # "gpt-5-chat-latest",
+ # "gpt-5",
+ "gpt-4o-mini",
+ # "open-mixtral-8x7b",
+ # "google/gemini-2.0-flash",
+ # "google/gemini-2.5-pro-preview-05-06",
+ # "google/gemini-2.5-pro-preview-06-05", # not yet on VertexAI
+ # "google/gemini-2.5-flash-preview-04-17",
+ # "google/gemini-2.5-flash-preview-05-20",
+ # "o1",
+ # "o4-mini",
+ # "bedrock-mistral-large",
+ # "sonar",
+ # "claude-3-7-sonnet-20250219",
+ # "claude-sonnet-4-20250514",
+ # "claude-opus-4-20250514",
+ # "claude-opus-4-1-20250805",
+ # "us.anthropic.claude-sonnet-4-20250514-v1:0",
+ # "us.anthropic.claude-opus-4-20250514-v1:0",
+ # "us.anthropic.claude-opus-4-1-20250805-v1:0",
+ # "sonar",
+ # "sonar-pro",
+ # "gemma3:4b",
+ # "llama4:scout",
+ # "mistral-small3.1:24b",
+ # "qwen3:8b",
+ # "blackboxai/openai/gpt-4o-mini",
+ # "pipelex/openai/gpt-4o-mini",
+ # "openai/gpt-4o-mini",
+ # "grok-3",
+ # "grok-3-mini",
+ # "pipelex/gpt-4o-mini",
+ # "pipelex/claude-3.7-sonnet",
+ # "vertex_ai/gemini-2.0-flash",
+ ]
+)
+def llm_id(request: pytest.FixtureRequest) -> str:
+ assert isinstance(request.param, str)
+ return request.param
+
+
+ params=[
+ LLMJobParams(
+ temperature=0.5,
+ max_tokens=None,
+ seed=None,
+ ),
+ ]
+)
+def llm_job_params(request: pytest.FixtureRequest) -> LLMJobParams:
+ assert isinstance(request.param, LLMJobParams)
+ return request.param
+
+
+ params=[
+ # ImggHandle.FLUX_1_PRO_LEGACY,
+ # ImggHandle.FLUX_1_1_PRO,
+ # ImggHandle.FLUX_1_1_ULTRA,
+ ImggHandle.SDXL_LIGHTNING,
+ # ImggHandle.OPENAI_GPT_IMAGE_1,
+ ]
+)
+def imgg_handle(request: pytest.FixtureRequest) -> ImggHandle:
+ assert isinstance(request.param, ImggHandle)
+ return request.param
+
+
+ params=[
+ OcrHandle.BASIC_OCR,
+ # OcrHandle.MISTRAL_OCR,
+ ]
+)
+def ocr_handle(request: pytest.FixtureRequest) -> OcrHandle:
+ assert isinstance(request.param, OcrHandle)
+ return request.param
diff --git a/tests/integration/pipelex/plugins/conftest.py b/tests/integration/pipelex/plugins/conftest.py
index b109439bf..19a5312f2 100644
--- a/tests/integration/pipelex/plugins/conftest.py
+++ b/tests/integration/pipelex/plugins/conftest.py
@@ -1,19 +1,5 @@
import pytest
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-
-
[email protected](params=LLMPlatform.list_openai_related())
-def llm_platform_for_openai_sdk(request: pytest.FixtureRequest) -> LLMPlatform:
- assert isinstance(request.param, LLMPlatform)
- return request.param
-
-
[email protected](params=LLMPlatform.list_anthropic_related())
-def llm_platform_for_anthropic_sdk(request: pytest.FixtureRequest) -> LLMPlatform:
- assert isinstance(request.param, LLMPlatform)
- return request.param
-
@pytest.fixture(params=["anthropic", "mistral", "meta", "amazon"])
def bedrock_provider(request: pytest.FixtureRequest) -> str:
diff --git a/tests/integration/pipelex/plugins/test_anthropic.py b/tests/integration/pipelex/plugins/test_anthropic.py
index afe087c69..fa1c4d979 100644
--- a/tests/integration/pipelex/plugins/test_anthropic.py
+++ b/tests/integration/pipelex/plugins/test_anthropic.py
@@ -4,9 +4,10 @@ from rich import box
from rich.console import Console
from rich.table import Table
-from pipelex.cogt.exceptions import LLMSDKError
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.hub import get_models_manager
+from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedError
from pipelex.plugins.anthropic.anthropic_llms import anthropic_list_anthropic_models
+from pipelex.plugins.plugin_sdk_registry import Plugin
# TODO: fix this: test works for Anthropic but not if you set peferred platform for Anthropic is Bedrock
@@ -20,13 +21,17 @@ class TestAnthropic:
async def test_anthropic_list_models(
self,
pytestconfig: pytest.Config,
- llm_platform_for_anthropic_sdk: LLMPlatform,
+ plugin_for_anthropic: Plugin,
):
try:
- anthropic_models_list = await anthropic_list_anthropic_models(llm_platform=llm_platform_for_anthropic_sdk)
+ backend = get_models_manager().get_required_inference_backend("anthropic")
+ anthropic_models_list = await anthropic_list_anthropic_models(
+ plugin=plugin_for_anthropic,
+ backend=backend,
+ )
except AuthenticationError as auth_exc:
pytest.fail(f"Authentication error for Anthropic: {auth_exc}")
- except LLMSDKError as exc:
+ except AnthropicSDKUnsupportedError as exc:
if "does not support listing models" in str(exc):
pytest.skip(f"Skipping: {exc}")
else:
diff --git a/tests/integration/pipelex/plugins/test_external_plugin.py b/tests/integration/pipelex/plugins/test_external_plugin.py
index 70fc1a067..5d4e9befb 100644
--- a/tests/integration/pipelex/plugins/test_external_plugin.py
+++ b/tests/integration/pipelex/plugins/test_external_plugin.py
@@ -8,9 +8,9 @@ from pipelex import log, pretty_print
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.cogt.llm.llm_models.llm_setting import LLMSetting
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
-from pipelex.cogt.llm.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.core.concepts.concept_native import NativeConceptEnum
from pipelex.hub import get_inference_manager, get_pipe_router, get_report_delegate
from pipelex.pipe_operators.llm.pipe_llm import PipeLLMOutput
diff --git a/tests/integration/pipelex/plugins/test_openai.py b/tests/integration/pipelex/plugins/test_openai.py
index 21d04cd9e..8d01566c1 100644
--- a/tests/integration/pipelex/plugins/test_openai.py
+++ b/tests/integration/pipelex/plugins/test_openai.py
@@ -1,10 +1,9 @@
import pytest
from pipelex import pretty_print
-from pipelex.cogt.exceptions import LLMSDKError
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
-from pipelex.hub import get_plugin_manager, get_secrets_provider
+from pipelex.hub import get_models_manager
from pipelex.plugins.openai.openai_llms import openai_list_available_models
+from pipelex.plugins.plugin_sdk_registry import Plugin
# make t VERBOSE=2 TEST=TestOpenAI
@@ -12,23 +11,19 @@ from pipelex.plugins.openai.openai_llms import openai_list_available_models
@pytest.mark.codex_disabled
@pytest.mark.asyncio(loop_scope="class")
class TestOpenAI:
- async def test_openai_api_key(self):
- openai_config = get_plugin_manager().plugin_configs.openai_config
- assert openai_config.get_api_key(secrets_provider=get_secrets_provider())
-
# pytest -k test_openai_list_available_models -s -vv
async def test_openai_list_available_models(
self,
pytestconfig: pytest.Config,
- llm_platform_for_openai_sdk: LLMPlatform,
+ plugin_for_openai: Plugin,
):
- try:
- openai_models_list = await openai_list_available_models(llm_platform=llm_platform_for_openai_sdk)
- except LLMSDKError as exc:
- if "does not support listing models" in str(exc):
- pytest.skip(f"Skipping: {exc}")
- else:
- raise exc
+ backend = get_models_manager().get_required_inference_backend(plugin_for_openai.backend)
+ openai_models_list = await openai_list_available_models(
+ plugin=plugin_for_openai,
+ backend=backend,
+ )
if pytestconfig.get_verbosity() >= 2:
list_of_ids = [model.id for model in openai_models_list]
- pretty_print(list_of_ids, title=f"models available for {llm_platform_for_openai_sdk}")
+ pretty_print(list_of_ids, title=f"models available for {plugin_for_openai}")
+
+ pretty_print(openai_models_list)
diff --git a/tests/integration/pipelex/plugins/test_openai_image_gen.py b/tests/integration/pipelex/plugins/test_openai_image_gen.py
index d539254ca..7fc2af006 100644
--- a/tests/integration/pipelex/plugins/test_openai_image_gen.py
+++ b/tests/integration/pipelex/plugins/test_openai_image_gen.py
@@ -1,8 +1,9 @@
import pytest
from pipelex.cogt.exceptions import ImggGenerationError
-from pipelex.cogt.llm.llm_models.llm_platform import LLMPlatform
+from pipelex.hub import get_models_manager
from pipelex.plugins.openai.openai_factory import OpenAIFactory
+from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.tools.misc.base_64_utils import save_base64_to_binary_file
from pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path
from tests.conftest import TEST_OUTPUTS_DIR
@@ -15,7 +16,11 @@ from tests.integration.pipelex.test_data import IMGGTestCases
class TestImggByOpenAIGpt:
@pytest.mark.parametrize("topic, image_desc", IMGGTestCases.IMAGE_DESC)
async def test_gpt_image_generation(self, topic: str, image_desc: str):
- client = OpenAIFactory.make_openai_client(LLMPlatform.OPENAI)
+ backend = get_models_manager().get_required_inference_backend("openai")
+ client = OpenAIFactory.make_openai_client(
+ Plugin(sdk="openai", backend="openai"),
+ backend=backend,
+ )
result = await client.images.generate(
prompt=image_desc,
model="gpt-image-1",
diff --git a/tests/pipelex_unit_test.toml b/tests/pipelex_unit_test.toml
index 6edfb9564..40ace2e55 100644
--- a/tests/pipelex_unit_test.toml
+++ b/tests/pipelex_unit_test.toml
@@ -1,9 +1,3 @@
-[cogt.llm_config.preferred_platforms]
-# These override the defaults set for any llm integration
-# Use at your convenience:
-# gpt-4o-mini = "azure_openai"
-# gpt-4o = "azure_openai"
-# claude-3-7-sonnet = "bedrock_anthropic"
[cogt.imgg_config]
default_imgg_handle = "fal-ai/fast-lightning-sdxl"
diff --git a/tests/unit/pipelex/cli/commands/test_init_cmd.py b/tests/unit/pipelex/cli/commands/test_init_cmd.py
new file mode 100644
index 000000000..69d4755fe
--- /dev/null
+++ b/tests/unit/pipelex/cli/commands/test_init_cmd.py
@@ -0,0 +1,197 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cli.commands.init_cmd import do_init_config
+from pipelex.exceptions import PipelexCLIError
+
+
+class TestInitCmd:
+ """Test cases for the init command functionality."""
+
+ def test_do_init_config_copies_all_files(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test that all files from config template are copied to target directory."""
+ # Setup directories
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+
+ target_dir = tmp_path / ".pipelex"
+ target_dir.mkdir()
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mock_echo = mocker.patch("typer.echo")
+
+ # Execute
+ do_init_config(reset=False)
+
+ # Verify
+ assert (target_dir / "pipelex.toml").exists()
+ mock_echo.assert_any_call(f"✅ Copied 1 files to {target_dir}:")
+
+ def test_do_init_config_skips_existing_files(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test that existing files are skipped when reset=False."""
+ # Setup directories with existing file
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+ (template_dir / "new_file.toml").write_text("[new]\nconfig = 'value'")
+
+ target_dir = tmp_path / ".pipelex"
+ target_dir.mkdir()
+ (target_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '0.9'")
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mock_echo = mocker.patch("typer.echo")
+
+ # Execute
+ do_init_config(reset=False)
+
+ # Verify existing file was not overwritten
+ content = (target_dir / "pipelex.toml").read_text()
+ assert "version = '0.9'" in content
+
+ # Verify new file was copied
+ assert (target_dir / "new_file.toml").exists()
+
+ # Verify messages
+ calls = [call.args[0] for call in mock_echo.call_args_list]
+ assert any("✅ Copied 1 files to" in call for call in calls)
+ assert any("ℹ️ Skipped 1 existing files" in call for call in calls)
+
+ def test_do_init_config_reset_overwrites_files(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test that reset=True overwrites existing files."""
+ # Setup directories with existing file
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+
+ target_dir = tmp_path / ".pipelex"
+ target_dir.mkdir()
+ (target_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '0.9'")
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mock_echo = mocker.patch("typer.echo")
+
+ # Execute
+ do_init_config(reset=True)
+
+ # Verify file was overwritten
+ content = (target_dir / "pipelex.toml").read_text()
+ assert "version = '1.0'" in content
+
+ # Verify no skipped files message
+ calls = [call.args[0] for call in mock_echo.call_args_list]
+ assert any("✅ Copied 1 files to" in call for call in calls)
+ assert not any("ℹ️ Skipped" in call for call in calls)
+
+ def test_do_init_config_nested_directory_structure(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test copying of nested directory structures."""
+ # Setup complex nested structure
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+
+ inference_dir = template_dir / "inference"
+ inference_dir.mkdir()
+ (inference_dir / "backends.toml").write_text("[backends]\nconfig = 'value'")
+
+ backends_dir = inference_dir / "backends"
+ backends_dir.mkdir()
+ (backends_dir / "openai.toml").write_text("[openai]\napi_key = 'test'")
+
+ target_dir = tmp_path / ".pipelex"
+ target_dir.mkdir()
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mock_echo = mocker.patch("typer.echo")
+
+ # Execute
+ do_init_config(reset=False)
+
+ # Verify all files and directories were created
+ assert (target_dir / "pipelex.toml").exists()
+ assert (target_dir / "inference").is_dir()
+ assert (target_dir / "inference" / "backends.toml").exists()
+ assert (target_dir / "inference" / "backends").is_dir()
+ assert (target_dir / "inference" / "backends" / "openai.toml").exists()
+
+ # Verify correct number of files copied
+ calls = [call.args[0] for call in mock_echo.call_args_list]
+ assert any("✅ Copied 3 files to" in call for call in calls)
+
+ def test_do_init_config_handles_permission_error(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test handling of permission errors during file copying."""
+ # Setup directories
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+
+ target_dir = tmp_path / ".pipelex"
+ target_dir.mkdir()
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mocker.patch("shutil.copy2", side_effect=PermissionError("Permission denied"))
+
+ # Execute and verify exception
+ with pytest.raises(PipelexCLIError) as exc_info:
+ do_init_config(reset=False)
+
+ assert "Failed to initialize configuration" in str(exc_info.value)
+
+ def test_do_init_config_creates_target_directory(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """Test that target directory is created if it doesn't exist."""
+ # Setup template directory only
+ template_dir = tmp_path / "config_template"
+ template_dir.mkdir()
+ (template_dir / "pipelex.toml").write_text("[tool.pipelex]\nversion = '1.0'")
+
+ target_dir = tmp_path / ".pipelex" # Don't create this
+
+ # Mock config manager
+ mock_config_manager = mocker.MagicMock()
+ mock_config_manager.pipelex_root_dir = str(tmp_path)
+ mock_config_manager.pipelex_config_dir = str(target_dir)
+
+ mocker.patch("pipelex.cli.commands.init_cmd.config_manager", mock_config_manager)
+ mock_echo = mocker.patch("typer.echo")
+
+ # Execute
+ do_init_config(reset=False)
+
+ # Verify directory was created and file was copied
+ assert target_dir.exists()
+ assert target_dir.is_dir()
+ assert (target_dir / "pipelex.toml").exists()
+
+ # Verify success message
+ calls = [call.args[0] for call in mock_echo.call_args_list]
+ assert any("✅ Copied 1 files to" in call for call in calls)
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck.py b/tests/unit/pipelex/cogt/models/test_model_deck.py
new file mode 100644
index 000000000..ee199cbef
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_model_deck.py
@@ -0,0 +1,286 @@
+"""Unit tests for ModelDeck class."""
+
+from typing import List, Optional
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaults
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.models.model_deck import ModelDeck
+from pipelex.cogt.usage.cost_category import CostCategory
+
+
+class TestModelDeckGetOptionalInferenceModel:
+ """Unit tests for get_optional_inference_model method."""
+
+ def _create_test_model_spec(self, name: str) -> InferenceModelSpec:
+ """Helper to create a test InferenceModelSpec."""
+ return InferenceModelSpec(
+ backend_name="test_backend",
+ name=name,
+ sdk="test_sdk",
+ model_id=f"test_model_{name}",
+ costs={CostCategory.INPUT: 0.001, CostCategory.OUTPUT: 0.002},
+ max_tokens=1000,
+ max_prompt_images=None,
+ )
+
+ def _create_test_model_deck(
+ self,
+ inference_models: Optional[dict[str, InferenceModelSpec]] = None,
+ aliases: Optional[dict[str, str | List[str]]] = None,
+ ) -> ModelDeck:
+ """Helper to create a test ModelDeck."""
+ return ModelDeck(
+ inference_models=inference_models or {},
+ aliases=aliases or {},
+ llm_presets={},
+ llm_choice_defaults=LLMSettingChoicesDefaults(
+ for_text=LLMSetting(llm_handle="default_text", temperature=0.7, max_tokens=1000),
+ for_object=LLMSetting(llm_handle="default_object", temperature=0.1, max_tokens=1000),
+ ),
+ )
+
+ def test_direct_model_lookup_success(self):
+ """Test successful direct model lookup."""
+ # Arrange
+ model_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec})
+
+ # Act
+ result = model_deck.get_optional_inference_model("gpt-4")
+
+ # Assert
+ assert result == model_spec
+
+ def test_direct_model_lookup_not_found(self, mocker: MockerFixture):
+ """Test direct model lookup when model doesn't exist."""
+ # Arrange
+ model_deck = self._create_test_model_deck()
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("nonexistent-model")
+
+ # Assert
+ assert result is None
+ mock_log.warning.assert_called_once_with("Skipping LLM handle 'nonexistent-model' because it's not found in deck")
+
+ def test_simple_string_alias_resolution_success(self, mocker: MockerFixture):
+ """Test successful resolution of a simple string alias."""
+ # Arrange
+ model_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-gpt": "gpt-4"})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-gpt")
+
+ # Assert
+ assert result == model_spec
+ mock_log.debug.assert_called_once_with("Redirection for 'best-gpt': gpt-4")
+
+ def test_simple_string_alias_resolution_not_found(self, mocker: MockerFixture):
+ """Test string alias resolution when target model doesn't exist."""
+ # Arrange
+ model_deck = self._create_test_model_deck(aliases={"best-gpt": "nonexistent-model"})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-gpt")
+
+ # Assert
+ assert result is None
+ mock_log.debug.assert_called_once_with("Redirection for 'best-gpt': nonexistent-model")
+ # The final warning is about the original alias, not the target
+ mock_log.warning.assert_called_with("Skipping LLM handle 'best-gpt' because it's not found in deck")
+
+ def test_list_alias_resolution_first_success(self, mocker: MockerFixture):
+ """Test list alias resolution where first model in list exists."""
+ # Arrange
+ model_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-model": ["gpt-4", "claude-3"]})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result == model_spec
+ mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['gpt-4', 'claude-3']")
+
+ def test_list_alias_resolution_second_success(self, mocker: MockerFixture):
+ """Test list alias resolution where second model in list exists."""
+ # Arrange
+ model_spec = self._create_test_model_spec("claude-3")
+ model_deck = self._create_test_model_deck(
+ inference_models={"claude-3": model_spec}, aliases={"best-model": ["nonexistent-model", "claude-3"]}
+ )
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result == model_spec
+ mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['nonexistent-model', 'claude-3']")
+
+ def test_list_alias_resolution_none_found(self, mocker: MockerFixture):
+ """Test list alias resolution where no models in list exist."""
+ # Arrange
+ model_deck = self._create_test_model_deck(aliases={"best-model": ["nonexistent-1", "nonexistent-2"]})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result is None
+ mock_log.debug.assert_called_once_with("Redirection for 'best-model': ['nonexistent-1', 'nonexistent-2']")
+ # Should have warning calls: one for each alias in the list + one for the original alias
+ assert mock_log.warning.call_count == 3
+
+ def test_recursive_alias_resolution_success(self, mocker: MockerFixture):
+ """Test recursive alias resolution where alias points to another alias."""
+ # Arrange
+ model_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, aliases={"best-model": "best-gpt", "best-gpt": "gpt-4"})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result == model_spec
+ # Should have debug calls for both redirections
+ assert mock_log.debug.call_count == 2
+
+ def test_recursive_alias_resolution_with_list(self, mocker: MockerFixture):
+ """Test recursive alias resolution where alias points to list containing another alias."""
+ # Arrange
+ model_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(
+ inference_models={"gpt-4": model_spec}, aliases={"best-model": ["nonexistent", "best-gpt"], "best-gpt": "gpt-4"}
+ )
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result == model_spec
+ # Should have debug calls for both redirections
+ assert mock_log.debug.call_count >= 2
+
+ def test_empty_alias_list(self, mocker: MockerFixture):
+ """Test behavior with empty alias list."""
+ # Arrange
+ model_deck = self._create_test_model_deck(aliases={"empty-alias": []})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("empty-alias")
+
+ # Assert
+ assert result is None
+ # Empty list evaluates to False, so no debug log is called
+ mock_log.debug.assert_not_called()
+ mock_log.warning.assert_called_once_with("Skipping LLM handle 'empty-alias' because it's not found in deck")
+
+ def test_circular_alias_prevention(self, mocker: MockerFixture):
+ """Test that circular aliases don't cause infinite loops."""
+ # Note: The current implementation doesn't have explicit circular reference detection,
+ # but Python's recursion limit should prevent infinite loops.
+ # This test ensures the method handles it gracefully.
+
+ # Arrange
+ model_deck = self._create_test_model_deck(aliases={"alias-a": "alias-b", "alias-b": "alias-a"})
+ mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act & Assert
+ # This should either return None or raise RecursionError (which would be caught by pytest)
+ # The important thing is it doesn't hang indefinitely
+ with pytest.raises((RecursionError, Exception)) or True:
+ result = model_deck.get_optional_inference_model("alias-a")
+ # If no exception, result should be None
+ assert result is None
+
+ def test_complex_waterfall_scenario(self, mocker: MockerFixture):
+ """Test complex waterfall scenario with multiple levels."""
+ # Arrange
+ model_spec = self._create_test_model_spec("claude-3")
+ model_deck = self._create_test_model_deck(
+ inference_models={"claude-3": model_spec},
+ aliases={"best-model": ["premium-gpt", "premium-claude"], "premium-gpt": ["gpt-4-turbo", "gpt-4"], "premium-claude": "claude-3"},
+ )
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model("best-model")
+
+ # Assert
+ assert result == model_spec
+ # Should have multiple debug calls for the waterfall resolution
+ assert mock_log.debug.call_count >= 2
+
+ def test_mixed_string_and_list_aliases(self, mocker: MockerFixture):
+ """Test scenario with mix of string and list aliases."""
+ # Arrange
+ model_spec1 = self._create_test_model_spec("gpt-4")
+ model_spec2 = self._create_test_model_spec("claude-3")
+ model_deck = self._create_test_model_deck(
+ inference_models={"gpt-4": model_spec1, "claude-3": model_spec2},
+ aliases={
+ "ai-model": "best-gpt", # string alias
+ "best-gpt": ["gpt-4-turbo", "gpt-4"], # list alias
+ "backup-model": ["claude-4", "claude-3"], # list alias
+ },
+ )
+ mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result1 = model_deck.get_optional_inference_model("ai-model")
+ result2 = model_deck.get_optional_inference_model("backup-model")
+
+ # Assert
+ assert result1 == model_spec1 # Should resolve to gpt-4
+ assert result2 == model_spec2 # Should resolve to claude-3
+
+ @pytest.mark.parametrize(
+ "llm_handle",
+ [
+ "", # empty string
+ " ", # whitespace only
+ "model-with-special-chars!@#", # special characters
+ "UPPERCASE-MODEL", # uppercase
+ ],
+ )
+ def test_edge_case_llm_handles(self, llm_handle: str, mocker: MockerFixture):
+ """Test edge cases for llm_handle parameter."""
+ # Arrange
+ model_deck = self._create_test_model_deck()
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ result = model_deck.get_optional_inference_model(llm_handle)
+
+ # Assert
+ assert result is None
+ mock_log.warning.assert_called_once_with(f"Skipping LLM handle '{llm_handle}' because it's not found in deck")
+
+ def test_logging_behavior(self, mocker: MockerFixture):
+ """Test that logging is called with correct messages."""
+ # Arrange
+ model_deck = self._create_test_model_deck(aliases={"test-alias": "target-model"})
+ mock_log = mocker.patch("pipelex.cogt.models.model_deck.log")
+
+ # Act
+ model_deck.get_optional_inference_model("test-alias")
+
+ # Assert
+ # Verify debug log for redirection
+ mock_log.debug.assert_called_with("Redirection for 'test-alias': target-model")
+
+ # The final warning is about the original alias, not the target
+ mock_log.warning.assert_called_with("Skipping LLM handle 'test-alias' because it's not found in deck")
diff --git a/tests/unit/pipelex/tools/config/test_config_manager.py b/tests/unit/pipelex/tools/config/test_config_manager.py
index 9d08c06cf..78c8c274e 100644
--- a/tests/unit/pipelex/tools/config/test_config_manager.py
+++ b/tests/unit/pipelex/tools/config/test_config_manager.py
@@ -10,6 +10,6 @@ class TestConfigManager:
pretty_print(project_name, title="project_name")
assert project_name == "pipelex"
- def test_load_pipelex_init_config(self):
+ def test_load_pipelex_template_config(self):
hub = get_pipelex_hub()
- hub.setup_config(config_cls=PipelexConfig, specific_config_path="pipelex/pipelex_template.toml")
+ hub.setup_config(config_cls=PipelexConfig, specific_config_path="pipelex/config_template/pipelex.toml")
diff --git a/tests/unit/pipelex/tools/misc/test_dict_utils.py b/tests/unit/pipelex/tools/misc/test_dict_utils.py
index 734f47cd5..8347db504 100644
--- a/tests/unit/pipelex/tools/misc/test_dict_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_dict_utils.py
@@ -1,7 +1,9 @@
"""Unit tests for dict_utils module."""
+from typing import Any, Dict
+
from pipelex.core.concepts.concept_native import NativeConceptEnum
-from pipelex.tools.misc.dict_utils import insert_before
+from pipelex.tools.misc.dict_utils import apply_to_strings_in_list, apply_to_strings_recursive, insert_before
class TestDictUtils:
@@ -47,3 +49,201 @@ class TestDictUtils:
expected_keys = ["type", "definition", "inputs", "output", "system_prompt"]
assert list(result.keys()) == expected_keys
assert result["inputs"] == "InputText"
+
+ def test_apply_to_strings_recursive_simple_dict(self) -> None:
+ """Test apply_to_strings_recursive with a simple dictionary."""
+ data = {"name": "Hello ${USER}", "age": 25, "active": True}
+
+ def uppercase_transform(s: str) -> str:
+ return s.upper()
+
+ result = apply_to_strings_recursive(data, uppercase_transform)
+
+ assert result["name"] == "HELLO ${USER}"
+ assert result["age"] == 25 # Non-string values unchanged
+ assert result["active"] is True # Non-string values unchanged
+
+ def test_apply_to_strings_recursive_nested_dict(self) -> None:
+ """Test apply_to_strings_recursive with nested dictionaries."""
+ data = {"user": {"name": "john ${SUFFIX}", "settings": {"theme": "dark ${MODE}", "count": 10}}, "version": "1.0 ${BUILD}"}
+
+ def env_substitute(s: str) -> str:
+ return s.replace("${SUFFIX}", "_doe").replace("${MODE}", "_theme").replace("${BUILD}", "_final")
+
+ result = apply_to_strings_recursive(data, env_substitute)
+
+ assert result["user"]["name"] == "john _doe"
+ assert result["user"]["settings"]["theme"] == "dark _theme"
+ assert result["user"]["settings"]["count"] == 10
+ assert result["version"] == "1.0 _final"
+
+ def test_apply_to_strings_recursive_with_lists(self) -> None:
+ """Test apply_to_strings_recursive with lists containing various types."""
+ data = {"items": ["hello ${WORLD}", 42, "goodbye ${WORLD}"], "config": {"values": [1, "test ${ENV}", True, {"nested": "value ${VAR}"}]}}
+
+ def substitute_vars(s: str) -> str:
+ return s.replace("${WORLD}", "earth").replace("${ENV}", "production").replace("${VAR}", "123")
+
+ result = apply_to_strings_recursive(data, substitute_vars)
+
+ assert result["items"] == ["hello earth", 42, "goodbye earth"]
+ assert result["config"]["values"][0] == 1
+ assert result["config"]["values"][1] == "test production"
+ assert result["config"]["values"][2] is True
+ assert result["config"]["values"][3]["nested"] == "value 123"
+
+ def test_apply_to_strings_recursive_empty_structures(self) -> None:
+ """Test apply_to_strings_recursive with empty dictionaries and lists."""
+ data: Dict[str, Any] = {"empty_dict": {}, "empty_list": [], "mixed": {"inner_empty": {}, "inner_list": []}}
+
+ def dummy_transform(s: str) -> str:
+ return s.upper()
+
+ result = apply_to_strings_recursive(data, dummy_transform)
+
+ assert result["empty_dict"] == {}
+ assert result["empty_list"] == []
+ assert result["mixed"]["inner_empty"] == {}
+ assert result["mixed"]["inner_list"] == []
+
+ def test_apply_to_strings_recursive_preserves_original(self) -> None:
+ """Test that apply_to_strings_recursive doesn't modify the original data."""
+ original = {"text": "hello ${USER}", "nested": {"value": "world ${ENV}"}, "list": ["item ${VAR}"]}
+ original_copy = {"text": "hello ${USER}", "nested": {"value": "world ${ENV}"}, "list": ["item ${VAR}"]}
+
+ def transform(s: str) -> str:
+ return s.replace("${USER}", "john").replace("${ENV}", "prod").replace("${VAR}", "test")
+
+ result = apply_to_strings_recursive(original, transform)
+
+ # Original should be unchanged
+ assert original == original_copy
+
+ # Result should be transformed
+ assert result["text"] == "hello john"
+ assert result["nested"]["value"] == "world prod"
+ assert result["list"] == ["item test"]
+
+ def test_apply_to_strings_in_list_basic(self) -> None:
+ """Test apply_to_strings_in_list with basic string transformations."""
+ data = ["hello ${USER}", "world", 42, True, "test ${ENV}"]
+
+ def substitute_vars(s: str) -> str:
+ return s.replace("${USER}", "john").replace("${ENV}", "production")
+
+ result = apply_to_strings_in_list(data, substitute_vars)
+
+ assert result == ["hello john", "world", 42, True, "test production"]
+ assert result is not data # Should return a new list
+
+ def test_apply_to_strings_in_list_nested_lists(self) -> None:
+ """Test apply_to_strings_in_list with nested lists."""
+ data = ["outer ${VAR}", [1, "inner ${NESTED}", ["deep ${DEEP}", 3.14]], "final ${END}"]
+
+ def transform(s: str) -> str:
+ return s.replace("${VAR}", "variable").replace("${NESTED}", "nested").replace("${DEEP}", "deep").replace("${END}", "end")
+
+ result = apply_to_strings_in_list(data, transform)
+
+ assert result[0] == "outer variable"
+ assert result[1][0] == 1
+ assert result[1][1] == "inner nested"
+ assert result[1][2][0] == "deep deep"
+ assert result[1][2][1] == 3.14
+ assert result[2] == "final end"
+
+ def test_apply_to_strings_in_list_with_dictionaries(self) -> None:
+ """Test apply_to_strings_in_list with dictionaries in the list."""
+ data = ["list item ${VAR}", {"key1": "dict value ${DICT}", "key2": 100}, 42, {"nested": {"deep": "deep value ${DEEP}"}}, "another ${ITEM}"]
+
+ def transform(s: str) -> str:
+ return s.replace("${VAR}", "var").replace("${DICT}", "dict").replace("${DEEP}", "deep").replace("${ITEM}", "item")
+
+ result = apply_to_strings_in_list(data, transform)
+
+ assert result[0] == "list item var"
+ assert result[1]["key1"] == "dict value dict"
+ assert result[1]["key2"] == 100
+ assert result[2] == 42
+ assert result[3]["nested"]["deep"] == "deep value deep"
+ assert result[4] == "another item"
+
+ def test_apply_to_strings_in_list_empty_list(self) -> None:
+ """Test apply_to_strings_in_list with empty list."""
+ data: Any = []
+
+ def dummy_transform(s: str) -> str:
+ return s.upper()
+
+ result = apply_to_strings_in_list(data, dummy_transform)
+
+ assert result == []
+ assert result is not data # Should return a new list
+
+ def test_apply_to_strings_in_list_no_strings(self) -> None:
+ """Test apply_to_strings_in_list with list containing no strings."""
+ data = [1, 2.5, True, None, {"key": "value"}, [1, 2, 3]]
+
+ def transform(s: str) -> str:
+ return s.upper()
+
+ result = apply_to_strings_in_list(data, transform)
+
+ assert result[0] == 1
+ assert result[1] == 2.5
+ assert result[2] is True
+ assert result[3] is None
+ assert result[4] == {"key": "VALUE"} # Dict should be transformed by apply_to_strings_recursive
+ assert result[5] == [1, 2, 3] # Nested list with no strings
+
+ def test_apply_to_strings_in_list_preserves_original(self) -> None:
+ """Test that apply_to_strings_in_list doesn't modify the original list."""
+ original = ["hello ${USER}", {"key": "value ${VAR}"}, ["nested ${NESTED}"]]
+ original_copy = ["hello ${USER}", {"key": "value ${VAR}"}, ["nested ${NESTED}"]]
+
+ def transform(s: str) -> str:
+ return s.replace("${USER}", "john").replace("${VAR}", "variable").replace("${NESTED}", "nested")
+
+ result = apply_to_strings_in_list(original, transform)
+
+ # Original should be unchanged
+ assert original == original_copy
+
+ # Result should be transformed
+ assert result[0] == "hello john"
+ assert result[1]["key"] == "value variable"
+ assert result[2][0] == "nested nested"
+
+ def test_apply_to_strings_in_list_complex_nesting(self) -> None:
+ """Test apply_to_strings_in_list with complex nested structures."""
+ data = [
+ "root ${ROOT}",
+ {"dict_key": "dict value ${DICT}", "nested_list": ["nested item ${NESTED}", {"deep": "deep value ${DEEP}"}], "number": 42},
+ ["list in list ${LIST}", [{"very_deep": "very deep value ${VERY_DEEP}"}, "deep list item ${DEEP_LIST}"]],
+ ]
+
+ def transform(s: str) -> str:
+ replacements = {
+ "${ROOT}": "root",
+ "${DICT}": "dict",
+ "${NESTED}": "nested",
+ "${DEEP}": "deep",
+ "${LIST}": "list",
+ "${VERY_DEEP}": "very_deep",
+ "${DEEP_LIST}": "deep_list",
+ }
+ result = s
+ for old, new in replacements.items():
+ result = result.replace(old, new)
+ return result
+
+ result = apply_to_strings_in_list(data, transform)
+
+ assert result[0] == "root root"
+ assert result[1]["dict_key"] == "dict value dict"
+ assert result[1]["nested_list"][0] == "nested item nested"
+ assert result[1]["nested_list"][1]["deep"] == "deep value deep"
+ assert result[1]["number"] == 42
+ assert result[2][0] == "list in list list"
+ assert result[2][1][0]["very_deep"] == "very deep value very_deep"
+ assert result[2][1][1] == "deep list item deep_list"
diff --git a/tests/unit/pipelex/tools/misc/test_string_utils.py b/tests/unit/pipelex/tools/misc/test_string_utils.py
index 786765f1e..80c3e7366 100644
--- a/tests/unit/pipelex/tools/misc/test_string_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_string_utils.py
@@ -7,6 +7,7 @@ from pipelex.tools.misc.string_utils import (
is_not_none_and_has_text,
is_pascal_case,
is_snake_case,
+ matches_wildcard_pattern,
pascal_case_to_sentence,
pascal_case_to_snake_case,
snake_to_capitalize_first_letter,
@@ -194,3 +195,99 @@ def test_is_snake_case(word: str, expected: bool) -> None:
)
def test_is_pascal_case(word: str, expected: bool) -> None:
assert is_pascal_case(word) is expected
+
+
+class TestMatchesWildcardPattern:
+ """Test class for the matches_wildcard_pattern function."""
+
+ @pytest.mark.parametrize(
+ "text, pattern, expected",
+ [
+ # Universal wildcard
+ ("any-text", "*", True),
+ ("", "*", True),
+ ("claude-3-sonnet", "*", True),
+ # Exact matches
+ ("claude-3-sonnet", "claude-3-sonnet", True),
+ ("gpt-4o-mini", "gpt-4o-mini", True),
+ ("mistral-large", "mistral-large", True),
+ ("exact-match", "different-text", False),
+ ("", "", True),
+ # Prefix wildcards (pattern ends with *)
+ ("claude-3-sonnet", "claude-*", True),
+ ("claude-3-haiku", "claude-*", True),
+ ("gpt-4o-mini", "claude-*", False),
+ ("claude", "claude-*", False), # No dash after claude
+ ("my-claude-model", "claude-*", False), # claude not at start
+ ("", "prefix-*", False),
+ ("prefix", "prefix-*", False), # No content after prefix
+ ("prefix-something", "prefix-*", True),
+ # Suffix wildcards (pattern starts with *)
+ ("gpt-4o-mini", "*mini", True),
+ ("claude-3-sonnet-mini", "*mini", True),
+ ("mistral-large", "*large", True),
+ ("mini-gpt", "*mini", False), # mini not at end
+ ("large-mistral", "*large", False), # large not at end
+ ("", "*suffix", False),
+ ("suffix", "*suffix", True),
+ ("my-suffix", "*suffix", True),
+ ("suffixnot", "*suffix", False),
+ # Contains wildcards (pattern starts and ends with *)
+ ("claude-3-sonnet", "*sonnet*", True),
+ ("my-sonnet-model", "*sonnet*", True),
+ ("sonnet-based", "*sonnet*", True),
+ ("sonnet", "*sonnet*", True),
+ ("gpt-4o-mini", "*sonnet*", False),
+ ("", "*middle*", False),
+ ("middle", "*middle*", True),
+ ("before-middle-after", "*middle*", True),
+ ("middlepart", "*middle*", True),
+ ("partmiddle", "*middle*", True),
+ # Edge cases with empty middle pattern
+ ("any-text", "**", True), # Empty middle matches anything
+ ("", "**", True),
+ # Case sensitivity
+ ("Claude-3-Sonnet", "claude-*", False), # Case sensitive
+ ("CLAUDE-3-SONNET", "*sonnet*", False), # Case sensitive
+ ("gpt-4O-MINI", "*mini", False), # Case sensitive
+ # Special characters in text and patterns
+ ("model-v1.2", "model-*", True),
+ ("my_model_name", "*_model_*", True),
+ ("api.v2.endpoint", "api.*", True),
+ ("[email protected]", "*@*", True),
+ ("file-name.ext", "*-name.*", True),
+ # Multiple wildcards (only first and last are treated as wildcards)
+ ("a-b-c-d", "*-b-*", True), # Treated as contains pattern
+ ("a*b*c", "*b*", True), # Asterisks in text are literal
+ ],
+ )
+ def test_matches_wildcard_pattern(self, text: str, pattern: str, expected: bool) -> None:
+ """Test wildcard pattern matching with comprehensive test cases."""
+ assert matches_wildcard_pattern(text, pattern) is expected
+
+ def test_matches_wildcard_pattern_model_routing_examples(self) -> None:
+ """Test with real-world model routing examples."""
+ # Claude models
+ assert matches_wildcard_pattern("claude-3-sonnet", "claude-*") is True
+ assert matches_wildcard_pattern("claude-3-haiku", "claude-*") is True
+ assert matches_wildcard_pattern("claude-3.5-sonnet", "claude-*") is True
+
+ # GPT models
+ assert matches_wildcard_pattern("gpt-4o-mini", "gpt-*") is True
+ assert matches_wildcard_pattern("gpt-4", "gpt-*") is True
+ assert matches_wildcard_pattern("gpt-3.5-turbo", "gpt-*") is True
+
+ # Mistral models
+ assert matches_wildcard_pattern("mistral-large", "*large") is True
+ assert matches_wildcard_pattern("mistral-medium", "*medium") is True
+ assert matches_wildcard_pattern("mistral-small", "*small") is True
+
+ # Contains patterns
+ assert matches_wildcard_pattern("claude-3-sonnet", "*sonnet*") is True
+ assert matches_wildcard_pattern("gpt-4o-mini", "*4o*") is True
+ assert matches_wildcard_pattern("mistral-large-instruct", "*large*") is True
+
+ # Negative cases
+ assert matches_wildcard_pattern("gemini-pro", "claude-*") is False
+ assert matches_wildcard_pattern("llama-2", "*gpt*") is False
+ assert matches_wildcard_pattern("anthropic-claude", "*sonnet") is False
diff --git a/tests/unit/pipelex/tools/misc/test_toml_utils.py b/tests/unit/pipelex/tools/misc/test_toml_utils.py
index 8124dbbdb..a19a20c18 100644
--- a/tests/unit/pipelex/tools/misc/test_toml_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_toml_utils.py
@@ -10,10 +10,10 @@ from pipelex.tools.misc.toml_utils import (
)
-class TestPLXUtils:
- def test_load_plx_from_path_valid_file(self, tmp_path: Path) -> None:
- """Test loading a valid PLX file without issues."""
- plx_content = """domain = "test_domain"
+class TestTOMLUtils:
+ def test_load_toml_from_path_valid_file(self, tmp_path: Path) -> None:
+ """Test loading a valid TOML file without issues."""
+ toml_content = """domain = "test_domain"
definition = "Test definition"
[concept]
@@ -27,10 +27,10 @@ prompt_template = '''
This is a test prompt
'''
"""
- plx_file = tmp_path / "valid.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "valid.toml"
+ toml_file.write_text(toml_content)
- result = load_toml_from_path(str(plx_file))
+ result = load_toml_from_path(str(toml_file))
assert isinstance(result, dict)
assert result["domain"] == "test_domain"
@@ -38,25 +38,25 @@ This is a test prompt
assert "concept" in result
assert "pipe" in result
- def test_validate_plx_file_trailing_whitespace(self, tmp_path: Path) -> None:
+ def test_validate_toml_file_trailing_whitespace(self, tmp_path: Path) -> None:
"""Test detection of trailing whitespace."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
definition = "Test"
"""
- plx_file = tmp_path / "trailing_space.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "trailing_space.toml"
+ toml_file.write_text(toml_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
assert "Trailing whitespace detected" in error_msg
assert "Line 1" in error_msg
assert "Line 2" in error_msg
- def test_validate_plx_file_trailing_space_after_triple_quotes(self, tmp_path: Path) -> None:
+ def test_validate_toml_file_trailing_space_after_triple_quotes(self, tmp_path: Path) -> None:
"""Test detection of trailing whitespace after triple quotes."""
- plx_content = '''domain = "test"
+ toml_content = '''domain = "test"
[pipe.test_pipe]
type = "PipeLLM"
@@ -65,92 +65,92 @@ prompt_template = """
Output this only: "test"
"""
'''
- plx_file = tmp_path / "trailing_quotes.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "trailing_quotes.toml"
+ toml_file.write_text(toml_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
assert "Trailing whitespace after triple quotes" in error_msg
assert "Line 8" in error_msg
@pytest.mark.skip(reason="Mixed line ending detection needs refinement - focusing on trailing whitespace detection")
- def test_validate_plx_file_mixed_line_endings(self, tmp_path: Path) -> None:
+ def test_validate_toml_file_mixed_line_endings(self, tmp_path: Path) -> None:
"""Test detection of mixed line endings."""
# Create content with explicit mixed line endings - use binary mode to ensure exact control
- plx_file = tmp_path / "mixed_endings.plx"
+ toml_file = tmp_path / "mixed_endings.toml"
# Write content with mixed line endings directly in binary mode
mixed_content = b'domain = "test"\r\ndefinition = "Test"\nextra = "value"\n'
- plx_file.write_bytes(mixed_content)
+ toml_file.write_bytes(mixed_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
assert "Mixed line endings detected" in error_msg
- def test_load_plx_from_path_no_validation_by_default(self, tmp_path: Path) -> None:
+ def test_load_toml_from_path_no_validation_by_default(self, tmp_path: Path) -> None:
"""Test that loading works normally without validation."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
definition = "Test with trailing space"
"""
- plx_file = tmp_path / "no_validation.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "no_validation.toml"
+ toml_file.write_text(toml_content)
# Should not raise - loading doesn't validate by default
- result = load_toml_from_path(str(plx_file))
+ result = load_toml_from_path(str(toml_file))
assert isinstance(result, dict)
assert result["domain"] == "test"
- def test_failable_load_plx_from_path_nonexistent_file(self) -> None:
+ def test_failable_load_toml_from_path_nonexistent_file(self) -> None:
"""Test failable loading with non-existent file."""
- result = failable_load_toml_from_path("/nonexistent/path.plx")
+ result = failable_load_toml_from_path("/nonexistent/path.toml")
assert result is None
- def test_failable_load_plx_from_path_valid_file(self, tmp_path: Path) -> None:
+ def test_failable_load_toml_from_path_valid_file(self, tmp_path: Path) -> None:
"""Test failable loading with valid file."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
definition = "Test definition"
"""
- plx_file = tmp_path / "valid.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "valid.toml"
+ toml_file.write_text(toml_content)
- result = failable_load_toml_from_path(str(plx_file))
+ result = failable_load_toml_from_path(str(toml_file))
assert result is not None
assert result["domain"] == "test"
- def test_failable_load_plx_from_path_with_whitespace_works(self, tmp_path: Path) -> None:
+ def test_failable_load_toml_from_path_with_whitespace_works(self, tmp_path: Path) -> None:
"""Test failable loading works normally with whitespace."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
"""
- plx_file = tmp_path / "with_whitespace.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "with_whitespace.toml"
+ toml_file.write_text(toml_content)
# Should work fine - loading doesn't validate by default
- result = failable_load_toml_from_path(str(plx_file))
+ result = failable_load_toml_from_path(str(toml_file))
assert result is not None
assert result["domain"] == "test"
- def test_validate_plx_file_valid_file_passes(self, tmp_path: Path) -> None:
+ def test_validate_toml_file_valid_file_passes(self, tmp_path: Path) -> None:
"""Test that validation passes for valid files."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
definition = "Test definition"
[concept]
TestConcept = "A test concept"
"""
- plx_file = tmp_path / "valid.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "valid.toml"
+ toml_file.write_text(toml_content)
# Should not raise
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
def test_validate_toml_file_multiple_validation_issues(self, tmp_path: Path) -> None:
"""Test that multiple validation issues are all reported."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
definition = "Test"
[pipe.test_pipe]
@@ -158,11 +158,11 @@ prompt_template = \"\"\"
Output: "test"
\"\"\"
"""
- plx_file = tmp_path / "multiple_issues.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "multiple_issues.toml"
+ toml_file.write_text(toml_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
# Should detect multiple lines with trailing whitespace
@@ -171,23 +171,23 @@ Output: "test"
assert "Line 7" in error_msg
assert "Trailing whitespace after triple quotes" in error_msg
- def test_validate_plx_file_error_contains_file_path(self, tmp_path: Path) -> None:
+ def test_validate_toml_file_error_contains_file_path(self, tmp_path: Path) -> None:
"""Test that validation error includes the file path."""
- plx_content = """domain = "test"
+ toml_content = """domain = "test"
"""
- plx_file = tmp_path / "path_test.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "path_test.toml"
+ toml_file.write_text(toml_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
- assert str(plx_file) in error_msg
+ assert str(toml_file) in error_msg
assert "TOML formatting issues" in error_msg
- def test_validate_plx_file_pipe_condition_real_case(self, tmp_path: Path) -> None:
- """Test the exact scenario from pipe_condition_2.plx with trailing space after triple quotes."""
- plx_content = '''domain = "test_pipe_condition_2"
+ def test_validate_toml_file_pipe_condition_real_case(self, tmp_path: Path) -> None:
+ """Test the exact scenario from pipe_condition_2.toml with trailing space after triple quotes."""
+ toml_content = '''domain = "test_pipe_condition_2"
definition = "Simple test for PipeCondition functionality using expression"
[concept]
@@ -213,19 +213,19 @@ output = "native.Text"
prompt_template = """
Output this only: "large"
""" '''
- plx_file = tmp_path / "pipe_condition_real_case.plx"
- plx_file.write_text(plx_content)
+ toml_file = tmp_path / "pipe_condition_real_case.toml"
+ toml_file.write_text(toml_content)
with pytest.raises(TOMLValidationError) as exc_info:
- validate_toml_file(str(plx_file))
+ validate_toml_file(str(toml_file))
error_msg = str(exc_info.value)
assert "Trailing whitespace after triple quotes" in error_msg
assert "Line 26" in error_msg # The line with """ followed by space
- def test_validate_plx_file_actual_problematic_file(self) -> None:
+ def test_validate_toml_file_actual_problematic_file(self) -> None:
"""Test validation on the actual problematic file from the codebase."""
- problematic_file = "tests/data/tools_data/problematic_test_cases.plx"
+ problematic_file = "tests/data/tools_data/problematic_test_cases.toml"
# This should catch multiple trailing whitespace issues
with pytest.raises(TOMLValidationError) as exc_info:
diff --git a/tests/unit/pipelex/tools/secrets/test_secrets_utils.py b/tests/unit/pipelex/tools/secrets/test_secrets_utils.py
new file mode 100644
index 000000000..0dd3b05ec
--- /dev/null
+++ b/tests/unit/pipelex/tools/secrets/test_secrets_utils.py
@@ -0,0 +1,225 @@
+import os
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+from pipelex.tools.secrets.secrets_utils import UnknownVarPrefixError, VarNotFoundError, substitute_vars
+
+
+def mock_secrets_provider(mocker: MockerFixture) -> Any:
+ """Mock secrets provider for testing."""
+ mock_provider = mocker.Mock(spec=SecretsProviderAbstract)
+ return mock_provider
+
+
+class TestSubstituteVars:
+ """Test cases for substitute_vars function."""
+
+ def test_simple_secret_substitution(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test basic secret substitution using default behavior."""
+ mock_secrets_provider.get_secret.return_value = "secret_value"
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Hello ${API_KEY}")
+
+ assert result == "Hello secret_value"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="API_KEY")
+
+ def test_explicit_env_substitution(self, mocker: MockerFixture) -> None:
+ """Test explicit environment variable substitution."""
+ mocker.patch.dict(os.environ, {"TEST_VAR": "env_value"})
+ result = substitute_vars("Value: ${env:TEST_VAR}")
+
+ assert result == "Value: env_value"
+
+ def test_explicit_secret_substitution(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test explicit secret substitution."""
+ mock_secrets_provider.get_secret.return_value = "secret_value"
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Value: ${secret:API_KEY}")
+
+ assert result == "Value: secret_value"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="API_KEY")
+
+ def test_fallback_env_to_secret(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test fallback from env to secret when env var doesn't exist."""
+ mock_secrets_provider.get_secret.return_value = "secret_fallback"
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Value: ${env:MISSING_VAR|secret:FALLBACK_KEY}")
+
+ assert result == "Value: secret_fallback"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="FALLBACK_KEY")
+
+ def test_env_takes_precedence_in_fallback(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that env var takes precedence when both env and secret are available."""
+ mocker.patch.dict(os.environ, {"EXISTING_VAR": "env_value"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Value: ${env:EXISTING_VAR|secret:FALLBACK_KEY}")
+
+ assert result == "Value: env_value"
+ # Secret should not be called since env var was found
+ mock_secrets_provider.get_secret.assert_not_called()
+
+ def test_fallback_secret_to_env(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test fallback from secret to env when secret doesn't exist."""
+ from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+
+ mock_secrets_provider.get_secret.side_effect = SecretNotFoundError("Secret not found")
+ mocker.patch.dict(os.environ, {"FALLBACK_ENV": "env_fallback"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Value: ${secret:MISSING_SECRET|env:FALLBACK_ENV}")
+
+ assert result == "Value: env_fallback"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="MISSING_SECRET")
+
+ def test_secret_takes_precedence_in_reverse_fallback(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that secret takes precedence when both secret and env are available in reverse order."""
+ mock_secrets_provider.get_secret.return_value = "secret_value"
+ mocker.patch.dict(os.environ, {"EXISTING_VAR": "env_value"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("Value: ${secret:EXISTING_SECRET|env:EXISTING_VAR}")
+
+ assert result == "Value: secret_value"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="EXISTING_SECRET")
+
+ def test_multiple_substitutions(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test multiple variable substitutions in the same content."""
+ mock_secrets_provider.get_secret.side_effect = lambda secret_id: f"secret_{secret_id}" # type: ignore
+ mocker.patch.dict(os.environ, {"ENV_VAR": "env_value"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ content = "API: ${API_KEY}, DB: ${env:ENV_VAR}, Token: ${secret:TOKEN}"
+ result = substitute_vars(content)
+
+ assert result == "API: secret_API_KEY, DB: env_value, Token: secret_TOKEN"
+
+ def test_no_placeholders(self) -> None:
+ """Test content with no placeholders remains unchanged."""
+ content = "This is just plain text with no variables"
+ result = substitute_vars(content)
+ assert result == content
+
+ def test_missing_env_var_raises_error(self) -> None:
+ """Test that missing required env var raises VarNotFoundError."""
+ with pytest.raises(VarNotFoundError, match="Environment variable 'MISSING_VAR' is required but not set"):
+ substitute_vars("Value: ${env:MISSING_VAR}")
+
+ def test_missing_secret_raises_error(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that missing required secret raises error."""
+ from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+
+ mock_secrets_provider.get_secret.side_effect = SecretNotFoundError("Secret not found")
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ with pytest.raises(VarNotFoundError, match="Could not get variable 'MISSING_SECRET': Secret not found"):
+ substitute_vars("Value: ${secret:MISSING_SECRET}")
+
+ def test_missing_default_secret_raises_error(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that missing default secret (no prefix) raises error."""
+ from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+
+ mock_secrets_provider.get_secret.side_effect = SecretNotFoundError("Secret not found")
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ with pytest.raises(VarNotFoundError, match="Could not get variable 'MISSING_SECRET': Secret not found"):
+ substitute_vars("Value: ${MISSING_SECRET}")
+
+ def test_fallback_both_missing_raises_error(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that when both env and secret are missing, error is raised."""
+ from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+
+ mock_secrets_provider.get_secret.side_effect = SecretNotFoundError("Secret not found")
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ with pytest.raises(VarNotFoundError, match="Variable not found in any source"):
+ substitute_vars("Value: ${env:MISSING_ENV|secret:MISSING_SECRET}")
+
+ def test_reverse_fallback_both_missing_raises_error(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that when both secret and env are missing in reverse order, error is raised."""
+ from pipelex.tools.secrets.secrets_errors import SecretNotFoundError
+
+ mock_secrets_provider.get_secret.side_effect = SecretNotFoundError("Secret not found")
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ with pytest.raises(VarNotFoundError, match="Variable not found in any source"):
+ substitute_vars("Value: ${secret:MISSING_SECRET|env:MISSING_ENV}")
+
+ def test_complex_variable_names(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test complex variable names with underscores, numbers, etc."""
+ mock_secrets_provider.get_secret.return_value = "complex_secret"
+ mocker.patch.dict(os.environ, {"API_KEY_V2": "complex_env"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ result = substitute_vars("${env:API_KEY_V2} and ${DB_PASSWORD_123}")
+
+ assert result == "complex_env and complex_secret"
+
+ def test_whitespace_handling(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that whitespace around variable names is handled correctly."""
+ mock_secrets_provider.get_secret.return_value = "secret_value"
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ # Note: whitespace should be preserved as part of the variable name
+ result = substitute_vars("Value: ${ API_KEY }")
+
+ assert result == "Value: secret_value"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id=" API_KEY ")
+
+ def test_nested_braces_not_matched(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test that nested braces are not matched by the pattern."""
+ # The content has nested braces that should not be matched as a single variable
+ # But ${nested} should be matched as a valid variable
+ mock_secrets_provider.get_secret.return_value = "secret_nested"
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ content = "This ${has ${nested} braces} should not match"
+ result = substitute_vars(content)
+
+ # The ${nested} part should be substituted, but ${has ... should not be matched
+ # because it contains a $ character which breaks the pattern
+ assert result == "This ${has secret_nested braces} should not match"
+ mock_secrets_provider.get_secret.assert_called_once_with(secret_id="nested")
+
+ def test_multiline_content(self, mocker: MockerFixture, mock_secrets_provider: Any) -> None:
+ """Test substitution in multiline content."""
+ mock_secrets_provider.get_secret.return_value = "secret_value"
+ mocker.patch.dict(os.environ, {"HOME": "/home/user"})
+ mocker.patch("pipelex.tools.secrets.secrets_utils.get_secrets_provider", return_value=mock_secrets_provider)
+
+ content = """Line 1: ${API_KEY}
+Line 2: Some text
+Line 3: ${env:HOME}"""
+ result = substitute_vars(content)
+
+ expected = """Line 1: secret_value
+Line 2: Some text
+Line 3: /home/user"""
+ assert result == expected
+
+ def test_pattern_does_not_match_across_newlines(self) -> None:
+ """Test that the pattern doesn't match variables that span multiple lines."""
+ content = """This ${VAR
+NAME} should not match"""
+ result = substitute_vars(content)
+ # Should remain unchanged
+ assert result == content
+
+ def test_pattern_does_not_match_with_quotes(self) -> None:
+ """Test that the pattern doesn't match variables containing quotes."""
+ content = "This ${VAR\"NAME} and ${VAR'NAME} should not match"
+ result = substitute_vars(content)
+ # Should remain unchanged
+ assert result == content
+
+ def test_unknown_prefix_raises_error(self) -> None:
+ """Test that unknown prefix raises UnknownVarPrefixError."""
+ with pytest.raises(UnknownVarPrefixError, match="Unknown variable prefix: 'foo'"):
+ substitute_vars("Value: ${foo:BAR}")
diff --git a/tests/unit/pipelex/tools/test_environment.py b/tests/unit/pipelex/tools/test_environment.py
index b66e56c20..84316ccb4 100644
--- a/tests/unit/pipelex/tools/test_environment.py
+++ b/tests/unit/pipelex/tools/test_environment.py
@@ -21,12 +21,12 @@ class TestGetRequiredEnv:
def test_get_required_env_missing_raises_error(self, mocker: MockerFixture):
mocker.patch.dict(os.environ, {}, clear=True)
- with pytest.raises(EnvVarNotFoundError, match="Missing 'MISSING_VAR 'in environment."):
+ with pytest.raises(EnvVarNotFoundError, match="Environment variable 'MISSING_VAR' is required but not set"):
get_required_env("MISSING_VAR")
def test_get_required_env_empty_string_raises_error(self, mocker: MockerFixture):
mocker.patch.dict(os.environ, {"EMPTY_VAR": ""})
- with pytest.raises(EnvVarNotFoundError, match="Missing 'EMPTY_VAR 'in environment."):
+ with pytest.raises(EnvVarNotFoundError, match="Environment variable 'EMPTY_VAR' is required but not set"):
get_required_env("EMPTY_VAR")