Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Handle-per-model-type · L-261007-0ff74d (#1374)

Louis Choquel17198c4e36d0
diff --git a/CHANGELOG.md b/CHANGELOG.md
index aef95a52f..bc11ef0f1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,13 @@
### Added
- **Claude Haiku 5.5 on the `anthropic` backend**: `claude-5.5-haiku` (model id `claude-haiku-5-5`) takes text, images and PDF, produces text and structured output, and reasons with adaptive thinking, with up to 128K output tokens. It costs $0.10 / $0.50 per million tokens for a prompt up to 100,000 tokens and $0.50 / $2.50 above, of which the backend file records the first. It refuses `temperature` but, unlike Sonnet 5.5 and Opus 5.5, accepts a forced `tool_choice`, so structured output keeps `instructor/anthropic_tools`.
+- **One handle, one model per model type**: two models of different model types may share a handle, so `gpt-6-luna` can be an LLM and a judgment model at once, and each pipe reaches the one its family asks for. A backend file declares the second model of a handle on a table of its own with the new `handle` key, which may not appear in `[defaults]`, and a file declaring one handle twice as the same type fails to load, naming both tables. A plugin may declare an internal model under a name `internal.toml` already declares as another type.
+
+### Changed
+
+- **A model served only as another type is refused when the method loads (Breaking)**: a pipe naming a handle the deck serves only as another model type, such as a `PipeJudge` naming an LLM, is refused by `pipelex validate`, `POST /v1/validate` and a run's load as an `unknown_model` item, whose suggestions list only models of the type the pipe asks for, where it passed validation and failed at its first call. A handle served as another type no longer hides an alias or a waterfall of the same name from a lookup of this type.
+- **Model specs are keyed by model type and handle (Breaking)**: `InferenceBackend.model_specs` and `ModelDeck.inference_models` are the new `ModelSpecIndex`, read through `get(model_type=…, handle=…)`, `handles_of_type`, `all_specs`, `all_handles` and `types_serving`, and `InferenceBackend.get_model_spec` takes `model_type` and `handle`. `InferenceBackend.list_model_names`, `InferenceBackendLibrary.list_all_model_names` and `InferenceBackendLibrary.get_all_models_and_possible_backends` are removed. `PluginModelDeclarations.internal_models` is a tuple of `PluginInternalModel`, which carries its `name`, and `DuplicateInternalModelError` names the `model_type` two plugins both declared.
+- **Usage is reported per model type and name (Breaking)**: a run calling both kinds of one handle reports two rows in the cost table and the agent CLI's cost summary, and two entries in the graph's usage attribution, where they merged into one row typed by whichever call came last. `AggregatedCosts.grouped_by_model` is keyed by the new `ModelUsageKey`, and its `model_types` is removed.
## [v0.77.0] - 2026-10-07
diff --git a/docs/configuration/config-technical/inference-backend-config.md b/docs/configuration/config-technical/inference-backend-config.md
index 6f1ba0f12..64db7ed53 100644
--- a/docs/configuration/config-technical/inference-backend-config.md
+++ b/docs/configuration/config-technical/inference-backend-config.md
@@ -207,6 +207,36 @@ costs = { input = 0.04, output = 0.0 }
The `[defaults]` table applies to every model of the file, and a model table overrides any key of it.
+#### One handle, several kinds of model
+
+A handle names one model per model type. The same name may be an LLM and a judgment model at once, and each pipe reaches the kind its family asks for: a `PipeLLM` naming `acme-one` gets the LLM, and a `PipeJudge` naming it gets the judgment model. A table's name is its handle, and TOML forbids declaring a table twice, so the second model of a handle gets a table name of its own and says which handle it serves with `handle`:
+
+```toml
+# acme.toml
+[defaults]
+model_type = "llm"
+sdk = "openai_responses"
+
+["acme-one"]
+inputs = ["text", "images"]
+outputs = ["text", "structured"]
+costs = { input = 0.1, output = 0.5 }
+
+["acme-one-judgment"]
+handle = "acme-one"
+model_type = "judgment"
+sdk = "acme_judgments" # the judgment SDK the provider's plugin registers
+inputs = ["text"]
+outputs = ["judgments"]
+costs = { input = 0.1, output = 0 }
+```
+
+The table name `acme-one-judgment` names the table and nothing else: methods, the model deck and the routing profiles all use the handle `acme-one`, and the model id defaults to the handle too. Two tables may share a handle only when their model types differ, so a file declaring one handle twice as the same type fails to load, naming both tables. `handle` cannot go in `[defaults]`, which every model of the file inherits, and like every key that describes the model rather than a value a call sends, it cannot reference a variable.
+
+A routing profile routes a handle's name, so its route applies to every kind of model of that name. A model type the routed backend does not serve is looked for along the profile's `fallback_order`, or in the internal backend when it has none, when the name reached that backend by default, is left out when it reached it through a wildcard pattern, and is not served at all when the profile routes the name to that backend exactly: an exact route pins the name to one backend.
+
+A method whose pipe names a handle the deck serves only as another kind of model is refused when it loads: a `PipeJudge` naming a handle served only as an LLM is told that no judgment model of that name exists, and the suggestions list judgment models only.
+
#### Input formats
`inputs` lists what a model reads. For files, its entries are format keys, the same keys the runtime derives from a file's MIME type to check, before a run starts, that every file reaches a model able to read it: every image type is the `image` family, and any other type is its extension.
diff --git a/docs/errors/duplicate-internal-model-error.md b/docs/errors/duplicate-internal-model-error.md
index e9213b2e3..2c3e22b20 100644
--- a/docs/errors/duplicate-internal-model-error.md
+++ b/docs/errors/duplicate-internal-model-error.md
@@ -7,7 +7,7 @@ description: "Reference for the `DuplicateInternalModelError` Pipelex error clas
# Duplicate internal model
-Two plugins declared a model of the same name in the internal backend.
+Two plugins declared a model of the same name and model type in the internal backend.
| Field | Value |
|---|---|
diff --git a/docs/under-the-hood/document-engine-plugins.md b/docs/under-the-hood/document-engine-plugins.md
index bcc578dd2..c2aaaa85c 100644
--- a/docs/under-the-hood/document-engine-plugins.md
+++ b/docs/under-the-hood/document-engine-plugins.md
@@ -89,7 +89,7 @@ class MyDocGenPlugin:
The model's `spec` is exactly the table a backend file would hold for it (`model_type`, `sdk`, `model_id`, `inputs`, `outputs`, `costs`, and any other model-spec field), and it is complete on its own: the `[defaults]` table of `internal.toml` is not applied to it. A default names a model, directly or through any reference the deck resolves.
-`register` stores these declarations and validates nothing. The model manager merges them at boot, before it builds the deck, and refuses what it cannot merge with `PluginModelDeclarationError`, naming the plugin: a model whose name the install's `internal.toml` already declares (the message names the file too), a table that is not a valid model spec, or a default for a format and source no step composes. Model names are not global across backends, so a name another backend declares is left to the routing profile, as it is for a model a file declares. When the install disables the `internal` backend, or declares none, the plugin's models are not merged, as a disabled backend's own models are not loaded, and its defaults are left out with them, so a plugin never makes a boot fail that would succeed without it. A second plugin declaring the same model fails at registration with `DuplicateInternalModelError`, and the same format and source with `DuplicateDocGenDefaultError`, each naming both plugins. `pipelex plugins list` shows each declaration on the plugin's row.
+`register` stores these declarations and validates nothing. The model manager merges them at boot, before it builds the deck, and refuses what it cannot merge with `PluginModelDeclarationError`, naming the plugin: a model whose name and model type the install's `internal.toml` already declares (the message names the file too), a table that is not a valid model spec, or a default for a format and source no step composes. A handle names one model per model type, so a plugin may declare another kind of a name the file declares. Model names are not global across backends, so a name another backend declares is left to the routing profile, as it is for a model a file declares. When the install disables the `internal` backend, or declares none, the plugin's models are not merged, as a disabled backend's own models are not loaded, and its defaults are left out with them, so a plugin never makes a boot fail that would succeed without it. A second plugin declaring the same model, the same name with the same model type, fails at registration with `DuplicateInternalModelError`, and the same format and source with `DuplicateDocGenDefaultError`, each naming both plugins. `pipelex plugins list` shows each declaration on the plugin's row.
A document engine is a kernel-layer capability, and so are its model and its defaults, so an out-of-tree engine publishes under the `pipelex.plugins.kernel` entry-point group (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for the groups). `make_worker` is called for each print, with the model's spec, and imports the engine's library inside it, so `register` stays import-light and a process that never prints never imports the library. An engine loads its library, fonts and styles once per process, in the module that holds its worker, rather than per worker. A second registration of the same sdk fails loud at boot with `DuplicateInferenceBackendError`, naming both plugins.
diff --git a/docs/under-the-hood/per-node-usage-attribution.md b/docs/under-the-hood/per-node-usage-attribution.md
index 5cc6af577..2797e55d8 100644
--- a/docs/under-the-hood/per-node-usage-attribution.md
+++ b/docs/under-the-hood/per-node-usage-attribution.md
@@ -90,7 +90,7 @@ A GraphSpec names a model in three places, and only one of them is the outcome:
`execution_data.resolved_model` is captured from `LLMSetting.model` (`pipe_llm.py`), which is a *handle*: alias resolution happens later, at inference time. The name overstates what it holds — roughly a third of the nodes in a typical corpus carry an unresolved `@alias` there. Only `by_model` survives alias resolution, deck defaults, and any fallback or retry that landed somewhere other than what was requested.
-**`by_model` is a list because one node routinely uses more than one model.** A `PipeLLM`'s text pass and its object-structuring pass resolve separately (that is what `resolved_model_for_object` is about), and a retry can land elsewhere again. Collapsing them into "the model" would be wrong in exactly the way a single `cost` across mixed rated and unrated calls is wrong. Entries are ordered most-used first, ties broken by name, so a consumer can take `by_model[0]` as the dominant model without sorting. Per-model `cost` follows invariant 2: `None` iff no call to that model was priced.
+**`by_model` is a list because one node routinely uses more than one model.** A `PipeLLM`'s text pass and its object-structuring pass resolve separately (that is what `resolved_model_for_object` is about), and a retry can land elsewhere again. Collapsing them into "the model" would be wrong in exactly the way a single `cost` across mixed rated and unrated calls is wrong. An entry is one model type and name, since a handle names one model per model type: a node calling `gpt-6-luna` as an LLM and as a judgment model has two entries. Entries are ordered most-used first, ties broken by name and then by model type, so a consumer can take `by_model[0]` as the dominant model without sorting. Per-model `cost` follows invariant 2: `None` iff no call to that model was priced.
The same breakdown rolls up: `subtree_by_model` tells you every model a controller's whole branch used.
diff --git a/docs/under-the-hood/writing-an-inference-plugin.md b/docs/under-the-hood/writing-an-inference-plugin.md
index 7b20707ea..74d7583a4 100644
--- a/docs/under-the-hood/writing-an-inference-plugin.md
+++ b/docs/under-the-hood/writing-an-inference-plugin.md
@@ -204,15 +204,14 @@ def list_hello_models(
backend's configuration file.
"""
console = get_console()
- model_names = backend.list_model_names()
+ # One entry per spec: a handle names one model per model type, so a name may appear once per type.
+ model_specs = [model_spec for model_spec in backend.model_specs.all_specs() if model_spec.sdk == sdk]
if flat:
if not any_listed:
console.print("model_id,sdk,backend")
- for model_name in model_names:
- model_spec = backend.get_model_spec(model_name)
- model_id = model_spec.model_id if model_spec else model_name
- console.print(f"{model_id},{sdk},{backend_name}")
+ for model_spec in model_specs:
+ console.print(f"{model_spec.model_id},{sdk},{backend_name}")
return
table = Table(
@@ -223,10 +222,7 @@ def list_hello_models(
)
table.add_column("Model ID", style="green")
table.add_column("Model Type", style="blue")
- for model_name in model_names:
- model_spec = backend.get_model_spec(model_name)
- if model_spec is None:
- continue
+ for model_spec in model_specs:
table.add_row(model_spec.model_id, model_spec.model_type)
console.print(table)
```
diff --git a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
index f9988c71e..3a6785b93 100644
--- a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
+++ b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
@@ -13,6 +13,7 @@ from rich.markup import escape
from rich.panel import Panel
from rich.table import Table
+from pipelex.cogt.model_backends.model_spec_document import list_declared_model_specs
from pipelex.runtime_hub import get_console
from pipelex.system.configuration.config_loader import config_manager
from pipelex.system.configuration.configs import ConfigPaths
@@ -33,49 +34,23 @@ TEST_PROFILES_OVERRIDE_PATH = Path(ConfigPaths.DEV_CONFIG_DIR_PATH) / "test_prof
# Model types we care about
MODEL_TYPES = ["llm", "img_gen", "text_extractor", "search", "judgment"]
-# Default model type from backend config
-DEFAULT_MODEL_TYPE = "llm"
+def _group_handles_by_model_type(*, document: dict[str, Any]) -> dict[str, list[str]]:
+ """Group the models a backend TOML declares by model type, reading the file as the loader reads it.
-def _group_handles_by_model_type(*, specs: dict[str, Any]) -> dict[str, list[str]]:
- """Group a table of model specs by model type.
-
- A backend TOML holds one table per model handle and an optional `defaults` table naming the
- model type the rest inherit.
+ A handle names one model per model type, so a handle appears once under each type it is declared as.
Args:
- specs: The specs table, keyed by model handle.
+ document: The backend TOML document.
Returns:
Dictionary mapping model_type to a sorted list of model handles.
"""
- defaults = specs.get("defaults", {})
- default_model_type: str = DEFAULT_MODEL_TYPE
- if isinstance(defaults, dict):
- defaults_dict = cast("dict[str, Any]", defaults)
- default_type_raw = defaults_dict.get("model_type")
- if default_type_raw:
- default_model_type = str(default_type_raw)
-
models_by_type: dict[str, list[str]] = {model_type: [] for model_type in MODEL_TYPES}
- for key, value in specs.items():
- # Skip defaults and non-dict entries
- if key == "defaults" or not isinstance(value, dict):
- continue
-
- # Skip rules sub-sections
- if ".rules" in key:
- continue
-
- # Determine model type (cast for type safety)
- value_dict = cast("dict[str, Any]", value)
- model_type_raw = value_dict.get("model_type", default_model_type)
- model_type = str(model_type_raw) if model_type_raw else default_model_type
-
- # Add to appropriate list
- if model_type in models_by_type:
- models_by_type[model_type].append(key)
+ for declared in list_declared_model_specs(document=document):
+ if declared.model_type in models_by_type:
+ models_by_type[declared.model_type].append(declared.handle)
# Sort model lists
for model_list in models_by_type.values():
@@ -98,7 +73,7 @@ def _extract_models_from_backend_toml(backend_path: Path) -> dict[str, list[str]
except (TomlError, OSError):
return {}
- return _group_handles_by_model_type(specs=config)
+ return _group_handles_by_model_type(document=config)
def _collect_all_model_availability() -> dict[str, Any]:
diff --git a/pipelex/cogt/model_backends/backend.py b/pipelex/cogt/model_backends/backend.py
index e5f3ae252..0e8657576 100644
--- a/pipelex/cogt/model_backends/backend.py
+++ b/pipelex/cogt/model_backends/backend.py
@@ -5,6 +5,8 @@ from pydantic import Field
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.tools.typing.pydantic_utils import empty_dict_factory_of, empty_list_factory_of
@@ -24,7 +26,7 @@ class InferenceBackend(ConfigModel):
listed_constraints: list[ListedConstraint] = Field(default_factory=empty_list_factory_of(ListedConstraint))
valued_constraints: dict[ValuedConstraint, Any] = Field(default_factory=empty_dict_factory_of(ValuedConstraint))
extra_config: dict[str, Any] = Field(default_factory=dict)
- model_specs: dict[str, InferenceModelSpec] = Field(default_factory=dict)
+ model_specs: ModelSpecIndex = Field(default_factory=ModelSpecIndex.make_empty)
# What a keyless load (`CredentialResolution.SKIP`) left unresolved: each field it left unset, or
# `model_specs` for the backend's own file, mapped to the names of the variables it references.
# A field minted at boot rather than read from a variable, as Vertex AI's token is, maps to no
@@ -36,13 +38,9 @@ class InferenceBackend(ConfigModel):
"""The names of every variable this backend's credentials reference and its load left unresolved, sorted."""
return sorted({var_name for var_names in self.unresolved_credentials.values() for var_name in var_names})
- def list_model_names(self) -> list[str]:
- """List the names of all models in the backend."""
- return list(self.model_specs.keys())
-
- def get_model_spec(self, model_name: str) -> InferenceModelSpec | None:
- """Get a model spec by name."""
- return self.model_specs.get(model_name)
+ def get_model_spec(self, *, model_type: ModelType, handle: str) -> InferenceModelSpec | None:
+ """The spec this backend declares for `handle` as `model_type`, or `None`."""
+ return self.model_specs.get(model_type=model_type, handle=handle)
def get_extra_config(self, key: str) -> Any | None:
"""Get an extra config by key."""
diff --git a/pipelex/cogt/model_backends/backend_factory.py b/pipelex/cogt/model_backends/backend_factory.py
index fb8f6f39e..328b095bd 100644
--- a/pipelex/cogt/model_backends/backend_factory.py
+++ b/pipelex/cogt/model_backends/backend_factory.py
@@ -5,7 +5,7 @@ from pydantic import Field, field_validator
from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
-from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.tools.typing.pydantic_utils import empty_dict_factory_of, empty_list_factory_of
@@ -47,7 +47,7 @@ class InferenceBackendFactory:
*,
blueprint: InferenceBackendBlueprint,
extra_config: dict[str, Any],
- model_specs: dict[str, InferenceModelSpec],
+ model_specs: ModelSpecIndex,
credentials: CredentialResolution,
unresolved_credentials: dict[str, list[str]],
) -> InferenceBackend:
diff --git a/pipelex/cogt/model_backends/backend_library.py b/pipelex/cogt/model_backends/backend_library.py
index 4c40e79fa..a0e9d2a40 100644
--- a/pipelex/cogt/model_backends/backend_library.py
+++ b/pipelex/cogt/model_backends/backend_library.py
@@ -22,13 +22,23 @@ from pipelex.cogt.model_backends.backend_factory import (
)
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
-from pipelex.cogt.model_backends.model_spec_document import MODEL_SPEC_DEFAULTS_TABLE
+from pipelex.cogt.model_backends.model_spec_document import (
+ MODEL_SPEC_DEFAULTS_TABLE,
+ MODEL_SPEC_HANDLE_FIELD,
+ describe_defaults_not_a_table,
+ describe_duplicate_declared_model,
+ describe_handle_in_defaults,
+ find_duplicate_declared_model,
+ list_declared_model_specs,
+)
from pipelex.cogt.model_backends.model_spec_factory import (
BackendModelSpecs,
InferenceModelSpecBlueprint,
InferenceModelSpecFactory,
)
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_spec_keys import describe_rejected_keys, split_model_spec_keys
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.migration.plan import MigrationPlan
from pipelex.plugins.plugin_model_declarations import PluginModelDeclarations
from pipelex.system.configuration.config_loader import config_manager
@@ -81,7 +91,7 @@ TEMPLATABLE_MODEL_SPEC_FIELDS = frozenset({"model_id", "endpoint_path"})
class RecoveredModelSpecs(NamedTuple):
"""One backend's model specs rebuilt from a migrated file, and what the ledger did to get there."""
- model_specs: "dict[str, InferenceModelSpec]"
+ model_specs: ModelSpecIndex
plans: list[MigrationPlan]
@@ -484,7 +494,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
backend_config_source: str,
backend_listed_constraints: list[ListedConstraint],
backend_valued_constraints: dict[ValuedConstraint, Any],
- ) -> "dict[str, InferenceModelSpec]":
+ ) -> ModelSpecIndex:
"""Turn one backend's raw tables into model specs: pop `[defaults]`, split, merge, validate.
Its own method so the boot-tolerance retry can run it a second time over a migrated document
@@ -493,8 +503,18 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
because the caller may still need the original tables when this raises.
"""
remaining_tables = dict(model_specs_dict)
- defaults_dict: dict[str, Any] = remaining_tables.pop(MODEL_SPEC_DEFAULTS_TABLE, {})
- backend_model_specs: dict[str, InferenceModelSpec] = {}
+ defaults_value: Any = remaining_tables.pop(MODEL_SPEC_DEFAULTS_TABLE, {})
+ if not isinstance(defaults_value, dict):
+ msg = f"Invalid model specs for backend '{backend_name}' from {backend_config_source}: {describe_defaults_not_a_table()}"
+ raise InferenceBackendLibraryError(msg, backend_name=backend_name)
+ defaults_dict = cast("dict[str, Any]", defaults_value)
+ if MODEL_SPEC_HANDLE_FIELD in defaults_dict:
+ msg = f"Invalid model specs for backend '{backend_name}' from {backend_config_source}: {describe_handle_in_defaults()}"
+ raise InferenceBackendLibraryError(msg, backend_name=backend_name)
+ # The handle each table serves and the type it declares, read as every other reader of the file reads them.
+ declared_model_specs = list_declared_model_specs(document=model_specs_dict)
+ handles_by_table = {declared.table_name: declared.handle for declared in declared_model_specs}
+ model_specs: list[InferenceModelSpec] = []
for model_spec_name, value in remaining_tables.items():
if not isinstance(value, dict):
msg = f"Model spec '{model_spec_name}' for backend '{backend_name}' from {backend_config_source} is not a dictionary"
@@ -519,13 +539,13 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
model_spec_blueprint = InferenceModelSpecBlueprint.model_validate(model_spec_blueprint_dict)
model_spec = InferenceModelSpecFactory.make_inference_model_spec(
backend_name=backend_name,
- name=model_spec_name,
+ name=handles_by_table[model_spec_name],
blueprint=model_spec_blueprint,
backend_listed_constraints=backend_listed_constraints,
backend_valued_constraints=backend_valued_constraints,
extra_headers=key_split.headers,
)
- backend_model_specs[model_spec_name] = model_spec
+ model_specs.append(model_spec)
except ValidationError as validation_error:
validation_error_msg = format_pydantic_validation_error(validation_error)
msg = (
@@ -536,7 +556,16 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
except InferenceModelSpecError as exc:
msg = f"Failed to load inference model spec '{model_spec_name}' for backend '{backend_name}' from {backend_config_source}"
raise InferenceBackendLibraryError(msg, backend_name=backend_name) from exc
- return backend_model_specs
+ # Last, as the document validator checks it: once every table is valid, a duplicate is one the file declares
+ # and never one a value the validation refuses would make.
+ if duplicate := find_duplicate_declared_model(declared_model_specs=declared_model_specs):
+ first, second = duplicate
+ msg = (
+ f"Invalid model specs for backend '{backend_name}' from {backend_config_source}: "
+ f"{describe_duplicate_declared_model(first=first, second=second)}"
+ )
+ raise InferenceBackendLibraryError(msg, backend_name=backend_name)
+ return ModelSpecIndex.make_from_specs(model_specs=model_specs)
def _local_model_specs_the_ledger_can_explain(
self,
@@ -685,11 +714,12 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
does not know is the plugin author's mistake and fatal, as it is in a local file. The table is complete on
its own, so the `[defaults]` of `internal.toml` is not applied to it.
- **A name the installation's `internal.toml` already declares is refused rather than overridden**, naming
- the plugin and the file: which of the two declarations a boot would run would otherwise depend on nothing
- the user can see. Model names are not global across backends — the same name in several backend files is
- the ordinary case, and the routing profile picks one — so a name another backend declares is left to the
- routing, exactly as it is for a model the file declares.
+ **A model the installation's `internal.toml` already declares, under the same name and model type, is refused
+ rather than overridden**, naming the plugin and the file: which of the two declarations a boot would run would
+ otherwise depend on nothing the user can see. A handle names one model per model type, so a plugin may declare
+ another kind of a name the file declares. Model names are not global across backends — the same name in several
+ backend files is the ordinary case, and the routing profile picks one — so a name another backend declares is
+ left to the routing, exactly as it is for a model the file declares.
Returns:
Whether this load has an internal backend. `False` when the installation disables it, or declares none:
@@ -704,19 +734,19 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
if internal_backend is None:
return False
internal_file_path = backend_toml_path(backends_dir_path=backends_dir_path, backend_name=PipelexBackend.INTERNAL)
- merged_model_specs = dict(internal_backend.model_specs)
- for model_name, plugin_model in plugin_model_declarations.internal_models.items():
+ merged_model_specs = ModelSpecIndex.make_from_specs(model_specs=internal_backend.model_specs.all_specs())
+ for plugin_model in plugin_model_declarations.internal_models:
+ model_name = plugin_model.name
if model_name == MODEL_SPEC_DEFAULTS_TABLE:
msg = (
f"Plugin '{plugin_model.plugin}' declares an internal model named '{model_name}', which is the name of a backend "
"file's table of defaults, so it cannot name a model."
)
raise PluginModelDeclarationError(msg, plugin=plugin_model.plugin)
- if model_name in internal_backend.model_specs:
+ if MODEL_SPEC_HANDLE_FIELD in plugin_model.spec:
msg = (
- f"Plugin '{plugin_model.plugin}' declares the internal model '{model_name}', which '{internal_file_path}' declares too. "
- "Remove it from that file, or run `pipelex update` to refresh the file from the kit: the plugin's declaration is the one "
- "that ships with its engine."
+ f"Plugin '{plugin_model.plugin}' declares the internal model '{model_name}' with a '{MODEL_SPEC_HANDLE_FIELD}' key. "
+ "A plugin names its model when it declares it, and may declare one name once per model type: leave the key out."
)
raise PluginModelDeclarationError(msg, plugin=plugin_model.plugin)
try:
@@ -730,29 +760,32 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
except InferenceBackendLibraryError as exc:
msg = f"Plugin '{plugin_model.plugin}' declares the internal model '{model_name}' with a table that is not a valid model spec: {exc}"
raise PluginModelDeclarationError(msg, plugin=plugin_model.plugin) from exc
- merged_model_specs.update(plugin_model_specs)
+ for plugin_model_spec in plugin_model_specs.all_specs():
+ if internal_backend.get_model_spec(model_type=plugin_model_spec.model_type, handle=plugin_model_spec.name) is not None:
+ msg = (
+ f"Plugin '{plugin_model.plugin}' declares the internal model '{model_name}' as "
+ f"{plugin_model_spec.model_type.indefinite_description}, which '{internal_file_path}' declares too. "
+ "Remove it from that file, or run `pipelex update` to refresh the file from the kit: the plugin's declaration is the one "
+ "that ships with its engine."
+ )
+ raise PluginModelDeclarationError(msg, plugin=plugin_model.plugin)
+ # The registrar has already refused two plugins declaring one pair.
+ merged_model_specs.add(plugin_model_spec)
self.root[PipelexBackend.INTERNAL] = internal_backend.model_copy(update={"model_specs": merged_model_specs})
return True
def list_backend_names(self) -> list[str]:
return list(self.root.keys())
- def list_all_model_names(self) -> list[str]:
- """List the names of all models in all backends."""
- all_model_names: set[str] = set()
- for backend in self.root.values():
- all_model_names.update(backend.list_model_names())
- return sorted(all_model_names)
-
- def get_all_models_and_possible_backends(self) -> dict[str, list[str]]:
- """Get a dictionary of all models and their possible backends."""
- all_models_and_possible_backends: dict[str, list[str]] = {}
+ def get_declared_types_by_handle(self) -> dict[str, list[ModelType]]:
+ """Every handle some loaded backend declares, with each model type some backend declares it as."""
+ declared_types_by_handle: dict[str, list[ModelType]] = {}
for backend in self.root.values():
- for model_name in backend.list_model_names():
- if model_name not in all_models_and_possible_backends:
- all_models_and_possible_backends[model_name] = []
- all_models_and_possible_backends[model_name].append(backend.name)
- return all_models_and_possible_backends
+ for model_spec in backend.model_specs.all_specs():
+ declared_types = declared_types_by_handle.setdefault(model_spec.name, [])
+ if model_spec.model_type not in declared_types:
+ declared_types.append(model_spec.model_type)
+ return declared_types_by_handle
def get_inference_backend(self, backend_name: str) -> InferenceBackend | None:
return self.root.get(backend_name)
diff --git a/pipelex/cogt/model_backends/model_lists.py b/pipelex/cogt/model_backends/model_lists.py
index 890deebb0..60c81ff7d 100644
--- a/pipelex/cogt/model_backends/model_lists.py
+++ b/pipelex/cogt/model_backends/model_lists.py
@@ -41,11 +41,12 @@ class ModelLister:
# Group models by SDK
models_by_sdk: dict[str, list[str]] = {}
- for model_name, model_spec in backend.model_specs.items():
+ # One entry per spec: a handle served as two model types counts twice, possibly under two SDKs.
+ for model_spec in backend.model_specs.all_specs():
sdk = model_spec.sdk
if sdk not in models_by_sdk:
models_by_sdk[sdk] = []
- models_by_sdk[sdk].append(model_name)
+ models_by_sdk[sdk].append(model_spec.name)
# Process each SDK separately, dispatching to the lister its backend plugin registered.
# A missing-extra guard lives inside each lister (it raises MissingDependencyError when
diff --git a/pipelex/cogt/model_backends/model_spec_document.py b/pipelex/cogt/model_backends/model_spec_document.py
index b44bf1c11..c30022798 100644
--- a/pipelex/cogt/model_backends/model_spec_document.py
+++ b/pipelex/cogt/model_backends/model_spec_document.py
@@ -21,17 +21,92 @@ on one model is rejected by name as `NOT_HEADER_SHAPED`. That is why one dead ke
breaks a whole backend and why the two halves need two different remedies in the migration ledger.
"""
-from typing import Any, cast
+from collections.abc import Mapping
+from typing import Any, NamedTuple, cast
from pydantic import ValidationError
from pipelex.cogt.model_backends.model_spec_factory import InferenceModelSpecBlueprint
from pipelex.cogt.model_backends.model_spec_keys import describe_rejected_keys, split_model_spec_keys
+from pipelex.cogt.model_backends.model_type import DEFAULT_MODEL_TYPE
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
MODEL_SPEC_DEFAULTS_TABLE = "defaults"
"""The one root table of a backend file that is not a model: what every model in it starts from."""
+MODEL_SPEC_HANDLE_FIELD = "handle"
+"""The model-table key naming the handle a table serves, when it is not the table's own name."""
+
+MODEL_SPEC_TYPE_FIELD = "model_type"
+
+
+class DeclaredModelSpec(NamedTuple):
+ """The model one table of a backend file declares: its table, the handle it serves and its model type, as written."""
+
+ table_name: str
+ handle: str
+ model_type: str
+
+
+def list_declared_model_specs(*, document: Mapping[str, Any]) -> list[DeclaredModelSpec]:
+ """Each model table of a backend document, with the handle it serves and the model type it declares.
+
+ The one statement of the file's shape that every reader shares — the loader, the document validator,
+ the kit's guards and the developer tools — so that none of them can disagree with another on what a
+ file declares. A handle names one model per model type, and the pair is what a file declares:
+
+ - every root table but `[defaults]` is one model;
+ - its handle is its `handle` key, else its table name;
+ - its model type is its own `model_type`, else the one `[defaults]` sets, else `DEFAULT_MODEL_TYPE`.
+
+ Nothing is validated here. A root value that is not a table declares nothing and is skipped, and a
+ value is read as the string a valid file holds: what is wrong with a file is the validation's to say,
+ and every reader that can meet an invalid file validates it.
+ """
+ defaults = document.get(MODEL_SPEC_DEFAULTS_TABLE, {})
+ typed_defaults: Mapping[str, Any] = cast("Mapping[str, Any]", defaults) if isinstance(defaults, Mapping) else {}
+ declared_model_specs: list[DeclaredModelSpec] = []
+ for table_name, value in document.items():
+ if table_name == MODEL_SPEC_DEFAULTS_TABLE or not isinstance(value, Mapping):
+ continue
+ model_table = cast("Mapping[str, Any]", value)
+ handle = model_table.get(MODEL_SPEC_HANDLE_FIELD, table_name)
+ model_type = model_table.get(MODEL_SPEC_TYPE_FIELD, typed_defaults.get(MODEL_SPEC_TYPE_FIELD, DEFAULT_MODEL_TYPE))
+ declared_model_specs.append(DeclaredModelSpec(table_name=table_name, handle=str(handle), model_type=str(model_type)))
+ return declared_model_specs
+
+
+def describe_defaults_not_a_table() -> str:
+ """Why a `defaults` that is not a table is refused, said once for the loader and the document validator."""
+ return f"'{MODEL_SPEC_DEFAULTS_TABLE}' is not a table"
+
+
+def describe_handle_in_defaults() -> str:
+ """Why a `[defaults]` table may not set `handle`, said once for the loader and the document validator."""
+ return (
+ f"'{MODEL_SPEC_HANDLE_FIELD}' is set in [{MODEL_SPEC_DEFAULTS_TABLE}], which every model of the file inherits, so it would "
+ f"give them all one handle: set it on the model table it renames"
+ )
+
+
+def find_duplicate_declared_model(*, declared_model_specs: list[DeclaredModelSpec]) -> tuple[DeclaredModelSpec, DeclaredModelSpec] | None:
+ """The first two tables declaring the same handle as the same model type, or `None` when every pair is declared once."""
+ first_declarations: dict[tuple[str, str], DeclaredModelSpec] = {}
+ for declared in declared_model_specs:
+ pair = (declared.model_type, declared.handle)
+ if first_declaration := first_declarations.get(pair):
+ return first_declaration, declared
+ first_declarations[pair] = declared
+ return None
+
+
+def describe_duplicate_declared_model(*, first: DeclaredModelSpec, second: DeclaredModelSpec) -> str:
+ """Why a file declaring one pair twice is refused, naming both tables."""
+ return (
+ f"models '{first.table_name}' and '{second.table_name}' both declare '{first.handle}' as model type '{first.model_type}': "
+ f"a handle names one model per model type"
+ )
+
class InferenceModelSpecFileNode(InferenceModelSpecBlueprint):
"""One root table of a backend file, on its own — every field optional.
@@ -77,8 +152,10 @@ def describe_model_spec_document_rejection(*, document: dict[str, Any]) -> str |
"""
defaults = document.get(MODEL_SPEC_DEFAULTS_TABLE, {})
if not isinstance(defaults, dict):
- return f"'{MODEL_SPEC_DEFAULTS_TABLE}' is not a table"
+ return describe_defaults_not_a_table()
typed_defaults = cast("dict[str, Any]", defaults)
+ if MODEL_SPEC_HANDLE_FIELD in typed_defaults:
+ return describe_handle_in_defaults()
for model_name, value in document.items():
if model_name == MODEL_SPEC_DEFAULTS_TABLE:
continue
@@ -92,4 +169,7 @@ def describe_model_spec_document_rejection(*, document: dict[str, Any]) -> str |
InferenceModelSpecBlueprint.model_validate({**typed_defaults, **key_split.fields})
except ValidationError as exc:
return f"model '{model_name}': {format_pydantic_validation_error(exc)}"
+ if duplicate := find_duplicate_declared_model(declared_model_specs=list_declared_model_specs(document=document)):
+ first, second = duplicate
+ return describe_duplicate_declared_model(first=first, second=second)
return None
diff --git a/pipelex/cogt/model_backends/model_spec_factory.py b/pipelex/cogt/model_backends/model_spec_factory.py
index 9c33b6520..a00cc012d 100644
--- a/pipelex/cogt/model_backends/model_spec_factory.py
+++ b/pipelex/cogt/model_backends/model_spec_factory.py
@@ -7,7 +7,7 @@ from pipelex.cogt.llm.structured_output import StructureMethod
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.model_backends.model_type import DEFAULT_MODEL_TYPE, ModelType
from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.tools.typing.pydantic_utils import empty_dict_factory_of, empty_list_factory_of
@@ -17,9 +17,13 @@ BackendModelSpecs = dict[str, Any]
class InferenceModelSpecBlueprint(ConfigModel):
enabled: bool = True
+ # The handle this table serves when it is not the table's own name: how one backend file declares a
+ # second model of one handle, of another model type, since TOML forbids declaring a table twice.
+ # Never in `[defaults]`, which every table of the file inherits.
+ handle: str | None = Field(default=None, min_length=1)
sdk: str
variant: str | None = None
- model_type: ModelType = Field(default=ModelType.LLM, strict=False)
+ model_type: ModelType = Field(default=DEFAULT_MODEL_TYPE, strict=False)
model_id: str | None = None
inputs: list[str] = Field(default_factory=list)
outputs: list[str] = Field(default_factory=list)
@@ -86,6 +90,11 @@ class InferenceModelSpecFactory(BaseModel):
backend_valued_constraints: dict[ValuedConstraint, Any],
extra_headers: dict[str, str] | None = None,
) -> InferenceModelSpec:
+ """Build a model spec from its validated blueprint.
+
+ `name` is the handle the spec serves: the table's `handle` key when it sets one, else the table's
+ own name. The model id defaults to it.
+ """
# Merge constraints: backend as base, model-level adds/overrides
# Listed constraints: union of backend + model (model can add, not remove)
merged_listed_constraints = list(set(backend_listed_constraints + blueprint.listed_constraints))
diff --git a/pipelex/cogt/model_backends/model_spec_index.py b/pipelex/cogt/model_backends/model_spec_index.py
new file mode 100644
index 000000000..c9537f595
--- /dev/null
+++ b/pipelex/cogt/model_backends/model_spec_index.py
@@ -0,0 +1,84 @@
+"""The model specs a backend declares, or the deck serves, keyed by model type and then by handle.
+
+A served model is identified by its model type and its handle together: one handle names one model
+per model type, so `gpt-6-luna` may be an LLM and a judgment model at once, and a `PipeLLM` and a
+`PipeJudge` naming it each reach their own spec. Type comes first because every lookup already
+carries one, the pipe's family having fixed it; "which types serve this handle" is the rare question.
+"""
+
+from collections.abc import Iterable
+from typing import Self
+
+from pydantic import Field, RootModel
+
+from pipelex.cogt.exceptions import InferenceModelSpecError
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.tools.typing.pydantic_utils import empty_dict_factory_of
+
+ModelSpecsByTypeAndHandle = dict[ModelType, dict[str, InferenceModelSpec]]
+
+
+class ModelSpecIndex(RootModel[ModelSpecsByTypeAndHandle]):
+ """Model specs keyed by model type, then by handle, the handle being the spec's `name`.
+
+ Both keys are read off the spec itself, so a key can never disagree with the spec it holds. The
+ methods are the questions the readers ask, and the length is the number of specs, not of handles.
+ """
+
+ root: ModelSpecsByTypeAndHandle = Field(default_factory=empty_dict_factory_of(ModelType))
+
+ @classmethod
+ def make_empty(cls) -> Self:
+ return cls(root={})
+
+ @classmethod
+ def make_from_specs(cls, *, model_specs: Iterable[InferenceModelSpec]) -> Self:
+ """An index of these specs.
+
+ Raises:
+ InferenceModelSpecError: Two of the specs share a model type and a handle.
+ """
+ index = cls.make_empty()
+ for model_spec in model_specs:
+ index.add(model_spec)
+ return index
+
+ def add(self, model_spec: InferenceModelSpec) -> None:
+ """Add a spec under its model type and its handle.
+
+ Raises:
+ InferenceModelSpecError: A spec of the same model type and handle is already here.
+ """
+ specs_of_type = self.root.setdefault(model_spec.model_type, {})
+ if existing_spec := specs_of_type.get(model_spec.name):
+ msg = (
+ f"Model '{model_spec.name}' is declared twice as {model_spec.model_type.indefinite_description}, "
+ f"by backend '{existing_spec.backend_name}' and by backend '{model_spec.backend_name}': "
+ f"a handle names one model per model type."
+ )
+ raise InferenceModelSpecError(msg, backend_name=model_spec.backend_name)
+ specs_of_type[model_spec.name] = model_spec
+
+ def get(self, *, model_type: ModelType, handle: str) -> InferenceModelSpec | None:
+ """The spec serving `handle` as `model_type`, or `None`."""
+ return self.root.get(model_type, {}).get(handle)
+
+ def handles_of_type(self, *, model_type: ModelType) -> list[str]:
+ """Every handle served as `model_type`, sorted."""
+ return sorted(self.root.get(model_type, {}))
+
+ def all_specs(self) -> list[InferenceModelSpec]:
+ """Every spec, of every model type."""
+ return [model_spec for specs_of_type in self.root.values() for model_spec in specs_of_type.values()]
+
+ def all_handles(self) -> list[str]:
+ """Every handle served as some model type, each once, sorted."""
+ return sorted({handle for specs_of_type in self.root.values() for handle in specs_of_type})
+
+ def types_serving(self, *, handle: str) -> list[ModelType]:
+ """The model types `handle` is served as, in the order of `ModelType`."""
+ return [model_type for model_type in ModelType if handle in self.root.get(model_type, {})]
+
+ def __len__(self) -> int:
+ return sum(len(specs_of_type) for specs_of_type in self.root.values())
diff --git a/pipelex/cogt/model_backends/model_type.py b/pipelex/cogt/model_backends/model_type.py
index 7e08c5215..cd3673654 100644
--- a/pipelex/cogt/model_backends/model_type.py
+++ b/pipelex/cogt/model_backends/model_type.py
@@ -1,4 +1,5 @@
from enum import StrEnum
+from typing import Final
class ModelType(StrEnum):
@@ -25,3 +26,7 @@ class ModelType(StrEnum):
return "a document-generation engine"
case ModelType.JUDGMENT:
return "a judgment model"
+
+
+DEFAULT_MODEL_TYPE: Final[ModelType] = ModelType.LLM
+"""The model type of a model spec that declares none, in its own table or in its file's `[defaults]`."""
diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.py
index 9ac4adeb4..7e07b4912 100644
--- a/pipelex/cogt/models/model_deck.py
+++ b/pipelex/cogt/models/model_deck.py
@@ -30,6 +30,7 @@ from pipelex.cogt.llm.llm_setting import (
LLMSettingChoicesDefaults,
)
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.exceptions import ModelReferenceParseError
from pipelex.cogt.models.model_reference import ModelReference, ModelReferenceKind, ensure_model_reference
@@ -124,7 +125,8 @@ class ModelDeckBlueprint(ConfigModel):
class ModelDeck(ConfigModel):
model_deck_config: ModelDeckConfig
- inference_models: dict[str, InferenceModelSpec] = Field(default_factory=dict)
+ # The models the deck serves, by model type and handle: a handle names one model per model type.
+ inference_models: ModelSpecIndex = Field(default_factory=ModelSpecIndex.make_empty)
# Track which model_handle fallback warnings have been logged to avoid duplicates
_logged_fallback_warnings: set[str] = PrivateAttr(default_factory=set[str])
@@ -208,13 +210,12 @@ class ModelDeck(ConfigModel):
# Presets are handled separately, not as direct handles
return False
case ModelReferenceKind.HANDLE:
- # Direct handle lookup - check inference_models, aliases, and waterfalls
- all_handles: set[str] = set()
- all_handles.update(self.inference_models.keys())
- all_handles.update(aliases.keys())
- if self.model_deck_config.is_model_fallback_enabled:
- all_handles.update(waterfalls.keys())
- return ref.name in all_handles
+ # Direct handle lookup - check the models served as this type, aliases, and waterfalls
+ if self.inference_models.get(model_type=model_type, handle=ref.name) is not None:
+ return True
+ if ref.name in aliases:
+ return True
+ return self.model_deck_config.is_model_fallback_enabled and ref.name in waterfalls
def _warn_if_ambiguous_llm(self, name: str) -> None:
"""Log a warning if a bare string handle matches presets/aliases/waterfalls."""
@@ -333,7 +334,7 @@ class ModelDeck(ConfigModel):
model_type=model_type,
model_choice=ref.raw,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(self.inference_models.keys()),
+ available_options=self.inference_models.handles_of_type(model_type=model_type),
)
def check_llm_choice(
@@ -392,7 +393,7 @@ class ModelDeck(ConfigModel):
model_type=ModelType.LLM,
model_choice=ref.raw,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(self.inference_models.keys()),
+ available_options=self.inference_models.handles_of_type(model_type=ModelType.LLM),
)
def get_llm_setting(self, llm_choice: LLMModelChoice) -> LLMSetting:
@@ -897,12 +898,12 @@ class ModelDeck(ConfigModel):
pass
def validate_inference_models(self):
- for model_handle, model_spec in self.inference_models.items():
- self.get_required_inference_model(model_handle=model_handle, model_type=model_spec.model_type)
+ for model_spec in self.inference_models.all_specs():
+ self.get_required_inference_model(model_handle=model_spec.name, model_type=model_spec.model_type)
def _get_enabled_backends(self) -> set[str]:
"""Return the set of backend names that have at least one model enabled."""
- return {model.backend_name for model in self.inference_models.values()}
+ return {model.backend_name for model in self.inference_models.all_specs()}
def _resolve_waterfall(
self,
@@ -1010,11 +1011,9 @@ class ModelDeck(ConfigModel):
# Direct handle - proceed with normal lookup
pass
- # For direct handles (HANDLE kind), try inference_models first
- if inference_model := self.inference_models.get(ref.name):
- if inference_model.model_type != model_type:
- log.warning(f"Model handle '{ref.name}' has type '{inference_model.model_type}' but was requested as '{model_type}'. Skipping.")
- return None
+ # For direct handles (HANDLE kind), try the models served as this type first: a handle served as
+ # another type only does not stop the lookup, since this type may have an alias or a waterfall of that name
+ if inference_model := self.inference_models.get(model_type=model_type, handle=ref.name):
return inference_model
# Then try aliases (without prefix)
if alias := aliases.get(ref.name):
@@ -1034,12 +1033,17 @@ class ModelDeck(ConfigModel):
fallback_list=fallback_list,
model_type=model_type,
)
+ if served_types := self.inference_models.types_serving(handle=ref.name):
+ served_types_description = ", ".join(f"'{served_type}'" for served_type in served_types)
+ log.warning(f"Model handle '{ref.name}' is served as {served_types_description} but was requested as '{model_type}'. Skipping.")
+ return None
log.verbose(f"Skipping model handle '{model_handle}' because it's was not found in the model deck, it could be an external plugin.")
return None
def is_handle_defined(self, model_handle: str, *, model_type: ModelType) -> bool:
aliases, waterfalls = self.get_aliases_and_waterfalls_for_type(model_type)
- return model_handle in self.inference_models or model_handle in aliases or model_handle in waterfalls
+ is_served = self.inference_models.get(model_type=model_type, handle=model_handle) is not None
+ return is_served or model_handle in aliases or model_handle in waterfalls
def _is_deck_model_reference(self, *, model_handle: str, model_type: ModelType) -> bool:
"""Whether this deck defines the model reference `model_handle` or names it in one of its own entries.
@@ -1061,10 +1065,8 @@ class ModelDeck(ConfigModel):
if ref is not None:
match ref.kind:
case ModelReferenceKind.HANDLE:
- # A model served as another type is not defined for this lookup, which refuses it.
- served_model = self.inference_models.get(ref.name)
- if served_model is not None and served_model.model_type != model_type:
- served_model = None
+ # A model served as another type only is not defined for this lookup, which refuses it.
+ served_model = self.inference_models.get(model_type=model_type, handle=ref.name)
if served_model is not None or ref.name in aliases or ref.name in waterfalls:
return True
case ModelReferenceKind.ALIAS:
@@ -1134,14 +1136,15 @@ class ModelDeck(ConfigModel):
model_not_found_error = ModelNotFoundError(message=msg, model_handle=model_handle)
if not self._is_deck_model_reference(model_handle=model_handle, model_type=model_type):
# The deck neither defines this reference for this type nor names it anywhere, so the
- # method being run named it, in an inline model setting the load-time check does not
- # look into, or as a model the deck serves as another type, which that check does not
- # tell apart: the caller's fault, which that check reports as
- # `ModelChoiceNotFoundError` for a reference it sees. The message and the next step
- # name nothing but the caller's own reference, as the method wrote it, and the type
- # its pipe asked for, never the type the deck serves it as. A reference the deck
- # itself names but cannot serve (a preset or an alias target on a backend that is not
- # enabled) stays the deployment's fault, and stays redacted, with no next step.
+ # method being run named it. The load-time check refuses every reference it sees that
+ # the deck does not serve as the type its pipe asks for, a model served only as another
+ # type included, as `ModelChoiceNotFoundError`; what is left to reach here is a reference
+ # in an inline model setting, which that check does not look into: the caller's fault.
+ # The message and the next step name nothing but the caller's own reference, as the
+ # method wrote it, and the type its pipe asked for, never the type the deck serves it
+ # as. A reference the deck itself names but cannot serve (a preset or an alias target on
+ # a backend that is not enabled) stays the deployment's fault, and stays redacted, with
+ # no next step.
model_not_found_error.as_caller_fault(
user_action=UserAction(
kind=UserActionKind.CHANGE_MODEL,
@@ -1149,6 +1152,6 @@ class ModelDeck(ConfigModel):
)
)
raise model_not_found_error
- if model_handle not in self.inference_models:
+ if self.inference_models.get(model_type=model_type, handle=model_handle) is None:
log.verbose(f"Model handle '{model_handle}' is an alias which resolves to '{inference_model.name}'")
return inference_model
diff --git a/pipelex/cogt/models/model_deck_check.py b/pipelex/cogt/models/model_deck_check.py
index 0b92ae1cc..cece1469a 100644
--- a/pipelex/cogt/models/model_deck_check.py
+++ b/pipelex/cogt/models/model_deck_check.py
@@ -100,7 +100,7 @@ def check_llm_choice_with_deck(llm_choice: LLMModelChoice) -> None:
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.LLM),
)
@@ -162,7 +162,7 @@ def check_extract_choice_with_deck(extract_choice: ExtractModelChoice) -> None:
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.TEXT_EXTRACTOR),
)
@@ -224,7 +224,7 @@ def check_search_choice_with_deck(search_choice: SearchModelChoice) -> None:
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.SEARCH),
)
@@ -286,7 +286,7 @@ def check_doc_gen_choice_with_deck(*, doc_gen_choice: DocGenModelChoice) -> None
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.DOC_GEN),
)
@@ -348,7 +348,7 @@ def check_judgment_choice_with_deck(judgment_choice: JudgmentModelChoice) -> Non
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.JUDGMENT),
)
@@ -410,5 +410,5 @@ def check_img_gen_choice_with_deck(img_gen_choice: ImgGenModelChoice) -> None:
raw_choice=ref.raw,
name=ref.name,
reference_kind=ModelReferenceKind.HANDLE,
- available_options=list(model_deck.inference_models.keys()),
+ available_options=model_deck.inference_models.handles_of_type(model_type=ModelType.IMG_GEN),
)
diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.py
index 99b25da19..0920f5d44 100644
--- a/pipelex/cogt/models/model_manager.py
+++ b/pipelex/cogt/models/model_manager.py
@@ -17,8 +17,9 @@ from pipelex.cogt.model_backends.backend import InferenceBackend, PipelexBackend
from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
-from pipelex.cogt.model_routing.routing_models import BackendMatchingMethod
+from pipelex.cogt.model_routing.routing_models import BackendMatchForModel, BackendMatchingMethod
from pipelex.cogt.model_routing.routing_profile import RoutingProfile
from pipelex.cogt.model_routing.routing_profile_loader import load_active_routing_profile
from pipelex.cogt.models.model_deck import ModelDeck, ModelDeckBlueprint
@@ -149,10 +150,21 @@ class ModelManager(ModelManagerAbstract):
return self._routing_profile
def build_deck(self, model_deck_blueprint: ModelDeckBlueprint, *, enabled_backends: list[str]) -> ModelDeck:
- all_models_and_possible_backends = self.inference_backend_library.get_all_models_and_possible_backends()
- inference_models: dict[str, InferenceModelSpec] = {}
+ """Build the deck over every pair of a handle and a model type some backend declares.
- for model_name in all_models_and_possible_backends:
+ A routing profile matches a handle's name, so the name is matched once, and the matched
+ backend is then asked for the spec of each type the name is declared as. Each matching method
+ keeps its meaning per pair: a pattern match skips a pair its backend lacks, a default match
+ looks for it along the fallback order, and an exact match pins the name to its backend, so a
+ type that backend does not serve is not served at all.
+
+ Raises:
+ ModelManagerError: A routed backend is not loaded, or a handle routed exactly to a backend
+ that declares it under no model type at all.
+ """
+ inference_models = ModelSpecIndex.make_empty()
+
+ for model_name, declared_model_types in self.inference_backend_library.get_declared_types_by_handle().items():
backend_match_for_model = self.routing_profile.get_backend_match_for_model(
enabled_backends=enabled_backends,
model_name=model_name,
@@ -160,54 +172,28 @@ class ModelManager(ModelManagerAbstract):
if backend_match_for_model is None:
continue
matched_backend_name = backend_match_for_model.backend_name
- backend = self.inference_backend_library.get_inference_backend(backend_name=matched_backend_name)
- if backend is None:
+ matched_backend = self.inference_backend_library.get_inference_backend(backend_name=matched_backend_name)
+ if matched_backend is None:
msg = f"Backend '{matched_backend_name}', requested for model '{model_name}', could not be found"
raise ModelManagerError(msg)
- model_spec = backend.get_model_spec(model_name)
- if model_spec is None:
- # Not finding the model spec can be an error or not according to the matching method
- match backend_match_for_model.matching_method:
- case BackendMatchingMethod.EXACT_MATCH:
- msg = (
- f"Model spec '{model_name}' not found in backend '{matched_backend_name}' "
- f"which was matched exactly in routing profile '{backend_match_for_model.routing_profile_name}'"
- )
- raise ModelManagerError(msg)
- case BackendMatchingMethod.PATTERN_MATCH:
- # We can skip it because it was only a pattern match
- continue
- case BackendMatchingMethod.DEFAULT:
- # We could not find the model spec, but it was a default match,
- # so we can look for it in the other available backends
- # Use fallback_order if specified, otherwise only try internal backend
- if backend_match_for_model.fallback_order:
- # Try fallback_order first, then any enabled backends not in fallback_order
- backends_to_try = backend_match_for_model.fallback_order + [
- b for b in enabled_backends if b not in backend_match_for_model.fallback_order
- ]
- else:
- # No fallback_order specified - only try the internal backend as a special case
- # Internal backend contains software-only models that should always be available
- # regardless of which AI provider routing profile is selected
- backends_to_try = [PipelexBackend.INTERNAL] if PipelexBackend.INTERNAL in enabled_backends else []
-
- for available_backend in backends_to_try:
- if available_backend == matched_backend_name:
- # we've already checked the matched_backend_name and it didn't have the model spec, that's why we're here
- continue
- backend = self.inference_backend_library.get_inference_backend(backend_name=available_backend)
- if backend is None:
- msg = f"Backend '{available_backend}' not found for model '{model_name}'"
- raise ModelManagerError(msg)
- model_spec = backend.get_model_spec(model_name)
- if model_spec is not None:
- break
- if model_spec is None:
- # Model not available in any of the searched backends - skip it
- # Not all models need to be available in the configured backends
- continue
- inference_models[model_name] = model_spec
+ if backend_match_for_model.matching_method == BackendMatchingMethod.EXACT_MATCH and not matched_backend.model_specs.types_serving(
+ handle=model_name
+ ):
+ msg = (
+ f"Model spec '{model_name}' not found in backend '{matched_backend_name}' "
+ f"which was matched exactly in routing profile '{backend_match_for_model.routing_profile_name}'"
+ )
+ raise ModelManagerError(msg)
+ for model_type in declared_model_types:
+ model_spec = self._find_routed_model_spec(
+ model_name=model_name,
+ model_type=model_type,
+ backend_match_for_model=backend_match_for_model,
+ matched_backend=matched_backend,
+ enabled_backends=enabled_backends,
+ )
+ if model_spec is not None:
+ inference_models.add(model_spec)
return ModelDeck(
inference_models=inference_models,
@@ -247,6 +233,66 @@ class ModelManager(ModelManagerAbstract):
model_deck_config=get_config().inference.model_deck,
)
+ def _find_routed_model_spec(
+ self,
+ *,
+ model_name: str,
+ model_type: ModelType,
+ backend_match_for_model: BackendMatchForModel,
+ matched_backend: InferenceBackend,
+ enabled_backends: list[str],
+ ) -> InferenceModelSpec | None:
+ """The spec serving one pair, found by the routing match of its name, or `None` when the pair is not served.
+
+ Raises:
+ ModelManagerError: A backend of the fallback order is not loaded.
+ """
+ if model_spec := matched_backend.get_model_spec(model_type=model_type, handle=model_name):
+ return model_spec
+ # Not finding the pair in the matched backend can be fine or not according to the matching method
+ match backend_match_for_model.matching_method:
+ case BackendMatchingMethod.EXACT_MATCH:
+ # The route pinned the name to this backend, which declares it as another type only
+ # (the caller refused a backend declaring it as no type at all): a call for this pair
+ # never goes elsewhere, so the pair is not served.
+ log.verbose(
+ f"Model '{model_name}' is routed exactly to backend '{matched_backend.name}' in routing profile "
+ f"'{backend_match_for_model.routing_profile_name}', which does not serve it as "
+ f"{model_type.indefinite_description}, so it is not served as one"
+ )
+ return None
+ case BackendMatchingMethod.PATTERN_MATCH:
+ # We can skip it because it was only a pattern match
+ return None
+ case BackendMatchingMethod.DEFAULT:
+ # We could not find the pair, but it was a default match,
+ # so we can look for it in the other available backends
+ # Use fallback_order if specified, otherwise only try internal backend
+ if backend_match_for_model.fallback_order:
+ # Try fallback_order first, then any enabled backends not in fallback_order
+ backends_to_try = backend_match_for_model.fallback_order + [
+ b for b in enabled_backends if b not in backend_match_for_model.fallback_order
+ ]
+ else:
+ # No fallback_order specified - only try the internal backend as a special case
+ # Internal backend contains software-only models that should always be available
+ # regardless of which AI provider routing profile is selected
+ backends_to_try = [PipelexBackend.INTERNAL] if PipelexBackend.INTERNAL in enabled_backends else []
+
+ for available_backend in backends_to_try:
+ if available_backend == matched_backend.name:
+ # we've already checked the matched backend and it didn't have the pair, that's why we're here
+ continue
+ backend = self.inference_backend_library.get_inference_backend(backend_name=available_backend)
+ if backend is None:
+ msg = f"Backend '{available_backend}' not found for model '{model_name}'"
+ raise ModelManagerError(msg)
+ if model_spec := backend.get_model_spec(model_type=model_type, handle=model_name):
+ return model_spec
+ # Pair not available in any of the searched backends - skip it
+ # Not all models need to be available in the configured backends
+ return None
+
@override
def get_inference_model(self, model_handle: str, *, model_type: ModelType) -> InferenceModelSpec:
if self.model_deck is None:
diff --git a/pipelex/cogt/models/model_suggestion.py b/pipelex/cogt/models/model_suggestion.py
index 36bc3735e..fc5a95506 100644
--- a/pipelex/cogt/models/model_suggestion.py
+++ b/pipelex/cogt/models/model_suggestion.py
@@ -81,7 +81,7 @@ def get_collection_keys(
case ModelType.JUDGMENT:
return list(model_deck.judgment_waterfalls.keys())
case ModelReferenceKind.HANDLE:
- return sorted(handle for handle, spec in model_deck.inference_models.items() if spec.model_type == model_type)
+ return model_deck.inference_models.handles_of_type(model_type=model_type)
def suggest_model_alternatives(
diff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.py
index cbfecc118..f2dee4e77 100644
--- a/pipelex/cogt/usage/cost_registry.py
+++ b/pipelex/cogt/usage/cost_registry.py
@@ -58,6 +58,17 @@ def compute_tokens_usage_cost(tokens_usage: TokensUsage) -> float | None:
)
+class ModelUsageKey(NamedTuple):
+ """One model a run called, as its usage is grouped: the kind of call, then the model's name.
+
+ A handle names one model per model type, so one run may call `gpt-6-luna` as an LLM and as a judgment
+ model: grouped by the name alone, the two would merge into one row typed by whichever call came last.
+ """
+
+ model_type: str
+ model_name: str
+
+
class AggregatedCosts(NamedTuple):
"""One run's token usage aggregated for reporting: flat records, per-model groups, and run totals.
@@ -67,8 +78,7 @@ class AggregatedCosts(NamedTuple):
"""
records: list[dict[str, Any]]
- grouped_by_model: dict[str, dict[str, float]]
- model_types: dict[str, str]
+ grouped_by_model: dict[ModelUsageKey, dict[str, float]]
total_cost: float
total_nb_tokens: int
@@ -136,7 +146,6 @@ class CostRegistry(RootModel[CostRegistryRoot]):
"""
records = aggregated.records
grouped_by_model = aggregated.grouped_by_model
- model_types = aggregated.model_types
# Use LLMTokenCostReportField for field names (same string values as ImgGenTokenCostReportField)
report_field = LLMTokenCostReportField
@@ -191,15 +200,15 @@ class CostRegistry(RootModel[CostRegistryRoot]):
table.add_column(f"Total Cost ({scale_str}$)", justify="right", style="bold yellow")
# Add rows for each model
- for model_name, aggregated_data in grouped_by_model.items():
+ for model_key, aggregated_data in grouped_by_model.items():
row_total_cost = cls.compute_total_cost(
input_non_cached_cost=aggregated_data[report_field.COST_INPUT_NON_CACHED],
input_cached_cost=aggregated_data[report_field.COST_INPUT_CACHED],
output_cost=aggregated_data[report_field.COST_OUTPUT],
)
table.add_row(
- model_name,
- model_types.get(model_name, "llm"),
+ model_key.model_name,
+ model_key.model_type,
f"{int(aggregated_data[report_field.NB_TOKENS_INPUT_CACHED]):,}",
f"{int(aggregated_data[report_field.NB_TOKENS_INPUT_NON_CACHED]):,}",
f"{int(aggregated_data[report_field.NB_TOKENS_INPUT_JOINED]):,}",
@@ -270,8 +279,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):
records = cost_registry.to_records()
report_field = LLMTokenCostReportField
- grouped_by_model: dict[str, dict[str, float]] = {}
- model_types: dict[str, str] = {}
+ grouped_by_model: dict[ModelUsageKey, dict[str, float]] = {}
for record in records:
model_name = (
record.get(report_field.LLM_NAME)
@@ -280,9 +288,9 @@ class CostRegistry(RootModel[CostRegistryRoot]):
or record.get(SearchTokenCostReportField.SEARCH_NAME)
or record.get(JudgmentTokenCostReportField.JUDGMENT_NAME, "unknown")
)
- model_types[model_name] = record.get(report_field.MODEL_TYPE, "llm")
- if model_name not in grouped_by_model:
- grouped_by_model[model_name] = {
+ model_key = ModelUsageKey(model_type=record.get(report_field.MODEL_TYPE, "llm"), model_name=model_name)
+ if model_key not in grouped_by_model:
+ grouped_by_model[model_key] = {
report_field.NB_TOKENS_INPUT_CACHED: 0,
report_field.NB_TOKENS_INPUT_NON_CACHED: 0,
report_field.NB_TOKENS_INPUT_JOINED: 0,
@@ -302,7 +310,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):
report_field.COST_INPUT_JOINED,
report_field.COST_OUTPUT,
]:
- grouped_by_model[model_name][field] += record.get(field, 0)
+ grouped_by_model[model_key][field] += record.get(field, 0)
total_cost = cls.compute_total_cost(
input_non_cached_cost=sum(record.get(report_field.COST_INPUT_NON_CACHED, 0) for record in records),
@@ -316,7 +324,6 @@ class CostRegistry(RootModel[CostRegistryRoot]):
return AggregatedCosts(
records=records,
grouped_by_model=grouped_by_model,
- model_types=model_types,
total_cost=total_cost,
total_nb_tokens=total_nb_tokens,
)
@@ -335,7 +342,7 @@ class CostRegistry(RootModel[CostRegistryRoot]):
report_field = LLMTokenCostReportField
by_model: list[dict[str, Any]] = []
- for model_name, aggregated_data in aggregated.grouped_by_model.items():
+ for model_key, aggregated_data in aggregated.grouped_by_model.items():
model_cost = cls.compute_total_cost(
input_non_cached_cost=aggregated_data[report_field.COST_INPUT_NON_CACHED],
input_cached_cost=aggregated_data[report_field.COST_INPUT_CACHED],
@@ -343,8 +350,8 @@ class CostRegistry(RootModel[CostRegistryRoot]):
)
by_model.append(
{
- "model": model_name,
- "model_type": aggregated.model_types.get(model_name, "llm"),
+ "model": model_key.model_name,
+ "model_type": model_key.model_type,
"nb_tokens_input": int(aggregated_data[report_field.NB_TOKENS_INPUT_JOINED]),
"nb_tokens_output": int(aggregated_data[report_field.NB_TOKENS_OUTPUT]),
"cost": model_cost,
diff --git a/pipelex/migration/goldens/inference-backend/[email protected] b/pipelex/migration/goldens/inference-backend/[email protected]
index fdddaf448..92f700f24 100644
--- a/pipelex/migration/goldens/inference-backend/[email protected]
+++ b/pipelex/migration/goldens/inference-backend/[email protected]
@@ -28,6 +28,14 @@
"required": false,
"open_node": false
},
+ "*.handle": {
+ "value_type": "str",
+ "required": false,
+ "open_node": false,
+ "constraints": {
+ "min_length": 1
+ }
+ },
"*.inputs": {
"value_type": "list[str]",
"required": false,
diff --git a/pipelex/pipeline/file_input_consumers.py b/pipelex/pipeline/file_input_consumers.py
index 44ae0fa64..00b6949f5 100644
--- a/pipelex/pipeline/file_input_consumers.py
+++ b/pipelex/pipeline/file_input_consumers.py
@@ -280,8 +280,10 @@ def _resolve_model_specs(*, model_reference: str, model_type: ModelType, visited
case ModelReferenceKind.WATERFALL:
waterfall_members = waterfalls.get(reference.name)
case ModelReferenceKind.HANDLE:
- if model_spec := model_deck.inference_models.get(reference.name):
- return [model_spec] if model_spec.model_type == model_type else []
+ # A handle served as another type only does not stop the lookup: this type's alias or
+ # waterfall of the same name is tried next, as the deck's own lookup does.
+ if model_spec := model_deck.inference_models.get(model_type=model_type, handle=reference.name):
+ return [model_spec]
alias_target = aliases.get(reference.name)
waterfall_members = None if alias_target else waterfalls.get(reference.name)
if alias_target:
diff --git a/pipelex/plugins/exceptions.py b/pipelex/plugins/exceptions.py
index eb46826ee..f9a937e0b 100644
--- a/pipelex/plugins/exceptions.py
+++ b/pipelex/plugins/exceptions.py
@@ -170,14 +170,15 @@ class DuplicateLogSinkError(PluginError):
class DuplicateInternalModelError(PluginError):
- """Two plugins declared a model of the same name in the internal backend."""
+ """Two plugins declared a model of the same name and model type in the internal backend."""
- def __init__(self, *, name: str, first_plugin: str, second_plugin: str):
+ def __init__(self, *, name: str, model_type: str, first_plugin: str, second_plugin: str):
self.name = name
+ self.model_type = model_type
self.first_plugin = first_plugin
self.second_plugin = second_plugin
message = (
- f"Internal model '{name}' is declared by both plugin '{first_plugin}' and plugin '{second_plugin}'. "
+ f"Internal model '{name}' of model type '{model_type}' is declared by both plugin '{first_plugin}' and plugin '{second_plugin}'. "
"Each internal model must be declared by a single plugin."
)
super().__init__(message)
diff --git a/pipelex/plugins/plugin_model_declarations.py b/pipelex/plugins/plugin_model_declarations.py
index f2b4c64eb..d3ef8e352 100644
--- a/pipelex/plugins/plugin_model_declarations.py
+++ b/pipelex/plugins/plugin_model_declarations.py
@@ -10,7 +10,7 @@ declaration as it merges it, and names the plugin when one is refused.
from typing import Any, Self
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource, doc_gen_choice_key
@@ -18,12 +18,15 @@ from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource, doc_
class PluginInternalModel(BaseModel):
"""One model a plugin declares in the internal backend, and the plugin that declared it.
- `spec` is exactly the table a backend file would hold for the model (`model_type`, `sdk`, `model_id`, `inputs`,
- `outputs`, `costs`…). It is complete on its own: no backend file's `[defaults]` table is applied to it.
+ `name` is the model's handle, and `spec` is exactly the table a backend file would hold for the model
+ (`model_type`, `sdk`, `model_id`, `inputs`, `outputs`, `costs`…). It is complete on its own: no backend file's
+ `[defaults]` table is applied to it. A handle names one model per model type, so two declarations may share a
+ name when their model types differ.
"""
model_config = ConfigDict(frozen=True)
+ name: str
spec: dict[str, Any]
plugin: str
@@ -49,8 +52,8 @@ class PluginModelDeclarations(BaseModel):
model_config = ConfigDict(frozen=True)
- internal_models: dict[str, PluginInternalModel] = Field(default_factory=dict)
- """The plugins' internal models, keyed by model name."""
+ internal_models: tuple[PluginInternalModel, ...] = ()
+ """The plugins' internal models, in the order they were declared."""
doc_gen_defaults: tuple[PluginDocGenDefault, ...] = ()
@classmethod
diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.py
index 87f0d7e3a..a434703b2 100644
--- a/pipelex/plugins/registrar.py
+++ b/pipelex/plugins/registrar.py
@@ -8,6 +8,7 @@ from pydantic import BaseModel, Field
from pipelex.base_exceptions import ErrorReport
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource, doc_gen_choice_key
from pipelex.cogt.inference.error_classification import RUNTIME_CLASSIFIED_ERROR_CODES
+from pipelex.cogt.model_backends.model_type import DEFAULT_MODEL_TYPE
from pipelex.plugins.bundle_validator_registry import BundleValidatorProtocol
from pipelex.plugins.exceptions import (
DuplicateBundleValidatorError,
@@ -175,7 +176,8 @@ class PluginRegistrar:
self.log_sinks: dict[str, LogSinkFactoryFn] = {}
self.pipe_func_executors: dict[str, PipeFuncExecutorFactoryFn] = {}
# Plain data, read once by ``make_model_declarations`` for the model manager to merge at boot.
- self.internal_models: dict[str, dict[str, Any]] = {}
+ # Keyed by the model's name and its model type as written, since a handle names one model per model type.
+ self.internal_models: dict[tuple[str, str], dict[str, Any]] = {}
self.doc_gen_defaults: dict[tuple[DocGenFormat, DocGenSource], str] = {}
self.service_error_codes: dict[str, ServiceErrorCode] = {}
# Ordered list (not a type-keyed dict) because the exception types are
@@ -193,7 +195,7 @@ class PluginRegistrar:
self._secrets_provider_sources: dict[str, str] = {}
self._log_sink_sources: dict[str, str] = {}
self._pipe_func_executor_sources: dict[str, str] = {}
- self._internal_model_sources: dict[str, str] = {}
+ self._internal_model_sources: dict[tuple[str, str], str] = {}
self._doc_gen_default_sources: dict[tuple[DocGenFormat, DocGenSource], str] = {}
self._service_error_code_sources: dict[str, str] = {}
self._slot_sources: dict[HubSlot, str] = {}
@@ -358,17 +360,20 @@ class PluginRegistrar:
``internal.toml``, and registers the engine's worker with ``add_inference_backend`` for the same sdk.
Plain data, so it is stored and nothing else: the model manager validates the table when it merges it into
- the internal backend at boot, and refuses one whose name the installation's ``internal.toml`` already
- declares. Fail-loud on a name another plugin declared, naming both plugins.
+ the internal backend at boot, and refuses one whose name and model type the installation's ``internal.toml``
+ already declares. A handle names one model per model type, so a model is identified by its name and by its
+ ``model_type`` as written, the default type when the table sets none. Fail-loud on a model another plugin
+ declared, naming both plugins.
"""
+ model_type = str(spec.get("model_type", DEFAULT_MODEL_TYPE))
self._add(
store=self.internal_models,
sources=self._internal_model_sources,
- key=name,
+ key=(name, model_type),
value=copy.deepcopy(dict(spec)),
- contribution=f"internal model {name}",
+ contribution=f"internal model {name} ({model_type})",
on_duplicate=lambda first_plugin, second_plugin: DuplicateInternalModelError(
- name=name, first_plugin=first_plugin, second_plugin=second_plugin
+ name=name, model_type=model_type, first_plugin=first_plugin, second_plugin=second_plugin
),
)
@@ -506,10 +511,10 @@ class PluginRegistrar:
A fresh value object with deep copies of the tables, so neither the model manager nor a plugin holding on to
the mapping it passed, or a list inside it, can change what the registrar recorded.
"""
- internal_models = {
- name: PluginInternalModel(spec=copy.deepcopy(spec), plugin=self._internal_model_sources[name])
- for name, spec in self.internal_models.items()
- }
+ internal_models = tuple(
+ PluginInternalModel(name=name, spec=copy.deepcopy(spec), plugin=self._internal_model_sources[name, model_type])
+ for (name, model_type), spec in self.internal_models.items()
+ )
doc_gen_defaults = tuple(
PluginDocGenDefault(
doc_gen_format=doc_gen_format, source=source, model=model, plugin=self._doc_gen_default_sources[doc_gen_format, source]
diff --git a/pipelex/tracing/usage_attribution.py b/pipelex/tracing/usage_attribution.py
index f39b8b498..2e22d5792 100644
--- a/pipelex/tracing/usage_attribution.py
+++ b/pipelex/tracing/usage_attribution.py
@@ -22,7 +22,7 @@ from collections.abc import Mapping, Sequence
from pipelex import log
from pipelex.cogt.llm.llm_report import LLMTokenCostReportField
-from pipelex.cogt.usage.cost_registry import AggregatedCosts, CostRegistry
+from pipelex.cogt.usage.cost_registry import AggregatedCosts, CostRegistry, ModelUsageKey
from pipelex.graph.graphspec import ModelUsageSpec, NodeUsageSpec
from pipelex.reporting.reporting_types import AnyTokensUsage
@@ -59,24 +59,25 @@ def _merge_token_categories(usages: Sequence[AnyTokensUsage]) -> dict[str, int]:
def _model_specs(*, usages: Sequence[AnyTokensUsage], aggregated: AggregatedCosts) -> list[ModelUsageSpec]:
"""Per-model breakdown, taken from the cost report's own grouping.
- ``AggregatedCosts.grouped_by_model`` already splits tokens and costs per model name;
- all this adds is the model id, the call counts, and the unrated ``None`` the engine
- cannot express. Ordered most-used first (ties by name) so a consumer can read
- ``by_model[0]`` as the dominant model without sorting.
+ ``AggregatedCosts.grouped_by_model`` already splits tokens and costs per model type
+ and name, since a handle names one model per model type; all this adds is the model
+ id, the call counts, and the unrated ``None`` the engine cannot express. Ordered
+ most-used first (ties by name, then type) so a consumer can read ``by_model[0]`` as
+ the dominant model without sorting.
"""
- calls_by_model: dict[str, int] = {}
- rated_by_model: dict[str, int] = {}
- model_ids: dict[str, str] = {}
+ calls_by_model: dict[ModelUsageKey, int] = {}
+ rated_by_model: dict[ModelUsageKey, int] = {}
+ model_ids: dict[ModelUsageKey, str] = {}
for tokens_usage in usages:
- model_name = tokens_usage.inference_model_name
- calls_by_model[model_name] = calls_by_model.get(model_name, 0) + 1
- model_ids.setdefault(model_name, tokens_usage.inference_model_id)
+ model_key = ModelUsageKey(model_type=tokens_usage.model_type, model_name=tokens_usage.inference_model_name)
+ calls_by_model[model_key] = calls_by_model.get(model_key, 0) + 1
+ model_ids.setdefault(model_key, tokens_usage.inference_model_id)
if tokens_usage.unit_costs:
- rated_by_model[model_name] = rated_by_model.get(model_name, 0) + 1
+ rated_by_model[model_key] = rated_by_model.get(model_key, 0) + 1
specs: list[ModelUsageSpec] = []
- for model_name, model_data in aggregated.grouped_by_model.items():
- rated_calls = rated_by_model.get(model_name, 0)
+ for model_key, model_data in aggregated.grouped_by_model.items():
+ rated_calls = rated_by_model.get(model_key, 0)
model_cost = CostRegistry.compute_total_cost(
input_non_cached_cost=model_data[LLMTokenCostReportField.COST_INPUT_NON_CACHED],
input_cached_cost=model_data[LLMTokenCostReportField.COST_INPUT_CACHED],
@@ -84,15 +85,15 @@ def _model_specs(*, usages: Sequence[AnyTokensUsage], aggregated: AggregatedCost
)
specs.append(
ModelUsageSpec(
- inference_model_name=model_name,
- inference_model_id=model_ids.get(model_name, ""),
- model_type=aggregated.model_types.get(model_name, "llm"),
- inference_calls=calls_by_model.get(model_name, 0),
+ inference_model_name=model_key.model_name,
+ inference_model_id=model_ids.get(model_key, ""),
+ model_type=model_key.model_type,
+ inference_calls=calls_by_model.get(model_key, 0),
rated_inference_calls=rated_calls,
cost=model_cost if rated_calls else None,
)
)
- specs.sort(key=lambda spec: (-spec.inference_calls, spec.inference_model_name, spec.inference_model_id))
+ specs.sort(key=lambda spec: (-spec.inference_calls, spec.inference_model_name, spec.model_type, spec.inference_model_id))
return specs
diff --git a/subject_grants.toml b/subject_grants.toml
index ac94dd903..0d604e9d5 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -1952,10 +1952,6 @@ rationale = "Verb–object: validates/checks the effort_to_level_map; single ope
param = "key"
rationale = "Keyed accessor by the key; single operand."
-["pipelex/cogt/model_backends/backend.py::InferenceBackend.get_model_spec"]
-param = "model_name"
-rationale = "Keyed accessor by the model_name; single operand."
-
["pipelex/cogt/model_backends/backend_credentials.py::BackendCredentialsErrorMsgFactory.make_comprehensive_error_msg"]
param = "backend_credential_reports"
rationale = "Factory from source: builds the result from the backend_credential_reports; single operand."
@@ -1984,6 +1980,10 @@ rationale = "Operates on the backend_name; single operand (qualifiers keyword-on
param = "backend_name"
rationale = "Factory from source: builds the result from the backend_name; single operand."
+["pipelex/cogt/model_backends/model_spec_index.py::ModelSpecIndex.add"]
+param = "model_spec"
+rationale = "add(model_spec) reads as a verb-object sentence: the spec is the one obvious operand"
+
["pipelex/cogt/model_routing/routing_profile_factory.py::RoutingProfileFactory.make_routing_profile"]
param = "name"
rationale = "Factory from source: builds the result from the name; single operand."
diff --git a/tests/integration/pipelex/cogt/test_handle_per_model_type_end_to_end.py b/tests/integration/pipelex/cogt/test_handle_per_model_type_end_to_end.py
new file mode 100644
index 000000000..669db66b6
--- /dev/null
+++ b/tests/integration/pipelex/cogt/test_handle_per_model_type_end_to_end.py
@@ -0,0 +1,168 @@
+"""One handle, one model per model type, end to end: backend files declaring a twin, then a method naming it from both families.
+
+The backends directory is the kit's, plus one backend whose file declares `acme-one` as an LLM and, through
+the `handle` key, as a judgment model. The deck that boot builds stands in for the session's for one test,
+and a method naming `acme-one` from a `PipeLLM` and from a `PipeJudge` loads and runs dry. A dry run calls
+no provider, so what each pipe reached is read from the deck's own lookups, which every resolution goes
+through.
+"""
+
+import shutil
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.models.model_deck import ModelDeck
+from pipelex.cogt.models.model_manager import ModelManager
+from pipelex.kit.paths import get_kit_configs_dir
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.plugins.plugin_model_declarations import PluginModelDeclarations
+from pipelex.runtime_hub import get_models_manager
+from pipelex.system.configuration.config_loader import INFERENCE_DIR_NAME
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+
+TWIN_HANDLE = "acme-one"
+LLM_ONLY_HANDLE = "acme-two"
+
+TWIN_BACKEND_FILE = f"""
+[defaults]
+model_type = "llm"
+sdk = "openai_responses"
+thinking_mode = "none"
+
+["{TWIN_HANDLE}"]
+model_id = "acme-one-chat"
+inputs = ["text"]
+outputs = ["text", "structured"]
+costs = {{ input = 0.1, output = 0.5 }}
+
+["{LLM_ONLY_HANDLE}"]
+inputs = ["text"]
+outputs = ["text"]
+costs = {{ input = 0.1, output = 0.5 }}
+
+["{TWIN_HANDLE}-judgment"]
+handle = "{TWIN_HANDLE}"
+model_type = "judgment"
+sdk = "typesafe"
+model_id = "acme-one-judge"
+inputs = ["text"]
+outputs = ["judgments"]
+costs = {{ input = 0.1, output = 0 }}
+"""
+
+ROUTING_PROFILES = f"""
+active = "twin"
+
+[profiles.twin]
+description = "Route the twin's name to the backend declaring it, and everything else to the internal backend"
+default = "internal"
+routes = {{ "{TWIN_HANDLE}" = "acme", "{LLM_ONLY_HANDLE}" = "acme" }}
+"""
+
+BUNDLE = f"""
+domain = "twin_handle"
+description = "One handle reached by two pipe families"
+main_pipe = "write_then_judge"
+
+[pipe.write_then_judge]
+type = "PipeSequence"
+description = "Write a reply, then judge it"
+inputs = {{ message = "Text" }}
+output = "YesNo"
+steps = [
+ {{ pipe = "write_it", result = "reply" }},
+ {{ pipe = "judge_it", result = "verdict" }},
+]
+
+[pipe.write_it]
+type = "PipeLLM"
+description = "Write a reply"
+inputs = {{ message = "Text" }}
+output = "Text"
+model = "{TWIN_HANDLE}"
+prompt = "Reply to this message: $message"
+
+[pipe.judge_it]
+type = "PipeJudge"
+description = "Judge the reply"
+inputs = {{ reply = "Text" }}
+output = "YesNo"
+model = "{TWIN_HANDLE}"
+question = "Is $reply polite?"
+"""
+
+
+def twin_model_manager(tmp_path: Path) -> ModelManager:
+ """A keyless boot over the kit's inference tree plus a backend whose file declares a twin."""
+ inference_dir = tmp_path / INFERENCE_DIR_NAME
+ shutil.copytree(Path(str(get_kit_configs_dir())) / INFERENCE_DIR_NAME, inference_dir)
+ backends_toml_path = inference_dir / "backends.toml"
+ backends_toml_path.write_text(
+ f'{backends_toml_path.read_text(encoding="utf-8")}\n[acme]\nenabled = true\napi_key = "not-a-real-key"\n', encoding="utf-8"
+ )
+ (inference_dir / "backends" / "acme.toml").write_text(TWIN_BACKEND_FILE, encoding="utf-8")
+ routing_profiles_path = inference_dir / "routing_profiles.toml"
+ routing_profiles_path.write_text(ROUTING_PROFILES, encoding="utf-8")
+
+ model_manager = ModelManager()
+ model_manager.setup(
+ secrets_provider=EnvSecretsProvider(),
+ plugin_model_declarations=PluginModelDeclarations.make_empty(),
+ needs_inference=False,
+ backends_library_paths=[backends_toml_path],
+ backends_dir_path=str(inference_dir / "backends"),
+ routing_profile_library_paths=[routing_profiles_path],
+ deck_dir_path=str(inference_dir / "deck"),
+ )
+ return model_manager
+
+
+def twin_deck_in_session(twin_model_manager: ModelManager, mocker: MockerFixture) -> ModelDeck:
+ """The twin boot's deck and backends, standing in for the session's for one test."""
+ session_model_manager = get_models_manager()
+ mocker.patch.object(session_model_manager, "model_deck", twin_model_manager.get_model_deck())
+ mocker.patch.object(session_model_manager, "inference_backend_library", twin_model_manager.inference_backend_library)
+ return twin_model_manager.get_model_deck()
+
+
+class TestHandlePerModelTypeEndToEnd:
+ def test_the_boot_serves_both_kinds_of_the_twin(self, twin_model_manager: ModelManager) -> None:
+ llm_spec = twin_model_manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM)
+ judgment_spec = twin_model_manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.JUDGMENT)
+
+ assert (llm_spec.backend_name, llm_spec.sdk, llm_spec.model_id) == ("acme", "openai_responses", "acme-one-chat")
+ assert (judgment_spec.backend_name, judgment_spec.sdk, judgment_spec.model_id) == ("acme", "typesafe", "acme-one-judge")
+
+ @pytest.mark.asyncio(loop_scope="function")
+ @pytest.mark.usefixtures("twin_deck_in_session")
+ async def test_a_method_naming_the_twin_from_both_families_runs_dry_each_reaching_its_own_spec(self, mocker: MockerFixture) -> None:
+ lookup_spy = mocker.spy(ModelDeck, "get_optional_inference_model")
+
+ await validate_bundle(mthds_contents=[BUNDLE])
+
+ reached: dict[ModelType, set[str]] = {}
+ for call, returned in zip(lookup_spy.call_args_list, lookup_spy.spy_return_list, strict=True):
+ if call.kwargs.get("model_handle") == TWIN_HANDLE and isinstance(returned, InferenceModelSpec):
+ reached.setdefault(call.kwargs["model_type"], set()).add(returned.model_id)
+ assert reached == {ModelType.LLM: {"acme-one-chat"}, ModelType.JUDGMENT: {"acme-one-judge"}}
+
+ @pytest.mark.asyncio(loop_scope="function")
+ async def test_a_judgment_naming_a_handle_served_only_as_an_llm_is_refused_at_load(self, twin_deck_in_session: ModelDeck) -> None:
+ assert twin_deck_in_session.inference_models.types_serving(handle=LLM_ONLY_HANDLE) == [ModelType.LLM]
+ judging_the_llm_only_model = BUNDLE.replace(f'output = "YesNo"\nmodel = "{TWIN_HANDLE}"', f'output = "YesNo"\nmodel = "{LLM_ONLY_HANDLE}"')
+ assert judging_the_llm_only_model != BUNDLE
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[judging_the_llm_only_model])
+
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert LLM_ONLY_HANDLE in report
+ assert "'unknown_model'" in report
+ assert "'model_type': 'judgment'" in report
diff --git a/tests/integration/pipelex/cogt/test_img_gen_param_support.py b/tests/integration/pipelex/cogt/test_img_gen_param_support.py
index 900db33cb..12c3462db 100644
--- a/tests/integration/pipelex/cogt/test_img_gen_param_support.py
+++ b/tests/integration/pipelex/cogt/test_img_gen_param_support.py
@@ -144,7 +144,9 @@ class TestImgGenParamSupportIntegration:
"""
deck = get_model_deck()
img_gen_models = [
- (handle, spec) for handle, spec in deck.inference_models.items() if spec.model_type == ModelType.IMG_GEN and spec.rules is not None
+ (handle, spec)
+ for handle in deck.inference_models.handles_of_type(model_type=ModelType.IMG_GEN)
+ if (spec := deck.inference_models.get(model_type=ModelType.IMG_GEN, handle=handle)) is not None and spec.rules is not None
]
assert img_gen_models, "Expected at least one img-gen model with rules in the deck"
for handle, spec in img_gen_models:
diff --git a/tests/integration/pipelex/cogt/test_setup_inference_workers.py b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
index 3d83353c8..3ebf02e61 100644
--- a/tests/integration/pipelex/cogt/test_setup_inference_workers.py
+++ b/tests/integration/pipelex/cogt/test_setup_inference_workers.py
@@ -6,7 +6,8 @@ from pipelex.runtime_hub import get_inference_manager, get_model_deck
class TestSetupInferenceWorkers:
def test_setup_inference_workers(self):
inference_manager = get_inference_manager()
- for model_handle, inference_model in get_model_deck().inference_models.items():
+ for inference_model in get_model_deck().inference_models.all_specs():
+ model_handle = inference_model.name
match inference_model.model_type:
case ModelType.LLM:
llm_worker = inference_manager.get_llm_worker(llm_handle=model_handle)
diff --git a/tests/integration/pipelex/pipeline/test_file_input_consumers.py b/tests/integration/pipelex/pipeline/test_file_input_consumers.py
index 11f7f57d4..b8ac60bc4 100644
--- a/tests/integration/pipelex/pipeline/test_file_input_consumers.py
+++ b/tests/integration/pipelex/pipeline/test_file_input_consumers.py
@@ -462,8 +462,9 @@ prompt = "Compare these transcripts: $transcripts, given their summaries: $summa
"""Every input is material to judge, so a document input is sent as a file whether or not the question names it."""
model_deck = get_model_deck()
judgment_model = model_deck.judgment_aliases["default-judgment"]
- booted_spec = model_deck.inference_models[judgment_model]
- mocker.patch.dict(model_deck.inference_models, {judgment_model: booted_spec.model_copy(update={"inputs": ["text", "images", "pdf"]})})
+ judgment_specs = model_deck.inference_models.root[ModelType.JUDGMENT]
+ booted_spec = judgment_specs[judgment_model]
+ mocker.patch.dict(judgment_specs, {judgment_model: booted_spec.model_copy(update={"inputs": ["text", "images", "pdf"]})})
pipes = """
[pipe.main]
type = "PipeJudge"
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py b/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
index 224759c57..dd93a96d1 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
@@ -2,6 +2,7 @@ from collections.abc import Iterator
import pytest
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.runtime_hub import get_model_deck
from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData
@@ -9,13 +10,13 @@ from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJu
@pytest.fixture
def judgment_model_reading_files() -> Iterator[None]:
"""The judgment model, for one test, as a backend that reads images and PDFs would declare it."""
- model_deck = get_model_deck()
- booted_spec = model_deck.inference_models[PipeJudgeLoadTestData.JUDGMENT_MODEL]
- model_deck.inference_models[PipeJudgeLoadTestData.JUDGMENT_MODEL] = booted_spec.model_copy(update={"inputs": ["text", "images", "pdf"]})
+ judgment_specs = get_model_deck().inference_models.root[ModelType.JUDGMENT]
+ booted_spec = judgment_specs[PipeJudgeLoadTestData.JUDGMENT_MODEL]
+ judgment_specs[PipeJudgeLoadTestData.JUDGMENT_MODEL] = booted_spec.model_copy(update={"inputs": ["text", "images", "pdf"]})
try:
yield
finally:
- model_deck.inference_models[PipeJudgeLoadTestData.JUDGMENT_MODEL] = booted_spec
+ judgment_specs[PipeJudgeLoadTestData.JUDGMENT_MODEL] = booted_spec
@pytest.fixture
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
index 46ac03ac4..af93d3544 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
@@ -2,8 +2,10 @@
import pytest
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.runtime_hub import get_model_deck
from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData
_MODEL = f'model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"'
@@ -28,6 +30,22 @@ class TestPipeJudgeLoadRefusals:
assert "jev-9.99.9" in report
assert "model" in report
+ async def test_a_model_the_deck_serves_only_as_an_llm_is_refused(self) -> None:
+ """A handle names one model per model type: an LLM of that name is no judgment model, and the load says so."""
+ served_models = get_model_deck().inference_models
+ llm_only_handle = next(
+ handle
+ for handle in served_models.handles_of_type(model_type=ModelType.LLM)
+ if ModelType.JUDGMENT not in served_models.types_serving(handle=handle)
+ )
+
+ report = await _refusal_report(PipeJudgeLoadTestData.bundle(step_fields=f'model = "{llm_only_handle}"'))
+
+ assert llm_only_handle in report
+ assert "field 'model'" in report
+ assert "'unknown_model'" in report
+ assert "'model_type': 'judgment'" in report
+
@pytest.mark.parametrize(
("inputs", "file_kind"),
[
diff --git a/tests/integration/pipelex/system/test_keyless_boot_model_metadata.py b/tests/integration/pipelex/system/test_keyless_boot_model_metadata.py
index 81be86b04..887d74881 100644
--- a/tests/integration/pipelex/system/test_keyless_boot_model_metadata.py
+++ b/tests/integration/pipelex/system/test_keyless_boot_model_metadata.py
@@ -107,10 +107,10 @@ def _model_manager() -> ModelManager:
return cast("ModelManager", get_models_manager())
-def _deck_models_with_their_constraints(model_deck: ModelDeck) -> dict[str, tuple[str, object, object]]:
+def _deck_models_with_their_constraints(model_deck: ModelDeck) -> dict[tuple[str, str], tuple[str, object, object]]:
return {
- name: (spec.backend_name, sorted(spec.listed_constraints), sorted(spec.valued_constraints.items()))
- for name, spec in model_deck.inference_models.items()
+ (spec.model_type, spec.name): (spec.backend_name, sorted(spec.listed_constraints), sorted(spec.valued_constraints.items()))
+ for spec in model_deck.inference_models.all_specs()
}
diff --git a/tests/unit/pipelex/cli/commands/init/test_recommended_init_backend.py b/tests/unit/pipelex/cli/commands/init/test_recommended_init_backend.py
index b71730c59..4c1ff7b46 100644
--- a/tests/unit/pipelex/cli/commands/init/test_recommended_init_backend.py
+++ b/tests/unit/pipelex/cli/commands/init/test_recommended_init_backend.py
@@ -9,9 +9,13 @@ from pathlib import Path
import pytest
from pipelex.cli.commands.init.ui.backends_ui import RECOMMENDED_INIT_BACKEND
+from pipelex.cogt.model_backends.model_spec_document import list_declared_model_specs
from pipelex.kit.paths import get_kit_configs_dir
from pipelex.tools.misc.toml_utils import load_toml_from_path
+# The model type each deck family asks its models for: a handle names one model per model type.
+FAMILY_MODEL_TYPES = {"llm": "llm", "img_gen": "img_gen"}
+
# The deck families a single provider key serves; search, extraction, judgment and document
# generation default to dedicated backends whichever key the user holds.
DECK_FILE_BY_FAMILY = {
@@ -24,9 +28,10 @@ def _inference_dir() -> Path:
return Path(str(get_kit_configs_dir())) / "inference"
-def _recommended_backend_handles() -> set[str]:
+def _recommended_backend_models() -> set[tuple[str, str]]:
+ """Every model the recommended backend declares, as its model type and its handle, read as the loader reads the file."""
backend_doc = load_toml_from_path(_inference_dir() / "backends" / f"{RECOMMENDED_INIT_BACKEND}.toml")
- return {handle for handle in backend_doc if handle != "defaults"}
+ return {(declared.model_type, declared.handle) for declared in list_declared_model_specs(document=backend_doc)}
class TestRecommendedInitBackend:
@@ -37,8 +42,8 @@ class TestRecommendedInitBackend:
default_targets = {alias: target for alias, target in aliases.items() if alias.startswith("default-")}
assert default_targets, f"the {family} deck declares no default alias"
- served = _recommended_backend_handles()
- unserved = {alias: target for alias, target in default_targets.items() if target not in served}
+ served = _recommended_backend_models()
+ unserved = {alias: target for alias, target in default_targets.items() if (FAMILY_MODEL_TYPES[family], target) not in served}
assert not unserved, f"'{RECOMMENDED_INIT_BACKEND}' serves none of these {family} defaults: {unserved}"
def test_is_a_backend_the_template_lists(self) -> None:
diff --git a/tests/unit/pipelex/cli/test_check_model_cmd.py b/tests/unit/pipelex/cli/test_check_model_cmd.py
index b2455d2fd..156a65401 100644
--- a/tests/unit/pipelex/cli/test_check_model_cmd.py
+++ b/tests/unit/pipelex/cli/test_check_model_cmd.py
@@ -11,6 +11,9 @@ if TYPE_CHECKING:
from pipelex.cli.agent_cli.commands.agent_output import CliOutputFormat
from pipelex.cli.agent_cli.commands.check_model_cmd import agent_check_model_cmd
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_listing import ModelCategory
@@ -25,11 +28,18 @@ class _FakeSetting:
self.description = description
-class _FakeInferenceModelSpec:
- """Minimal stand-in for InferenceModelSpec."""
-
- def __init__(self, model_type: ModelType):
- self.model_type = model_type
+def _llm_spec(name: str) -> InferenceModelSpec:
+ return InferenceModelSpec(
+ backend_name="test_backend",
+ name=name,
+ sdk="test_sdk",
+ model_type=ModelType.LLM,
+ model_id=name,
+ costs={},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
class TestData:
@@ -48,13 +58,9 @@ class TestData:
LLM_WATERFALLS: ClassVar[dict[str, list[str]]] = {
"robust-llm": ["claude-sonnet", "gpt-4o"],
}
- INFERENCE_MODELS: ClassVar[dict[str, _FakeInferenceModelSpec]] = {
- "claude-4.5-sonnet": _FakeInferenceModelSpec(model_type=ModelType.LLM),
- "claude-4-sonnet": _FakeInferenceModelSpec(model_type=ModelType.LLM),
- "claude-4.6-opus": _FakeInferenceModelSpec(model_type=ModelType.LLM),
- "gpt-4o": _FakeInferenceModelSpec(model_type=ModelType.LLM),
- "gpt-4o-mini": _FakeInferenceModelSpec(model_type=ModelType.LLM),
- }
+ INFERENCE_MODELS: ClassVar[ModelSpecIndex] = ModelSpecIndex.make_from_specs(
+ model_specs=[_llm_spec(name) for name in ("claude-4.5-sonnet", "claude-4-sonnet", "claude-4.6-opus", "gpt-4o", "gpt-4o-mini")]
+ )
EXTRACT_PRESETS: ClassVar[dict[str, Any]] = {}
EXTRACT_ALIASES: ClassVar[dict[str, str]] = {}
diff --git a/tests/unit/pipelex/cogt/model_backends/backend_file_handle_key_utils.py b/tests/unit/pipelex/cogt/model_backends/backend_file_handle_key_utils.py
new file mode 100644
index 000000000..7f6ba7817
--- /dev/null
+++ b/tests/unit/pipelex/cogt/model_backends/backend_file_handle_key_utils.py
@@ -0,0 +1,115 @@
+"""Backend files declaring twins through the `handle` key, and the two ways their tests read one: loaded, and as a document."""
+
+from pathlib import Path
+from typing import Any
+
+from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+from tests.helpers.backend_library_loading import load_library
+from tests.unit.pipelex.cogt.model_backends.test_data import BackendLibraryTomls
+
+TWIN_FILE = """
+[defaults]
+model_type = "llm"
+sdk = "openai_responses"
+
+["gpt-6-luna"]
+inputs = ["text"]
+outputs = ["text"]
+
+["gpt-6-luna-judgment"]
+handle = "gpt-6-luna"
+model_type = "judgment"
+sdk = "openai_decisions"
+inputs = ["text"]
+outputs = ["judgments"]
+"""
+
+NON_STRING_HANDLE_FILE = """
+[defaults]
+sdk = "openai_responses"
+
+["1"]
+
+["gpt-6-luna"]
+handle = 1
+"""
+
+EMPTY_HANDLES_FILE = """
+[defaults]
+sdk = "openai_responses"
+
+["gpt-6-luna"]
+handle = ""
+
+["gpt-6-luna-again"]
+handle = ""
+"""
+
+HANDLE_IN_DEFAULTS_FILE = """
+[defaults]
+sdk = "openai_responses"
+handle = "everything"
+
+["gpt-6-luna"]
+"""
+
+# A `defaults` holding a string or a number rather than a table, which no reader may read as a table.
+DEFAULTS_A_STRING_FILE = """
+defaults = "handle-everything"
+
+["gpt-6-luna"]
+sdk = "openai_responses"
+"""
+
+DEFAULTS_A_NUMBER_FILE = """
+defaults = 1
+
+["gpt-6-luna"]
+sdk = "openai_responses"
+"""
+
+DUPLICATE_PAIR_FILE = """
+[defaults]
+sdk = "openai_responses"
+
+["gpt-6-luna"]
+model_type = "judgment"
+
+["gpt-6-luna-again"]
+handle = "gpt-6-luna"
+model_type = "judgment"
+"""
+
+# The table leaves its type to `[defaults]`, the twin sets its own, and a file setting none at all gets the default type.
+TYPE_FROM_DEFAULTS_FILE = """
+[defaults]
+model_type = "judgment"
+sdk = "openai_decisions"
+
+["gpt-6-luna"]
+
+["gpt-6-luna-llm"]
+handle = "gpt-6-luna"
+model_type = "llm"
+sdk = "openai_responses"
+"""
+
+TYPE_FROM_BLUEPRINT_FILE = """
+[defaults]
+sdk = "openai_responses"
+
+["gpt-6-luna"]
+"""
+
+
+def load_backend_file(tmp_path: Path, *, model_specs_toml: str) -> Any:
+ return load_library(
+ tmp_path, backends_toml=BackendLibraryTomls.BACKENDS_TOML, model_specs_toml=model_specs_toml, credentials=CredentialResolution.REQUIRE
+ )
+
+
+def read_backend_document(tmp_path: Path, *, model_specs_toml: str) -> dict[str, Any]:
+ document_path = tmp_path / "document.toml"
+ document_path.write_text(model_specs_toml, encoding="utf-8")
+ return load_toml_from_path(str(document_path))
diff --git a/tests/unit/pipelex/cogt/model_backends/test_backend_file_handle_key.py b/tests/unit/pipelex/cogt/model_backends/test_backend_file_handle_key.py
new file mode 100644
index 000000000..9714433e0
--- /dev/null
+++ b/tests/unit/pipelex/cogt/model_backends/test_backend_file_handle_key.py
@@ -0,0 +1,83 @@
+"""A backend file declares a second model of one handle with the `handle` key.
+
+A root table's name is its handle unless it sets `handle`, and TOML forbids declaring one table twice, so
+`handle` is how one file says that `gpt-6-luna` is an LLM and a judgment model at once.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from pipelex.cogt.exceptions import InferenceBackendLibraryError, InferenceBackendLibraryValidationError
+from pipelex.cogt.model_backends.model_spec_document import (
+ list_declared_model_specs,
+)
+from pipelex.cogt.model_backends.model_type import ModelType
+from tests.helpers.backend_library_loading import required_backend
+from tests.unit.pipelex.cogt.model_backends.backend_file_handle_key_utils import (
+ DUPLICATE_PAIR_FILE,
+ HANDLE_IN_DEFAULTS_FILE,
+ TWIN_FILE,
+ TYPE_FROM_BLUEPRINT_FILE,
+ TYPE_FROM_DEFAULTS_FILE,
+ load_backend_file,
+ read_backend_document,
+)
+
+
+class TestTheHandleKey:
+ def test_a_twin_declared_through_handle_loads_both_specs(self, tmp_path: Path) -> None:
+ backend = required_backend(load_backend_file(tmp_path, model_specs_toml=TWIN_FILE))
+
+ llm_spec = backend.get_model_spec(model_type=ModelType.LLM, handle="gpt-6-luna")
+ judgment_spec = backend.get_model_spec(model_type=ModelType.JUDGMENT, handle="gpt-6-luna")
+ assert llm_spec is not None
+ assert judgment_spec is not None
+ assert llm_spec.sdk == "openai_responses"
+ assert judgment_spec.sdk == "openai_decisions"
+ # The model id defaults to the handle the table serves, never to the table's own name.
+ assert judgment_spec.model_id == "gpt-6-luna"
+ assert backend.model_specs.all_handles() == ["gpt-6-luna"]
+
+ def test_handle_in_defaults_is_refused(self, tmp_path: Path) -> None:
+ with pytest.raises(InferenceBackendLibraryError) as exc_info:
+ load_backend_file(tmp_path, model_specs_toml=HANDLE_IN_DEFAULTS_FILE)
+
+ assert "'handle'" in str(exc_info.value)
+ assert "[defaults]" in str(exc_info.value)
+
+ def test_a_duplicate_pair_is_refused_naming_both_tables(self, tmp_path: Path) -> None:
+ with pytest.raises(InferenceBackendLibraryError) as exc_info:
+ load_backend_file(tmp_path, model_specs_toml=DUPLICATE_PAIR_FILE)
+
+ message = str(exc_info.value)
+ assert "'gpt-6-luna'" in message
+ assert "'gpt-6-luna-again'" in message
+ assert "model type 'judgment'" in message
+
+ def test_a_placeholder_may_not_stand_in_for_a_handle(self, tmp_path: Path) -> None:
+ """A handle describes the model, which a boot without inference must know without resolving anything."""
+ templated_file = TWIN_FILE.replace('handle = "gpt-6-luna"', 'handle = "${LUNA_HANDLE}"')
+
+ with pytest.raises(InferenceBackendLibraryValidationError, match="'handle' references LUNA_HANDLE"):
+ load_backend_file(tmp_path, model_specs_toml=templated_file)
+
+ @pytest.mark.parametrize(
+ ("model_specs_toml", "expected_pairs"),
+ [
+ pytest.param(TYPE_FROM_DEFAULTS_FILE, [(ModelType.LLM, "gpt-6-luna"), (ModelType.JUDGMENT, "gpt-6-luna")], id="table_then_defaults"),
+ pytest.param(TYPE_FROM_BLUEPRINT_FILE, [(ModelType.LLM, "gpt-6-luna")], id="blueprint_default"),
+ ],
+ )
+ def test_the_type_comes_from_the_table_then_the_defaults_then_the_blueprint(
+ self, tmp_path: Path, model_specs_toml: str, expected_pairs: list[tuple[ModelType, str]]
+ ) -> None:
+ backend = required_backend(load_backend_file(tmp_path, model_specs_toml=model_specs_toml))
+ loaded_pairs = sorted((spec.model_type, spec.name) for spec in backend.model_specs.all_specs())
+ declared_pairs = sorted(
+ (ModelType(declared.model_type), declared.handle)
+ for declared in list_declared_model_specs(document=read_backend_document(tmp_path, model_specs_toml=model_specs_toml))
+ )
+
+ assert loaded_pairs == sorted(expected_pairs)
+ assert declared_pairs == loaded_pairs
diff --git a/tests/unit/pipelex/cogt/model_backends/test_backend_file_readers_agree.py b/tests/unit/pipelex/cogt/model_backends/test_backend_file_readers_agree.py
new file mode 100644
index 000000000..e8e4bedce
--- /dev/null
+++ b/tests/unit/pipelex/cogt/model_backends/test_backend_file_readers_agree.py
@@ -0,0 +1,99 @@
+"""Every reader of a backend file agrees on what it declares and whether it is refused.
+
+The loader, the document validator and the declared-model reader the kit's guards use read one file's shape
+through one function, so none of them can disagree with another on a file declaring a twin.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from pipelex.cogt.exceptions import InferenceBackendLibraryError
+from pipelex.cogt.model_backends.model_spec_document import (
+ DeclaredModelSpec,
+ describe_model_spec_document_rejection,
+ list_declared_model_specs,
+)
+from tests.unit.pipelex.cogt.model_backends.backend_file_handle_key_utils import (
+ DEFAULTS_A_NUMBER_FILE,
+ DEFAULTS_A_STRING_FILE,
+ DUPLICATE_PAIR_FILE,
+ EMPTY_HANDLES_FILE,
+ HANDLE_IN_DEFAULTS_FILE,
+ NON_STRING_HANDLE_FILE,
+ TWIN_FILE,
+ TYPE_FROM_BLUEPRINT_FILE,
+ TYPE_FROM_DEFAULTS_FILE,
+ load_backend_file,
+ read_backend_document,
+)
+
+
+class TestEveryReaderAgrees:
+ def test_the_declared_reader_names_each_table_with_its_handle_and_type(self, tmp_path: Path) -> None:
+ declared = list_declared_model_specs(document=read_backend_document(tmp_path, model_specs_toml=TWIN_FILE))
+
+ assert declared == [
+ DeclaredModelSpec(table_name="gpt-6-luna", handle="gpt-6-luna", model_type="llm"),
+ DeclaredModelSpec(table_name="gpt-6-luna-judgment", handle="gpt-6-luna", model_type="judgment"),
+ ]
+
+ @pytest.mark.parametrize(
+ ("model_specs_toml", "is_accepted"),
+ [
+ pytest.param(TWIN_FILE, True, id="twin"),
+ pytest.param(TYPE_FROM_DEFAULTS_FILE, True, id="type_from_defaults"),
+ pytest.param(TYPE_FROM_BLUEPRINT_FILE, True, id="type_from_blueprint"),
+ pytest.param(HANDLE_IN_DEFAULTS_FILE, False, id="handle_in_defaults"),
+ pytest.param(DUPLICATE_PAIR_FILE, False, id="duplicate_pair"),
+ pytest.param(DEFAULTS_A_STRING_FILE, False, id="defaults_a_string"),
+ pytest.param(DEFAULTS_A_NUMBER_FILE, False, id="defaults_a_number"),
+ ],
+ )
+ def test_the_document_validator_agrees_with_the_loader(self, tmp_path: Path, model_specs_toml: str, is_accepted: bool) -> None:
+ rejection = describe_model_spec_document_rejection(document=read_backend_document(tmp_path, model_specs_toml=model_specs_toml))
+ loader_error: InferenceBackendLibraryError | None = None
+ try:
+ load_backend_file(tmp_path, model_specs_toml=model_specs_toml)
+ except InferenceBackendLibraryError as exc:
+ loader_error = exc
+
+ assert (rejection is None) is is_accepted, rejection
+ assert (loader_error is None) is is_accepted, loader_error
+
+ @pytest.mark.parametrize(
+ "model_specs_toml",
+ [
+ pytest.param(NON_STRING_HANDLE_FILE, id="non_string_handle"),
+ pytest.param(EMPTY_HANDLES_FILE, id="empty_handles"),
+ ],
+ )
+ def test_an_invalid_handle_is_refused_for_its_value_and_never_as_a_duplicate(self, tmp_path: Path, model_specs_toml: str) -> None:
+ """A handle the validation refuses can read as another table's handle once coerced: both readers name the bad value instead."""
+ rejection = describe_model_spec_document_rejection(document=read_backend_document(tmp_path, model_specs_toml=model_specs_toml))
+ with pytest.raises(InferenceBackendLibraryError) as exc_info:
+ load_backend_file(tmp_path, model_specs_toml=model_specs_toml)
+
+ assert rejection is not None
+ assert "'gpt-6-luna'" in rejection
+ assert "handle" in rejection
+ assert "both declare" not in rejection
+ assert "'gpt-6-luna'" in str(exc_info.value)
+ assert "handle" in str(exc_info.value)
+ assert "both declare" not in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ "model_specs_toml",
+ [
+ pytest.param(DEFAULTS_A_STRING_FILE, id="defaults_a_string"),
+ pytest.param(DEFAULTS_A_NUMBER_FILE, id="defaults_a_number"),
+ ],
+ )
+ def test_a_defaults_that_is_not_a_table_is_refused_as_such(self, tmp_path: Path, model_specs_toml: str) -> None:
+ """A string `defaults` holding the word "handle" is not a `[defaults]` setting `handle`, and a number is not iterable at all."""
+ rejection = describe_model_spec_document_rejection(document=read_backend_document(tmp_path, model_specs_toml=model_specs_toml))
+ with pytest.raises(InferenceBackendLibraryError) as exc_info:
+ load_backend_file(tmp_path, model_specs_toml=model_specs_toml)
+
+ assert rejection == "'defaults' is not a table"
+ assert "'defaults' is not a table" in str(exc_info.value)
diff --git a/tests/unit/pipelex/cogt/model_backends/test_backend_library_malformed_configuration.py b/tests/unit/pipelex/cogt/model_backends/test_backend_library_malformed_configuration.py
index 2530585a5..74531cef7 100644
--- a/tests/unit/pipelex/cogt/model_backends/test_backend_library_malformed_configuration.py
+++ b/tests/unit/pipelex/cogt/model_backends/test_backend_library_malformed_configuration.py
@@ -17,6 +17,7 @@ from pipelex.cogt.exceptions import (
InferenceBackendLibraryValidationError,
)
from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
+from pipelex.cogt.model_backends.model_type import ModelType
from tests.helpers.backend_library_loading import BOTH_MODES, load_library, required_backend
from tests.helpers.recording_secrets_provider import RecordingSecretsProvider
from tests.unit.pipelex.cogt.model_backends.test_data import LITERAL_FIELD_VAR, BackendLibraryTomls
@@ -140,7 +141,8 @@ class TestMalformedConfigurationIsFatalInBothModes:
credentials=CredentialResolution.REQUIRE,
)
- model_spec = required_backend(library).model_specs["acme-one"]
+ model_spec = required_backend(library).get_model_spec(model_type=ModelType.LLM, handle="acme-one")
+ assert model_spec is not None
assert model_spec.extra_headers == {"x-portkey-provider": "@openai"}
assert model_spec.model_id == "acme-one"
diff --git a/tests/unit/pipelex/cogt/model_backends/test_backend_library_skip_credentials.py b/tests/unit/pipelex/cogt/model_backends/test_backend_library_skip_credentials.py
index c6733f14b..adbc38e1d 100644
--- a/tests/unit/pipelex/cogt/model_backends/test_backend_library_skip_credentials.py
+++ b/tests/unit/pipelex/cogt/model_backends/test_backend_library_skip_credentials.py
@@ -13,6 +13,7 @@ from pytest_mock import MockerFixture
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.credential_resolution import CredentialResolution
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.providers.openai.vertexai_factory import VertexAIFactory
from tests.helpers.backend_library_loading import load_library, required_backend
from tests.helpers.recording_secrets_provider import RecordingSecretsProvider
@@ -34,7 +35,7 @@ class TestSkipKeepsEveryBackendAndResolvesNothing:
)
backend = required_backend(library)
- assert list(backend.model_specs) == ["acme-one"]
+ assert backend.model_specs.all_handles() == ["acme-one"]
assert backend.listed_constraints == [ListedConstraint.TEMPERATURE_UNSUPPORTED]
assert backend.valued_constraints == {ValuedConstraint.FIXED_TEMPERATURE: 1}
assert secrets_provider.looked_up == []
@@ -100,7 +101,9 @@ class TestSkipKeepsEveryBackendAndResolvesNothing:
)
backend = required_backend(library)
- assert backend.model_specs["acme-one"].model_id.startswith("${")
+ model_spec = backend.get_model_spec(model_type=ModelType.LLM, handle="acme-one")
+ assert model_spec is not None
+ assert model_spec.model_id.startswith("${")
assert backend.unresolved_credentials == {"model_specs": expected_vars}
def test_a_missing_backend_credential_keeps_the_backend(self, tmp_path: Path) -> None:
@@ -130,6 +133,6 @@ class TestSkipKeepsEveryBackendAndResolvesNothing:
make_endpoint_and_api_key.assert_not_called()
assert backend.endpoint is None
assert backend.api_key is None
- assert list(backend.model_specs) == ["gemini-one"]
+ assert backend.model_specs.all_handles() == ["gemini-one"]
# No variable names this token: it is minted from the service-account file, so the field alone is recorded.
assert backend.unresolved_credentials == {"endpoint": [], "api_key": []}
diff --git a/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py b/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py
index b61468de5..db3069e65 100644
--- a/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py
+++ b/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py
@@ -7,28 +7,46 @@ across SDKs, treats a registry miss and the soft ``ModelListingUnsupportedError`
from __future__ import annotations
from types import SimpleNamespace
-from typing import TYPE_CHECKING, cast
+from typing import TYPE_CHECKING
import pytest
from pipelex.cli.exceptions import PipelexCLIError
from pipelex.cogt.exceptions import ModelListingUnsupportedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.model_lists import ModelLister
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.plugins.model_lister_registry import ListModelsFn, ModelListerRegistry
if TYPE_CHECKING:
from pytest_mock import MockerFixture
- from pipelex.cogt.model_backends.backend import InferenceBackend
-
_MODULE = "pipelex.cogt.model_backends.model_lists"
def _fake_backend(model_specs: dict[str, str]) -> InferenceBackend:
- # The loop only reads ``.model_specs`` and each spec's ``.sdk``.
- return cast(
- "InferenceBackend",
- SimpleNamespace(model_specs={name: SimpleNamespace(sdk=sdk) for name, sdk in model_specs.items()}),
+ # The loop only reads ``.model_specs`` and each spec's ``.name`` and ``.sdk``.
+ return InferenceBackend(
+ name="acme",
+ model_specs=ModelSpecIndex.make_from_specs(
+ model_specs=[
+ InferenceModelSpec(
+ backend_name="acme",
+ name=name,
+ sdk=sdk,
+ model_type=ModelType.LLM,
+ model_id=name,
+ costs={},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+ for name, sdk in model_specs.items()
+ ]
+ ),
)
diff --git a/tests/unit/pipelex/cogt/model_backends/test_model_spec_index.py b/tests/unit/pipelex/cogt/model_backends/test_model_spec_index.py
new file mode 100644
index 000000000..06769c28a
--- /dev/null
+++ b/tests/unit/pipelex/cogt/model_backends/test_model_spec_index.py
@@ -0,0 +1,75 @@
+"""A handle names one model per model type: the index of specs keyed by type, then by handle."""
+
+import pytest
+
+from pipelex.cogt.exceptions import InferenceModelSpecError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
+from pipelex.cogt.model_backends.model_type import ModelType
+
+
+def _spec(*, name: str, model_type: ModelType, backend_name: str = "openai") -> InferenceModelSpec:
+ return InferenceModelSpec(
+ backend_name=backend_name,
+ name=name,
+ sdk="test_sdk",
+ model_type=model_type,
+ model_id=name,
+ costs={},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+
+
+class TestModelSpecIndex:
+ def test_one_handle_holds_one_spec_per_model_type(self) -> None:
+ llm_spec = _spec(name="gpt-6-luna", model_type=ModelType.LLM)
+ judgment_spec = _spec(name="gpt-6-luna", model_type=ModelType.JUDGMENT, backend_name="openai_decisions")
+
+ index = ModelSpecIndex.make_from_specs(model_specs=[llm_spec, judgment_spec])
+
+ assert index.get(model_type=ModelType.LLM, handle="gpt-6-luna") is llm_spec
+ assert index.get(model_type=ModelType.JUDGMENT, handle="gpt-6-luna") is judgment_spec
+ assert index.get(model_type=ModelType.SEARCH, handle="gpt-6-luna") is None
+ assert index.types_serving(handle="gpt-6-luna") == [ModelType.LLM, ModelType.JUDGMENT]
+ assert len(index) == 2
+ assert index.all_handles() == ["gpt-6-luna"]
+
+ def test_the_same_pair_twice_is_refused_naming_both_backends(self) -> None:
+ index = ModelSpecIndex.make_from_specs(model_specs=[_spec(name="gpt-6-luna", model_type=ModelType.JUDGMENT, backend_name="first")])
+
+ with pytest.raises(InferenceModelSpecError) as exc_info:
+ index.add(_spec(name="gpt-6-luna", model_type=ModelType.JUDGMENT, backend_name="second"))
+
+ message = str(exc_info.value)
+ assert "'gpt-6-luna'" in message
+ assert "a judgment model" in message
+ assert "'first'" in message
+ assert "'second'" in message
+
+ def test_the_questions_of_one_type_see_that_type_only(self) -> None:
+ index = ModelSpecIndex.make_from_specs(
+ model_specs=[
+ _spec(name="gpt-6-luna", model_type=ModelType.LLM),
+ _spec(name="claude", model_type=ModelType.LLM),
+ _spec(name="gpt-6-luna", model_type=ModelType.JUDGMENT),
+ _spec(name="verdict", model_type=ModelType.JUDGMENT),
+ ]
+ )
+
+ assert index.handles_of_type(model_type=ModelType.LLM) == ["claude", "gpt-6-luna"]
+ assert index.handles_of_type(model_type=ModelType.JUDGMENT) == ["gpt-6-luna", "verdict"]
+ assert index.handles_of_type(model_type=ModelType.SEARCH) == []
+ assert sorted((spec.model_type, spec.name) for spec in index.all_specs()) == [
+ (ModelType.JUDGMENT, "gpt-6-luna"),
+ (ModelType.JUDGMENT, "verdict"),
+ (ModelType.LLM, "claude"),
+ (ModelType.LLM, "gpt-6-luna"),
+ ]
+
+ def test_an_empty_index_is_falsy(self) -> None:
+ """Readers test a backend's specs for emptiness, so the length must count the specs."""
+ assert not ModelSpecIndex.make_empty()
+ assert ModelSpecIndex.make_from_specs(model_specs=[_spec(name="x", model_type=ModelType.SEARCH)])
diff --git a/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py b/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py
new file mode 100644
index 000000000..d89674a13
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py
@@ -0,0 +1,73 @@
+"""Shared builders for the per-pair deck tests: in-memory backends, a deck blueprint naming the twin, and a deck built from them."""
+
+from pipelex.cogt.extract.extract_setting import ExtractModelChoice
+from pipelex.cogt.img_gen.img_gen_job_components import Quality
+from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaults
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.backend import InferenceBackend
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from pipelex.cogt.models.model_deck import (
+ ExtractDeckBlueprint,
+ ImgGenDeckBlueprint,
+ JudgmentDeckBlueprint,
+ LLMDeckBlueprint,
+ ModelDeckBlueprint,
+ SearchDeckBlueprint,
+)
+from pipelex.cogt.models.model_manager import ModelManager
+
+TWIN_HANDLE = "gpt-6-luna"
+GET_MODEL_DECK_TARGET = "pipelex.cogt.models.model_deck_check.get_model_deck"
+
+
+def make_spec(*, name: str, model_type: ModelType, backend_name: str) -> InferenceModelSpec:
+ return InferenceModelSpec(
+ backend_name=backend_name,
+ name=name,
+ sdk=f"{backend_name}_sdk",
+ model_type=model_type,
+ model_id=name,
+ costs={},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+
+
+def make_backend(name: str, *served: tuple[str, ModelType]) -> InferenceBackend:
+ return InferenceBackend(
+ name=name,
+ model_specs=ModelSpecIndex.make_from_specs(
+ model_specs=[make_spec(name=handle, model_type=model_type, backend_name=name) for handle, model_type in served]
+ ),
+ )
+
+
+def make_deck_blueprint(*, extract_choice_default: ExtractModelChoice = "extractor") -> ModelDeckBlueprint:
+ return ModelDeckBlueprint(
+ llm=LLMDeckBlueprint(
+ choice_defaults=LLMSettingChoicesDefaults(
+ default_temperature=0.7,
+ for_text=LLMSetting(model=TWIN_HANDLE, temperature=0.7),
+ for_object=LLMSetting(model=TWIN_HANDLE, temperature=0.1),
+ ),
+ ),
+ extract=ExtractDeckBlueprint(choice_default=extract_choice_default),
+ img_gen=ImgGenDeckBlueprint(default_quality=Quality.MEDIUM, choice_default="painter"),
+ search=SearchDeckBlueprint(choice_default="searcher"),
+ judgment=JudgmentDeckBlueprint(aliases={"fast": "verdict"}),
+ )
+
+
+def build_deck_manager(*backends: InferenceBackend, routing_profile: RoutingProfile) -> ModelManager:
+ manager = ModelManager()
+ manager.inference_backend_library.root = {backend.name: backend for backend in backends}
+ manager._routing_profile = routing_profile # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+ manager.model_deck = manager.build_deck(
+ model_deck_blueprint=make_deck_blueprint(),
+ enabled_backends=[backend.name for backend in backends],
+ )
+ return manager
diff --git a/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py b/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
index 6746b745b..b9f947868 100644
--- a/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
+++ b/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
@@ -12,7 +12,7 @@ def find_invalid_alias_targets(
aliases: dict[str, str],
all_aliases: dict[str, str],
all_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
expected_model_type: ModelType,
) -> list[tuple[str, str, str]]:
"""Find aliases that reference invalid targets.
@@ -21,7 +21,7 @@ def find_invalid_alias_targets(
aliases: The aliases dict to validate
all_aliases: All available aliases for this model type
all_waterfalls: All available waterfalls for this model type
- known_model_handles: Mapping of model handle -> ModelType
+ known_model_handles: Mapping of model handle -> the model types the backends declare it as
expected_model_type: The expected model type for this deck (LLM, TEXT_EXTRACTOR, IMG_GEN)
Returns:
@@ -44,10 +44,10 @@ def find_invalid_alias_targets(
case ModelReferenceKind.HANDLE:
if ref.name not in known_model_handles:
invalid_refs.append((alias_name, target_value, f"model handle '{ref.name}' not found in backends"))
- elif known_model_handles[ref.name] != expected_model_type:
- actual_type = known_model_handles[ref.name]
+ elif expected_model_type not in known_model_handles[ref.name]:
+ actual_types = ", ".join(f"'{actual_type}'" for actual_type in sorted(known_model_handles[ref.name]))
invalid_refs.append(
- (alias_name, target_value, f"model handle '{ref.name}' has type '{actual_type}' but expected '{expected_model_type}'")
+ (alias_name, target_value, f"model handle '{ref.name}' has type {actual_types} but expected '{expected_model_type}'")
)
return invalid_refs
@@ -56,7 +56,7 @@ def find_invalid_alias_targets(
def find_invalid_waterfall_entries(
waterfalls: dict[str, list[str]],
all_aliases: dict[str, str],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
expected_model_type: ModelType,
) -> list[tuple[str, int, str, str]]:
"""Find waterfall entries that reference invalid targets.
@@ -64,7 +64,7 @@ def find_invalid_waterfall_entries(
Args:
waterfalls: The waterfalls dict to validate
all_aliases: All available aliases for this model type
- known_model_handles: Mapping of model handle -> ModelType
+ known_model_handles: Mapping of model handle -> the model types the backends declare it as
expected_model_type: The expected model type for this deck (LLM, TEXT_EXTRACTOR, IMG_GEN)
Returns:
@@ -92,14 +92,14 @@ def find_invalid_waterfall_entries(
case ModelReferenceKind.HANDLE:
if ref.name not in known_model_handles:
invalid_refs.append((waterfall_name, index, entry_value, f"model handle '{ref.name}' not found in backends"))
- elif known_model_handles[ref.name] != expected_model_type:
- actual_type = known_model_handles[ref.name]
+ elif expected_model_type not in known_model_handles[ref.name]:
+ actual_types = ", ".join(f"'{actual_type}'" for actual_type in sorted(known_model_handles[ref.name]))
invalid_refs.append(
(
waterfall_name,
index,
entry_value,
- f"model handle '{ref.name}' has type '{actual_type}' but expected '{expected_model_type}'",
+ f"model handle '{ref.name}' has type {actual_types} but expected '{expected_model_type}'",
)
)
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck.py b/tests/unit/pipelex/cogt/models/test_model_deck.py
index 1449b8973..6bc8f3ce0 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck.py
@@ -6,6 +6,7 @@ from pipelex.cogt.img_gen.img_gen_job_components import Quality
from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaults
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import ModelDeck
from pipelex.cogt.usage.cost_category import CostCategory
@@ -28,13 +29,13 @@ class TestModelDeckGetOptionalInferenceModel:
def _create_test_model_deck(
self,
- inference_models: dict[str, InferenceModelSpec] | None = None,
+ inference_models: list[InferenceModelSpec] | None = None,
llm_aliases: dict[str, str] | None = None,
llm_waterfalls: dict[str, list[str]] | None = None,
is_model_fallback_enabled: bool = False,
) -> ModelDeck:
return ModelDeck(
- inference_models=inference_models or {},
+ inference_models=ModelSpecIndex.make_from_specs(model_specs=inference_models or []),
# LLM-specific
llm_default_temperature=0.7,
llm_aliases=llm_aliases or {},
@@ -64,7 +65,7 @@ class TestModelDeckGetOptionalInferenceModel:
def test_direct_model_lookup_success(self):
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
- model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec})
+ model_deck = self._create_test_model_deck(inference_models=[model_spec])
# Act
result = model_deck.get_optional_inference_model("gpt-4", model_type=ModelType.LLM)
@@ -85,7 +86,7 @@ class TestModelDeckGetOptionalInferenceModel:
def test_simple_string_alias_resolution_success(self):
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
- model_deck = self._create_test_model_deck(inference_models={"gpt-4": model_spec}, llm_aliases={"best-gpt": "gpt-4"})
+ model_deck = self._create_test_model_deck(inference_models=[model_spec], llm_aliases={"best-gpt": "gpt-4"})
# Act
result = model_deck.get_optional_inference_model("best-gpt", model_type=ModelType.LLM)
@@ -107,7 +108,7 @@ class TestModelDeckGetOptionalInferenceModel:
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_waterfalls={"dummy-model-handle": ["gpt-4", "claude-3"]},
is_model_fallback_enabled=True,
)
@@ -122,7 +123,7 @@ class TestModelDeckGetOptionalInferenceModel:
# Arrange
model_spec = self._create_test_model_spec("claude-3")
model_deck = self._create_test_model_deck(
- inference_models={"claude-3": model_spec},
+ inference_models=[model_spec],
llm_waterfalls={"dummy-model-handle": ["nonexistent-model", "claude-3"]},
is_model_fallback_enabled=True,
)
@@ -150,9 +151,7 @@ class TestModelDeckGetOptionalInferenceModel:
def test_recursive_alias_resolution_success(self):
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
- model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec}, llm_aliases={"dummy-model-handle": "best-gpt", "best-gpt": "gpt-4"}
- )
+ model_deck = self._create_test_model_deck(inference_models=[model_spec], llm_aliases={"dummy-model-handle": "best-gpt", "best-gpt": "gpt-4"})
# Act
result = model_deck.get_optional_inference_model("dummy-model-handle", model_type=ModelType.LLM)
@@ -164,7 +163,7 @@ class TestModelDeckGetOptionalInferenceModel:
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_aliases={"best-gpt": "gpt-4"},
llm_waterfalls={"dummy-model-handle": ["nonexistent", "best-gpt"]},
is_model_fallback_enabled=True,
@@ -200,10 +199,8 @@ class TestModelDeckGetOptionalInferenceModel:
# Assert - cycle detection returns None instead of causing RecursionError
assert result is None
- def test_model_type_mismatch_returns_none(self):
- """Test that requesting a model with wrong model_type returns None."""
- # Arrange - create a TEXT_EXTRACTOR model
- extractor_spec = InferenceModelSpec(
+ def _create_extractor_spec(self) -> InferenceModelSpec:
+ return InferenceModelSpec(
backend_name="test_backend",
name="mistral-extractor",
sdk="test_sdk",
@@ -214,19 +211,29 @@ class TestModelDeckGetOptionalInferenceModel:
max_tokens=1000,
max_prompt_images=None,
)
- model_deck = self._create_test_model_deck(inference_models={"mistral-extractor": extractor_spec})
- # Act - request it as LLM
- result = model_deck.get_optional_inference_model("mistral-extractor", model_type=ModelType.LLM)
+ def test_a_handle_served_as_another_type_only_is_not_served_as_this_one(self):
+ """A handle names one model per model type: an extractor is no LLM, whatever its name."""
+ model_deck = self._create_test_model_deck(inference_models=[self._create_extractor_spec()])
- # Assert - should return None due to model_type mismatch
- assert result is None
+ assert model_deck.get_optional_inference_model("mistral-extractor", model_type=ModelType.LLM) is None
+ assert model_deck.get_optional_inference_model("mistral-extractor", model_type=ModelType.TEXT_EXTRACTOR) is not None
+
+ def test_a_handle_served_as_another_type_does_not_shadow_this_types_alias(self):
+ """The LLM half of the deck may name an alias like a handle the deck serves as an extractor: the alias resolves."""
+ gpt_spec = self._create_test_model_spec("gpt-4")
+ model_deck = self._create_test_model_deck(
+ inference_models=[self._create_extractor_spec(), gpt_spec],
+ llm_aliases={"mistral-extractor": "gpt-4"},
+ )
+
+ assert model_deck.get_optional_inference_model("mistral-extractor", model_type=ModelType.LLM) == gpt_spec
def test_complex_waterfall_scenario(self):
# Arrange
model_spec = self._create_test_model_spec("claude-3")
model_deck = self._create_test_model_deck(
- inference_models={"claude-3": model_spec},
+ inference_models=[model_spec],
llm_aliases={"premium-claude": "claude-3"},
llm_waterfalls={"dummy-model-handle": ["premium-gpt", "premium-claude"]},
is_model_fallback_enabled=True,
@@ -243,7 +250,7 @@ class TestModelDeckGetOptionalInferenceModel:
model_spec1 = self._create_test_model_spec("gpt-4")
model_spec2 = self._create_test_model_spec("claude-3")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec1, "claude-3": model_spec2},
+ inference_models=[model_spec1, model_spec2],
llm_aliases={
"ai-model": "best-gpt", # string alias
},
@@ -304,14 +311,14 @@ class TestModelDeckPrefixedAliasReferences:
def _create_test_model_deck(
self,
- inference_models: dict[str, InferenceModelSpec] | None = None,
+ inference_models: list[InferenceModelSpec] | None = None,
llm_aliases: dict[str, str] | None = None,
llm_waterfalls: dict[str, list[str]] | None = None,
llm_presets: dict[str, LLMSetting] | None = None,
is_model_fallback_enabled: bool = False,
) -> ModelDeck:
return ModelDeck(
- inference_models=inference_models or {},
+ inference_models=ModelSpecIndex.make_from_specs(model_specs=inference_models or []),
# LLM-specific
llm_default_temperature=0.7,
llm_aliases=llm_aliases or {},
@@ -343,7 +350,7 @@ class TestModelDeckPrefixedAliasReferences:
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_aliases={"best-gpt": "gpt-4"},
)
@@ -358,7 +365,7 @@ class TestModelDeckPrefixedAliasReferences:
# Arrange
model_spec = self._create_test_model_spec("gpt-4")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_aliases={"best-gpt": "gpt-4"},
)
@@ -374,7 +381,7 @@ class TestModelDeckPrefixedAliasReferences:
# Arrange
model_spec = self._create_test_model_spec("gpt-4-mini")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4-mini": model_spec},
+ inference_models=[model_spec],
llm_waterfalls={"small-llm": ["gpt-4-mini", "claude-instant"]},
is_model_fallback_enabled=True,
)
@@ -390,7 +397,7 @@ class TestModelDeckPrefixedAliasReferences:
# Arrange
model_spec = self._create_test_model_spec("gpt-4-mini")
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4-mini": model_spec},
+ inference_models=[model_spec],
llm_waterfalls={"small-llm": ["gpt-4-mini", "claude-instant"]},
is_model_fallback_enabled=True,
)
@@ -412,7 +419,7 @@ class TestModelDeckPrefixedAliasReferences:
model_spec = self._create_test_model_spec("gpt-4")
preset_with_alias = LLMSetting(model="@best-gpt", temperature=0.5)
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_aliases={"best-gpt": "gpt-4"},
llm_presets={"my-preset": preset_with_alias},
)
@@ -426,7 +433,7 @@ class TestModelDeckPrefixedAliasReferences:
model_spec = self._create_test_model_spec("claude-4.5-opus")
preset_with_alias = LLMSetting(model="@default-premium", temperature=0.1)
model_deck = self._create_test_model_deck(
- inference_models={"claude-4.5-opus": model_spec},
+ inference_models=[model_spec],
llm_aliases={
"default-premium": "claude-4.5-opus", # this is what @default-premium should resolve to
},
@@ -459,12 +466,12 @@ class TestModelDeckGetLLMSettingWithPresets:
def _create_test_model_deck(
self,
- inference_models: dict[str, InferenceModelSpec] | None = None,
+ inference_models: list[InferenceModelSpec] | None = None,
llm_aliases: dict[str, str] | None = None,
llm_presets: dict[str, LLMSetting] | None = None,
) -> ModelDeck:
return ModelDeck(
- inference_models=inference_models or {},
+ inference_models=ModelSpecIndex.make_from_specs(model_specs=inference_models or []),
# LLM-specific
llm_default_temperature=0.7,
llm_aliases=llm_aliases or {},
@@ -497,7 +504,7 @@ class TestModelDeckGetLLMSettingWithPresets:
model_spec = self._create_test_model_spec("gpt-4")
preset = LLMSetting(model="gpt-4", temperature=0.5, max_tokens=500)
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_presets={"testing-text": preset},
)
@@ -520,7 +527,7 @@ class TestModelDeckGetLLMSettingWithPresets:
model_spec = self._create_test_model_spec("gpt-4")
preset = LLMSetting(model="gpt-4", temperature=0.5, max_tokens=500)
model_deck = self._create_test_model_deck(
- inference_models={"gpt-4": model_spec},
+ inference_models=[model_spec],
llm_presets={"testing-text": preset},
)
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_build_per_pair.py b/tests/unit/pipelex/cogt/models/test_model_deck_build_per_pair.py
new file mode 100644
index 000000000..b82c98401
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_build_per_pair.py
@@ -0,0 +1,82 @@
+"""One handle, one model per model type: the deck build routes each pair of a name, by the rule its route matched.
+
+The backends are built in memory, so these tests say nothing of how a backend file declares a twin.
+"""
+
+import pytest
+
+from pipelex.cogt.exceptions import ModelManagerError
+from pipelex.cogt.model_backends.backend import PipelexBackend
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from tests.unit.pipelex.cogt.models.model_deck_per_pair_utils import TWIN_HANDLE, build_deck_manager, make_backend
+
+
+class TestDeckBuildPerPair:
+ def test_a_twin_in_one_backend_is_served_whole(self) -> None:
+ manager = build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM), (TWIN_HANDLE, ModelType.JUDGMENT)),
+ routing_profile=RoutingProfile(name="profile", default="openai"),
+ )
+
+ llm_spec = manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM)
+ judgment_spec = manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.JUDGMENT)
+ assert llm_spec.model_type == ModelType.LLM
+ assert judgment_spec.model_type == ModelType.JUDGMENT
+ assert llm_spec is not judgment_spec
+
+ def test_a_twin_split_across_backends_is_served_whole_under_a_default_match_with_a_fallback_order(self) -> None:
+ """The primary backend lacks the judgment kind, so the pair is looked for along the fallback order."""
+ manager = build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),
+ make_backend("openai_decisions", (TWIN_HANDLE, ModelType.JUDGMENT)),
+ routing_profile=RoutingProfile(name="profile", default="openai", fallback_order=["openai", "openai_decisions"]),
+ )
+
+ assert manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM).backend_name == "openai"
+ assert manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.JUDGMENT).backend_name == "openai_decisions"
+
+ def test_a_default_match_without_a_fallback_order_looks_in_the_internal_backend_per_pair(self) -> None:
+ manager = build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),
+ make_backend(PipelexBackend.INTERNAL, (TWIN_HANDLE, ModelType.DOC_GEN)),
+ make_backend("elsewhere", (TWIN_HANDLE, ModelType.JUDGMENT)),
+ routing_profile=RoutingProfile(name="profile", default="openai"),
+ )
+ model_deck = manager.get_model_deck()
+
+ assert model_deck.inference_models.types_serving(handle=TWIN_HANDLE) == [ModelType.LLM, ModelType.DOC_GEN]
+ assert manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.DOC_GEN).backend_name == PipelexBackend.INTERNAL
+
+ def test_a_pattern_match_skips_the_pair_its_backend_lacks(self) -> None:
+ manager = build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),
+ make_backend("openai_decisions", (TWIN_HANDLE, ModelType.JUDGMENT)),
+ routing_profile=RoutingProfile(name="profile", routes={"gpt-*": "openai"}, fallback_order=["openai", "openai_decisions"]),
+ )
+ model_deck = manager.get_model_deck()
+
+ assert model_deck.inference_models.types_serving(handle=TWIN_HANDLE) == [ModelType.LLM]
+
+ def test_an_exact_route_to_a_backend_serving_the_name_as_another_type_only_boots_and_leaves_the_pair_unserved(self) -> None:
+ """A route pins a name to a backend, so a call for that name never goes elsewhere."""
+ manager = build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),
+ make_backend("openai_decisions", (TWIN_HANDLE, ModelType.JUDGMENT)),
+ routing_profile=RoutingProfile(name="profile", routes={TWIN_HANDLE: "openai"}, fallback_order=["openai", "openai_decisions"]),
+ )
+ model_deck = manager.get_model_deck()
+
+ assert model_deck.inference_models.types_serving(handle=TWIN_HANDLE) == [ModelType.LLM]
+ assert manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM).backend_name == "openai"
+
+ def test_an_exact_route_to_a_backend_not_declaring_the_name_at_all_fails_the_boot(self) -> None:
+ with pytest.raises(ModelManagerError) as exc_info:
+ build_deck_manager(
+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),
+ make_backend("anthropic", ("claude", ModelType.LLM)),
+ routing_profile=RoutingProfile(name="profile", routes={TWIN_HANDLE: "anthropic"}, default="openai"),
+ )
+
+ assert f"'{TWIN_HANDLE}'" in str(exc_info.value)
+ assert "'anthropic'" in str(exc_info.value)
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_check.py b/tests/unit/pipelex/cogt/models/test_model_deck_check.py
index 420fb2ddf..7c5711ffa 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_check.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_check.py
@@ -12,6 +12,7 @@ from pipelex.cogt.judgment.judgment_setting import JudgmentSetting
from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaults
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import ModelDeck
from pipelex.cogt.models.model_deck_check import (
@@ -48,13 +49,15 @@ class TestModelDeckCheck:
def _create_test_model_deck(self) -> ModelDeck:
"""Build a deck with one preset, alias, waterfall and handle per model type."""
return ModelDeck(
- inference_models={
- "gpt-4o-mini": self._create_model_spec("gpt-4o-mini", ModelType.LLM),
- "extract-engine": self._create_model_spec("extract-engine", ModelType.TEXT_EXTRACTOR),
- "img-painter": self._create_model_spec("img-painter", ModelType.IMG_GEN),
- "web-searcher": self._create_model_spec("web-searcher", ModelType.SEARCH),
- "verdict-giver": self._create_model_spec("verdict-giver", ModelType.JUDGMENT),
- },
+ inference_models=ModelSpecIndex.make_from_specs(
+ model_specs=[
+ self._create_model_spec("gpt-4o-mini", ModelType.LLM),
+ self._create_model_spec("extract-engine", ModelType.TEXT_EXTRACTOR),
+ self._create_model_spec("img-painter", ModelType.IMG_GEN),
+ self._create_model_spec("web-searcher", ModelType.SEARCH),
+ self._create_model_spec("verdict-giver", ModelType.JUDGMENT),
+ ]
+ ),
# LLM-specific
llm_default_temperature=0.7,
llm_aliases={"best-gpt": "gpt-4o-mini"},
@@ -297,8 +300,14 @@ class TestModelDeckCheck:
assert error.model_choice == model_choice
assert error.reference_kind == expected_kind
assert error.model_type == expected_model_type
- expected_options: dict[str, Any] = getattr(model_deck, options_attr)
- assert error.available_options == list(expected_options.keys())
+ if options_attr == "inference_models":
+ # A handle names one model per model type: only the handles served as the type the pipe asks for are options.
+ served_handles = model_deck.inference_models.handles_of_type(model_type=expected_model_type)
+ assert len(served_handles) == 1
+ assert error.available_options == served_handles
+ else:
+ expected_options: dict[str, Any] = getattr(model_deck, options_attr)
+ assert error.available_options == list(expected_options.keys())
def test_near_miss_handle_yields_fuzzy_suggestion(self, mocker: MockerFixture) -> None:
"""A typo'd handle gets a fuzzy suggestion pointing at the close-by real handle."""
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_load_time_checks_per_pair.py b/tests/unit/pipelex/cogt/models/test_model_deck_load_time_checks_per_pair.py
new file mode 100644
index 000000000..bbd375f54
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_load_time_checks_per_pair.py
@@ -0,0 +1,57 @@
+"""One handle, one model per model type: a method naming a model the deck serves only as another type is refused when it loads."""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ModelChoiceNotFoundError
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from pipelex.cogt.models.model_deck import (
+ ModelDeck,
+)
+from pipelex.cogt.models.model_deck_check import check_judgment_choice_with_deck, check_llm_choice_with_deck
+from tests.unit.pipelex.cogt.models.model_deck_per_pair_utils import GET_MODEL_DECK_TARGET, TWIN_HANDLE, build_deck_manager, make_backend
+
+
+class TestLoadTimeChecksPerPair:
+ @staticmethod
+ def _deck() -> ModelDeck:
+ return build_deck_manager(
+ make_backend(
+ "openai",
+ ("llm-only", ModelType.LLM),
+ ("verdict", ModelType.JUDGMENT),
+ ("verdicts", ModelType.JUDGMENT),
+ (TWIN_HANDLE, ModelType.LLM),
+ (TWIN_HANDLE, ModelType.JUDGMENT),
+ ),
+ routing_profile=RoutingProfile(name="profile", default="openai"),
+ ).get_model_deck()
+
+ def test_a_judgment_naming_an_llm_only_handle_is_refused_at_load(self, mocker: MockerFixture) -> None:
+ model_deck = self._deck()
+ mocker.patch(GET_MODEL_DECK_TARGET, return_value=model_deck)
+
+ with pytest.raises(ModelChoiceNotFoundError) as exc_info:
+ check_judgment_choice_with_deck("llm-only")
+
+ error = exc_info.value
+ assert error.model_type == ModelType.JUDGMENT
+ assert sorted(error.available_options) == [TWIN_HANDLE, "verdict", "verdicts"]
+ assert "llm-only" not in error.available_options
+ assert all(suggestion in {TWIN_HANDLE, "verdict", "verdicts"} for suggestion in error.suggestions)
+
+ def test_the_deck_refuses_a_judgment_handle_served_only_as_an_llm(self) -> None:
+ model_deck = self._deck()
+
+ assert model_deck.is_model_handle_defined(model_handle="llm-only", model_type=ModelType.LLM)
+ assert not model_deck.is_model_handle_defined(model_handle="llm-only", model_type=ModelType.JUDGMENT)
+ with pytest.raises(ModelChoiceNotFoundError) as exc_info:
+ model_deck.get_judgment_setting("llm-only")
+ assert sorted(exc_info.value.available_options) == [TWIN_HANDLE, "verdict", "verdicts"]
+
+ def test_both_families_accept_a_twin(self, mocker: MockerFixture) -> None:
+ mocker.patch(GET_MODEL_DECK_TARGET, return_value=self._deck())
+
+ check_llm_choice_with_deck(TWIN_HANDLE)
+ check_judgment_choice_with_deck(TWIN_HANDLE)
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_lookups_per_pair.py b/tests/unit/pipelex/cogt/models/test_model_deck_lookups_per_pair.py
new file mode 100644
index 000000000..3b0a71b3b
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_lookups_per_pair.py
@@ -0,0 +1,59 @@
+"""One handle, one model per model type: every lookup of the deck goes by the pair, so each pipe family reaches its own spec."""
+
+import pytest
+
+from pipelex.cogt.exceptions import ModelNotFoundError
+from pipelex.cogt.inference.error_classification import UserActionKind
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.model_routing.routing_profile import RoutingProfile
+from pipelex.cogt.models.model_manager import ModelManager
+from tests.unit.pipelex.cogt.models.model_deck_per_pair_utils import TWIN_HANDLE, build_deck_manager, make_backend
+
+
+class TestLookupsPerPair:
+ @staticmethod
+ def _twin_manager() -> ModelManager:
+ return build_deck_manager(
+ make_backend(
+ "openai",
+ (TWIN_HANDLE, ModelType.LLM),
+ (TWIN_HANDLE, ModelType.JUDGMENT),
+ ("verdict", ModelType.JUDGMENT),
+ ("fast", ModelType.LLM),
+ ),
+ routing_profile=RoutingProfile(name="profile", default="openai"),
+ )
+
+ def test_each_family_reaches_its_own_spec_through_the_model_manager(self) -> None:
+ manager = self._twin_manager()
+
+ llm_spec = manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM)
+ judgment_spec = manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.JUDGMENT)
+
+ assert (llm_spec.model_type, llm_spec.name) == (ModelType.LLM, TWIN_HANDLE)
+ assert (judgment_spec.model_type, judgment_spec.name) == (ModelType.JUDGMENT, TWIN_HANDLE)
+
+ def test_an_alias_resolves_when_another_type_serves_a_handle_of_the_same_name(self) -> None:
+ """`fast` is an LLM handle and a judgment alias: the judgment lookup reaches the alias's target."""
+ manager = self._twin_manager()
+
+ assert manager.get_inference_model("fast", model_type=ModelType.JUDGMENT).name == "verdict"
+ assert manager.get_inference_model("fast", model_type=ModelType.LLM).model_type == ModelType.LLM
+
+ def test_an_inline_setting_naming_a_handle_served_as_another_type_only_is_the_callers_fault(self) -> None:
+ """The load-time check does not look into an inline setting, so the run-time lookup is what reports it.
+
+ `verdict` is served as a judgment model only, and nothing of the deck's LLM half names it.
+ The report names the type the pipe asked for, never the type the deck serves the handle as.
+ """
+ inline_setting = LLMSetting(model="verdict", temperature=0.5)
+
+ with pytest.raises(ModelNotFoundError) as exc_info:
+ self._twin_manager().get_inference_model(inline_setting.model, model_type=ModelType.LLM)
+
+ report = exc_info.value.to_error_report()
+ assert report.user_action is not None
+ assert report.user_action.kind is UserActionKind.CHANGE_MODEL
+ assert "an LLM" in report.user_action.detail
+ assert "judgment" not in report.user_action.detail
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_references.py b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
index 0056abda6..bfea7c778 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_references.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
@@ -13,6 +13,7 @@ from pipelex.cogt.extract.extract_setting import ExtractSetting
from pipelex.cogt.img_gen.img_gen_setting import ImgGenSetting
from pipelex.cogt.judgment.judgment_setting import JudgmentSetting
from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.cogt.model_backends.model_spec_document import list_declared_model_specs
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import (
ModelDeckBlueprint,
@@ -41,7 +42,7 @@ class TestModelDeckReferences:
return load_model_deck_blueprint(model_deck_paths=model_deck_paths)
@pytest.fixture(scope="class")
- def all_known_model_handles(self) -> dict[str, ModelType]:
+ def all_known_model_handles(self) -> dict[str, set[ModelType]]:
"""Collect all valid model handles with their types from the local backend TOML files.
Returns:
@@ -53,35 +54,25 @@ class TestModelDeckReferences:
# Helper methods
# ============================================================
- def _get_local_backend_models(self) -> dict[str, ModelType]:
- """Parse all local backend TOML files to collect model handles with their types.
+ def _get_local_backend_models(self) -> dict[str, set[ModelType]]:
+ """Parse all local backend TOML files to collect model handles with the model types they are declared as.
- Model names are the top-level keys in each backend TOML file,
- excluding the 'defaults' section which contains shared config.
- Model types are determined from the model's 'model_type' field or the defaults section.
+ Read as the loader reads a backend file: every root table but `[defaults]` is a model, its handle is its
+ `handle` key or else its table name, and its type is its own, else the `[defaults]` one, else the default.
+ A handle names one model per model type, so one handle may be declared as several.
Returns:
- Mapping of model_handle -> ModelType
+ Mapping of model_handle -> the model types the backends declare it as
"""
- known_handles: dict[str, ModelType] = {}
+ known_handles: dict[str, set[ModelType]] = {}
backends_dir = config_manager.backends_dir_path
toml_files = find_files_in_dir(backends_dir, pattern="*.toml", is_recursive=False)
for toml_path in toml_files:
backend_data = load_toml_from_path_if_exists(str(toml_path))
if backend_data:
- # Get default model_type from defaults section if present
- defaults = backend_data.get("defaults", {})
- default_model_type_str = defaults.get("model_type")
-
- # Model names are top-level keys except "defaults"
- for key in backend_data:
- if key != "defaults":
- model_config = backend_data[key]
- # Model can override the default model_type
- model_type_str = model_config.get("model_type", default_model_type_str)
- if model_type_str:
- known_handles[key] = ModelType(model_type_str)
+ for declared in list_declared_model_specs(document=backend_data):
+ known_handles.setdefault(declared.handle, set()).add(ModelType(declared.model_type))
return known_handles
@@ -90,7 +81,7 @@ class TestModelDeckReferences:
presets: Mapping[str, LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | DocGenSetting | JudgmentSetting],
all_aliases: dict[str, str],
all_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
expected_model_type: ModelType,
) -> list[tuple[str, str, str]]:
"""Find presets that reference invalid model targets.
@@ -99,7 +90,7 @@ class TestModelDeckReferences:
presets: The presets dict to validate
all_aliases: All available aliases for this model type
all_waterfalls: All available waterfalls for this model type
- known_model_handles: Mapping of model handle -> ModelType
+ known_model_handles: Mapping of model handle -> the model types the backends declare it as
expected_model_type: The expected model type for this deck (LLM, TEXT_EXTRACTOR, IMG_GEN)
Returns:
@@ -123,10 +114,10 @@ class TestModelDeckReferences:
case ModelReferenceKind.HANDLE:
if ref.name not in known_model_handles:
invalid_refs.append((preset_name, model_value, f"model handle '{ref.name}' not found in backends"))
- elif known_model_handles[ref.name] != expected_model_type:
- actual_type = known_model_handles[ref.name]
+ elif expected_model_type not in known_model_handles[ref.name]:
+ actual_types = ", ".join(f"'{actual_type}'" for actual_type in sorted(known_model_handles[ref.name]))
invalid_refs.append(
- (preset_name, model_value, f"model handle '{ref.name}' has type '{actual_type}' but expected '{expected_model_type}'")
+ (preset_name, model_value, f"model handle '{ref.name}' has type {actual_types} but expected '{expected_model_type}'")
)
return invalid_refs
@@ -169,7 +160,7 @@ class TestModelDeckReferences:
def test_aliases_reference_valid_targets(
self,
model_deck_blueprint: ModelDeckBlueprint,
- all_known_model_handles: dict[str, ModelType],
+ all_known_model_handles: dict[str, set[ModelType]],
model_type: ModelType,
deck_name: str,
):
@@ -204,7 +195,7 @@ class TestModelDeckReferences:
def test_presets_reference_valid_models(
self,
model_deck_blueprint: ModelDeckBlueprint,
- all_known_model_handles: dict[str, ModelType],
+ all_known_model_handles: dict[str, set[ModelType]],
model_type: ModelType,
deck_name: str,
):
@@ -239,7 +230,7 @@ class TestModelDeckReferences:
def test_waterfalls_contain_valid_models(
self,
model_deck_blueprint: ModelDeckBlueprint,
- all_known_model_handles: dict[str, ModelType],
+ all_known_model_handles: dict[str, set[ModelType]],
model_type: ModelType,
deck_name: str,
):
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_unknown_reference.py b/tests/unit/pipelex/cogt/models/test_model_deck_unknown_reference.py
index fa63a2129..96fb60218 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_unknown_reference.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_unknown_reference.py
@@ -9,6 +9,7 @@ from pipelex.cogt.inference.error_classification import UserAction, UserActionKi
from pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoices, LLMSettingChoicesDefaults
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_spec_index import ModelSpecIndex
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import ModelDeck
from pipelex.cogt.usage.cost_category import CostCategory
@@ -29,7 +30,7 @@ def _make_deck() -> ModelDeck:
max_prompt_images=None,
)
return ModelDeck(
- inference_models={"served-model": served_model},
+ inference_models=ModelSpecIndex.make_from_specs(model_specs=[served_model]),
llm_default_temperature=0.7,
llm_aliases={"deck-alias": "deck-alias-target", "served-alias": "served-model"},
llm_waterfalls={"deck-waterfall": ["deck-waterfall-member", "served-model"]},
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_validation_helpers.py b/tests/unit/pipelex/cogt/models/test_model_deck_validation_helpers.py
index 53e526131..2ec496af3 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_validation_helpers.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_validation_helpers.py
@@ -21,14 +21,14 @@ class TestModelDeckValidationHelpers:
# ============================================================
@pytest.fixture
- def known_model_handles(self) -> dict[str, ModelType]:
+ def known_model_handles(self) -> dict[str, set[ModelType]]:
"""A small set of valid model handles with their types for testing."""
return {
- "gpt-4o": ModelType.LLM,
- "claude-3-opus": ModelType.LLM,
- "gemini-pro": ModelType.LLM,
- "mistral-ocr": ModelType.TEXT_EXTRACTOR,
- "dall-e-3": ModelType.IMG_GEN,
+ "gpt-4o": {ModelType.LLM},
+ "claude-3-opus": {ModelType.LLM},
+ "gemini-pro": {ModelType.LLM},
+ "mistral-ocr": {ModelType.TEXT_EXTRACTOR},
+ "dall-e-3": {ModelType.IMG_GEN},
}
@pytest.fixture
@@ -54,7 +54,7 @@ class TestModelDeckValidationHelpers:
self,
valid_aliases: dict[str, str],
valid_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Alias pointing to non-existent model handle is detected."""
bad_aliases = {"broken": "nonexistent-model"}
@@ -75,7 +75,7 @@ class TestModelDeckValidationHelpers:
self,
valid_aliases: dict[str, str],
valid_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Alias pointing to a preset ($ prefix) is detected as invalid."""
bad_aliases = {"broken": "$some_preset"}
@@ -95,7 +95,7 @@ class TestModelDeckValidationHelpers:
self,
valid_aliases: dict[str, str],
valid_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Alias pointing to a model with wrong type is detected (e.g., LLM alias -> IMG_GEN model)."""
# LLM alias pointing to an IMG_GEN model
@@ -117,7 +117,7 @@ class TestModelDeckValidationHelpers:
self,
valid_aliases: dict[str, str],
valid_waterfalls: dict[str, list[str]],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Extract alias pointing to an LLM model is detected."""
# TEXT_EXTRACTOR alias pointing to an LLM model
@@ -142,7 +142,7 @@ class TestModelDeckValidationHelpers:
def test_detects_waterfall_containing_another_waterfall(
self,
valid_aliases: dict[str, str],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Waterfall containing another waterfall (~ prefix) is detected."""
bad_waterfalls = {"broken": ["gpt-4o", "~other_waterfall"]}
@@ -161,7 +161,7 @@ class TestModelDeckValidationHelpers:
def test_detects_waterfall_containing_preset(
self,
valid_aliases: dict[str, str],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Waterfall containing a preset ($ prefix) is detected."""
bad_waterfalls = {"broken": ["$some_preset"]}
@@ -179,7 +179,7 @@ class TestModelDeckValidationHelpers:
def test_detects_waterfall_containing_wrong_model_type(
self,
valid_aliases: dict[str, str],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Waterfall containing a model with wrong type is detected."""
# LLM waterfall containing an IMG_GEN model
@@ -199,7 +199,7 @@ class TestModelDeckValidationHelpers:
def test_detects_empty_waterfall(
self,
valid_aliases: dict[str, str],
- known_model_handles: dict[str, ModelType],
+ known_model_handles: dict[str, set[ModelType]],
):
"""Empty waterfall is detected (would cause IndexError at runtime)."""
bad_waterfalls: dict[str, list[str]] = {"empty-waterfall": []}
diff --git a/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py b/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py
index 9f5c7e0a9..d3ad83b52 100644
--- a/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py
+++ b/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py
@@ -126,6 +126,25 @@ class TestPluginModelMerge:
assert "'pipelex-xlsx'" in str(exc_info.value)
assert str(internal_toml_path) in str(exc_info.value)
+ def test_a_plugin_may_declare_another_kind_of_a_name_internal_toml_declares(self, inference_dir: Path) -> None:
+ """A handle names one model per model type: the file's document engine and the plugin's extractor share a name."""
+ declarations = _declarations(
+ internal_models={"reportlab-pdf": {**PIPELEX_XLSX_SPEC, "model_type": "text_extractor", "model_id": "read-pdf"}}, doc_gen_defaults={}
+ )
+
+ model_deck = self._setup(inference_dir=inference_dir, plugin_model_declarations=declarations).get_model_deck()
+
+ assert model_deck.inference_models.types_serving(handle="reportlab-pdf") == [ModelType.TEXT_EXTRACTOR, ModelType.DOC_GEN]
+ assert model_deck.get_required_inference_model(model_handle="reportlab-pdf", model_type=ModelType.TEXT_EXTRACTOR).model_id == "read-pdf"
+ assert model_deck.get_required_inference_model(model_handle="reportlab-pdf", model_type=ModelType.DOC_GEN).model_id == "print-pdf"
+
+ def test_a_plugin_table_may_not_set_handle(self, inference_dir: Path) -> None:
+ """A plugin names its model when it declares it, so a `handle` key in its table could only contradict that name."""
+ declarations = _declarations(internal_models={"pipelex-xlsx": {**PIPELEX_XLSX_SPEC, "handle": "other-name"}}, doc_gen_defaults={})
+
+ with pytest.raises(PluginModelDeclarationError, match="Plugin 'doc-gen' declares the internal model 'pipelex-xlsx' with a 'handle' key"):
+ self._setup(inference_dir=inference_dir, plugin_model_declarations=declarations)
+
def test_a_plugin_table_takes_no_file_defaults(self, inference_dir: Path) -> None:
"""The plugin's table is complete on its own: `internal.toml`'s `[defaults]` reaches the file's models only."""
internal_toml_path = inference_dir / "backends" / "internal.toml"
@@ -168,5 +187,5 @@ class TestPluginModelMerge:
model_deck = self._setup(inference_dir=inference_dir, plugin_model_declarations=_xlsx_plugin_declarations()).get_model_deck()
- assert "pipelex-xlsx" not in model_deck.inference_models
+ assert not model_deck.inference_models.types_serving(handle="pipelex-xlsx")
assert model_deck.get_doc_gen_choice_default(doc_gen_format=DocGenFormat.XLSX, source=DocGenSource.LAYOUT) is None
diff --git a/tests/unit/pipelex/cogt/usage/test_cost_registry.py b/tests/unit/pipelex/cogt/usage/test_cost_registry.py
index 29fd3c3fc..46fc83f67 100644
--- a/tests/unit/pipelex/cogt/usage/test_cost_registry.py
+++ b/tests/unit/pipelex/cogt/usage/test_cost_registry.py
@@ -12,7 +12,7 @@ from pipelex.cogt.judgment.judgment_report import JudgmentTokenCostReport, Judgm
from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage
from pipelex.cogt.search.search_report import SearchTokenCostReport
from pipelex.cogt.usage.cost_category import CostCategory
-from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.cost_registry import CostRegistry, ModelUsageKey
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.reporting.reporting_types import AnyTokensUsage
from pipelex.system.exceptions import MissingDependencyError
@@ -669,5 +669,4 @@ class TestCostRegistry:
aggregated = CostRegistry.aggregate_costs(tokens_usages=[UsageFixtures.judgment_usage()])
assert cost_report.as_flat_dictionary()[JudgmentTokenCostReportField.JUDGMENT_NAME] == "jev"
- assert "jev" in aggregated.grouped_by_model
- assert aggregated.model_types["jev"] == "judgment"
+ assert ModelUsageKey(model_type="judgment", model_name="jev") in aggregated.grouped_by_model
diff --git a/tests/unit/pipelex/cogt/usage/test_usage_per_model_pair.py b/tests/unit/pipelex/cogt/usage/test_usage_per_model_pair.py
new file mode 100644
index 000000000..235b55100
--- /dev/null
+++ b/tests/unit/pipelex/cogt/usage/test_usage_per_model_pair.py
@@ -0,0 +1,97 @@
+"""A run calling both kinds of one handle reports two models, never one merged row.
+
+A handle names one model per model type, so `gpt-6-luna` may be called as an LLM and as a judgment model
+in one run. The cost table, the agent summary and the graph's usage attribution all group by the model
+type and the name together: grouping by the name alone would merge the two kinds into one row and type
+it by whichever call came last.
+"""
+
+from pipelex.cogt.judgment.judgment_report import JudgmentTokensUsage
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_registry import CostRegistry, ModelUsageKey
+from pipelex.graph.graphspec import ModelUsageSpec
+from pipelex.system.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.tracing.usage_attribution import make_self_contained_spec
+from tests.unit.pipelex.cogt.usage.test_data import RATED_NB_TOKENS, RATED_UNIT_COSTS, UsageFixtures
+
+TWIN_HANDLE = "gpt-6-luna"
+
+
+def _twin_usages() -> list[LLMTokensUsage | JudgmentTokensUsage]:
+ llm_job_metadata: JobMetadata = UsageFixtures.full_job_metadata(unit_job_id=UnitJobId.LLM_GEN_TEXT, job_category=JobCategory.LLM_JOB)
+ judgment_job_metadata: JobMetadata = UsageFixtures.full_job_metadata(unit_job_id=UnitJobId.JUDGMENT_ANSWER, job_category=JobCategory.JUDGMENT_JOB)
+ return [
+ LLMTokensUsage(
+ job_metadata=llm_job_metadata,
+ inference_model_name=TWIN_HANDLE,
+ inference_model_id="gpt-6-luna-llm",
+ nb_tokens_by_category=dict(RATED_NB_TOKENS),
+ unit_costs=dict(RATED_UNIT_COSTS),
+ ),
+ LLMTokensUsage(
+ job_metadata=llm_job_metadata,
+ inference_model_name=TWIN_HANDLE,
+ inference_model_id="gpt-6-luna-llm",
+ nb_tokens_by_category=dict(RATED_NB_TOKENS),
+ unit_costs=dict(RATED_UNIT_COSTS),
+ ),
+ JudgmentTokensUsage(
+ job_metadata=judgment_job_metadata,
+ inference_model_name=TWIN_HANDLE,
+ inference_model_id="gpt-6-luna-decisions",
+ nb_tokens_by_category=dict(RATED_NB_TOKENS),
+ unit_costs=dict(RATED_UNIT_COSTS),
+ ),
+ ]
+
+
+class TestUsagePerModelPair:
+ def test_the_aggregation_keeps_one_group_per_kind(self) -> None:
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=_twin_usages())
+
+ assert set(aggregated.grouped_by_model) == {
+ ModelUsageKey(model_type="llm", model_name=TWIN_HANDLE),
+ ModelUsageKey(model_type="judgment", model_name=TWIN_HANDLE),
+ }
+
+ def test_the_cost_summary_reports_two_rows(self) -> None:
+ summary = CostRegistry.build_cost_summary(_twin_usages())
+
+ assert summary is not None
+ rows = sorted((row["model"], row["model_type"]) for row in summary["by_model"])
+ assert rows == [(TWIN_HANDLE, "judgment"), (TWIN_HANDLE, "llm")]
+ llm_row = next(row for row in summary["by_model"] if row["model_type"] == "llm")
+ judgment_row = next(row for row in summary["by_model"] if row["model_type"] == "judgment")
+ assert llm_row["cost"] == 2 * judgment_row["cost"]
+
+ def test_the_usage_attribution_reports_two_entries(self) -> None:
+ by_model = make_self_contained_spec(_twin_usages()).by_model
+
+ assert [(spec.inference_model_name, spec.model_type, spec.inference_calls, spec.inference_model_id) for spec in by_model] == [
+ (TWIN_HANDLE, "llm", 2, "gpt-6-luna-llm"),
+ (TWIN_HANDLE, "judgment", 1, "gpt-6-luna-decisions"),
+ ]
+ assert all(isinstance(spec, ModelUsageSpec) for spec in by_model)
+
+ def test_entries_of_one_name_and_one_call_count_are_ordered_by_type(self) -> None:
+ """The sort breaks a tie on the name with the type, so the order never depends on the order of the calls."""
+ usages = _twin_usages()
+ one_call_each = [usages[0], usages[2]]
+
+ forward = [spec.model_type for spec in make_self_contained_spec(one_call_each).by_model]
+ backward = [spec.model_type for spec in make_self_contained_spec(list(reversed(one_call_each))).by_model]
+
+ assert forward == backward == ["judgment", "llm"]
+
+ def test_the_cost_summary_is_the_same_in_either_arrival_order(self) -> None:
+ """Grouped by the name alone, the merged row took the type of whichever call came last."""
+ usages = _twin_usages()
+ judgment_last = CostRegistry.build_cost_summary(usages)
+ judgment_first = CostRegistry.build_cost_summary([usages[2], usages[0], usages[1]])
+
+ assert judgment_last is not None
+ assert judgment_first is not None
+ by_type_last = {row["model_type"]: row for row in judgment_last["by_model"]}
+ by_type_first = {row["model_type"]: row for row in judgment_first["by_model"]}
+ assert by_type_last == by_type_first
+ assert set(by_type_last) == {"llm", "judgment"}
diff --git a/tests/unit/pipelex/kit/test_deck_variants.py b/tests/unit/pipelex/kit/test_deck_variants.py
index f485aa683..0ffdf223a 100644
--- a/tests/unit/pipelex/kit/test_deck_variants.py
+++ b/tests/unit/pipelex/kit/test_deck_variants.py
@@ -5,10 +5,12 @@ every variant through the same loader the runtime uses and holds it to the shipp
- the same alias, preset and waterfall names, per model family, except for the names the shipped
deck deliberately dropped, which are listed here by variant;
-- every model handle the variant names and the shipped deck does not is still declared by one of
- the kit's backend files, because handle retirement is how a parked deck actually goes stale.
+- every model the variant names and the shipped deck does not is still declared by one of the
+ kit's backend files, as the model type of the family that names it, because handle retirement is
+ how a parked deck actually goes stale. A handle names one model per model type, so a handle
+ declared only as another type does not count.
-The second check is scoped to the handles only the variant names. A handle the shipped deck names
+The second check is scoped to the models only the variant names. A handle the shipped deck names
too is not held to them here: some are served only through Pipelex, with no backend section
of their own, and the shipped deck's default aliases have their own guard in `test_shipped_deck_defaults.py`.
"""
@@ -21,12 +23,16 @@ import pytest
from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice, DocGenSetting
from pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSetting
from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting
+from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice, JudgmentSetting
from pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSetting
+from pipelex.cogt.model_backends.model_spec_document import list_declared_model_specs
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.deck_manifest import KitManagedArea, kit_deck_dir, list_managed_kit_files
from pipelex.cogt.models.model_deck import (
DocGenDeckBlueprint,
ExtractDeckBlueprint,
ImgGenDeckBlueprint,
+ JudgmentDeckBlueprint,
LLMDeckBlueprint,
ModelDeckBlueprint,
SearchDeckBlueprint,
@@ -38,7 +44,21 @@ from pipelex.kit.paths import get_kit_configs_dir, get_kit_deck_variants_dir
from pipelex.tools.misc.toml_utils import load_toml_from_path
# A vocabulary coordinate: the model family, then the kind of name within it.
-DeckFamilyBlueprint = LLMDeckBlueprint | ExtractDeckBlueprint | ImgGenDeckBlueprint | SearchDeckBlueprint | DocGenDeckBlueprint
+DeckFamilyBlueprint = (
+ LLMDeckBlueprint | ExtractDeckBlueprint | ImgGenDeckBlueprint | SearchDeckBlueprint | DocGenDeckBlueprint | JudgmentDeckBlueprint
+)
+# A model a deck names or a backend declares: a handle names one model per model type, so the pair is the identity.
+DeclaredModel = tuple[ModelType, str]
+
+# The model type each family of the deck asks its models for.
+FAMILY_MODEL_TYPES: dict[str, ModelType] = {
+ "llm": ModelType.LLM,
+ "extract": ModelType.TEXT_EXTRACTOR,
+ "img_gen": ModelType.IMG_GEN,
+ "search": ModelType.SEARCH,
+ "doc_gen": ModelType.DOC_GEN,
+ "judgment": ModelType.JUDGMENT,
+}
VocabularyKey = tuple[str, str]
Vocabulary = dict[VocabularyKey, set[str]]
PermittedDrops = Mapping[VocabularyKey, frozenset[str]]
@@ -68,6 +88,7 @@ def extract_vocabulary(blueprint: ModelDeckBlueprint) -> Vocabulary:
"img_gen": blueprint.img_gen,
"search": blueprint.search,
"doc_gen": blueprint.doc_gen,
+ "judgment": blueprint.judgment,
}
vocabulary: Vocabulary = {}
for family, family_blueprint in family_blueprints.items():
@@ -103,63 +124,76 @@ def load_deck_from_dir(deck_dir: Path, *, filenames: list[str]) -> ModelDeckBlue
return load_model_deck_blueprint([str(deck_dir / filename) for filename in sorted(filenames)])
-def list_declared_backend_handles() -> set[str]:
- """Every model handle the kit's backend files declare. Each top-level table is one, bar `defaults`."""
+def list_declared_backend_models() -> set[DeclaredModel]:
+ """Every model the kit's backend files declare, as its model type and its handle, read as the loader reads a file."""
backends_dir = Path(str(get_kit_configs_dir())) / "inference" / "backends"
- handles: set[str] = set()
+ declared_models: set[DeclaredModel] = set()
for backend_path in sorted(backends_dir.glob("*.toml")):
backend_dict = load_toml_from_path(str(backend_path))
- handles.update(name for name, value in backend_dict.items() if isinstance(value, dict) and name != "defaults")
- return handles
-
-
-def list_choice_default_references(blueprint: ModelDeckBlueprint) -> list[str]:
- """The reference each family's default choice names: a setting's model, or the reference the deck wrote."""
- choices: list[LLMModelChoice | ExtractModelChoice | ImgGenModelChoice | SearchModelChoice | DocGenModelChoice] = [
- blueprint.llm.choice_defaults.for_text,
- blueprint.llm.choice_defaults.for_object,
- blueprint.extract.choice_default,
- blueprint.img_gen.choice_default,
- blueprint.search.choice_default,
- *blueprint.doc_gen.choice_defaults.values(),
+ declared_models.update((ModelType(declared.model_type), declared.handle) for declared in list_declared_model_specs(document=backend_dict))
+ return declared_models
+
+
+def list_choice_default_references(blueprint: ModelDeckBlueprint) -> list[tuple[ModelType, str]]:
+ """The reference each family's default choice names, with that family's model type: a setting's model, or the reference the deck wrote."""
+ choices: list[
+ tuple[ModelType, LLMModelChoice | ExtractModelChoice | ImgGenModelChoice | SearchModelChoice | DocGenModelChoice | JudgmentModelChoice]
+ ] = [
+ (ModelType.LLM, blueprint.llm.choice_defaults.for_text),
+ (ModelType.LLM, blueprint.llm.choice_defaults.for_object),
+ (ModelType.TEXT_EXTRACTOR, blueprint.extract.choice_default),
+ (ModelType.IMG_GEN, blueprint.img_gen.choice_default),
+ (ModelType.SEARCH, blueprint.search.choice_default),
+ *((ModelType.DOC_GEN, choice) for choice in blueprint.doc_gen.choice_defaults.values()),
]
- references: list[str] = []
- for choice in choices:
+ # The judgment family alone may have no default, since no judgment model is served out of the box.
+ if blueprint.judgment.choice_default is not None:
+ choices.append((ModelType.JUDGMENT, blueprint.judgment.choice_default))
+ references: list[tuple[ModelType, str]] = []
+ for model_type, choice in choices:
match choice:
- case LLMSetting() | ExtractSetting() | ImgGenSetting() | SearchSetting() | DocGenSetting():
- references.append(choice.model)
+ case LLMSetting() | ExtractSetting() | ImgGenSetting() | SearchSetting() | DocGenSetting() | JudgmentSetting():
+ references.append((model_type, choice.model))
case ModelReference():
- references.append(choice.raw)
+ references.append((model_type, choice.raw))
case str():
- references.append(choice)
+ references.append((model_type, choice))
return references
-def extract_model_handles(blueprint: ModelDeckBlueprint) -> set[str]:
- """Every concrete handle a deck names, from its alias targets, its waterfall entries, its presets' models and its default choices.
+def extract_named_models(blueprint: ModelDeckBlueprint) -> set[DeclaredModel]:
+ """Every concrete model a deck names, from its alias targets, its waterfall entries, its presets' models and its default choices.
- A reference naming an alias, a preset or a waterfall carries no handle of its own: it resolves
- through one of the collections this function reads directly. A waterfall's own entries do
- carry handles, which is why they are read here and not only through whatever names the waterfall,
- and so does a default choice that names a model directly.
+ Each handle comes with the model type of the family that names it, which is the type a lookup asks
+ for. A reference naming an alias, a preset or a waterfall carries no handle of its own: it resolves
+ through one of the collections this function reads directly. A waterfall's own entries do carry
+ handles, which is why they are read here and not only through whatever names the waterfall, and so
+ does a default choice that names a model directly.
"""
- family_blueprints: list[DeckFamilyBlueprint] = [blueprint.llm, blueprint.extract, blueprint.img_gen, blueprint.search, blueprint.doc_gen]
- references: list[str] = []
- for family_blueprint in family_blueprints:
- references.extend(family_blueprint.aliases.values())
+ family_blueprints: list[tuple[ModelType, DeckFamilyBlueprint]] = [
+ (ModelType.LLM, blueprint.llm),
+ (ModelType.TEXT_EXTRACTOR, blueprint.extract),
+ (ModelType.IMG_GEN, blueprint.img_gen),
+ (ModelType.SEARCH, blueprint.search),
+ (ModelType.DOC_GEN, blueprint.doc_gen),
+ (ModelType.JUDGMENT, blueprint.judgment),
+ ]
+ references: list[tuple[ModelType, str]] = []
+ for model_type, family_blueprint in family_blueprints:
+ references.extend((model_type, target) for target in family_blueprint.aliases.values())
for waterfall_entries in family_blueprint.waterfalls.values():
- references.extend(waterfall_entries)
- references.extend(setting.model for setting in family_blueprint.presets.values())
+ references.extend((model_type, entry) for entry in waterfall_entries)
+ references.extend((model_type, setting.model) for setting in family_blueprint.presets.values())
references.extend(list_choice_default_references(blueprint))
- handles: set[str] = set()
- for reference in references:
+ named_models: set[DeclaredModel] = set()
+ for model_type, reference in references:
parsed = ModelReference.parse(reference)
match parsed.kind:
case ModelReferenceKind.HANDLE:
- handles.add(parsed.name)
+ named_models.add((model_type, parsed.name))
case ModelReferenceKind.ALIAS | ModelReferenceKind.WATERFALL | ModelReferenceKind.PRESET:
continue
- return handles
+ return named_models
def make_vocabulary(*, aliases: set[str], presets: set[str]) -> Vocabulary:
@@ -196,15 +230,24 @@ class TestDeckVariants:
def test_variant_only_handles_are_still_declared_by_a_backend(self, variant_dir: Path):
"""Name parity says nothing about a handle that was retired from the backends underneath it."""
managed_filenames = list(list_managed_kit_files(area=KitManagedArea.DECK))
- shipped_handles = extract_model_handles(load_deck_from_dir(kit_deck_dir(), filenames=managed_filenames))
- variant_handles = extract_model_handles(load_deck_from_dir(variant_dir, filenames=managed_filenames))
+ shipped_models = extract_named_models(load_deck_from_dir(kit_deck_dir(), filenames=managed_filenames))
+ variant_models = extract_named_models(load_deck_from_dir(variant_dir, filenames=managed_filenames))
+
+ variant_only_models = variant_models - shipped_models
+ assert variant_only_models, f"Variant '{variant_dir.name}' names no handle of its own, which makes this guard vacuous"
- variant_only_handles = variant_handles - shipped_handles
- assert variant_only_handles, f"Variant '{variant_dir.name}' names no handle of its own, which makes this guard vacuous"
+ declared_models = list_declared_backend_models()
+ undeclared = sorted(f"{handle} ({model_type})" for model_type, handle in variant_only_models if (model_type, handle) not in declared_models)
+ assert not undeclared, f"Variant '{variant_dir.name}' names models no backend file declares any more: {', '.join(undeclared)}"
- declared_handles = list_declared_backend_handles()
- undeclared = sorted(handle for handle in variant_only_handles if handle not in declared_handles)
- assert not undeclared, f"Variant '{variant_dir.name}' names handles no backend file declares any more: {', '.join(undeclared)}"
+ def test_a_handle_declared_as_another_type_only_does_not_count(self):
+ """A handle names one model per model type: an image model's name does not serve an LLM lookup."""
+ declared_models = list_declared_backend_models()
+ img_gen_handles = sorted(handle for model_type, handle in declared_models if model_type == ModelType.IMG_GEN)
+ img_gen_only_handle = next(handle for handle in img_gen_handles if (ModelType.LLM, handle) not in declared_models)
+
+ assert (ModelType.IMG_GEN, img_gen_only_handle) in declared_models
+ assert (ModelType.LLM, img_gen_only_handle) not in declared_models
def test_handle_collection_reads_waterfall_entries(self):
"""A handle a deck names only inside a waterfall must still reach the retirement check above.
@@ -213,20 +256,30 @@ class TestDeckVariants:
go quietly blind to every handle a variant names only inside a waterfall.
"""
blueprint = load_deck_from_dir(kit_deck_dir(), filenames=list(list_managed_kit_files(area=KitManagedArea.DECK)))
- probe_handle = "handle-named-only-by-a-waterfall"
- assert probe_handle not in extract_model_handles(blueprint)
+ probe_model = (ModelType.LLM, "handle-named-only-by-a-waterfall")
+ assert probe_model not in extract_named_models(blueprint)
- blueprint.llm.waterfalls["waterfall-parity-probe"] = [probe_handle]
- assert probe_handle in extract_model_handles(blueprint), "A handle named inside a waterfall escaped the handle collection"
+ blueprint.llm.waterfalls["waterfall-parity-probe"] = [probe_model[1]]
+ assert probe_model in extract_named_models(blueprint), "A handle named inside a waterfall escaped the handle collection"
def test_handle_collection_reads_default_choices(self):
"""A handle a deck names only as a family's default choice must still reach the retirement check above."""
blueprint = load_deck_from_dir(kit_deck_dir(), filenames=list(list_managed_kit_files(area=KitManagedArea.DECK)))
- probe_handle = "handle-named-only-by-a-default"
- assert probe_handle not in extract_model_handles(blueprint)
+ probe_model = (ModelType.DOC_GEN, "handle-named-only-by-a-default")
+ assert probe_model not in extract_named_models(blueprint)
+
+ blueprint.doc_gen.choice_defaults["xlsx.layout"] = ModelReference.parse(probe_model[1])
+ assert probe_model in extract_named_models(blueprint), "A handle named as a default choice escaped the handle collection"
+
+ def test_handle_collection_reads_the_judgment_family(self):
+ """A handle a deck names only as a judgment alias must still reach the retirement check above, as a judgment model."""
+ blueprint = load_deck_from_dir(kit_deck_dir(), filenames=list(list_managed_kit_files(area=KitManagedArea.DECK)))
+ probe_model = (ModelType.JUDGMENT, "handle-named-only-by-a-judgment-alias")
+ assert probe_model not in extract_named_models(blueprint)
- blueprint.doc_gen.choice_defaults["xlsx.layout"] = ModelReference.parse(probe_handle)
- assert probe_handle in extract_model_handles(blueprint), "A handle named as a default choice escaped the handle collection"
+ blueprint.judgment.aliases["judgment-parity-probe"] = probe_model[1]
+ assert probe_model in extract_named_models(blueprint), "A handle named by a judgment alias escaped the handle collection"
+ assert ("judgment", "aliases") in extract_vocabulary(blueprint)
def test_comparator_reports_a_preset_the_variant_is_missing(self):
differences = compare_vocabularies(
diff --git a/tests/unit/pipelex/kit/test_shipped_deck_defaults.py b/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
index 0bb548890..562976894 100644
--- a/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
+++ b/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
@@ -1,17 +1,26 @@
-"""Every default alias of the shipped deck reaches a model that one of the kit's own backend files declares.
+"""Every default alias of the shipped deck reaches a model that one of the kit's own backend files declares, as its family's type.
Boot only checks that an alias exists, not that anything serves what it names, so a default whose every
candidate is served through a Pipelex-managed service alone loads cleanly and then fails the first pipe
that relies on it, for everyone running on their own keys. A waterfall passes when any one of its entries
-is declared, which is how a default can put a managed model first and still resolve without it.
+is declared, which is how a default can put a managed model first and still resolve without it. A handle
+names one model per model type, so a default reaches a model only through a handle declared as the type
+its family asks for: a judgment default naming a handle the kit declares only as an LLM serves nothing.
"""
import pytest
+from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.deck_manifest import KitManagedArea, kit_deck_dir, list_managed_kit_files
from pipelex.cogt.models.model_deck import JudgmentDeckBlueprint
from pipelex.cogt.models.model_reference import ModelReference, ModelReferenceKind
-from tests.unit.pipelex.kit.test_deck_variants import DeckFamilyBlueprint, list_declared_backend_handles, load_deck_from_dir
+from tests.unit.pipelex.kit.test_deck_variants import (
+ FAMILY_MODEL_TYPES,
+ DeckFamilyBlueprint,
+ DeclaredModel,
+ list_declared_backend_models,
+ load_deck_from_dir,
+)
DEFAULT_ALIAS_PREFIX = "default-"
@@ -50,6 +59,15 @@ def _candidate_handles(reference: str, *, family_blueprint: AnyFamilyBlueprint,
return {parsed.name}
+def _unserved_default(*, family: str, alias: str, family_blueprint: AnyFamilyBlueprint, declared_models: set[DeclaredModel]) -> str | None:
+ """Why a default alias of this family reaches no model a kit backend declares as the family's type, or `None` when it reaches one."""
+ model_type = FAMILY_MODEL_TYPES[family]
+ candidates = _candidate_handles(f"@{alias}", family_blueprint=family_blueprint, visited=frozenset())
+ if any((model_type, candidate) in declared_models for candidate in candidates):
+ return None
+ return f"{family} alias '{alias}' can only resolve to {sorted(candidates)}, which no backend file of the kit declares as '{model_type}'"
+
+
def _default_aliases() -> list[tuple[str, str]]:
return [
(family, alias)
@@ -62,11 +80,25 @@ def _default_aliases() -> list[tuple[str, str]]:
class TestShippedDeckDefaults:
@pytest.mark.parametrize(("family", "alias"), _default_aliases())
def test_default_alias_reaches_a_declared_model(self, family: str, alias: str) -> None:
- family_blueprint = _shipped_family_blueprints()[family]
- candidates = _candidate_handles(f"@{alias}", family_blueprint=family_blueprint, visited=frozenset())
- assert candidates & list_declared_backend_handles(), (
- f"{family} alias '{alias}' can only resolve to {sorted(candidates)}, which no backend file of the kit declares"
+ unserved = _unserved_default(
+ family=family, alias=alias, family_blueprint=_shipped_family_blueprints()[family], declared_models=list_declared_backend_models()
)
+ assert unserved is None, unserved
+
+ def test_a_judgment_default_naming_a_handle_the_kit_declares_only_as_an_llm_is_refused(self) -> None:
+ """The guard must not count a handle's LLM spec for the judgment family, which asks for a judgment model."""
+ declared_models = list_declared_backend_models()
+ llm_only_handle = next(
+ handle
+ for model_type, handle in sorted(declared_models)
+ if model_type == ModelType.LLM and (ModelType.JUDGMENT, handle) not in declared_models
+ )
+ judgment_blueprint = JudgmentDeckBlueprint(aliases={"default-probe": llm_only_handle})
+
+ unserved = _unserved_default(family="judgment", alias="default-probe", family_blueprint=judgment_blueprint, declared_models=declared_models)
+ assert unserved is not None
+ assert llm_only_handle in unserved
+ assert _unserved_default(family="llm", alias="default-probe", family_blueprint=judgment_blueprint, declared_models=declared_models) is None
def test_a_waterfall_is_read_through_every_entry(self) -> None:
"""The waterfall branch must see past a first entry no backend file declares, or the guard reads it as unserved."""
diff --git a/tests/unit/pipelex/plugins/test_plugin_model_declarations.py b/tests/unit/pipelex/plugins/test_plugin_model_declarations.py
index 1cd8a2148..8b7ebe40b 100644
--- a/tests/unit/pipelex/plugins/test_plugin_model_declarations.py
+++ b/tests/unit/pipelex/plugins/test_plugin_model_declarations.py
@@ -45,8 +45,8 @@ class TestPluginModelDeclarations:
registrar.add_internal_model(name="pipelex-xlsx", spec=PIPELEX_XLSX_SPEC)
- assert registrar.internal_models == {"pipelex-xlsx": PIPELEX_XLSX_SPEC}
- assert "internal model pipelex-xlsx" in discovery.contributions
+ assert registrar.internal_models == {("pipelex-xlsx", "doc_gen"): PIPELEX_XLSX_SPEC}
+ assert "internal model pipelex-xlsx (doc_gen)" in discovery.contributions
def test_a_doc_gen_default_is_stored_and_recorded_as_a_contribution(self) -> None:
registrar = _make_registrar()
@@ -66,11 +66,40 @@ class TestPluginModelDeclarations:
with pytest.raises(DuplicateInternalModelError) as exc_info:
registrar.add_internal_model(name="pipelex-xlsx", spec=PIPELEX_XLSX_SPEC)
+ assert exc_info.value.model_type == "doc_gen"
assert exc_info.value.first_plugin == "alpha"
assert exc_info.value.second_plugin == "beta"
assert "'alpha'" in str(exc_info.value)
assert "'beta'" in str(exc_info.value)
+ def test_one_name_may_be_declared_once_per_model_type(self) -> None:
+ """A handle names one model per model type, so two plugins may declare two kinds of one name."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=PluginGroup.KERNEL)
+ registrar.add_internal_model(name="pipelex-xlsx", spec=PIPELEX_XLSX_SPEC)
+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=PluginGroup.KERNEL)
+ registrar.add_internal_model(name="pipelex-xlsx", spec=dict(PIPELEX_XLSX_SPEC, model_type="text_extractor"))
+
+ declarations = registrar.make_model_declarations()
+
+ assert [(model.name, model.spec["model_type"], model.plugin) for model in declarations.internal_models] == [
+ ("pipelex-xlsx", "doc_gen", "alpha"),
+ ("pipelex-xlsx", "text_extractor", "beta"),
+ ]
+
+ def test_a_table_without_a_model_type_is_keyed_by_the_default_type(self) -> None:
+ """Two tables that both leave the type out declare the same pair, as their specs would."""
+ registrar = _make_registrar()
+ spec_without_type = {key: value for key, value in PIPELEX_XLSX_SPEC.items() if key != "model_type"}
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=PluginGroup.KERNEL)
+ registrar.add_internal_model(name="shared", spec=spec_without_type)
+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=PluginGroup.KERNEL)
+
+ with pytest.raises(DuplicateInternalModelError) as exc_info:
+ registrar.add_internal_model(name="shared", spec=dict(spec_without_type, model_type="llm"))
+
+ assert exc_info.value.model_type == "llm"
+
def test_a_doc_gen_default_declared_twice_names_both_plugins(self) -> None:
registrar = _make_registrar()
registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION, group=PluginGroup.KERNEL)
@@ -100,7 +129,7 @@ class TestPluginModelDeclarations:
registrar.add_internal_model(name="broken", spec={"not_a_field": True})
registrar.add_doc_gen_default(doc_gen_format=DocGenFormat.PPTX, source=DocGenSource.LAYOUT, model="pipelex-pptx")
- assert "broken" in registrar.internal_models
+ assert ("broken", "llm") in registrar.internal_models
assert (DocGenFormat.PPTX, DocGenSource.LAYOUT) in registrar.doc_gen_defaults
def test_the_declarations_carry_each_entry_and_the_plugin_that_declared_it(self) -> None:
@@ -111,8 +140,10 @@ class TestPluginModelDeclarations:
declarations = registrar.make_model_declarations()
- assert declarations.internal_models["pipelex-xlsx"].spec == PIPELEX_XLSX_SPEC
- assert declarations.internal_models["pipelex-xlsx"].plugin == "doc-gen"
+ (internal_model,) = declarations.internal_models
+ assert internal_model.name == "pipelex-xlsx"
+ assert internal_model.spec == PIPELEX_XLSX_SPEC
+ assert internal_model.plugin == "doc-gen"
(doc_gen_default,) = declarations.doc_gen_defaults
assert doc_gen_default.choice_key == "xlsx.layout"
assert doc_gen_default.model == "pipelex-xlsx"
@@ -128,9 +159,9 @@ class TestPluginModelDeclarations:
spec["sdk"] = "changed"
spec["inputs"].append("html")
- declarations.internal_models["pipelex-xlsx"].spec["inputs"].append("html")
+ declarations.internal_models[0].spec["inputs"].append("html")
- recorded_spec = registrar.make_model_declarations().internal_models["pipelex-xlsx"].spec
+ recorded_spec = registrar.make_model_declarations().internal_models[0].spec
assert recorded_spec["sdk"] == "openpyxl"
assert recorded_spec["inputs"] == ["layout", "template_file"]
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
index 4bc5bb018..0cf16d92c 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
@@ -30,7 +30,7 @@ from pipelex.cogt.img_gen.img_gen_job_components import (
from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
-from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.cost_registry import CostRegistry, ModelUsageKey
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.config import get_config
from pipelex.reporting.reporting_manager import ReportingManager
@@ -148,7 +148,7 @@ class TestEmitRunnerFallbackNonLLM:
aggregated = CostRegistry.aggregate_costs(tokens_usages=UsageAggregator.aggregate(events))
assert aggregated.total_nb_tokens == 300 # 100 input + 200 output
- assert aggregated.model_types == {"img-model": "img_gen"}
+ assert list(aggregated.grouped_by_model) == [ModelUsageKey(model_type="img_gen", model_name="img-model")]
assert aggregated.has_reportable_usage is True
def test_extract_usage_emitted_via_runner_fallback(self, tmp_path: Path, mocker: MockerFixture) -> None:
@@ -167,5 +167,5 @@ class TestEmitRunnerFallbackNonLLM:
aggregated = CostRegistry.aggregate_costs(tokens_usages=UsageAggregator.aggregate(events))
assert aggregated.total_nb_tokens == 700 # 300 input + 400 output
- assert aggregated.model_types == {"extract-model": "extract"}
+ assert list(aggregated.grouped_by_model) == [ModelUsageKey(model_type="extract", model_name="extract-model")]
assert aggregated.has_reportable_usage is True
diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
index 32e72c105..3cb51081d 100644
--- a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
@@ -17,7 +17,7 @@ from pipelex.cogt.extract.extract_report import ExtractTokensUsage
from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
from pipelex.cogt.search.search_report import SearchTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
-from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.cost_registry import CostRegistry, ModelUsageKey
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.reporting.reporting_types import AnyTokensUsage
from pipelex.system.job_metadata import JobMetadata, RunMetadata
@@ -93,9 +93,8 @@ class TestNonLLMUsageAggregation:
# search: 10*0.005 + 20*0.005 = 0.15
assert abs(aggregated.total_cost - 1.35) < 1e-9
assert aggregated.has_reportable_usage is True
- assert set(aggregated.grouped_by_model.keys()) == {"img-gen-model", "extract-model", "search-model"}
- assert aggregated.model_types == {
- "img-gen-model": "img_gen",
- "extract-model": "extract",
- "search-model": "search",
+ assert set(aggregated.grouped_by_model.keys()) == {
+ ModelUsageKey(model_type="img_gen", model_name="img-gen-model"),
+ ModelUsageKey(model_type="extract", model_name="extract-model"),
+ ModelUsageKey(model_type="search", model_name="search-model"),
}