Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Internal-backend-last-resort · L-261008-71170d (#1380)
Louis Choquelec21020c36d8
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9f369e685..545c6f8e7 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -17,6 +17,7 @@- **The CLIs identify themselves as `pipelex-cli` to the hosted API**: every request `pipelex run --hosted`, `pipelex-agent run --hosted` and `pipelex login` send now leads its `User-Agent` with `pipelex-cli/<version>`, the token the client-identification spec reserves for these CLIs, so the hosted API counts their runs on its CLI surface instead of as calls from an SDK.- **A home configuration directory holding only some files gets the kit's missing ones**: every boot now copies into `~/.pipelex/`, or the directory `PIPELEX_HOME` names, each kit file it lacks, never overwriting one it holds or writing through a symbolic link, so a kit file deleted from it comes back. A directory holding anything at all, such as the `pipelex_service.toml` an earlier release wrote, used to be left as it was, and every boot then failed with "Config files are missing for the inference backend library". The inference files are copied as one unit, only when the directory has no `inference/backends.toml`, each file and kit manifest is written whole or not at all, so a fill cut short is completed by the next boot, and a directory Pipelex cannot write to is read as it is.+- **The internal backend's models are served whatever the routing profile**: a boot whose active profile sends a model's name to no enabled backend served none of the internal models (`reportlab-pdf`, `pypdfium2-extract-pdf`, `docling-extract-text` and the ones a plugin adds), so every local extraction and document step was refused at load. The default `all_enabled_backends` profile did this whenever only the internal backend was enabled, or when the only other enabled backend sat outside its `fallback_order`, such as `typesafe`. Such a name is now served by the internal backend for each model type it declares there; an exact or wildcard route sending the name elsewhere still decides, and disabling the internal backend still withholds its models.- **`pipelex-agent init --help` keeps `list[str]`**: the `--config` help printed the schema's `"backends": list[str]` as `"backends": list`, because Typer renders help as Rich markup and Rich took `[str]` for a style; the bracket is now escaped and printed.## [v0.78.0] - 2026-10-08diff --git a/docs/configuration/config-technical/inference-backend-config.md b/docs/configuration/config-technical/inference-backend-config.mdindex ca29b2e79..29171a167 100644--- a/docs/configuration/config-technical/inference-backend-config.md+++ b/docs/configuration/config-technical/inference-backend-config.md@@ -235,7 +235,7 @@ costs = { input = 0.1, output = 0 }The table name `acme-one-judgment` names the table and nothing else: methods, the model deck and the routing profiles all use the handle `acme-one`, and the model id defaults to the handle too. Two tables may share a handle only when their model types differ, so a file declaring one handle twice as the same type fails to load, naming both tables. `handle` cannot go in `[defaults]`, which every model of the file inherits, and like every key that describes the model rather than a value a call sends, it cannot reference a variable.-A routing profile routes a handle's name, so its route applies to every kind of model of that name. A model type the routed backend does not serve is looked for along the profile's `fallback_order`, or in the internal backend when it has none, when the name reached that backend by default, is left out when it reached it through a wildcard pattern, and is not served at all when the profile routes the name to that backend exactly: an exact route pins the name to one backend.+A routing profile routes a handle's name, so its route applies to every kind of model of that name. A model type the routed backend does not serve is looked for along the profile's `fallback_order`, or in the internal backend when it has none, when the name reached that backend by default, is left out when it reached it through a wildcard pattern, and is not served at all when the profile routes the name to that backend exactly: an exact route pins the name to one backend. A name the profile sends to no enabled backend, because no route matches it and neither its `default` nor any backend of its `fallback_order` is enabled, is served by the internal backend for each model type that backend declares under it, and by no other backend.A method whose pipe names a handle the deck serves only as another kind of model is refused when it loads: a `PipeJudge` naming a handle served only as an LLM is told that the deck serves the handle, but not as a judgment model, and the suggestions list judgment models only.@@ -425,7 +425,9 @@ Common scenarios for hybrid routing:The **internal backend** is a special backend containing software-only models that run locally without requiring AI services. These include models for PDF text extraction, local document parsing, and other processing tasks that don't need external API calls.-Unlike other backends, internal backend models are **always available** regardless of which routing profile you select. This means you can use these models even when your routing profile is focused on a specific AI provider (e.g., `all_anthropic` or `all_openai`).+Unlike other backends, internal backend models are **always available** regardless of which routing profile you select. This means you can use these models even when your routing profile is focused on a specific AI provider (e.g., `all_anthropic` or `all_openai`), and even when no backend your profile routes to is enabled: with the internal backend alone enabled, the shipped `all_enabled_backends` profile still serves every internal model.++Two things withhold an internal model. A route that sends its name to another backend on purpose, whether an exact route or a wildcard pattern that catches it, decides where that name goes, as it does for any model. Disabling the internal backend turns off all of its models, including the ones a plugin adds.This behavior is automatic and requires no additional configuration. To see which models are available from the internal backend, check `.pipelex/inference/backends/internal.toml`.diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.pyindex 6d24e8120..15664213d 100644--- a/pipelex/cogt/models/model_manager.py+++ b/pipelex/cogt/models/model_manager.py@@ -169,7 +169,8 @@ class ModelManager(ModelManagerAbstract):backend is then asked for the spec of each type the name is declared as. Each matching methodkeeps its meaning per pair: a pattern match skips a pair its backend lacks, a default matchlooks for it along the fallback order, and an exact match pins the name to its backend, so a- type that backend does not serve is not served at all.+ type that backend does not serve is not served at all. A name the profile sends to no enabled+ backend is served by the internal backend, for each type that backend declares under it.Raises:ModelManagerError: A routed backend is not loaded, or a handle routed exactly to a backend@@ -183,6 +184,12 @@ class ModelManager(ModelManagerAbstract):model_name=model_name,)if backend_match_for_model is None:+ # The profile sends the name nowhere, as a profile routing only to providers does when none of+ # them is enabled. The internal backend's models need no provider, so they are served all the same:+ # only a route sending their name elsewhere, or disabling the internal backend, withholds them.+ for model_type in declared_model_types:+ if model_spec := self._find_internal_model_spec(model_name=model_name, model_type=model_type, enabled_backends=enabled_backends):+ inference_models.add(model_spec)continuematched_backend_name = backend_match_for_model.backend_namematched_backend = self.inference_backend_library.get_inference_backend(backend_name=matched_backend_name)@@ -258,7 +265,7 @@ class ModelManager(ModelManagerAbstract):"""The spec serving one pair, found by the routing match of its name, or `None` when the pair is not served.Raises:- ModelManagerError: A backend of the fallback order is not loaded.+ ModelManagerError: A backend of the fallback order, or the enabled internal backend, is not loaded."""if model_spec := matched_backend.get_model_spec(model_type=model_type, handle=model_name):return model_spec@@ -280,18 +287,16 @@ class ModelManager(ModelManagerAbstract):case BackendMatchingMethod.DEFAULT:# We could not find the pair, but it was a default match,# so we can look for it in the other available backends- # Use fallback_order if specified, otherwise only try internal backend- if backend_match_for_model.fallback_order:- # Try fallback_order first, then any enabled backends not in fallback_order- backends_to_try = backend_match_for_model.fallback_order + [- b for b in enabled_backends if b not in backend_match_for_model.fallback_order- ]- else:- # No fallback_order specified - only try the internal backend as a special case- # Internal backend contains software-only models that should always be available- # regardless of which AI provider routing profile is selected- backends_to_try = [PipelexBackend.INTERNAL] if PipelexBackend.INTERNAL in enabled_backends else []-+ if not backend_match_for_model.fallback_order:+ # No fallback_order specified - only try the internal backend, whose software-only models+ # are served whatever the routing profile, unless it is the matched backend we just checked+ if matched_backend.name == PipelexBackend.INTERNAL:+ return None+ return self._find_internal_model_spec(model_name=model_name, model_type=model_type, enabled_backends=enabled_backends)+ # Try fallback_order first, then any enabled backends not in fallback_order+ backends_to_try = backend_match_for_model.fallback_order + [+ b for b in enabled_backends if b not in backend_match_for_model.fallback_order+ ]for available_backend in backends_to_try:if available_backend == matched_backend.name:# we've already checked the matched backend and it didn't have the pair, that's why we're here@@ -306,6 +311,23 @@ class ModelManager(ModelManagerAbstract):# Not all models need to be available in the configured backendsreturn None+ def _find_internal_model_spec(self, *, model_name: str, model_type: ModelType, enabled_backends: list[str]) -> InferenceModelSpec | None:+ """The internal backend's spec for one pair, or `None` when the internal backend is not enabled or does not declare it.++ The internal backend runs software-only models, with no provider behind them, so whatever the active profile+ it is looked in for a name the profile sends to no enabled backend, and for a default match with no fallback order.++ Raises:+ ModelManagerError: The internal backend is enabled but not loaded.+ """+ if PipelexBackend.INTERNAL not in enabled_backends:+ return None+ internal_backend = self.inference_backend_library.get_inference_backend(backend_name=PipelexBackend.INTERNAL)+ if internal_backend is None:+ msg = f"Backend '{PipelexBackend.INTERNAL}' not found for model '{model_name}'"+ raise ModelManagerError(msg)+ return internal_backend.get_model_spec(model_type=model_type, handle=model_name)+@overridedef get_inference_model(self, model_handle: str, *, model_type: ModelType) -> InferenceModelSpec:if self.model_deck is None:diff --git a/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py b/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.pyindex d89674a13..09c746fcf 100644--- a/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py+++ b/tests/unit/pipelex/cogt/models/model_deck_per_pair_utils.py@@ -62,12 +62,13 @@ def make_deck_blueprint(*, extract_choice_default: ExtractModelChoice = "extract)-def build_deck_manager(*backends: InferenceBackend, routing_profile: RoutingProfile) -> ModelManager:+def build_deck_manager(*backends: InferenceBackend, routing_profile: RoutingProfile, enabled_backends: list[str] | None = None) -> ModelManager:+ """A manager whose deck is built over these backends, every one of them enabled unless `enabled_backends` names a subset."""manager = ModelManager()manager.inference_backend_library.root = {backend.name: backend for backend in backends}manager._routing_profile = routing_profile # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]manager.model_deck = manager.build_deck(model_deck_blueprint=make_deck_blueprint(),- enabled_backends=[backend.name for backend in backends],+ enabled_backends=enabled_backends if enabled_backends is not None else [backend.name for backend in backends],)return managerdiff --git a/tests/unit/pipelex/cogt/models/test_internal_models_under_the_kit_profile.py b/tests/unit/pipelex/cogt/models/test_internal_models_under_the_kit_profile.pynew file mode 100644index 000000000..9eebce27e--- /dev/null+++ b/tests/unit/pipelex/cogt/models/test_internal_models_under_the_kit_profile.py@@ -0,0 +1,77 @@+"""The internal backend's models, served by the kit's own routing profile whichever provider backends are enabled.++The kit's active profile routes along a `fallback_order` of provider backends and has no `default`, so with none of+them enabled it sends every name nowhere; the internal backend serves its models all the same. Booted the way an+installation boots, through the global `config_manager` over the kit's inference tree in a faked home, from a project+root with no `.pipelex/`, and keyless (`needs_inference=False`) so no credential on this machine is a precondition.+"""++from __future__ import annotations++import shutil+from pathlib import Path+from typing import TYPE_CHECKING++import pytest++from pipelex.cogt.model_backends.backend import PipelexBackend+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.models.model_manager import ModelManager+from pipelex.kit.paths import get_kit_configs_dir+from pipelex.plugins.plugin_model_declarations import PluginModelDeclarations+from pipelex.system.configuration.config_loader import INFERENCE_DIR_NAME+from pipelex.system.environment import CONFIG_DIR_NAME+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider++if TYPE_CHECKING:+ from pytest_mock import MockerFixture+++pytestmark = pytest.mark.usefixtures("no_pipelex_home")++INTERNAL_ONLY_BACKENDS = "[internal]\nenabled = true\n"+# TypeSafe is reached only through the kit profile's optional route for its own models, outside its `fallback_order`.+INTERNAL_AND_TYPESAFE_BACKENDS = f'{INTERNAL_ONLY_BACKENDS}\n[typesafe]\nenabled = true\napi_key = "${{TYPESAFE_API_KEY}}"\n'+# The control: Mistral is in the `fallback_order`, so the profile matches every name by default.+INTERNAL_AND_MISTRAL_BACKENDS = f'{INTERNAL_ONLY_BACKENDS}\n[mistral]\nenabled = true\napi_key = "${{MISTRAL_API_KEY}}"\n'++KIT_INTERNAL_MODELS = [+ ("reportlab-pdf", ModelType.DOC_GEN),+ ("pypdfium2-extract-pdf", ModelType.TEXT_EXTRACTOR),+ ("docling-extract-text", ModelType.TEXT_EXTRACTOR),+]+++class TestInternalModelsUnderTheKitProfile:+ @pytest.fixture+ def global_inference_dir(self, tmp_path: Path, mocker: MockerFixture) -> Path:+ """A faked home carrying the kit's inference tree, and a project root with no `.pipelex/` to shadow it."""+ fake_home = tmp_path / "home"+ inference_dir = fake_home / CONFIG_DIR_NAME / INFERENCE_DIR_NAME+ shutil.copytree(Path(str(get_kit_configs_dir())) / INFERENCE_DIR_NAME, inference_dir)+ project_root = tmp_path / "project"+ (project_root / ".git").mkdir(parents=True)+ mocker.patch.object(Path, "home", return_value=fake_home)+ mocker.patch.object(Path, "cwd", return_value=project_root)+ return inference_dir++ @pytest.mark.parametrize(+ "backends_toml",+ [INTERNAL_ONLY_BACKENDS, INTERNAL_AND_TYPESAFE_BACKENDS, INTERNAL_AND_MISTRAL_BACKENDS],+ ids=["internal_only", "internal_and_typesafe", "internal_and_mistral"],+ )+ def test_the_kit_profile_serves_every_internal_model(self, global_inference_dir: Path, backends_toml: str) -> None:+ (global_inference_dir / "backends.toml").write_text(backends_toml, encoding="utf-8")++ models_manager = ModelManager()+ models_manager.setup(+ secrets_provider=EnvSecretsProvider(),+ plugin_model_declarations=PluginModelDeclarations.make_empty(),+ needs_inference=False,+ )++ assert models_manager.routing_profile.name == "all_enabled_backends"+ model_deck = models_manager.get_model_deck()+ for model_handle, model_type in KIT_INTERNAL_MODELS:+ inference_model = model_deck.get_required_inference_model(model_handle=model_handle, model_type=model_type)+ assert inference_model.backend_name == PipelexBackend.INTERNALdiff --git a/tests/unit/pipelex/cogt/models/test_model_deck_build_internal_backend_under_no_match.py b/tests/unit/pipelex/cogt/models/test_model_deck_build_internal_backend_under_no_match.pynew file mode 100644index 000000000..bbe331c54--- /dev/null+++ b/tests/unit/pipelex/cogt/models/test_model_deck_build_internal_backend_under_no_match.py@@ -0,0 +1,85 @@+"""The internal backend as the last resort of the deck build: a name the profile sends to no enabled backend.++The backends are built in memory, so these tests say nothing of how a backend file declares a model.+"""++import pytest++from pipelex.cogt.model_backends.backend import PipelexBackend+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.model_routing.routing_profile import RoutingProfile+from tests.unit.pipelex.cogt.models.model_deck_per_pair_utils import TWIN_HANDLE, build_deck_manager, make_backend++INTERNAL_HANDLE = "reportlab-pdf"+++class TestDeckBuildInternalBackendUnderNoMatch:+ """A name the profile sends to no enabled backend is served by the internal backend, for each type it declares there."""++ def test_a_fallback_order_reaching_no_enabled_backend_serves_the_internal_pairs_from_the_internal_backend(self) -> None:+ manager = build_deck_manager(+ make_backend(PipelexBackend.INTERNAL, (INTERNAL_HANDLE, ModelType.DOC_GEN), (TWIN_HANDLE, ModelType.DOC_GEN)),+ make_backend("elsewhere", (TWIN_HANDLE, ModelType.LLM), ("claude", ModelType.LLM)),+ routing_profile=RoutingProfile(name="profile", fallback_order=["openai", "anthropic"]),+ )+ model_deck = manager.get_model_deck()++ assert manager.get_inference_model(INTERNAL_HANDLE, model_type=ModelType.DOC_GEN).backend_name == PipelexBackend.INTERNAL+ # The internal backend serves the types it declares, and a type another backend declares under the name stays unserved.+ assert model_deck.inference_models.types_serving(handle=TWIN_HANDLE) == [ModelType.DOC_GEN]+ assert not model_deck.inference_models.types_serving(handle="claude")++ def test_a_profile_made_only_of_routes_serves_an_internal_name_its_routes_do_not_mention(self) -> None:+ manager = build_deck_manager(+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),+ make_backend("anthropic", ("claude", ModelType.LLM)),+ make_backend(PipelexBackend.INTERNAL, (INTERNAL_HANDLE, ModelType.DOC_GEN)),+ routing_profile=RoutingProfile(name="profile", routes={TWIN_HANDLE: "openai"}),+ )+ model_deck = manager.get_model_deck()++ assert manager.get_inference_model(INTERNAL_HANDLE, model_type=ModelType.DOC_GEN).backend_name == PipelexBackend.INTERNAL+ assert manager.get_inference_model(TWIN_HANDLE, model_type=ModelType.LLM).backend_name == "openai"+ # Only the internal backend is a last resort: another backend's unrouted name stays unserved.+ assert not model_deck.inference_models.types_serving(handle="claude")++ @pytest.mark.parametrize(+ "routing_profile",+ [+ RoutingProfile(name="unreachable_fallback", fallback_order=["openai", "anthropic"]),+ RoutingProfile(name="routes_only", routes={TWIN_HANDLE: "elsewhere"}),+ ],+ )+ def test_with_the_internal_backend_not_enabled_its_models_stay_unserved_and_the_deck_builds(self, routing_profile: RoutingProfile) -> None:+ """A disabled internal backend withholds its models, whatever the profile; the internal backend is here but not enabled."""+ manager = build_deck_manager(+ make_backend(PipelexBackend.INTERNAL, (INTERNAL_HANDLE, ModelType.DOC_GEN)),+ make_backend("elsewhere", (TWIN_HANDLE, ModelType.LLM)),+ routing_profile=routing_profile,+ enabled_backends=["elsewhere"],+ )+ model_deck = manager.get_model_deck()++ assert not model_deck.inference_models.types_serving(handle=INTERNAL_HANDLE)++ def test_a_pattern_route_catching_an_internal_name_to_a_backend_lacking_the_pair_leaves_it_out(self) -> None:+ """A wildcard is a routing statement: the internal backend rescues only a name the profile sends nowhere."""+ manager = build_deck_manager(+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),+ make_backend(PipelexBackend.INTERNAL, (INTERNAL_HANDLE, ModelType.DOC_GEN)),+ routing_profile=RoutingProfile(name="profile", routes={"reportlab-*": "openai"}),+ )+ model_deck = manager.get_model_deck()++ assert not model_deck.inference_models.types_serving(handle=INTERNAL_HANDLE)++ def test_an_exact_route_of_an_internal_name_to_another_backend_still_pins_it(self) -> None:+ """The route says where the name goes, so the internal backend's type of it is not served."""+ manager = build_deck_manager(+ make_backend("openai", (TWIN_HANDLE, ModelType.LLM)),+ make_backend(PipelexBackend.INTERNAL, (TWIN_HANDLE, ModelType.DOC_GEN)),+ routing_profile=RoutingProfile(name="profile", routes={TWIN_HANDLE: "openai"}),+ )+ model_deck = manager.get_model_deck()++ assert model_deck.inference_models.types_serving(handle=TWIN_HANDLE) == [ModelType.LLM]diff --git a/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py b/tests/unit/pipelex/cogt/models/test_plugin_model_merge.pyindex d3ad83b52..0f43dcff9 100644--- a/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py+++ b/tests/unit/pipelex/cogt/models/test_plugin_model_merge.py@@ -189,3 +189,23 @@ class TestPluginModelMerge:assert not model_deck.inference_models.types_serving(handle="pipelex-xlsx")assert model_deck.get_doc_gen_choice_default(doc_gen_format=DocGenFormat.XLSX, source=DocGenSource.LAYOUT) is None++ def test_a_plugin_model_is_served_under_a_profile_reaching_no_enabled_backend(self, inference_dir: Path) -> None:+ """With the internal backend alone enabled, the kit's own profile sends the plugin's name nowhere, and the internal backend serves it."""+ (inference_dir / "backends.toml").write_text("[internal]\nenabled = true\n", encoding="utf-8")+ routing_profiles_path = inference_dir / "routing_profiles.toml"+ routing_profiles_path.write_text(+ routing_profiles_path.read_text(encoding="utf-8").replace('active = "all_internal"', 'active = "all_enabled_backends"', 1),+ encoding="utf-8",+ )++ models_manager = self._setup(inference_dir=inference_dir, plugin_model_declarations=_xlsx_plugin_declarations())++ # Under `all_internal`, the fixture's profile, the name matches by default and the no-match path is never taken.+ assert models_manager.routing_profile.name == "all_enabled_backends"+ model_deck = models_manager.get_model_deck()+ inference_model = model_deck.get_required_inference_model(model_handle="pipelex-xlsx", model_type=ModelType.DOC_GEN)+ assert inference_model.backend_name == PipelexBackend.INTERNAL+ choice = model_deck.get_doc_gen_choice_default(doc_gen_format=DocGenFormat.XLSX, source=DocGenSource.LAYOUT)+ assert choice is not None+ assert model_deck.get_doc_gen_setting(doc_gen_choice=choice).model == "pipelex-xlsx"