Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Image-judgment-alias · L-261011-8a98d3 (#1419)

Louis Choquelae514dd36b65
diff --git a/.pipelex/inference/deck/6_judgment_deck.toml b/.pipelex/inference/deck/6_judgment_deck.toml
index 5fec6dd59..0370c0ee2 100644
--- a/.pipelex/inference/deck/6_judgment_deck.toml
+++ b/.pipelex/inference/deck/6_judgment_deck.toml
@@ -39,9 +39,18 @@
#
# `jev-1.13.0` is TypeSafe's. Its next version is found on the provider's models page and not through
# its API, whose model listing names only the moving aliases — moving this pin is a documentation read.
+#
+# Jev reads text alone, so a `PipeJudge` whose prompt presents an image names `@default-judgment-vision`,
+# a judgment model that reads images, as a `PipeExtract` over a web page names `@default-extract-web-page`.
+# It is OpenAI's `gpt-6-luna`, judging through OpenAI's Decisions API, and the one alias here that is not
+# versioned, because nothing versioned exists to pin: the Decisions API serves it under that name alone.
+# Pin the first dated snapshot OpenAI publishes. No backend file of the kit declares it as a judgment model
+# yet, since the openai plugin does not call the Decisions API, so on your own keys this alias resolves to
+# no model until it does.
[judgment.aliases]
default-judgment = "jev-1.13.0"
+default-judgment-vision = "gpt-6-luna"
####################################################################################################
# Judgment Presets
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 684880d9a..7830f514f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@
- **`pipelex usage-stats`**: `status` (with `--format json`), `disable`, `enable` and `reset-id` show and change the statistics of every Pipelex tool on the machine, through the state they share in `~/.mthds/usage-stats/`. `PIPELEX_USAGE_STATS_DEBUG` prints each event to stderr as a JSON line instead of sending it, and `PIPELEX_USAGE_STATS_ENDPOINT` sends to a loopback origin for tests. Each queued file names the endpoint its events were queued for, and a sender posts the files naming its own origin and those naming none, which earlier builds wrote, so events queued under the variable never reach PostHog.
- **`[usage_stats] enabled`**: a new `pipelex.toml` section, `true` in the packaged configuration. Set to `false` in a project's `.pipelex/pipelex.toml` it turns statistics off for everyone working in the project; it is read from the project's own file, never from `~/.pipelex/`, and a file that cannot be read turns them off too.
- **Token usage names the backend that served the call**: `LLMTokensUsage`, `ImgGenTokensUsage`, `ExtractTokensUsage`, `SearchTokensUsage` and `JudgmentTokensUsage` gain `backend_name`, set by the job from the backend whose worker ran it, so a run's usage events credit each call to its own backend even when two backends call the model by the same id, as after a fallback. A synthetic report no backend made, a dry run's, leaves it `None`.
+- **`@default-judgment-vision`**: the judgment deck names a second alias, for a judgment model that reads images, beside `@default-judgment`, which stays TypeSafe's text-only `jev-1.13.0`. It points at OpenAI's `gpt-6-luna`, which judges through OpenAI's Decisions API; no inference backend of Pipelex calls that API yet, so on your own keys a `PipeJudge` naming the alias finds no model when it runs. Run `pipelex update` to bring the alias into an installed deck.
### Changed
diff --git a/api/.pipelex/inference/deck/6_judgment_deck.toml b/api/.pipelex/inference/deck/6_judgment_deck.toml
index 5fec6dd59..0370c0ee2 100644
--- a/api/.pipelex/inference/deck/6_judgment_deck.toml
+++ b/api/.pipelex/inference/deck/6_judgment_deck.toml
@@ -39,9 +39,18 @@
#
# `jev-1.13.0` is TypeSafe's. Its next version is found on the provider's models page and not through
# its API, whose model listing names only the moving aliases — moving this pin is a documentation read.
+#
+# Jev reads text alone, so a `PipeJudge` whose prompt presents an image names `@default-judgment-vision`,
+# a judgment model that reads images, as a `PipeExtract` over a web page names `@default-extract-web-page`.
+# It is OpenAI's `gpt-6-luna`, judging through OpenAI's Decisions API, and the one alias here that is not
+# versioned, because nothing versioned exists to pin: the Decisions API serves it under that name alone.
+# Pin the first dated snapshot OpenAI publishes. No backend file of the kit declares it as a judgment model
+# yet, since the openai plugin does not call the Decisions API, so on your own keys this alias resolves to
+# no model until it does.
[judgment.aliases]
default-judgment = "jev-1.13.0"
+default-judgment-vision = "gpt-6-luna"
####################################################################################################
# Judgment Presets
diff --git a/docs/building-methods/pipes/pipe-operators/PipeJudge.md b/docs/building-methods/pipes/pipe-operators/PipeJudge.md
index be08330ee..381f1a2c8 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeJudge.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeJudge.md
@@ -32,7 +32,7 @@ The output must be that native or a concept refining it, and `Dynamic` is neithe
**The question is plain text.** `question` is a template as well, with the same shorthand, but it presents no file: it suits the instruction itself and a short parameter such as `"Is the message about $topic?"`. The evidence belongs in the prompt, and a question that reads an `Image` or a `Document` is refused when the method loads, since only the prompt presents files.
-**Images and documents are sent as files.** An `Image` or a `Document` the prompt reads, a single value or a list, reaches the model as a file rather than as text, and a numbered token such as `[Image 1]` marks the place in the rendered prompt where the prompt reads it. Whether a model reads files is stated in its model configuration, and a `PipeJudge` whose prompt reads an image or a document is refused when the method loads if its model does not read that kind of file, with a message naming the variable and what the model reads. An input declared `Dynamic` names no kind of value, so the prompt renders it as text, as a `PipeLLM`'s does, and an image it holds reaches the model as its address rather than as a file. A document given to the run in a format the model does not read is refused when the run starts, before any step spends anything. TypeSafe's model reads text only, so turn a document into text with a [`PipeExtract`](PipeExtract.md) step first.
+**Images and documents are sent as files.** An `Image` or a `Document` the prompt reads, a single value or a list, reaches the model as a file rather than as text, and a numbered token such as `[Image 1]` marks the place in the rendered prompt where the prompt reads it. Whether a model reads files is stated in its model configuration, and a `PipeJudge` whose prompt reads an image or a document is refused when the method loads if its model does not read that kind of file, with a message naming the variable and what the model reads. An input declared `Dynamic` names no kind of value, so the prompt renders it as text, as a `PipeLLM`'s does, and an image it holds reaches the model as its address rather than as a file. A document given to the run in a format the model does not read is refused when the run starts, before any step spends anything. TypeSafe's model reads text only: a prompt that presents an image names `@default-judgment-vision`, the deck's alias for a judgment model that reads images (see [the judgment model](#the-judgment-model)), and a document is turned into text with a [`PipeExtract`](PipeExtract.md) step first.
**The verdict.** A yes/no question's verdict follows from its probability: yes when the probability of yes is at least the `threshold`, which defaults to `0.5`. A model that reports no probability gives its own verdict, and a declared threshold then cannot apply, which the run logs as a warning. A choice's verdict is the key of the option chosen, with the model's `confidence` and the `probabilities` of every option when it reports them. A rating's verdict is the index of the level chosen, `0` being the first level declared, with the model's `confidence`, the `probabilities` of every level keyed by its index written as text, and a continuous `position` on the scale, when it reports them. When the scale's levels carry labels, the verdict's `label` is the label of the level chosen, copied from the pipe's declaration.
@@ -94,6 +94,8 @@ levels = [{ label = "Minor" }, { label = "Major" }, { label = "Blocking" }]
No judgment model is served out of the box: the model deck names no default for judgments, so a `PipeJudge` names its model in its `model` field, and a `PipeJudge` that names none is refused when the method loads. The deck's `@default-judgment` alias points at TypeSafe's model, which works once a `TYPESAFE_API_KEY` is set (see [the inference backend configuration](../../../configuration/config-technical/inference-backend-config.md)). To let your pipes leave `model` out, set a `choice_default` under `[judgment]` in your model deck.
+TypeSafe's model reads text alone, so the deck also names `@default-judgment-vision` for a `PipeJudge` whose prompt presents an image, as a `PipeExtract` over a web page names `@default-extract-web-page`. It points at OpenAI's `gpt-6-luna`, a judgment model that reads images through OpenAI's Decisions API. No inference backend of Pipelex calls that API yet, so on your own keys a step naming the alias finds no model when it runs.
+
### MTHDS Parameters
| Parameter | Type | Description | Required |
diff --git a/docs/features/judgment.md b/docs/features/judgment.md
index 2537f8e5b..723694a71 100644
--- a/docs/features/judgment.md
+++ b/docs/features/judgment.md
@@ -26,7 +26,7 @@ Many steps of a method come down to a closed question: is this message urgent, w
Put a PipeJudge where a method needs a decision, then route on the verdict: a `PipeCondition` whose expression is `routing.choice` sends each item to the branch its judgment chose, and one comparing `approved.probability` with a bound gates a step on the model's certainty. To judge every item of a list, map the PipeJudge over it with a `PipeBatch`.
-No judgment model is served out of the box: a judgment names its model, such as the deck's `@default-judgment` alias for TypeSafe's model, which needs a `TYPESAFE_API_KEY`.
+No judgment model is served out of the box: a judgment names its model, such as the deck's `@default-judgment` alias for TypeSafe's model, which needs a `TYPESAFE_API_KEY`. TypeSafe's model reads text alone, and a judgment whose prompt presents an image names the deck's `@default-judgment-vision` alias instead, for OpenAI's `gpt-6-luna`, which no inference backend of Pipelex serves on your own keys yet.
## Related Documentation
diff --git a/docs/get-started/configure-ai-providers.md b/docs/get-started/configure-ai-providers.md
index 93ba674c8..3cce0ef5d 100644
--- a/docs/get-started/configure-ai-providers.md
+++ b/docs/get-started/configure-ai-providers.md
@@ -150,6 +150,7 @@ The deck files `pipelex init` installs resolve their defaults to these models, a
- **Image generation** — the general and premium tiers are GPT Image 2, the small tier is GPT Image 1 mini, both served by `openai` and `azure_openai`.
- **Document extraction** — `default-extract-document` tries Mistral OCR, which needs your Mistral key, and otherwise reads a PDF's text layer locally with pypdfium2, which needs no key but recovers no text from a scanned page. `default-extract-image` and `default-premium` are Mistral OCR only. `default-text-from-pdf` and `default-no-inference` always read the PDF locally and call no model. See [Document Extraction](../features/document-extraction.md#the-default-extractor).
- **Web search and web pages** — everything in `4_search_deck.toml`, and `default-extract-web-page` in `3_extract_deck.toml`, resolve to Linkup, so a method that searches the web or extracts from a web page needs a Linkup key.
+- **Judgment** — the deck names no default judgment model, so a `PipeJudge` names its model. `default-judgment` is TypeSafe's `jev-1.13.0`, which needs a TypeSafe key and reads text alone. `default-judgment-vision`, for a judgment whose prompt presents an image, is OpenAI's `gpt-6-luna`, which no backend of Pipelex serves as a judgment model yet. See [PipeJudge](../building-methods/pipes/pipe-operators/PipeJudge.md#the-judgment-model).
With an OpenAI or Azure OpenAI key alone, the language and image defaults run as shipped. With any other provider, OpenRouter included, point the aliases at models it serves, as the next paragraph describes; the models each provider serves are listed under `~/.pipelex/inference/backends/`. A few presets name a model directly rather than through an alias, `retrieval-cheap`, `retrieval-premium`, `engineering-code-cheap` and `engineering-code-cheaper`, so they need overriding too. On OpenRouter, the presets that set a reasoning effort, `deep-analysis`, `quick-reasoning` and `retrieval-premium`, also need overriding without it, because its models are declared without reasoning support.
diff --git a/pipelex/kit/configs/inference/deck/6_judgment_deck.toml b/pipelex/kit/configs/inference/deck/6_judgment_deck.toml
index 5fec6dd59..0370c0ee2 100644
--- a/pipelex/kit/configs/inference/deck/6_judgment_deck.toml
+++ b/pipelex/kit/configs/inference/deck/6_judgment_deck.toml
@@ -39,9 +39,18 @@
#
# `jev-1.13.0` is TypeSafe's. Its next version is found on the provider's models page and not through
# its API, whose model listing names only the moving aliases — moving this pin is a documentation read.
+#
+# Jev reads text alone, so a `PipeJudge` whose prompt presents an image names `@default-judgment-vision`,
+# a judgment model that reads images, as a `PipeExtract` over a web page names `@default-extract-web-page`.
+# It is OpenAI's `gpt-6-luna`, judging through OpenAI's Decisions API, and the one alias here that is not
+# versioned, because nothing versioned exists to pin: the Decisions API serves it under that name alone.
+# Pin the first dated snapshot OpenAI publishes. No backend file of the kit declares it as a judgment model
+# yet, since the openai plugin does not call the Decisions API, so on your own keys this alias resolves to
+# no model until it does.
[judgment.aliases]
default-judgment = "jev-1.13.0"
+default-judgment-vision = "gpt-6-luna"
####################################################################################################
# Judgment Presets
diff --git a/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml b/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml
index 5fec6dd59..0370c0ee2 100644
--- a/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml
+++ b/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml
@@ -39,9 +39,18 @@
#
# `jev-1.13.0` is TypeSafe's. Its next version is found on the provider's models page and not through
# its API, whose model listing names only the moving aliases — moving this pin is a documentation read.
+#
+# Jev reads text alone, so a `PipeJudge` whose prompt presents an image names `@default-judgment-vision`,
+# a judgment model that reads images, as a `PipeExtract` over a web page names `@default-extract-web-page`.
+# It is OpenAI's `gpt-6-luna`, judging through OpenAI's Decisions API, and the one alias here that is not
+# versioned, because nothing versioned exists to pin: the Decisions API serves it under that name alone.
+# Pin the first dated snapshot OpenAI publishes. No backend file of the kit declares it as a judgment model
+# yet, since the openai plugin does not call the Decisions API, so on your own keys this alias resolves to
+# no model until it does.
[judgment.aliases]
default-judgment = "jev-1.13.0"
+default-judgment-vision = "gpt-6-luna"
####################################################################################################
# Judgment Presets
diff --git a/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py b/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
index b9f947868..2d51f2631 100644
--- a/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
+++ b/tests/unit/pipelex/cogt/models/model_deck_validation_utils.py
@@ -7,6 +7,14 @@ and are used by multiple test modules.
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_reference import ModelReference, ModelReferenceKind
+# Deck aliases that knowingly name a model no backend file of the kit declares as their family's type,
+# because the backend that would serve it on the user's own keys does not exist yet, each with the reason.
+# The deck's own comment beside the alias says the same to the person reading the deck. The shipped deck's
+# default guard holds each entry to staying unserved, so it ends with the gap it explains.
+ALIASES_AWAITING_A_KIT_BACKEND: dict[tuple[ModelType, str], str] = {
+ (ModelType.JUDGMENT, "default-judgment-vision"): "gpt-6-luna judges through OpenAI's Decisions API, which the openai plugin does not call yet",
+}
+
def find_invalid_alias_targets(
aliases: dict[str, str],
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_references.py b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
index bfea7c778..a0f0d8cf6 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_references.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
@@ -26,6 +26,7 @@ from pipelex.system.configuration.config_loader import config_manager
from pipelex.tools.misc.file_utils import find_files_in_dir
from pipelex.tools.misc.toml_utils import load_toml_from_path_if_exists
from tests.unit.pipelex.cogt.models.model_deck_validation_utils import (
+ ALIASES_AWAITING_A_KIT_BACKEND,
find_circular_aliases,
find_invalid_alias_targets,
find_invalid_waterfall_entries,
@@ -164,11 +165,19 @@ class TestModelDeckReferences:
model_type: ModelType,
deck_name: str,
):
- """Verify aliases point to valid targets (other aliases, waterfalls, or direct handles)."""
+ """Verify aliases point to valid targets (other aliases, waterfalls, or direct handles).
+
+ An alias knowingly awaiting a kit backend is left out: `test_shipped_deck_defaults.py` holds it to staying unserved.
+ """
deck = self._get_deck_for_model_type(model_deck_blueprint, model_type)
+ checked_aliases = {
+ alias_name: target_value
+ for alias_name, target_value in deck.aliases.items()
+ if (model_type, alias_name) not in ALIASES_AWAITING_A_KIT_BACKEND
+ }
invalid_refs = find_invalid_alias_targets(
- aliases=deck.aliases,
+ aliases=checked_aliases,
all_aliases=deck.aliases,
all_waterfalls=deck.waterfalls,
known_model_handles=all_known_model_handles,
diff --git a/tests/unit/pipelex/kit/test_shipped_deck_defaults.py b/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
index 562976894..dc09a99ba 100644
--- a/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
+++ b/tests/unit/pipelex/kit/test_shipped_deck_defaults.py
@@ -6,6 +6,11 @@ that relies on it, for everyone running on their own keys. A waterfall passes wh
is declared, which is how a default can put a managed model first and still resolve without it. A handle
names one model per model type, so a default reaches a model only through a handle declared as the type
its family asks for: a judgment default naming a handle the kit declares only as an LLM serves nothing.
+
+A default that knowingly reaches no declared model, because the backend that would serve it on the user's
+own keys does not exist yet, is listed with the reason in `ALIASES_AWAITING_A_KIT_BACKEND`, and must stay
+unserved: its entry fails the day a backend file declares what it names, so the exception ends with the gap
+it explains.
"""
import pytest
@@ -14,6 +19,7 @@ from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.deck_manifest import KitManagedArea, kit_deck_dir, list_managed_kit_files
from pipelex.cogt.models.model_deck import JudgmentDeckBlueprint
from pipelex.cogt.models.model_reference import ModelReference, ModelReferenceKind
+from tests.unit.pipelex.cogt.models.model_deck_validation_utils import ALIASES_AWAITING_A_KIT_BACKEND
from tests.unit.pipelex.kit.test_deck_variants import (
FAMILY_MODEL_TYPES,
DeckFamilyBlueprint,
@@ -83,7 +89,22 @@ class TestShippedDeckDefaults:
unserved = _unserved_default(
family=family, alias=alias, family_blueprint=_shipped_family_blueprints()[family], declared_models=list_declared_backend_models()
)
- assert unserved is None, unserved
+ awaited_backend = ALIASES_AWAITING_A_KIT_BACKEND.get((FAMILY_MODEL_TYPES[family], alias))
+ if awaited_backend is None:
+ assert unserved is None, unserved
+ else:
+ assert unserved is not None, (
+ f"{family} alias '{alias}' is listed as awaiting a kit backend ({awaited_backend}), but a backend file of the kit now "
+ "declares what it names: remove its entry from ALIASES_AWAITING_A_KIT_BACKEND, and the deck comment that explains the gap"
+ )
+
+ @pytest.mark.parametrize(("model_type", "alias"), sorted(ALIASES_AWAITING_A_KIT_BACKEND))
+ def test_an_alias_awaiting_a_kit_backend_is_a_shipped_default(self, model_type: ModelType, alias: str) -> None:
+ """An entry for an alias the shipped deck no longer defines is stale, and would excuse the next alias given its name."""
+ shipped_defaults = {(FAMILY_MODEL_TYPES[family], default_alias) for family, default_alias in _default_aliases()}
+ assert (model_type, alias) in shipped_defaults, (
+ f"{model_type} alias '{alias}' is listed as awaiting a kit backend, but the shipped deck defines no such default"
+ )
def test_a_judgment_default_naming_a_handle_the_kit_declares_only_as_an_llm_is_refused(self) -> None:
"""The guard must not count a handle's LLM spec for the judgment family, which asks for a judgment model."""