Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Judgment-cogt-family · L-260919-502f36 — judgment cogt family, typesafe worker (#1215)
Louis Choquel9d54f072b47e
diff --git a/.drift/acks/cli-docs.toml b/.drift/acks/cli-docs.tomlindex 7272d2ce2..ebb116a37 100644--- a/.drift/acks/cli-docs.toml+++ b/.drift/acks/cli-docs.toml@@ -1,8 +1,8 @@contract = "cli-docs"-digest = "sha256:edfad164dcb9959a90b014f8779348db0d43a1caaa23efa75b3eea3503b6f3ab"-reviewed_by = "Louis Choquel"-reviewed_at = "2026-09-30T16:40:04Z"-rationale = "clean-pass: the main-output file copy now compares the names already in the results folder without case on every file system, which the CI failure on Linux exposed; docs/tools/cli/run.md already states the rule (a taken name gets a -1 suffix) and the agent CLI CLAUDE.md names none of this behaviour, so no doc change is needed."+digest = "sha256:9ba08b783d4a8cbd140a7e64af16b43ccc7987be97ea9ee0bcfe76d30da7fae2"+reviewed_by = "Claude (Opus 5.5)"+reviewed_at = "2026-10-02T21:24:09Z"+rationale = "clean-pass: merging dev into the judgment cogt branch leaves one trigger change against dev's ack, check-model's --type help naming judgment, which matches ModelCategory's members as pipelex/builder/operations/models_ops.py defines them; no page under docs/tools/cli/ enumerates check-model's categories, the agent CLI CLAUDE.md describes --type without listing them, and docs/tools/cli/update.md lists 6_judgment_deck.toml among the managed deck files."[trigger_files]"pipelex/cli/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"@@ -14,7 +14,7 @@ rationale = "clean-pass: the main-output file copy now compares the names alread"pipelex/cli/agent_cli/commands/agent_cli_factory.py" = "blob:49b8bab889d8673e7d4f1d683efe0928e43914b9""pipelex/cli/agent_cli/commands/agent_output.py" = "blob:30ecf4a3b1cb778864ed821a7b9918f62407a7f4""pipelex/cli/agent_cli/commands/bundle_path_resolver.py" = "blob:30339c5d91048f4e67312d5691ed9a65d57472e0"-"pipelex/cli/agent_cli/commands/check_model_cmd.py" = "blob:91589e517509248a4639a37e82f46a69538f8071"+"pipelex/cli/agent_cli/commands/check_model_cmd.py" = "blob:fbf29c499d8856f06736656ba17953457d04ca46""pipelex/cli/agent_cli/commands/codegen/__init__.py" = "blob:e69de29bb2d1d6434b8b29ae775ad8c2e48c5391""pipelex/cli/agent_cli/commands/codegen/app.py" = "blob:56130b89fe9efec5ad3d014ed6bc16bef1bd1645""pipelex/cli/agent_cli/commands/codegen/check_cmd.py" = "blob:9e8beb084a9f04f39e8acca9e6ae2554d40cb667"diff --git a/.pipelex-dev/test_profiles.toml b/.pipelex-dev/test_profiles.tomlindex c7535edd3..f202a27f1 100644--- a/.pipelex-dev/test_profiles.toml+++ b/.pipelex-dev/test_profiles.toml@@ -280,6 +280,7 @@ llm_models = ["gpt-4o-mini"]img_gen_models = []extract_models = []search_models = []+judgment_models = []################################################################################# Dev Profile@@ -296,6 +297,7 @@ llm_models = [img_gen_models = ["gpt-image-1-mini"]extract_models = ["@from_pdf", "linkup-fetch"]search_models = ["linkup-standard"]+judgment_models = []################################################################################# Manifold Profile@@ -321,6 +323,7 @@ llm_models = [img_gen_models = ["gpt-image-1-mini"]extract_models = ["azure-document-intelligence", "linkup-fetch"]search_models = ["linkup-standard"]+judgment_models = []################################################################################# Coverage Profile@@ -337,6 +340,7 @@ llm_models = [img_gen_models = ["gpt-image-1", "nano-banana"]extract_models = ["pypdfium2-extract-pdf"]search_models = ["linkup-standard"]+judgment_models = []################################################################################# Claude Latest Profile@@ -348,6 +352,7 @@ llm_models = ["claude-4.5-*"]img_gen_models = []extract_models = []search_models = []+judgment_models = []################################################################################# GPT No Codex Profile@@ -359,6 +364,7 @@ llm_models = ["gpt-*", "!*-codex", "!*-codex-*"]img_gen_models = ["gpt-image-*"]extract_models = []search_models = []+judgment_models = []################################################################################# All Backends Profile - Test across all backends with select models@@ -370,6 +376,7 @@ llm_models = ["claude-4.5-haiku", "gpt-4o-mini", "gemini-2.5-flash-lite"]img_gen_models = ["gpt-image-1-mini"]extract_models = ["@from_pdf"]search_models = ["@linkup"]+judgment_models = []################################################################################# Full Profile - All available models (use with caution)@@ -404,3 +411,4 @@ extract_models = [search_models = ["linkup-standard",]+judgment_models = []diff --git a/.pipelex/inference/deck/6_judgment_deck.toml b/.pipelex/inference/deck/6_judgment_deck.tomlnew file mode 100644index 000000000..3692726eb--- /dev/null+++ b/.pipelex/inference/deck/6_judgment_deck.toml@@ -0,0 +1,46 @@+####################################################################################################+# Pipelex Model Deck - Judgment Configuration+####################################################################################################+#+# This file defines model aliases and presets for Judgment models — models that answer+# closed-form questions (yes/no, choice, rating) over some material, with a measure of+# how sure they are.+#+# Managed by `pipelex update`. To customize without conflict, create an+# x_custom_*.toml file in this directory — pipelex never tracks or+# overwrites x_custom_* files. Local edits to this file are backed up+# to <file>.bak.<timestamp> on update but will not survive future runs.+#+# Model Reference Syntax:+# - Preset: $preset_name or preset:preset_name+# - Alias: @alias_name or alias:alias_name+#+# Documentation: https://docs.pipelex.com+# Support: https://go.pipelex.com/discord+#+####################################################################################################++####################################################################################################+# Judgment Default Choice+####################################################################################################+#+# There is deliberately no `choice_default` here, and that is what tells you no judgment model is+# served out of the box: a judgment backend is one you bring, so a pipe names the model it judges+# with until the Pipelex Gateway serves the family. Set one here once you have a backend configured.++[judgment]++####################################################################################################+# Aliases+####################################################################################################+#+# A judgment alias points at a VERSIONED model id, never at a provider's `latest` alias: a verdict+# model that changes under a method silently changes what every threshold in that method means.++[judgment.aliases]++####################################################################################################+# Judgment Presets+####################################################################################################++[judgment.presets]diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 888ce4abc..e2500e82b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,10 @@## [Unreleased]+### Added++- **A `judgment` inference family**: a new family beside `llm`, `img_gen`, `extract`, `search` and `doc_gen`, for models that answer a closed-form question over some material — yes or no, one option out of a declared set, or a position on a declared scale — and report how sure they are. The worker contract is batch-shaped (one state, a map of questions, a map of answers), every measure of uncertainty is optional so a backend that measures none can still serve the family honestly, and usage is reported as real input and output tokens priced per million. The deck gains a `[judgment]` section in a new managed file, `6_judgment_deck.toml`, which `pipelex update` installs, and a deck installed before it keeps loading with no judgment model; it ships with no `choice_default`, because no backend serves the family yet, so a judgment names its own model. `pipelex-agent models` lists the new category, and a `judgment` pytest marker joins the inference markers.+### Changed- **The method-app template and the SDK references link `Pipelex/pipelex-sdk`**: the SDKs and the method-app templates now live in the `Pipelex/pipelex-sdk` repository, so the README and the documentation's **Quick Start** link the method-app template to its `method-apps/` directory, the webapp's by-hand commands clone it and copy `method-apps/webapp-js/`, and the `@pipelex/sdk` and `pipelex-sdk` references point at its `js/docs/` and `python/docs/` directories.diff --git a/Makefile b/Makefileindex f82618121..c7cd6ccbd 100644--- a/Makefile+++ b/Makefile@@ -37,7 +37,7 @@ TS_TOOLCHAIN_ZOD_VERSION ?= 4.5.4UV_MIN_VERSION = $(shell grep -m1 'required-version' pyproject.toml | sed -E 's/.*= *"([^<>=, ]+).*/\1/')-USUAL_PYTEST_MARKERS := "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api"+USUAL_PYTEST_MARKERS := "(dry_runnable or not (inference or llm or img_gen or extract or search or judgment)) and not pipelex_api"# The selection the sharded CI job runs. Shared by gha-tests and store-test-durations so the map# can never be balanced against a different set of tests than the one it is measured from.GHA_PYTEST_MARKERS := "(dry_runnable or not inference) and not (gha_disabled or pipelex_api)"diff --git a/api/.pipelex/inference/deck/6_judgment_deck.toml b/api/.pipelex/inference/deck/6_judgment_deck.tomlnew file mode 100644index 000000000..3692726eb--- /dev/null+++ b/api/.pipelex/inference/deck/6_judgment_deck.toml@@ -0,0 +1,46 @@+####################################################################################################+# Pipelex Model Deck - Judgment Configuration+####################################################################################################+#+# This file defines model aliases and presets for Judgment models — models that answer+# closed-form questions (yes/no, choice, rating) over some material, with a measure of+# how sure they are.+#+# Managed by `pipelex update`. To customize without conflict, create an+# x_custom_*.toml file in this directory — pipelex never tracks or+# overwrites x_custom_* files. Local edits to this file are backed up+# to <file>.bak.<timestamp> on update but will not survive future runs.+#+# Model Reference Syntax:+# - Preset: $preset_name or preset:preset_name+# - Alias: @alias_name or alias:alias_name+#+# Documentation: https://docs.pipelex.com+# Support: https://go.pipelex.com/discord+#+####################################################################################################++####################################################################################################+# Judgment Default Choice+####################################################################################################+#+# There is deliberately no `choice_default` here, and that is what tells you no judgment model is+# served out of the box: a judgment backend is one you bring, so a pipe names the model it judges+# with until the Pipelex Gateway serves the family. Set one here once you have a backend configured.++[judgment]++####################################################################################################+# Aliases+####################################################################################################+#+# A judgment alias points at a VERSIONED model id, never at a provider's `latest` alias: a verdict+# model that changes under a method silently changes what every threshold in that method means.++[judgment.aliases]++####################################################################################################+# Judgment Presets+####################################################################################################++[judgment.presets]diff --git a/api/tests/unit/test_build_and_agent_routes.py b/api/tests/unit/test_build_and_agent_routes.pyindex fbfb9fd4f..e8fd71843 100644--- a/api/tests/unit/test_build_and_agent_routes.py+++ b/api/tests/unit/test_build_and_agent_routes.py@@ -5,6 +5,8 @@ from typing import Anyimport pytestfrom fastapi import FastAPIfrom fastapi.testclient import TestClient+from mthds.protocol.models import ModelCategory as MthdsModelCategory+from pipelex.builder.operations.models_ops import ModelCategoryfrom pipelex.interpreter_hub import get_library_managerfrom pipelex.pipe_run.exceptions import DryRunErrorfrom pipelex.pipeline.bundle_validator import DryRunOutput, DryRunStatus@@ -161,7 +163,8 @@ class TestBuildAndAgentRoutes:# per-category payload caused in the SDK — F2) plus this implementation's routing# extensions, keyed by category (the same alias name exists in several categories —# a flat map would silently drop entries on collision). The old raw keys (`presets`- # by category, `success`) are gone.+ # by category, `success`) are gone. The flat list speaks the protocol's categories only,+ # while the extensions are keyed by every category this runtime serves, `judgment` included.client = _build_client()response = client.get("/v1/models")assert response.status_code == 200, response.text@@ -170,7 +173,8 @@ class TestBuildAndAgentRoutes:first_model = body["models"][0]assert first_model["name"]assert first_model["type"]- valid_categories = {"llm", "extract", "img_gen", "search"}+ assert {model["type"] for model in body["models"]} <= {category.value for category in MthdsModelCategory}+ valid_categories = {category.value for category in ModelCategory}assert set(body["aliases"]) <= valid_categoriesassert all(isinstance(category_aliases, dict) for category_aliases in body["aliases"].values())assert set(body["waterfalls"]) <= valid_categoriesdiff --git a/docs/building-methods/configure-ai-llm-to-optimize-methods.md b/docs/building-methods/configure-ai-llm-to-optimize-methods.mdindex 0c6428a71..a2aa1fafa 100644--- a/docs/building-methods/configure-ai-llm-to-optimize-methods.md+++ b/docs/building-methods/configure-ai-llm-to-optimize-methods.md@@ -115,6 +115,7 @@ The Model Deck is your central configuration hub for all LLM-related settings. I- `3_extract_deck.toml`: Document extraction configuration- `4_search_deck.toml`: Search configuration- `5_doc_gen_deck.toml`: The document engines `PipeDocGen` prints with, by format and source+- `6_judgment_deck.toml`: Judgment configuration- `x_custom_extract_deck.toml`: Custom extraction waterfalls/overrides (loaded last)- `x_custom_llm_deck.toml`: Custom LLM waterfalls/overrides (loaded last)@@ -135,6 +136,7 @@ The Model Deck is your central configuration hub for all LLM-related settings. I├── 3_extract_deck.toml # Document extraction config├── 4_search_deck.toml # Search config├── 5_doc_gen_deck.toml # Document engines, by format and source+ ├── 6_judgment_deck.toml # Judgment config├── x_custom_extract_deck.toml # Custom extraction waterfalls/overrides└── x_custom_llm_deck.toml # Custom LLM waterfalls/overrides```diff --git a/docs/configuration/config-technical/inference-backend-config.md b/docs/configuration/config-technical/inference-backend-config.mdindex 68f6f4eb2..1d73b9858 100644--- a/docs/configuration/config-technical/inference-backend-config.md+++ b/docs/configuration/config-technical/inference-backend-config.md@@ -80,6 +80,7 @@ All inference backend configurations are stored in the `.pipelex/inference/` dir├── 3_extract_deck.toml # Document extraction config├── 4_search_deck.toml # Web search config├── 5_doc_gen_deck.toml # Document engines, by format and source+ ├── 6_judgment_deck.toml # Judgment config├── x_custom_llm_deck.toml # Custom LLM waterfalls/overrides└── x_custom_extract_deck.toml # Custom extract waterfalls```@@ -87,7 +88,7 @@ All inference backend configurations are stored in the `.pipelex/inference/` dirDeck files are loaded in order by their numeric prefix (`1_`, `2_`, `3_`), with custom/override files (`x_` prefix) loaded last.!!! tip "Numbered files are pipelex-managed; overrides go in `x_custom_*.toml`"- The numbered deck files (`1_llm_deck.toml`...`5_doc_gen_deck.toml`) are refreshed by `pipelex update` when a new release ships an updated deck. Local edits to those files are preserved with a timestamped `.bak` backup but will not survive future updates.+ The numbered deck files (`1_llm_deck.toml`...`6_judgment_deck.toml`) are refreshed by `pipelex update` when a new release ships an updated deck. Local edits to those files are preserved with a timestamped `.bak` backup but will not survive future updates.To customize aliases, presets, or default choices without conflict, edit (or create) any file in this directory whose name starts with `x_custom_` — Pipelex never tracks or overwrites those. See [`pipelex update`](../../tools/cli/update.md) for the full workflow.diff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.mdindex b65b2ea9a..bed8c44f8 100644--- a/docs/errors/inference-and-providers.md+++ b/docs/errors/inference-and-providers.md@@ -42,6 +42,10 @@ own page. Classes are grouped by subsystem.- [`InferenceBackendLibraryNotFoundError`](inference-backend-library-not-found-error.md) — Inference backend library not found- [`InferenceBackendLibraryValidationError`](inference-backend-library-validation-error.md) — Inference backend library validation- [`InferenceModelSpecError`](inference-model-spec-error.md) — Inference model spec+- [`JudgmentAnswerMismatchError`](judgment-answer-mismatch-error.md) — Judgment answer mismatch+- [`JudgmentHandleNotFoundError`](judgment-handle-not-found-error.md) — Judgment handle not found+- [`JudgmentJobFailureError`](judgment-job-failure-error.md) — Judgment job failure+- [`JudgmentModelNotFoundError`](judgment-model-not-found-error.md) — Judgment model not found- [`LLMAssignmentError`](llm-assignment-error.md) — LLM assignment- [`LLMCapabilityError`](llm-capability-error.md) — LLM capability- [`LLMCompletionError`](llm-completion-error.md) — LLM completiondiff --git a/docs/errors/judgment-answer-mismatch-error.md b/docs/errors/judgment-answer-mismatch-error.mdnew file mode 100644index 000000000..10e83e9cc--- /dev/null+++ b/docs/errors/judgment-answer-mismatch-error.md@@ -0,0 +1,21 @@+---+title: "Judgment answer mismatch"+description: "Reference for the `JudgmentAnswerMismatchError` Pipelex error class."+---++<!-- pipelex:generated -->++# Judgment answer mismatch++A judgment worker answered questions nobody asked, or answered one in the wrong shape.++| Field | Value |+|---|---|+| `error_type` | `JudgmentAnswerMismatchError` |+| `title` | Judgment answer mismatch |+| `type_uri` | `https://docs.pipelex.com/latest/errors/judgment-answer-mismatch-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.cogt.exceptions` |+| Parent class | [`CogtError`](cogt-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/judgment-handle-not-found-error.md b/docs/errors/judgment-handle-not-found-error.mdnew file mode 100644index 000000000..9b64ac7ce--- /dev/null+++ b/docs/errors/judgment-handle-not-found-error.md@@ -0,0 +1,19 @@+---+title: "Judgment handle not found"+description: "Reference for the `JudgmentHandleNotFoundError` Pipelex error class."+---++<!-- pipelex:generated -->++# Judgment handle not found++| Field | Value |+|---|---|+| `error_type` | `JudgmentHandleNotFoundError` |+| `title` | Judgment handle not found |+| `type_uri` | `https://docs.pipelex.com/latest/errors/judgment-handle-not-found-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.cogt.exceptions` |+| Parent class | [`CogtError`](cogt-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/judgment-job-failure-error.md b/docs/errors/judgment-job-failure-error.mdnew file mode 100644index 000000000..1cd4bbc36--- /dev/null+++ b/docs/errors/judgment-job-failure-error.md@@ -0,0 +1,19 @@+---+title: "Judgment job failure"+description: "Reference for the `JudgmentJobFailureError` Pipelex error class."+---++<!-- pipelex:generated -->++# Judgment job failure++| Field | Value |+|---|---|+| `error_type` | `JudgmentJobFailureError` |+| `title` | Judgment job failure |+| `type_uri` | `https://docs.pipelex.com/latest/errors/judgment-job-failure-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.cogt.exceptions` |+| Parent class | [`CogtError`](cogt-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/judgment-model-not-found-error.md b/docs/errors/judgment-model-not-found-error.mdnew file mode 100644index 000000000..5ce0479a6--- /dev/null+++ b/docs/errors/judgment-model-not-found-error.md@@ -0,0 +1,19 @@+---+title: "Judgment model not found"+description: "Reference for the `JudgmentModelNotFoundError` Pipelex error class."+---++<!-- pipelex:generated -->++# Judgment model not found++| Field | Value |+|---|---|+| `error_type` | `JudgmentModelNotFoundError` |+| `title` | Judgment model not found |+| `type_uri` | `https://docs.pipelex.com/latest/errors/judgment-model-not-found-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.cogt.exceptions` |+| Parent class | [`ModelNotFoundError`](model-not-found-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/get-started/configure-ai-providers.md b/docs/get-started/configure-ai-providers.mdindex 24701f260..532f0794f 100644--- a/docs/get-started/configure-ai-providers.md+++ b/docs/get-started/configure-ai-providers.md@@ -159,6 +159,7 @@ By default, this creates the global `~/.pipelex/` directory with:│ ├── 3_extract_deck.toml # Document extraction config│ ├── 4_search_deck.toml # Search config│ ├── 5_doc_gen_deck.toml # Document engines, by format and source+ │ ├── 6_judgment_deck.toml # Judgment config│ ├── x_custom_llm_deck.toml # Custom LLM configurations│ └── x_custom_extract_deck.toml # Custom extract configurations└── routing_profiles.toml # Model routing configurationdiff --git a/docs/tools/cli/update.md b/docs/tools/cli/update.mdindex c8b9ef59d..e5650ef0f 100644--- a/docs/tools/cli/update.md+++ b/docs/tools/cli/update.md@@ -54,7 +54,7 @@ Locally-modified files are preserved as `<file>.bak.<UTC-timestamp>` before thePipelex draws a sharp line between the files it owns and the files you own.-- **Managed (pipelex-owned)** — the numbered deck files, `1_llm_deck.toml`, `2_img_gen_deck.toml`, `3_extract_deck.toml`, `4_search_deck.toml` and `5_doc_gen_deck.toml`, and `backends/internal.toml`. These are refreshed by `pipelex update`. Local edits are preserved with a `.bak.<timestamp>` backup but will not survive future updates.+- **Managed (pipelex-owned)** — the numbered deck files, `1_llm_deck.toml`, `2_img_gen_deck.toml`, `3_extract_deck.toml`, `4_search_deck.toml`, `5_doc_gen_deck.toml` and `6_judgment_deck.toml`, and `backends/internal.toml`. These are refreshed by `pipelex update`. Local edits are preserved with a `.bak.<timestamp>` backup but will not survive future updates.- **User overrides** — any file in the deck directory whose name starts with `x_custom_` (e.g. `x_custom_llm_deck.toml`, `x_custom_extract_deck.toml`). `pipelex update` never tracks, hashes, copies, or removes these, while `pipelex init`, a full reset, rewrites the two the kit ships. Add new ones whenever you need to override aliases, presets, or default choices for a backend.- **Every other backend file** — `openai.toml`, `anthropic.toml`, a backend of your own, and the rest of `backends/`. `pipelex update` never touches them; `pipelex init`, a full reset, rewrites the ones the kit ships, such as `openai.toml`, and leaves a backend of your own in place. Declare models of your own in a backend of your own rather than in `internal.toml`.diff --git a/docs/under-the-hood/dry-run-mock-generation.md b/docs/under-the-hood/dry-run-mock-generation.mdindex 769163dc9..d640010ea 100644--- a/docs/under-the-hood/dry-run-mock-generation.md+++ b/docs/under-the-hood/dry-run-mock-generation.md@@ -47,7 +47,7 @@ flowchart TDJ --> K[model_validate with resolved values]```-A dry run does not swap in a special content generator: `run_mode=DRY` rides `CogtRunParams` on every cogt assignment, and each inference leaf (`llm_generate`, the `*_and_store` image/extract leaves, search, templating) branches to its `dry_*` mock helper in `dry_mock.py` — identically whether the leaf runs inline or inside a Temporal activity.+A dry run does not swap in a special content generator: `run_mode=DRY` rides `CogtRunParams` on every cogt assignment, and each inference leaf (`llm_generate`, the `*_and_store` image/extract leaves, search, judgment, templating) branches to its `dry_*` mock helper in `dry_mock.py` — identically whether the leaf runs inline or inside a Temporal activity.| Trigger | Entry Point | Mock Generation ||---------|-------------|-----------------|diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.mdindex 54ef49c0d..b7910e3c3 100644--- a/docs/under-the-hood/inference-backend-plugins.md+++ b/docs/under-the-hood/inference-backend-plugins.md@@ -5,7 +5,7 @@ description: "How Pipelex discovers inference backends through the plugin seam,# Inference Backend Plugins-Every model call in Pipelex — an LLM completion, an image generation, a document extraction, a web search — is served by an **inference worker**. Which worker handles a given model is decided entirely by data: a model's `sdk` field selects a backend, and a **backend plugin** is what teaches Pipelex how to build the worker for that `sdk`.+Every model call in Pipelex — an LLM completion, an image generation, a document extraction, a web search, a judgment — is served by an **inference worker**. Which worker handles a given model is decided entirely by data: a model's `sdk` field selects a backend, and a **backend plugin** is what teaches Pipelex how to build the worker for that `sdk`.Core names no backend by import or by string. The built-in drivers (OpenAI, Gateway, Anthropic, Mistral, Bedrock, Google, FAL, HuggingFace, Docling, …) are plugins too — they ride the exact same seam an out-of-tree plugin would. This page documents that seam, the **Inference SPI** a plugin compiles against, and how to write one.@@ -29,7 +29,7 @@ run time (e.g. LLMWorkerFactory.make_llm_worker)└─ worker = make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)```-The worker factories (`LLMWorkerFactory`, `ImgGenWorkerFactory`, `ExtractWorkerFactory`, `SearchWorkerFactory`, `DocGenWorkerFactory`) hold **no** `match` over SDK strings. They build a `ModelHandle`, resolve the `InferenceBackend` config, look up the backend's `make_worker` by `(family, sdk)`, and call it. A lookup miss raises a friendly `InferenceBackendNotFoundError` ("… Is its plugin installed and enabled?").+The worker factories (`LLMWorkerFactory`, `ImgGenWorkerFactory`, `ExtractWorkerFactory`, `SearchWorkerFactory`, `DocGenWorkerFactory`, `JudgmentWorkerFactory`) hold **no** `match` over SDK strings. They build a `ModelHandle`, resolve the `InferenceBackend` config, look up the backend's `make_worker` by `(family, sdk)`, and call it. A lookup miss raises a friendly `InferenceBackendNotFoundError` ("… Is its plugin installed and enabled?").---@@ -57,7 +57,7 @@ A backend plugin's `register` calls one menu method per `(family, sdk)` it serve```pythonregistrar.add_inference_backend(- family=InferenceFamily.LLM, # LLM | IMG_GEN | EXTRACT | SEARCH | DOC_GEN+ family=InferenceFamily.LLM, # LLM | IMG_GEN | EXTRACT | SEARCH | DOC_GEN | JUDGMENTsdk="acme", # the model's `sdk` stringmake_worker=_make_acme_worker, # a MakeWorkerFn (a plain callable))@@ -227,7 +227,7 @@ What an out-of-tree backend plugin imports *is* the contract. The published surf| `SdkClientRegistry` | `pipelex.plugins.sdk_client_registry` | per-handle client memoization || `InferenceModelSpec` | `pipelex.cogt.model_backends.model_spec` | the resolved model record || `InferenceBackend` | `pipelex.cogt.model_backends.backend` | the backend config record (api key, extras) |-| `InferenceWorkerAbstract` and the `{LLM,ImgGen,Extract,Search}WorkerAbstract` subclasses | `pipelex.cogt.…` | the worker contracts a plugin returns |+| `InferenceWorkerAbstract` and the `{LLM,ImgGen,Extract,Search,Judgment}WorkerAbstract` subclasses | `pipelex.cogt.…` | the worker contracts a plugin returns || `MissingDependencyError` | `pipelex.exceptions` | raised by `require_sdk` |The SPI is a documented, versioned **module/symbol list** gated by `PLUGIN_API_VERSION` — not an `__init__.py` re-export shim (the repo bans re-exports; import by full path). Anything a plugin needs to import outside this surface is a design gap to resolve, not an accident to live with.diff --git a/docs/under-the-hood/tokens-usage-wire-records.md b/docs/under-the-hood/tokens-usage-wire-records.mdindex bb2cfe72a..2b39d9431 100644--- a/docs/under-the-hood/tokens-usage-wire-records.md+++ b/docs/under-the-hood/tokens-usage-wire-records.md@@ -20,12 +20,12 @@ One record per inference call, in completion order:| field | type | semantics ||---|---|---|-| `model_type` | string | Kind of inference: known values `llm`, `img_gen`, `extract`, `search`. Open set. |+| `model_type` | string | Kind of inference: known values `llm`, `img_gen`, `extract`, `search`, `judgment`. Open set. || `inference_model_name` | string | Human model name (e.g. `gpt-4o`). || `inference_model_id` | string | Provider/platform model id (e.g. `gpt-4o-2024-11-20`). || `pipe_code` | string \| null | Pipe that made the call — per-pipe cost attribution. |-| `job_category` | string \| null | Known values: `llm_job`, `img_gen_job`, `extract_job`, `search_job`, `jinja2_job`, `mock_job`. Open set. |-| `unit_job_id` | string \| null | Known values: `llm_gen_text`, `llm_gen_object`, `img_gen_text_to_image`, `extract_pages`, `search_sourced_answer`, `search_structured`. Open set. |+| `job_category` | string \| null | Known values: `llm_job`, `img_gen_job`, `extract_job`, `search_job`, `judgment_job`, `jinja2_job`, `mock_job`. Open set. |+| `unit_job_id` | string \| null | Known values: `llm_gen_text`, `llm_gen_object`, `img_gen_text_to_image`, `extract_pages`, `search_sourced_answer`, `search_structured`, `judgment_answer`. Open set. || `nb_tokens_by_category` | object<string, int> | Raw provider-reported token counts, keyed by `TokenCategory` values (`input`, `input_cached`, `output`, ...). Open set. `input` is the joined total; `input_cached` is a subset of it, not additive. || `cost` | number \| null | Computed USD cost of this call. `null` when the model has no rate table (own-GPU, mock, dry-run). || `started_at` | string \| null | ISO 8601. |diff --git a/pipelex/builder/operations/models_ops.py b/pipelex/builder/operations/models_ops.pyindex 55e819aa4..0c933ea84 100644--- a/pipelex/builder/operations/models_ops.py+++ b/pipelex/builder/operations/models_ops.py@@ -15,6 +15,7 @@ class ModelCategory(StrEnum):EXTRACT = "extract"IMG_GEN = "img_gen"SEARCH = "search"+ JUDGMENT = "judgment"CATEGORY_TO_MODEL_TYPE: dict[ModelCategory, ModelType] = {@@ -22,6 +23,7 @@ CATEGORY_TO_MODEL_TYPE: dict[ModelCategory, ModelType] = {ModelCategory.EXTRACT: ModelType.TEXT_EXTRACTOR,ModelCategory.IMG_GEN: ModelType.IMG_GEN,ModelCategory.SEARCH: ModelType.SEARCH,+ ModelCategory.JUDGMENT: ModelType.JUDGMENT,}@@ -116,6 +118,8 @@ def _build_presets_for_category(presets_dict = model_deck.img_gen_presetscase ModelCategory.SEARCH:presets_dict = model_deck.search_presets+ case ModelCategory.JUDGMENT:+ presets_dict = model_deck.judgment_presetspresets_list: list[dict[str, Any]] = []for preset_name, setting in presets_dict.items():@@ -150,6 +154,8 @@ def _build_aliases_for_category(aliases = model_deck.img_gen_aliasescase ModelCategory.SEARCH:aliases = model_deck.search_aliases+ case ModelCategory.JUDGMENT:+ aliases = model_deck.judgment_aliasesif backend is not None:aliases = _filter_aliases_by_backend(aliases, model_deck=model_deck, model_type=model_type, backend=backend)@@ -175,6 +181,8 @@ def _build_waterfalls_for_category(waterfalls = model_deck.img_gen_waterfallscase ModelCategory.SEARCH:waterfalls = model_deck.search_waterfalls+ case ModelCategory.JUDGMENT:+ waterfalls = model_deck.judgment_waterfallsif backend is not None:waterfalls = _filter_waterfalls_by_backend(waterfalls, model_deck=model_deck, model_type=model_type, backend=backend)@@ -221,6 +229,10 @@ def list_models(presets["search"] = _build_presets_for_category(model_deck=model_deck, category=ModelCategory.SEARCH, backend=backend)aliases["search"] = _build_aliases_for_category(model_deck=model_deck, category=ModelCategory.SEARCH, backend=backend)waterfalls["search"] = _build_waterfalls_for_category(model_deck=model_deck, category=ModelCategory.SEARCH, backend=backend)+ if _should_include(ModelCategory.JUDGMENT, categories=categories):+ presets["judgment"] = _build_presets_for_category(model_deck=model_deck, category=ModelCategory.JUDGMENT, backend=backend)+ aliases["judgment"] = _build_aliases_for_category(model_deck=model_deck, category=ModelCategory.JUDGMENT, backend=backend)+ waterfalls["judgment"] = _build_waterfalls_for_category(model_deck=model_deck, category=ModelCategory.JUDGMENT, backend=backend)return {"presets": presets,@@ -234,6 +246,7 @@ CATEGORY_DISPLAY_NAMES: dict[str, str] = {"img_gen": "Image Generation","extract": "Extract","search": "Search",+ "judgment": "Judgment",}diff --git a/pipelex/cli/agent_cli/commands/check_model_cmd.py b/pipelex/cli/agent_cli/commands/check_model_cmd.pyindex 91589e517..fbf29c499 100644--- a/pipelex/cli/agent_cli/commands/check_model_cmd.py+++ b/pipelex/cli/agent_cli/commands/check_model_cmd.py@@ -58,7 +58,7 @@ def agent_check_model_cmd(],model_type: Annotated[ModelCategory,- typer.Option("--type", "-t", help="Model category: llm, extract, img_gen, search"),+ typer.Option("--type", "-t", help="Model category: llm, extract, img_gen, search, judgment"),],output_format: Annotated[CliOutputFormat,diff --git a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.pyindex 9b28098f4..2ccb767db 100644--- a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py+++ b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py@@ -34,7 +34,7 @@ TEST_PROFILES_PATH = Path(ConfigPaths.DEV_CONFIG_DIR_PATH) / "test_profiles.tomlTEST_PROFILES_OVERRIDE_PATH = Path(ConfigPaths.DEV_CONFIG_DIR_PATH) / "test_profiles_override.toml"# Model types we care about-MODEL_TYPES = ["llm", "img_gen", "text_extractor", "search"]+MODEL_TYPES = ["llm", "img_gen", "text_extractor", "search", "judgment"]# Default model type from backend configDEFAULT_MODEL_TYPE = "llm"@@ -164,6 +164,7 @@ def _collect_all_model_availability() -> dict[str, Any]:"img_gen": {},"text_extractor": {},"search": {},+ "judgment": {},}backends_dir = config_manager.backends_dir_path@@ -392,6 +393,7 @@ def _filter_models_by_profile("img_gen": [],"text_extractor": [],"search": [],+ "judgment": [],}# If include_all is set, return all valid pairs@@ -442,6 +444,7 @@ def _filter_models_by_profile("img_gen_models": "img_gen","extract_models": "text_extractor","search_models": "search",+ "judgment_models": "judgment",}# Model type to collection type mapping (internal name -> TOML section name)@@ -450,6 +453,7 @@ def _filter_models_by_profile("img_gen": "img_gen","text_extractor": "extract","search": "search",+ "judgment": "judgment",}# Resolve model lists from profile using advanced specifiers@@ -555,6 +559,14 @@ def _generate_fixtures_python(lines.append("]")lines.append("")+ # Judgment combos+ judgment_pairs = combo_pairs.get("judgment", [])+ lines.append("JUDGMENT_COMBOS: list[ModelCombo] = [")+ for model, backend in sorted(judgment_pairs):+ lines.append(f" ModelCombo({model!r}, {backend!r}),")+ lines.append("]")+ lines.append("")+return "\n".join(lines)@@ -596,11 +608,13 @@ def _display_summary(img_gen_total = sum(len(models) for models in availability.get("img_gen", {}).values())extract_total = sum(len(models) for models in availability.get("text_extractor", {}).values())search_total = sum(len(models) for models in availability.get("search", {}).values())+ judgment_total = sum(len(models) for models in availability.get("judgment", {}).values())table.add_row("LLM Models", str(llm_total), str(len(combo_pairs.get("llm", []))))table.add_row("Image Gen Models", str(img_gen_total), str(len(combo_pairs.get("img_gen", []))))table.add_row("Extract Models", str(extract_total), str(len(combo_pairs.get("text_extractor", []))))table.add_row("Search Models", str(search_total), str(len(combo_pairs.get("search", []))))+ table.add_row("Judgment Models", str(judgment_total), str(len(combo_pairs.get("judgment", []))))table.add_row("", "", "")table.add_row("Total Backends", str(len(total_backends)), "-")table.add_row("Total Model/Backend Pairs", str(total_models), str(filtered_models))diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.pyindex 3b74fde42..10b4714c1 100644--- a/pipelex/cogt/content_generation/assignment_models.py+++ b/pipelex/cogt/content_generation/assignment_models.py@@ -10,6 +10,8 @@ from pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt+from pipelex.cogt.judgment.judgment_models import JudgmentQuestion, JudgmentState+from pipelex.cogt.judgment.judgment_setting import JudgmentSettingfrom pipelex.cogt.llm.llm_job_components import LLMJobParamsfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_setting import LLMSetting@@ -235,3 +237,30 @@ class SearchObjectAssignment(BaseModel):output_class_schema=output_class.model_json_schema(),search_assignment=search_assignment,)+++class JudgmentAssignment(BaseModel):+ """Serializable unit for a single judgment leaf call.++ Carries everything the framework-agnostic ``judgment_generate`` core needs to rebuild the+ ``JudgmentJob`` on the other side of a distributed boundary: the state the questions are asked+ over, the questions themselves, and the fully resolved ``judgment_setting`` (its ``model`` is the+ resolved provider handle, also the routing key). Mirrors ``SearchAssignment`` for the search leaf.++ Unlike the structured-search leaf, there is no schema to ship and no dynamic class to rebuild:+ a judgment's answers are plain models of this package's own, so they cross a boundary as they are.+ """++ job_metadata: JobMetadata+ cogt_run_params: CogtRunParams+ state: JudgmentState+ questions: dict[str, JudgmentQuestion] = Field(min_length=1)+ judgment_setting: JudgmentSetting++ @property+ def judgment_handle(self) -> str:+ return self.judgment_setting.model++ def referenced_uris(self) -> list[UriReference]:+ """None: a judgment hands its state to the provider as data and reads no URL the state carries."""+ return []diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.pyindex 92e065ea5..934098c75 100644--- a/pipelex/cogt/content_generation/content_generator.py+++ b/pipelex/cogt/content_generation/content_generator.py@@ -6,6 +6,7 @@ from pipelex import logfrom pipelex.cogt.content_generation.assignment_models import (ExtractAssignment,ImgGenAssignment,+ JudgmentAssignment,LLMAssignment,ObjectAssignment,RenderDocumentAssignment,@@ -19,6 +20,7 @@ from pipelex.cogt.content_generation.doc_gen_generate import render_document_andfrom pipelex.cogt.content_generation.extract_generate import extract_gen_pages_and_storefrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list_and_store, img_gen_single_image_and_store+from pipelex.cogt.content_generation.judgment_generate import judgment_gen_answersfrom pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_textfrom pipelex.cogt.content_generation.object_revalidation import revalidate_leaf_objectfrom pipelex.cogt.content_generation.render_generate import render_page_views_and_store@@ -30,6 +32,7 @@ from pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt+from pipelex.cogt.judgment.judgment_models import JudgmentAnswerfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.config import get_config@@ -329,3 +332,10 @@ class ContentGenerator(ContentGeneratorProtocol):# `SearchObjectAssignment` is built here: that wire model ships the class's schema across a# boundary, and this arm has none to cross.return await search_gen_structured_object(search_assignment, output_class=output_structure_class)++ @override+ async def make_judgment_answers(+ self,+ judgment_assignment: JudgmentAssignment,+ ) -> dict[str, JudgmentAnswer]:+ return await judgment_gen_answers(judgment_assignment)diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.pyindex b67e6365c..1e901c592 100644--- a/pipelex/cogt/content_generation/content_generator_protocol.py+++ b/pipelex/cogt/content_generation/content_generator_protocol.py@@ -2,7 +2,7 @@ from collections.abc import Awaitable, Callable, Coroutinefrom functools import wrapsfrom typing import Any, ParamSpec, Protocol, TypeVar-from pipelex.cogt.content_generation.assignment_models import SearchAssignment+from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment, SearchAssignmentfrom pipelex.cogt.content_generation.cogt_run_params import CogtRunParamsfrom pipelex.cogt.doc_gen.doc_gen_setting import DocGenSettingfrom pipelex.cogt.doc_gen.document_composition import DocumentComposition@@ -10,6 +10,7 @@ from pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParamsfrom pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt+from pipelex.cogt.judgment.judgment_models import JudgmentAnswerfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.core.stuffs.document_content import DocumentContent@@ -164,3 +165,8 @@ class ContentGeneratorProtocol(Protocol):output_structure_class: type[BaseModelTypeVar],search_assignment: SearchAssignment,) -> Coroutine[Any, Any, BaseModelTypeVar]: ...++ def make_judgment_answers(+ self,+ judgment_assignment: JudgmentAssignment,+ ) -> Coroutine[Any, Any, dict[str, JudgmentAnswer]]: ...diff --git a/pipelex/cogt/content_generation/dry_mock.py b/pipelex/cogt/content_generation/dry_mock.pyindex 849c1b2a2..0547a4e54 100644--- a/pipelex/cogt/content_generation/dry_mock.py+++ b/pipelex/cogt/content_generation/dry_mock.py@@ -45,6 +45,7 @@ from pipelex import logfrom pipelex.cogt.content_generation.assignment_models import (ExtractAssignment,ImgGenAssignment,+ JudgmentAssignment,LLMAssignment,ObjectAssignment,RenderDocumentAssignment,@@ -58,6 +59,15 @@ from pipelex.cogt.content_generation.dry_run_factory import DryRunFactoryfrom pipelex.cogt.content_generation.exceptions import DryRunMockBuildError, OutputStructureSchemaErrorfrom pipelex.cogt.content_generation.object_class_resolution import resolve_object_classfrom pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory+from pipelex.cogt.judgment.judgment_models import (+ ChoiceAnswer,+ ChoiceQuestion,+ JudgmentAnswer,+ RatingAnswer,+ RatingQuestion,+ YesNoAnswer,+ YesNoQuestion,+)from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReportfrom pipelex.cogt.llm.llm_prompt import LLMPrompt@@ -427,6 +437,28 @@ def dry_search_gen_sourced_answer(search_assignment: SearchAssignment) -> Searchreturn build_mock_object(SearchResultContent, sources=mock_sources)+def dry_judgment_gen_answers(judgment_assignment: JudgmentAssignment) -> dict[str, JudgmentAnswer]:+ """Dry leaf for a judgment: a deterministic verdict per question, with no uncertainty at all.++ Every answer is the first thing its question allows — yes, the first option, the lowest level —+ because a dry run must be reproducible and nothing here measured anything. The uncertainty+ members are left absent for the same reason: the contract makes them optional precisely so a+ producer that measured nothing can say so, and a mock probability would be the one number in the+ whole family that nobody could tell apart from a real one.+ """+ log.verbose(f"🤡 DRY RUN: judgment_gen_answers for '{judgment_assignment.judgment_handle}'")+ answers: dict[str, JudgmentAnswer] = {}+ for question_key, question in judgment_assignment.questions.items():+ match question:+ case YesNoQuestion():+ answers[question_key] = YesNoAnswer(yes_no=True)+ case ChoiceQuestion():+ answers[question_key] = ChoiceAnswer(choice=next(iter(question.options)))+ case RatingQuestion():+ answers[question_key] = RatingAnswer(level=0)+ return answers++def dry_search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:"""Dry leaf for structured search at the *boundary*: schema-built mock dumped to a dict, no provider.diff --git a/pipelex/cogt/content_generation/judgment_generate.py b/pipelex/cogt/content_generation/judgment_generate.pynew file mode 100644index 000000000..7ef7befd0--- /dev/null+++ b/pipelex/cogt/content_generation/judgment_generate.py@@ -0,0 +1,50 @@+"""Framework-agnostic judgment leaf, sibling of ``search_generate``.++This coroutine is the single place a judgment's provider spend happens. It takes a serializable+``JudgmentAssignment``, rebuilds the ``JudgmentJob``, resolves the worker from the model handle, and+runs it. The direct ``ContentGenerator`` calls it inline; a distributed orchestrator calls it inside+an activity so the result is recorded in the run's history and any failure is converted to a+terminal error.++There is only one entry point here, where the search leaf has three. A judgment's answers are plain+models of this package's own — no caller class travels down and no schema has to be shipped — so the+in-process arm and the boundary arm would be the same function, and there is nothing to split.+"""++from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment+from pipelex.cogt.content_generation.dry_mock import dry_judgment_gen_answers+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads+from pipelex.cogt.judgment.judgment_job import JudgmentJob+from pipelex.cogt.judgment.judgment_job_factory import JudgmentJobFactory+from pipelex.cogt.judgment.judgment_models import JudgmentAnswer+from pipelex.cogt.judgment.judgment_worker_abstract import JudgmentWorkerAbstract+from pipelex.cogt.judgment.judgment_worker_factory import JudgmentWorkerFactory+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.runtime_hub import get_model_deck, get_report_delegate+++def _make_judgment_worker(judgment_assignment: JudgmentAssignment) -> JudgmentWorkerAbstract:+ model_deck = get_model_deck()+ inference_model = model_deck.get_required_inference_model(+ model_handle=judgment_assignment.judgment_setting.model,+ model_type=ModelType.JUDGMENT,+ )+ return JudgmentWorkerFactory.make_judgment_worker(inference_model, reporting_delegate=get_report_delegate())+++def _make_judgment_job(judgment_assignment: JudgmentAssignment) -> JudgmentJob:+ return JudgmentJobFactory.make_judgment_job(+ state=judgment_assignment.state,+ questions=judgment_assignment.questions,+ judgment_setting=judgment_assignment.judgment_setting,+ job_metadata=judgment_assignment.job_metadata,+ )+++async def judgment_gen_answers(judgment_assignment: JudgmentAssignment) -> dict[str, JudgmentAnswer]:+ authorize_assignment_reads(job_metadata=judgment_assignment.job_metadata, uri_references=judgment_assignment.referenced_uris())+ if judgment_assignment.cogt_run_params.run_mode.is_dry:+ return dry_judgment_gen_answers(judgment_assignment)+ worker = _make_judgment_worker(judgment_assignment)+ judgment_job = _make_judgment_job(judgment_assignment)+ return await worker.judge(judgment_job)diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.pyindex 572a5d98d..7dc113913 100644--- a/pipelex/cogt/exceptions.py+++ b/pipelex/cogt/exceptions.py@@ -358,6 +358,15 @@ class DocGenHandleNotFoundError(CogtError):super().__init__(message)+class JudgmentHandleNotFoundError(CogtError):+ error_category = InferenceErrorCategory.CONFIGURATION++ def __init__(self, message: str, preset_id: str, model_handle: str):+ self.preset_id = preset_id+ self.model_handle = model_handle+ super().__init__(message)++class ExtractOutputError(CogtError):pass@@ -442,6 +451,18 @@ class SearchModelNotFoundError(ModelNotFoundError):pass+class JudgmentJobFailureError(CogtError):+ pass+++class JudgmentAnswerMismatchError(CogtError):+ """A judgment worker answered questions nobody asked, or answered one in the wrong shape."""+++class JudgmentModelNotFoundError(ModelNotFoundError):+ pass++class RoutingProfileLibraryNotFoundError(CogtError):passdiff --git a/pipelex/cogt/inference/error_render.py b/pipelex/cogt/inference/error_render.pyindex 6145f84e3..b5e1e5dfd 100644--- a/pipelex/cogt/inference/error_render.py+++ b/pipelex/cogt/inference/error_render.py@@ -15,6 +15,8 @@ from pipelex.cogt.exceptions import (ExtractModelNotFoundError,ImgGenGenerationError,ImgGenModelNotFoundError,+ JudgmentJobFailureError,+ JudgmentModelNotFoundError,LLMCompletionError,LLMModelNotFoundError,ModelNotFoundError,@@ -39,6 +41,7 @@ class InferenceErrorFamily(StrEnum):IMG_GEN = "img_gen"EXTRACT = "extract"SEARCH = "search"+ JUDGMENT = "judgment"# Generic failure error class per family — used when the error is not a missing model.@@ -47,6 +50,7 @@ _FAILURE_CLASSES: dict[InferenceErrorFamily, type[CogtError]] = {InferenceErrorFamily.IMG_GEN: ImgGenGenerationError,InferenceErrorFamily.EXTRACT: ExtractJobFailureError,InferenceErrorFamily.SEARCH: SearchJobFailureError,+ InferenceErrorFamily.JUDGMENT: JudgmentJobFailureError,}# Model-not-found error class per family — used when ``is_model_not_found`` is set.@@ -55,6 +59,7 @@ _NOT_FOUND_CLASSES: dict[InferenceErrorFamily, type[ModelNotFoundError]] = {InferenceErrorFamily.IMG_GEN: ImgGenModelNotFoundError,InferenceErrorFamily.EXTRACT: ExtractModelNotFoundError,InferenceErrorFamily.SEARCH: SearchModelNotFoundError,+ InferenceErrorFamily.JUDGMENT: JudgmentModelNotFoundError,}diff --git a/pipelex/cogt/judgment/__init__.py b/pipelex/cogt/judgment/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/pipelex/cogt/judgment/judgment_job.py b/pipelex/cogt/judgment/judgment_job.pynew file mode 100644index 000000000..fb3dac50f--- /dev/null+++ b/pipelex/cogt/judgment/judgment_job.py@@ -0,0 +1,55 @@+from datetime import datetime++from pydantic import BaseModel, Field+from typing_extensions import override++from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract+from pipelex.cogt.judgment.judgment_models import JudgmentQuestion, JudgmentState+from pipelex.cogt.judgment.judgment_report import JudgmentTokensUsage+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+++class JudgmentJobParams(BaseModel):+ judgment_setting: JudgmentSetting+++class JudgmentJobReport(BaseModel):+ judgment_tokens_usage: JudgmentTokensUsage | None = None+++class JudgmentJob(InferenceJobAbstract):+ """One state, a map of questions over it, and the answers they get.++ The shape is batch-first because batching is the backend's whole economy: the spike measured+ three questions over one state at 506 input tokens against 1138 for the same three sent+ separately. A caller with one question sends a batch of one; a contract that could not express+ the batch would have to be broken to gain it later.++ The question keys are the caller's own — the model never sees them — and they are what the+ answers come back under.+ """++ state: JudgmentState+ questions: dict[str, JudgmentQuestion] = Field(min_length=1)+ job_params: JudgmentJobParams+ job_report: JudgmentJobReport = JudgmentJobReport()++ @override+ def validate_before_execution(self):+ pass++ def judgment_job_before_start(self, *, inference_model: InferenceModelSpec):+ self.job_metadata.started_at = datetime.now()++ self.job_report = JudgmentJobReport()+ self.job_report.judgment_tokens_usage = JudgmentTokensUsage(+ job_metadata=self.job_metadata,+ inference_model_name=inference_model.name,+ unit_costs=inference_model.costs,+ inference_model_id=inference_model.model_id,+ nb_tokens_by_category={},+ )++ def judgment_job_after_complete(self):+ self.job_metadata.completed_at = datetime.now()diff --git a/pipelex/cogt/judgment/judgment_job_factory.py b/pipelex/cogt/judgment/judgment_job_factory.pynew file mode 100644index 000000000..314580c72--- /dev/null+++ b/pipelex/cogt/judgment/judgment_job_factory.py@@ -0,0 +1,24 @@+from pipelex.cogt.judgment.judgment_job import JudgmentJob, JudgmentJobParams+from pipelex.cogt.judgment.judgment_models import JudgmentQuestion, JudgmentState+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting+from pipelex.system.job_metadata import JobCategory, JobMetadata+++class JudgmentJobFactory:+ @classmethod+ def make_judgment_job(+ cls,+ *,+ state: JudgmentState,+ questions: dict[str, JudgmentQuestion],+ judgment_setting: JudgmentSetting,+ job_metadata: JobMetadata,+ ) -> JudgmentJob:+ job_metadata.job_category = JobCategory.JUDGMENT_JOB+ job_params = JudgmentJobParams(judgment_setting=judgment_setting)+ return JudgmentJob(+ job_metadata=job_metadata,+ state=state,+ questions=questions,+ job_params=job_params,+ )diff --git a/pipelex/cogt/judgment/judgment_models.py b/pipelex/cogt/judgment/judgment_models.pynew file mode 100644index 000000000..a4b9e2d26--- /dev/null+++ b/pipelex/cogt/judgment/judgment_models.py@@ -0,0 +1,116 @@+"""The questions a judgment asks and the answers it gets back.++Both are discriminated unions over :class:`JudgmentKind`, and the three kinds are the language's+own words — yes/no, choice, rating — never a vendor's. A backend translates into its own vocabulary+inside its worker and nowhere else.++**Every uncertainty member is optional, and that is a contract rather than an oversight.** A backend+that measures a probability reports it; a backend that cannot — a generative model emulating a+judgment, say — leaves it absent rather than inventing a number. Only the verdict itself is required,+and :class:`YesNoAnswer` requires at least one of its two forms of verdict.+"""++from enum import StrEnum+from typing import Annotated, Any, Literal, Self, TypeAlias++from pydantic import BaseModel, Field, model_validator++# The material a judgment is passed, as a JSON object with one member per name the questions refer+# to. An object rather than free text or an array on purpose: the vendor reads named fields as+# relationships, and the spike measured an array state — which drops the names — answering+# measurably worse. `Any` is the JSON value space, and the kernel is what fills it.+JudgmentState: TypeAlias = dict[str, Any]+++class JudgmentKind(StrEnum):+ YES_NO = "yes_no"+ CHOICE = "choice"+ RATING = "rating"+++class YesNoQuestion(BaseModel):+ """Does this condition hold? The answer's probability is its confidence."""++ kind: Literal[JudgmentKind.YES_NO] = JudgmentKind.YES_NO+ instructions: str+ yes_criterion: str | None = None+ no_criterion: str | None = None+++class ChoiceQuestion(BaseModel):+ """Which one of these options? An option may be declared without a description."""++ kind: Literal[JudgmentKind.CHOICE] = JudgmentKind.CHOICE+ instructions: str+ options: dict[str, str | None] = Field(min_length=1)+++class RatingQuestion(BaseModel):+ """Where on this scale? The levels are ordered from low to high, each describing a situation.++ One level is the floor here because one is what a judgment needs to be answerable at all. The+ language's own minimum is two, and it belongs to the blueprint that authors write against; a+ backend's maximum is that backend's, and belongs to its worker.+ """++ kind: Literal[JudgmentKind.RATING] = JudgmentKind.RATING+ instructions: str+ levels: list[str] = Field(min_length=1)+++JudgmentQuestion: TypeAlias = Annotated[+ YesNoQuestion | ChoiceQuestion | RatingQuestion,+ Field(discriminator="kind"),+]+++class YesNoAnswer(BaseModel):+ """A yes/no verdict, as a probability, as a boolean, or as both.++ The model validator is the contract: an answer carrying neither is not an answer. A backend that+ measured a probability leaves ``yes_no`` to whoever holds the threshold; a backend that could+ only decide reports ``yes_no`` alone, and a threshold applied to it has nothing to work on —+ which the caller reports rather than silently ignoring.+ """++ kind: Literal[JudgmentKind.YES_NO] = JudgmentKind.YES_NO+ probability: float | None = Field(default=None, ge=0, le=1)+ yes_no: bool | None = None++ @model_validator(mode="after")+ def validate_has_a_verdict(self) -> Self:+ if self.probability is None and self.yes_no is None:+ msg = "A YesNoAnswer must carry a probability, a yes_no, or both"+ raise ValueError(msg)+ return self+++class ChoiceAnswer(BaseModel):+ """The option chosen, with the distribution it was chosen from when the backend measured one."""++ kind: Literal[JudgmentKind.CHOICE] = JudgmentKind.CHOICE+ choice: str+ confidence: float | None = Field(default=None, ge=0, le=1)+ probabilities: dict[str, float] | None = None+++class RatingAnswer(BaseModel):+ """The level reached, as a zero-based index into the question's levels.++ ``position`` is the probability-weighted position over those indices, which is what a backend+ that measured a distribution actually knows; ``level`` is the level a verdict has to name, and+ deriving it from the distribution is the worker's translation work, since only the worker knows+ what its backend measured.+ """++ kind: Literal[JudgmentKind.RATING] = JudgmentKind.RATING+ level: int = Field(ge=0)+ position: float | None = Field(default=None, ge=0)+ confidence: float | None = Field(default=None, ge=0, le=1)+ probabilities: dict[int, float] | None = None+++JudgmentAnswer: TypeAlias = Annotated[+ YesNoAnswer | ChoiceAnswer | RatingAnswer,+ Field(discriminator="kind"),+]diff --git a/pipelex/cogt/judgment/judgment_report.py b/pipelex/cogt/judgment/judgment_report.pynew file mode 100644index 000000000..ca9496408--- /dev/null+++ b/pipelex/cogt/judgment/judgment_report.py@@ -0,0 +1,79 @@+from enum import StrEnum+from typing import Any, Literal++from pydantic import BaseModel++from pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory+from pipelex.system.job_metadata import JobMetadata+++class JudgmentTokenCostReportField(StrEnum):+ MODEL_TYPE = "model_type"+ JUDGMENT_NAME = "judgment_name"+ PLATFORM_JUDGMENT_ID = "platform_judgment_id"+ NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"+ NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"+ NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"+ NB_TOKENS_OUTPUT = "nb_tokens_output"+ COST_INPUT_CACHED = "cost_input_cached"+ COST_INPUT_NON_CACHED = "cost_input_non_cached"+ COST_INPUT_JOINED = "cost_input_joined"+ COST_OUTPUT = "cost_output"++ @staticmethod+ def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str:+ return f"nb_tokens_{token_category}"++ @staticmethod+ def report_field_for_cost_by_category(token_category: CostCategory) -> str:+ return f"cost_{token_category}"+++class JudgmentTokenCostReport(BaseModel):+ model_type: Literal["judgment"] = "judgment"+ job_metadata: JobMetadata+ inference_model_name: str+ platform_model_id: str++ nb_tokens_by_category: NbTokensByCategoryDict+ costs_by_token_category: CostsByCategoryDict++ def as_flat_dictionary(self) -> dict[str, Any]:+ the_dict: dict[str, Any] = {}+ dict_for_job_metadata = self.job_metadata.model_dump(serialize_as_any=True)+ the_dict.update(dict_for_job_metadata)+ dict_for_model: dict[str, Any] = {+ JudgmentTokenCostReportField.MODEL_TYPE: self.model_type,+ JudgmentTokenCostReportField.JUDGMENT_NAME: self.inference_model_name,+ JudgmentTokenCostReportField.PLATFORM_JUDGMENT_ID: self.platform_model_id,+ }+ the_dict.update(dict_for_model)+ dict_for_nb_tokens = {+ JudgmentTokenCostReportField.report_field_for_nb_tokens_by_category(token_category): nb_tokens+ for token_category, nb_tokens in self.nb_tokens_by_category.items()+ }+ the_dict.update(dict_for_nb_tokens)+ dict_for_costs = {+ JudgmentTokenCostReportField.report_field_for_cost_by_category(token_category): cost+ for token_category, cost in self.costs_by_token_category.items()+ }+ the_dict.update(dict_for_costs)+ return the_dict+++class JudgmentTokensUsage(BaseModel):+ """Real input and output token counts, priced per million like an LLM's.++ This is where the family parts company with search, which encodes per-request billing as a token+ count: a judgment backend reports the tokens it actually read. A backend that reports no usage+ leaves ``nb_tokens_by_category`` empty rather than guessing, and the call reports as the+ zero-token attempt it was.+ """++ model_type: Literal["judgment"] = "judgment"+ job_metadata: JobMetadata+ inference_model_name: str+ unit_costs: CostsByCategoryDict+ inference_model_id: str+ nb_tokens_by_category: NbTokensByCategoryDictdiff --git a/pipelex/cogt/judgment/judgment_setting.py b/pipelex/cogt/judgment/judgment_setting.pynew file mode 100644index 000000000..d8327ad7e--- /dev/null+++ b/pipelex/cogt/judgment/judgment_setting.py@@ -0,0 +1,24 @@+from typing import Annotated, Union++from pydantic import BeforeValidator++from pipelex.cogt.models.model_reference import ModelReference, parse_model_reference+from pipelex.system.configuration.config_model import ConfigModel+++class JudgmentSetting(ConfigModel):+ model: str+ description: str | None = None++ def desc(self) -> str:+ return f"JudgmentSetting(model={self.model})"+++# JudgmentModelChoice accepts JudgmentSetting, ModelReference, or a string (which gets parsed to+# ModelReference). The BeforeValidator ensures that strings are automatically converted to+# ModelReference during validation; ModelReference.model_serializer handles serialization back to+# the raw string value.+JudgmentModelChoice = Union[+ JudgmentSetting,+ Annotated[str | ModelReference, BeforeValidator(parse_model_reference)],+]diff --git a/pipelex/cogt/judgment/judgment_worker_abstract.py b/pipelex/cogt/judgment/judgment_worker_abstract.pynew file mode 100644index 000000000..0c5ece47e--- /dev/null+++ b/pipelex/cogt/judgment/judgment_worker_abstract.py@@ -0,0 +1,116 @@+from abc import abstractmethod++from typing_extensions import override++from pipelex import log+from pipelex.cogt.exceptions import CogtError, JudgmentAnswerMismatchError+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.judgment.judgment_job import JudgmentJob+from pipelex.cogt.judgment.judgment_models import ChoiceAnswer, ChoiceQuestion, JudgmentAnswer, JudgmentQuestion, RatingAnswer, RatingQuestion+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.reporting.reporting_protocol import ReportingProtocol+from pipelex.system.job_metadata import UnitJobId+++class JudgmentWorkerAbstract(InferenceWorkerAbstract):+ def __init__(+ self,+ inference_model: InferenceModelSpec,+ reporting_delegate: ReportingProtocol | None = None,+ ):+ InferenceWorkerAbstract.__init__(self, reporting_delegate=reporting_delegate)+ self.inference_model = inference_model++ @property+ @override+ def desc(self) -> str:+ return f"Judgment using {self.inference_model.desc}"++ async def judge(+ self,+ judgment_job: JudgmentJob,+ ) -> dict[str, JudgmentAnswer]:+ """Answer every question in the job over its state, keyed as the questions were."""+ log.dev(f"✨ {self.desc} ✨")+ judgment_job.validate_before_execution()+ judgment_job.job_metadata.unit_job_id = UnitJobId.JUDGMENT_ANSWER+ judgment_job.judgment_job_before_start(inference_model=self.inference_model)++ try:+ answers = await self._judge(judgment_job=judgment_job)+ _check_answers_match_questions(judgment_job=judgment_job, answers=answers)+ except CogtError as exc:+ exc.fill_model_and_provider(model_handle=self.inference_model.name, backend_name=self.inference_model.backend_name)+ raise+ finally:+ # Completion and reporting belong on *every* way out, not just the happy one. The+ # answer-shape guard above raises after the provider already answered and after usage was+ # recorded, so that call is billed — reporting only on success would make the spend vanish+ # from the run's cost report. A failure that never reached the provider recorded no tokens+ # (`judgment_job_before_start` initialises `nb_tokens_by_category` empty), so it reports as+ # the zero-cost attempt it was rather than inventing a charge.+ judgment_job.judgment_job_after_complete()+ if self.reporting_delegate:+ self.reporting_delegate.report_inference_job(inference_job=judgment_job)++ return answers++ @abstractmethod+ async def _judge(+ self,+ judgment_job: JudgmentJob,+ ) -> dict[str, JudgmentAnswer]:+ pass+++def _check_answers_match_questions(*, judgment_job: JudgmentJob, answers: dict[str, JudgmentAnswer]) -> None:+ """Refuse an answer set that does not correspond, question for question, to what was asked.++ Three ways a backend can get this wrong, and all are silent without this check: answering a+ question nobody asked (or dropping one), answering the right question in the wrong shape — a+ choice where a rating was asked for — and naming a verdict the question never offered, such as+ an option it does not list or a level beyond its scale. The caller reads the answers by key, by+ kind and by the option or level they name, so any of them would surface much later, as a+ missing key, a verdict of the wrong type or a verdict routed nowhere.+ """+ asked = set(judgment_job.questions)+ answered = set(answers)+ if asked != answered:+ missing = sorted(asked - answered)+ unasked = sorted(answered - asked)+ msg = f"Judgment worker answered the wrong set of questions: missing {missing}, unasked {unasked}"+ raise JudgmentAnswerMismatchError(msg)+ for question_key, question in judgment_job.questions.items():+ answer = answers[question_key]+ if answer.kind != question.kind:+ msg = f"Judgment worker answered question '{question_key}' with a '{answer.kind}' answer, but it asked for '{question.kind}'"+ raise JudgmentAnswerMismatchError(msg)+ _check_answer_is_offered(question_key=question_key, question=question, answer=answer)+++def _check_answer_is_offered(*, question_key: str, question: JudgmentQuestion, answer: JudgmentAnswer) -> None:+ """Refuse a verdict the question did not offer, in the answer itself or in its distribution.++ The answer models cannot check this on their own, since they never see their question: this is+ the one place that holds both.+ """+ match question, answer:+ case ChoiceQuestion(), ChoiceAnswer():+ named_options = {answer.choice, *(answer.probabilities or {})}+ unoffered = sorted(named_options - set(question.options))+ if unoffered:+ msg = f"Judgment worker answered question '{question_key}' with options it does not offer: {unoffered}"+ raise JudgmentAnswerMismatchError(msg)+ case RatingQuestion(), RatingAnswer():+ nb_levels = len(question.levels)+ named_levels = {answer.level, *(answer.probabilities or {})}+ out_of_scale = sorted(level for level in named_levels if not 0 <= level < nb_levels)+ if out_of_scale:+ msg = (+ f"Judgment worker answered question '{question_key}' with levels {out_of_scale}, "+ f"outside its scale of {nb_levels} levels (0 to {nb_levels - 1})"+ )+ raise JudgmentAnswerMismatchError(msg)+ case _:+ # A yes/no answer names no option and no level, and its probability is bounded by its model.+ passdiff --git a/pipelex/cogt/judgment/judgment_worker_factory.py b/pipelex/cogt/judgment/judgment_worker_factory.pynew file mode 100644index 000000000..bbd49e338--- /dev/null+++ b/pipelex/cogt/judgment/judgment_worker_factory.py@@ -0,0 +1,41 @@+from typing import cast++from pipelex.cogt.judgment.judgment_worker_abstract import JudgmentWorkerAbstract+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.reporting.reporting_protocol import ReportingProtocol+from pipelex.runtime_hub import get_inference_backend_registry, get_models_manager, get_sdk_client_manager+++class JudgmentWorkerFactory:+ @classmethod+ def make_judgment_worker(+ cls,+ inference_model: InferenceModelSpec,+ *,+ reporting_delegate: ReportingProtocol | None = None,+ ) -> JudgmentWorkerAbstract:+ """Create a judgment worker for the given inference model via the inference-backend registry.++ Args:+ inference_model: The model spec from the backend configuration.+ reporting_delegate: The reporting delegate passed through to the worker+ (supplied by the caller — the factory no longer reaches into the hub).++ Returns:+ A JudgmentWorkerAbstract instance.+ """+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.JUDGMENT, sdk=model_handle.sdk)+ worker = make_worker(+ inference_model=inference_model,+ backend=backend,+ sdk_clients=get_sdk_client_manager().sdk_client_registry,+ reporting_delegate=reporting_delegate,+ )+ # The JUDGMENT family registry only ever holds judgment workers (the+ # (family, sdk) key guarantees it); the uniform MakeWorkerFn return type is+ # widened to InferenceWorkerAbstract, so narrow it back here.+ return cast("JudgmentWorkerAbstract", worker)diff --git a/pipelex/cogt/model_backends/model_type.py b/pipelex/cogt/model_backends/model_type.pyindex e7344fedc..7e08c5215 100644--- a/pipelex/cogt/model_backends/model_type.py+++ b/pipelex/cogt/model_backends/model_type.py@@ -7,6 +7,7 @@ class ModelType(StrEnum):IMG_GEN = "img_gen"SEARCH = "search"DOC_GEN = "doc_gen"+ JUDGMENT = "judgment"@propertydef indefinite_description(self) -> str:@@ -22,3 +23,5 @@ class ModelType(StrEnum):return "a search model"case ModelType.DOC_GEN:return "a document-generation engine"+ case ModelType.JUDGMENT:+ return "a judgment model"diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.pyindex 931f46d65..2d5706f3e 100644--- a/pipelex/cogt/models/model_deck.py+++ b/pipelex/cogt/models/model_deck.py@@ -10,6 +10,7 @@ from pipelex.cogt.exceptions import (DocGenHandleNotFoundError,ExtractHandleNotFoundError,ImgGenHandleNotFoundError,+ JudgmentHandleNotFoundError,LLMHandleNotFoundError,ModelChoiceNotFoundError,ModelDeckPresetValidatonError,@@ -21,6 +22,7 @@ from pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSettfrom pipelex.cogt.img_gen.img_gen_job_components import Qualityfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSettingfrom pipelex.cogt.inference.error_classification import UserAction, UserActionKind+from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice, JudgmentSettingfrom pipelex.cogt.llm.llm_setting import (LLMModelChoice,LLMSetting,@@ -92,6 +94,23 @@ class DocGenDeckBlueprint(ConfigModel):return choice_defaults+class JudgmentDeckBlueprint(ConfigModel):+ """The judgment family's half of the deck.++ ``choice_default`` is optional here where every other family requires one, and the reason is+ that no judgment model is served by default. The other families' defaults name a handle the+ Pipelex Gateway carries, so they always resolve; a judgment backend is brought by the user, and+ a deck naming a default that nothing serves fails the gateway's own membership check at boot —+ for every user, whether or not they ever ask for a judgment. Absent says the honest thing: no+ model is the default, so a judgment names its own until one is served.+ """++ aliases: dict[str, str] = Field(default_factory=dict)+ waterfalls: dict[str, list[str]] = Field(default_factory=dict)+ presets: dict[str, JudgmentSetting] = Field(default_factory=dict)+ choice_default: JudgmentModelChoice | None = None++class ModelDeckBlueprint(ConfigModel):llm: LLMDeckBlueprintextract: ExtractDeckBlueprint@@ -100,6 +119,10 @@ class ModelDeckBlueprint(ConfigModel):# Optional, unlike the other families: a project whose deck predates it still boots, and only its# `PipeDocGen` steps are refused, naming the missing default, until `pipelex update` adds the deck file.doc_gen: DocGenDeckBlueprint = Field(default_factory=DocGenDeckBlueprint)+ # Defaulted where the other families are required: a deck installed before this family+ # existed has no judgment file, only `pipelex update` installs one, and its absence means+ # exactly what the kit's own empty section means — no judgment model.+ judgment: JudgmentDeckBlueprint = Field(default_factory=JudgmentDeckBlueprint)class ModelDeck(ConfigModel):@@ -145,6 +168,13 @@ class ModelDeck(ConfigModel):doc_gen_presets: dict[str, DocGenSetting] = Field(default_factory=dict)doc_gen_choice_defaults: dict[str, DocGenModelChoice] = Field(default_factory=dict)+ # Judgment-specific+ judgment_aliases: dict[str, str] = Field(default_factory=dict)+ judgment_waterfalls: dict[str, list[str]] = Field(default_factory=dict)+ judgment_presets: dict[str, JudgmentSetting] = Field(default_factory=dict)+ # Optional, unlike every other family's — see JudgmentDeckBlueprint for why.+ judgment_choice_default: JudgmentModelChoice | None = None+def get_aliases_and_waterfalls_for_type(self, model_type: ModelType) -> tuple[dict[str, str], dict[str, list[str]]]:"""Return the type-specific aliases and waterfalls dictionaries."""match model_type:@@ -158,6 +188,8 @@ class ModelDeck(ConfigModel):return self.search_aliases, self.search_waterfallscase ModelType.DOC_GEN:return self.doc_gen_aliases, self.doc_gen_waterfalls+ case ModelType.JUDGMENT:+ return self.judgment_aliases, self.judgment_waterfallsdef is_model_handle_defined(self, model_handle: str, *, model_type: ModelType) -> bool:"""Check if a model handle is defined in the model deck.@@ -257,12 +289,31 @@ class ModelDeck(ConfigModel):f"Bare string '{name}' matches: {', '.join(matches)}. Using it as a direct model handle. Add explicit prefix to avoid ambiguity.")+ def _warn_if_ambiguous_judgment(self, name: str) -> None:+ """Log a warning if a bare string handle matches presets/aliases/waterfalls."""+ matches: list[str] = []+ if name in self.judgment_presets:+ matches.append(f"judgment preset (use ${name} or preset:{name})")+ if name in self.judgment_aliases:+ matches.append(f"alias (use @{name} or alias:{name})")+ if name in self.judgment_waterfalls:+ matches.append(f"waterfall (use ~{name} or waterfall:{name})")+ if matches:+ log.warning(+ f"Bare string '{name}' matches: {', '.join(matches)}. Using it as a direct model handle. Add explicit prefix to avoid ambiguity."+ )+def _raise_handle_not_found_error(self,ref: ModelReference,*,model_type: ModelType,- presets: dict[str, LLMSetting] | dict[str, ExtractSetting] | dict[str, ImgGenSetting] | dict[str, SearchSetting] | dict[str, DocGenSetting],+ presets: dict[str, LLMSetting]+ | dict[str, ExtractSetting]+ | dict[str, ImgGenSetting]+ | dict[str, SearchSetting]+ | dict[str, DocGenSetting]+ | dict[str, JudgmentSetting],) -> NoReturn:"""Raise ModelChoiceNotFoundError with migration hints if applicable."""msg = f"Model handle '{ref.name}' was not found in the model deck"@@ -551,6 +602,55 @@ class ModelDeck(ConfigModel):presets=self.doc_gen_presets,)+ def get_judgment_setting(self, judgment_choice: JudgmentModelChoice) -> JudgmentSetting:+ if isinstance(judgment_choice, JudgmentSetting):+ return judgment_choice++ ref = ensure_model_reference(judgment_choice)+ match ref.kind:+ case ModelReferenceKind.PRESET:+ if preset := self.judgment_presets.get(ref.name):+ return preset+ msg = f"Judgment preset '{ref.name}' was not found in the model deck"+ raise ModelChoiceNotFoundError(+ message=msg,+ model_type=ModelType.JUDGMENT,+ model_choice=ref.raw,+ reference_kind=ModelReferenceKind.PRESET,+ available_options=list(self.judgment_presets.keys()),+ )+ case ModelReferenceKind.ALIAS:+ if alias_target := self.judgment_aliases.get(ref.name):+ return JudgmentSetting(model=alias_target)+ msg = f"Alias '{ref.name}' was not found in the model deck"+ raise ModelChoiceNotFoundError(+ message=msg,+ model_type=ModelType.JUDGMENT,+ model_choice=ref.raw,+ reference_kind=ModelReferenceKind.ALIAS,+ available_options=list(self.judgment_aliases.keys()),+ )+ case ModelReferenceKind.WATERFALL:+ if ref.name in self.judgment_waterfalls:+ return JudgmentSetting(model=ref.name)+ msg = f"Waterfall '{ref.name}' was not found in the model deck"+ raise ModelChoiceNotFoundError(+ message=msg,+ model_type=ModelType.JUDGMENT,+ model_choice=ref.raw,+ reference_kind=ModelReferenceKind.WATERFALL,+ available_options=list(self.judgment_waterfalls.keys()),+ )+ case ModelReferenceKind.HANDLE:+ self._warn_if_ambiguous_judgment(ref.name)+ if self.is_model_handle_defined(model_handle=ref.name, model_type=ModelType.JUDGMENT):+ return JudgmentSetting(model=ref.name)+ self._raise_handle_not_found_error(+ ref=ref,+ model_type=ModelType.JUDGMENT,+ presets=self.judgment_presets,+ )+def get_img_gen_setting(self, img_gen_choice: ImgGenModelChoice) -> ImgGenSetting:if isinstance(img_gen_choice, ImgGenSetting):return img_gen_choice@@ -688,6 +788,17 @@ class ModelDeck(ConfigModel):)return self+ def validate_judgment_presets(self) -> Self:+ for judgment_preset_id, judgment_setting in self.judgment_presets.items():+ if not self.is_model_handle_defined(model_handle=judgment_setting.model, model_type=ModelType.JUDGMENT):+ msg = f"Judgment handle '{judgment_setting.model}' for judgment preset '{judgment_preset_id}' was not found in the model deck"+ raise JudgmentHandleNotFoundError(+ message=msg,+ preset_id=judgment_preset_id,+ model_handle=judgment_setting.model,+ )+ return self+def validate_registered_models(self):self.validate_inference_models()try:@@ -771,6 +882,22 @@ class ModelDeck(ConfigModel):log.warning(f"DocGen handle not found: {exc}")case ProblemReaction.NONE:pass+ try:+ self.validate_judgment_presets()+ except JudgmentHandleNotFoundError as exc:+ match self.model_deck_config.missing_presets_reaction:+ case ProblemReaction.RAISE:+ msg = f"Failed to validate all Judgment presets: {exc}"+ raise ModelDeckPresetValidatonError(+ message=msg,+ model_type=ModelType.JUDGMENT,+ preset_id=exc.preset_id,+ model_handle=exc.model_handle,+ ) from exc+ case ProblemReaction.LOG:+ log.warning(f"Judgment handle not found: {exc}")+ case ProblemReaction.NONE:+ passdef validate_inference_models(self):for model_handle, model_spec in self.inference_models.items():@@ -996,8 +1123,10 @@ class ModelDeck(ConfigModel):named_references: set[str] = set(aliases.values())for fallback_list in waterfalls.values():named_references.update(fallback_list)- deck_settings: list[LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | DocGenSetting]- deck_choices: list[LLMModelChoice | ExtractModelChoice | ImgGenModelChoice | SearchModelChoice | DocGenModelChoice | None]+ deck_settings: list[LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | DocGenSetting | JudgmentSetting]+ deck_choices: list[+ LLMModelChoice | ExtractModelChoice | ImgGenModelChoice | SearchModelChoice | DocGenModelChoice | JudgmentModelChoice | None+ ]match model_type:case ModelType.LLM:deck_settings = list(self.llm_presets.values())@@ -1019,9 +1148,12 @@ class ModelDeck(ConfigModel):case ModelType.DOC_GEN:deck_settings = list(self.doc_gen_presets.values())deck_choices = list(self.doc_gen_choice_defaults.values())+ case ModelType.JUDGMENT:+ deck_settings = list(self.judgment_presets.values())+ deck_choices = [self.judgment_choice_default]named_references.update(deck_setting.model for deck_setting in deck_settings)for deck_choice in deck_choices:- if isinstance(deck_choice, (LLMSetting, ExtractSetting, ImgGenSetting, SearchSetting, DocGenSetting)):+ if isinstance(deck_choice, (LLMSetting, ExtractSetting, ImgGenSetting, SearchSetting, DocGenSetting, JudgmentSetting)):named_references.add(deck_choice.model)elif isinstance(deck_choice, ModelReference):match deck_choice.kind:diff --git a/pipelex/cogt/models/model_deck_check.py b/pipelex/cogt/models/model_deck_check.pyindex 930df875e..0b92ae1cc 100644--- a/pipelex/cogt/models/model_deck_check.py+++ b/pipelex/cogt/models/model_deck_check.py@@ -6,6 +6,7 @@ from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice, DocGenSettinfrom pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSettingfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting+from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice, JudgmentSettingfrom pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSettingfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck import ModelDeck@@ -289,6 +290,68 @@ def check_doc_gen_choice_with_deck(*, doc_gen_choice: DocGenModelChoice) -> None)+def check_judgment_choice_with_deck(judgment_choice: JudgmentModelChoice) -> None:+ if isinstance(judgment_choice, JudgmentSetting):+ return++ model_deck = get_model_deck()+ ref = ensure_model_reference(judgment_choice)++ match ref.kind:+ case ModelReferenceKind.PRESET:+ if ref.name in model_deck.judgment_presets:+ return+ msg = f"Judgment preset '{ref.name}' was not found in the model deck"+ _raise_model_choice_not_found(+ msg,+ model_deck=model_deck,+ model_type=ModelType.JUDGMENT,+ raw_choice=ref.raw,+ name=ref.name,+ reference_kind=ModelReferenceKind.PRESET,+ available_options=list(model_deck.judgment_presets.keys()),+ )+ case ModelReferenceKind.ALIAS:+ if ref.name in model_deck.judgment_aliases:+ return+ msg = f"Alias '{ref.name}' was not found in the model deck"+ _raise_model_choice_not_found(+ msg,+ model_deck=model_deck,+ model_type=ModelType.JUDGMENT,+ raw_choice=ref.raw,+ name=ref.name,+ reference_kind=ModelReferenceKind.ALIAS,+ available_options=list(model_deck.judgment_aliases.keys()),+ )+ case ModelReferenceKind.WATERFALL:+ if ref.name in model_deck.judgment_waterfalls:+ return+ msg = f"Waterfall '{ref.name}' was not found in the model deck"+ _raise_model_choice_not_found(+ msg,+ model_deck=model_deck,+ model_type=ModelType.JUDGMENT,+ raw_choice=ref.raw,+ name=ref.name,+ reference_kind=ModelReferenceKind.WATERFALL,+ available_options=list(model_deck.judgment_waterfalls.keys()),+ )+ case ModelReferenceKind.HANDLE:+ if model_deck.is_model_handle_defined(model_handle=ref.name, model_type=ModelType.JUDGMENT):+ return+ msg = f"Model handle '{ref.name}' was not found in the model deck"+ _raise_model_choice_not_found(+ msg,+ model_deck=model_deck,+ model_type=ModelType.JUDGMENT,+ raw_choice=ref.raw,+ name=ref.name,+ reference_kind=ModelReferenceKind.HANDLE,+ available_options=list(model_deck.inference_models.keys()),+ )++def check_img_gen_choice_with_deck(img_gen_choice: ImgGenModelChoice) -> None:if isinstance(img_gen_choice, ImgGenSetting):returndiff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.pyindex 46e6fe4f1..83f246adc 100644--- a/pipelex/cogt/models/model_manager.py+++ b/pipelex/cogt/models/model_manager.py@@ -9,6 +9,7 @@ from pipelex.cogt.doc_gen.doc_gen_format import parse_doc_gen_choice_keyfrom pipelex.cogt.exceptions import GatewayUnknownModelError, ModelManagerError, PluginModelDeclarationErrorfrom pipelex.cogt.extract.extract_setting import ExtractSettingfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenSetting+from pipelex.cogt.judgment.judgment_setting import JudgmentSettingfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.model_backends.backend import InferenceBackend, PipelexBackendfrom pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary@@ -268,10 +269,18 @@ class ModelManager(ModelManagerAbstract):search_default_handle = cls._extract_choice_handle(deck.search_choice_default)if search_default_handle is not None:references.append((search_default_handle, ModelType.SEARCH))+ for judgment_setting in deck.judgment_presets.values():+ references.append((judgment_setting.model, ModelType.JUDGMENT))+ judgment_default_handle = cls._extract_choice_handle(deck.judgment_choice_default)+ if judgment_default_handle is not None:+ references.append((judgment_default_handle, ModelType.JUDGMENT))return references@classmethod- def _extract_choice_handle(cls, choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | ModelReference | str | None) -> str | None:+ def _extract_choice_handle(+ cls,+ choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | JudgmentSetting | ModelReference | str | None,+ ) -> str | None:"""Normalise a ``*ModelChoice`` union (LLMModelChoice etc.) to a raw handle string.Choice defaults can be a typed setting object, a parsed ``ModelReference``, or a raw@@ -480,6 +489,11 @@ class ModelManager(ModelManagerAbstract):doc_gen_waterfalls=model_deck_blueprint.doc_gen.waterfalls,doc_gen_presets=model_deck_blueprint.doc_gen.presets,doc_gen_choice_defaults=model_deck_blueprint.doc_gen.choice_defaults,+ # Judgment+ judgment_aliases=model_deck_blueprint.judgment.aliases,+ judgment_waterfalls=model_deck_blueprint.judgment.waterfalls,+ judgment_presets=model_deck_blueprint.judgment.presets,+ judgment_choice_default=model_deck_blueprint.judgment.choice_default,model_deck_config=get_config().inference.model_deck,)diff --git a/pipelex/cogt/models/model_suggestion.py b/pipelex/cogt/models/model_suggestion.pyindex 9934fc683..36bc3735e 100644--- a/pipelex/cogt/models/model_suggestion.py+++ b/pipelex/cogt/models/model_suggestion.py@@ -50,6 +50,8 @@ def get_collection_keys(return list(model_deck.search_presets.keys())case ModelType.DOC_GEN:return list(model_deck.doc_gen_presets.keys())+ case ModelType.JUDGMENT:+ return list(model_deck.judgment_presets.keys())case ModelReferenceKind.ALIAS:match model_type:case ModelType.LLM:@@ -62,6 +64,8 @@ def get_collection_keys(return list(model_deck.search_aliases.keys())case ModelType.DOC_GEN:return list(model_deck.doc_gen_aliases.keys())+ case ModelType.JUDGMENT:+ return list(model_deck.judgment_aliases.keys())case ModelReferenceKind.WATERFALL:match model_type:case ModelType.LLM:@@ -74,6 +78,8 @@ def get_collection_keys(return list(model_deck.search_waterfalls.keys())case ModelType.DOC_GEN:return list(model_deck.doc_gen_waterfalls.keys())+ case ModelType.JUDGMENT:+ return list(model_deck.judgment_waterfalls.keys())case ModelReferenceKind.HANDLE:return sorted(handle for handle, spec in model_deck.inference_models.items() if spec.model_type == model_type)diff --git a/pipelex/cogt/usage/cost_registry.py b/pipelex/cogt/usage/cost_registry.pyindex 2174dbf6f..47a8b0a44 100644--- a/pipelex/cogt/usage/cost_registry.py+++ b/pipelex/cogt/usage/cost_registry.py@@ -1,7 +1,7 @@import csvfrom collections.abc import Sequencefrom pathlib import Path-from typing import Any, NamedTuple+from typing import Any, NamedTuple, assert_neverfrom pydantic import Field, RootModel@@ -9,6 +9,7 @@ from pipelex import logfrom pipelex.cogt.exceptions import CostRegistryErrorfrom pipelex.cogt.extract.extract_report import ExtractTokenCostReport, ExtractTokenCostReportField, ExtractTokensUsagefrom pipelex.cogt.img_gen.img_gen_report import ImgGenTokenCostReport, ImgGenTokenCostReportField, ImgGenTokensUsage+from pipelex.cogt.judgment.judgment_report import JudgmentTokenCostReport, JudgmentTokenCostReportField, JudgmentTokensUsagefrom pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsagefrom pipelex.cogt.search.search_report import SearchTokenCostReport, SearchTokenCostReportField, SearchTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict@@ -17,8 +18,8 @@ from pipelex.cogt.usage.token_category import TokenCategoryfrom pipelex.runtime_hub import get_consolefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage-TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport+TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | JudgmentTokensUsage+TokenCostReport = LLMTokenCostReport | ImgGenTokenCostReport | ExtractTokenCostReport | SearchTokenCostReport | JudgmentTokenCostReportCostRegistryRoot = list[TokenCostReport]@@ -276,7 +277,8 @@ class CostRegistry(RootModel[CostRegistryRoot]):record.get(report_field.LLM_NAME)or record.get(ImgGenTokenCostReportField.IMG_GEN_NAME)or record.get(ExtractTokenCostReportField.EXTRACT_NAME)- or record.get(SearchTokenCostReportField.SEARCH_NAME, "unknown")+ or record.get(SearchTokenCostReportField.SEARCH_NAME)+ or record.get(JudgmentTokenCostReportField.JUDGMENT_NAME, "unknown"))model_types[model_name] = record.get(report_field.MODEL_TYPE, "llm")if model_name not in grouped_by_model:@@ -360,41 +362,60 @@ class CostRegistry(RootModel[CostRegistryRoot]):)costs_by_token_category[token_type.to_cost_category] = cost_per_token * nb_tokens- if isinstance(tokens_usage, LLMTokensUsage):- return LLMTokenCostReport(- model_type=tokens_usage.model_type,- job_metadata=tokens_usage.job_metadata,- inference_model_name=tokens_usage.inference_model_name,- platform_model_id=tokens_usage.inference_model_id,- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,- costs_by_token_category=costs_by_token_category,- )- if isinstance(tokens_usage, ImgGenTokensUsage):- return ImgGenTokenCostReport(- model_type=tokens_usage.model_type,- job_metadata=tokens_usage.job_metadata,- inference_model_name=tokens_usage.inference_model_name,- platform_model_id=tokens_usage.inference_model_id,- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,- costs_by_token_category=costs_by_token_category,- )- if isinstance(tokens_usage, SearchTokensUsage):- return SearchTokenCostReport(- model_type=tokens_usage.model_type,- job_metadata=tokens_usage.job_metadata,- inference_model_name=tokens_usage.inference_model_name,- platform_model_id=tokens_usage.inference_model_id,- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,- costs_by_token_category=costs_by_token_category,- )- return ExtractTokenCostReport(- model_type=tokens_usage.model_type,- job_metadata=tokens_usage.job_metadata,- inference_model_name=tokens_usage.inference_model_name,- platform_model_id=tokens_usage.inference_model_id,- nb_tokens_by_category=tokens_usage.nb_tokens_by_category,- costs_by_token_category=costs_by_token_category,- )+ # A match over the usage classes rather than an if-chain ending in a bare return, and the+ # difference is the last arm. The dispatch used to let everything it did not recognise fall+ # through to an extract report, so a new family's usage silently billed as an extraction,+ # under the wrong model-type label. The trailing `case _` is not a catch-all that hides a+ # missing arm — it is the opposite: `assert_never` fails the type check the day the union+ # grows an arm this match does not handle, which is how the fallthrough is now impossible.+ match tokens_usage:+ case LLMTokensUsage():+ return LLMTokenCostReport(+ model_type=tokens_usage.model_type,+ job_metadata=tokens_usage.job_metadata,+ inference_model_name=tokens_usage.inference_model_name,+ platform_model_id=tokens_usage.inference_model_id,+ nb_tokens_by_category=tokens_usage.nb_tokens_by_category,+ costs_by_token_category=costs_by_token_category,+ )+ case ImgGenTokensUsage():+ return ImgGenTokenCostReport(+ model_type=tokens_usage.model_type,+ job_metadata=tokens_usage.job_metadata,+ inference_model_name=tokens_usage.inference_model_name,+ platform_model_id=tokens_usage.inference_model_id,+ nb_tokens_by_category=tokens_usage.nb_tokens_by_category,+ costs_by_token_category=costs_by_token_category,+ )+ case SearchTokensUsage():+ return SearchTokenCostReport(+ model_type=tokens_usage.model_type,+ job_metadata=tokens_usage.job_metadata,+ inference_model_name=tokens_usage.inference_model_name,+ platform_model_id=tokens_usage.inference_model_id,+ nb_tokens_by_category=tokens_usage.nb_tokens_by_category,+ costs_by_token_category=costs_by_token_category,+ )+ case JudgmentTokensUsage():+ return JudgmentTokenCostReport(+ model_type=tokens_usage.model_type,+ job_metadata=tokens_usage.job_metadata,+ inference_model_name=tokens_usage.inference_model_name,+ platform_model_id=tokens_usage.inference_model_id,+ nb_tokens_by_category=tokens_usage.nb_tokens_by_category,+ costs_by_token_category=costs_by_token_category,+ )+ case ExtractTokensUsage():+ return ExtractTokenCostReport(+ model_type=tokens_usage.model_type,+ job_metadata=tokens_usage.job_metadata,+ inference_model_name=tokens_usage.inference_model_name,+ platform_model_id=tokens_usage.inference_model_id,+ nb_tokens_by_category=tokens_usage.nb_tokens_by_category,+ costs_by_token_category=costs_by_token_category,+ )+ case _:+ assert_never(tokens_usage)@classmethoddef complete_cost_report(cls, tokens_usage: TokensUsage) -> TokenCostReport:diff --git a/pipelex/kit/configs/inference/deck/6_judgment_deck.toml b/pipelex/kit/configs/inference/deck/6_judgment_deck.tomlnew file mode 100644index 000000000..3692726eb--- /dev/null+++ b/pipelex/kit/configs/inference/deck/6_judgment_deck.toml@@ -0,0 +1,46 @@+####################################################################################################+# Pipelex Model Deck - Judgment Configuration+####################################################################################################+#+# This file defines model aliases and presets for Judgment models — models that answer+# closed-form questions (yes/no, choice, rating) over some material, with a measure of+# how sure they are.+#+# Managed by `pipelex update`. To customize without conflict, create an+# x_custom_*.toml file in this directory — pipelex never tracks or+# overwrites x_custom_* files. Local edits to this file are backed up+# to <file>.bak.<timestamp> on update but will not survive future runs.+#+# Model Reference Syntax:+# - Preset: $preset_name or preset:preset_name+# - Alias: @alias_name or alias:alias_name+#+# Documentation: https://docs.pipelex.com+# Support: https://go.pipelex.com/discord+#+####################################################################################################++####################################################################################################+# Judgment Default Choice+####################################################################################################+#+# There is deliberately no `choice_default` here, and that is what tells you no judgment model is+# served out of the box: a judgment backend is one you bring, so a pipe names the model it judges+# with until the Pipelex Gateway serves the family. Set one here once you have a backend configured.++[judgment]++####################################################################################################+# Aliases+####################################################################################################+#+# A judgment alias points at a VERSIONED model id, never at a provider's `latest` alias: a verdict+# model that changes under a method silently changes what every threshold in that method means.++[judgment.aliases]++####################################################################################################+# Judgment Presets+####################################################################################################++[judgment.presets]diff --git a/pipelex/kit/deck_variants/README.md b/pipelex/kit/deck_variants/README.mdindex 4a893b2ca..a37825664 100644--- a/pipelex/kit/deck_variants/README.md+++ b/pipelex/kit/deck_variants/README.md@@ -1,6 +1,6 @@# Deck variants-A deck variant is a complete set of the numbered model deck files (`1_llm_deck.toml`, `2_img_gen_deck.toml`, `3_extract_deck.toml`, `4_search_deck.toml`, `5_doc_gen_deck.toml`) kept as an alternative edition of the deck that the kit ships in `pipelex/kit/configs/inference/deck/`. Nothing loads a variant: `pipelex init` does not copy it, `pipelex update` and `pipelex doctor` do not diff against it, and the deck manifest does not know it exists. It is source code that is kept correct, not configuration that is applied.+A deck variant is a complete set of the numbered model deck files (`1_llm_deck.toml`, `2_img_gen_deck.toml`, `3_extract_deck.toml`, `4_search_deck.toml`, `5_doc_gen_deck.toml`, `6_judgment_deck.toml`) kept as an alternative edition of the deck that the kit ships in `pipelex/kit/configs/inference/deck/`. Nothing loads a variant: `pipelex init` does not copy it, `pipelex update` and `pipelex doctor` do not diff against it, and the deck manifest does not know it exists. It is source code that is kept correct, not configuration that is applied.`multi_provider/` is the deck Pipelex shipped before the default moved to the models the Pipelex Gateway serves from Azure. It resolves the default aliases and presets to Anthropic, Google and OpenAI models, and it still defines the three provider-named aliases (`best-claude`, `best-gemini`, `best-mistral`) that the shipped deck no longer carries.diff --git a/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml b/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.tomlnew file mode 100644index 000000000..3692726eb--- /dev/null+++ b/pipelex/kit/deck_variants/multi_provider/6_judgment_deck.toml@@ -0,0 +1,46 @@+####################################################################################################+# Pipelex Model Deck - Judgment Configuration+####################################################################################################+#+# This file defines model aliases and presets for Judgment models — models that answer+# closed-form questions (yes/no, choice, rating) over some material, with a measure of+# how sure they are.+#+# Managed by `pipelex update`. To customize without conflict, create an+# x_custom_*.toml file in this directory — pipelex never tracks or+# overwrites x_custom_* files. Local edits to this file are backed up+# to <file>.bak.<timestamp> on update but will not survive future runs.+#+# Model Reference Syntax:+# - Preset: $preset_name or preset:preset_name+# - Alias: @alias_name or alias:alias_name+#+# Documentation: https://docs.pipelex.com+# Support: https://go.pipelex.com/discord+#+####################################################################################################++####################################################################################################+# Judgment Default Choice+####################################################################################################+#+# There is deliberately no `choice_default` here, and that is what tells you no judgment model is+# served out of the box: a judgment backend is one you bring, so a pipe names the model it judges+# with until the Pipelex Gateway serves the family. Set one here once you have a backend configured.++[judgment]++####################################################################################################+# Aliases+####################################################################################################+#+# A judgment alias points at a VERSIONED model id, never at a provider's `latest` alias: a verdict+# model that changes under a method silently changes what every threshold in that method means.++[judgment.aliases]++####################################################################################################+# Judgment Presets+####################################################################################################++[judgment.presets]diff --git a/pipelex/migration/goldens/inference-backend/[email protected] b/pipelex/migration/goldens/inference-backend/[email protected]index 55a738271..fb9e10a20 100644--- a/pipelex/migration/goldens/inference-backend/[email protected]+++ b/pipelex/migration/goldens/inference-backend/[email protected]@@ -64,6 +64,7 @@"enum_members": ["doc_gen","img_gen",+ "judgment","llm","search","text_extractor"diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex 4db7a63df..a013df6ff 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -477,7 +477,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):them would silently drop entries on collision.Args:- category: Optional deck filter (`llm`, `extract`, `img_gen`, `search`).+ category: Optional deck filter, one of the protocol's own categories.Returns:PipelexModelDeck with the flat model list and the category-keyed@@ -490,6 +490,13 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):aliases_by_category: dict[str, dict[str, str]] = deck_raw["aliases"]waterfalls_by_category: dict[str, dict[str, list[str]]] = deck_raw["waterfalls"]for category_key, category_presets in presets_by_category.items():+ if category_key not in MthdsModelCategory.__members__.values():+ # A category this runtime serves that the standard's flat list has no word for —+ # `judgment` today. Its presets are left out of `models` rather than raising, because+ # one unrepresentable category must not take the whole deck down; its aliases and+ # waterfalls still travel under the category-keyed routing extensions below. This+ # branch disappears when the protocol's `ModelCategory` learns the member.+ continuefor preset in category_presets:models.append(MthdsModelInfo(name=preset["name"], type=MthdsModelCategory(category_key)))return PipelexModelDeck(models=models, aliases=aliases_by_category, waterfalls=waterfalls_by_category)diff --git a/pipelex/plugins/inference_backend_registry.py b/pipelex/plugins/inference_backend_registry.pyindex b09386bf8..edcf4a347 100644--- a/pipelex/plugins/inference_backend_registry.py+++ b/pipelex/plugins/inference_backend_registry.py@@ -14,6 +14,7 @@ class InferenceFamily(StrEnum):EXTRACT = "extract"SEARCH = "search"DOC_GEN = "doc_gen"+ JUDGMENT = "judgment"# The uniform inference-backend factory. A backend plugin registers one of thesediff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex fbd77230d..4bd925042 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -8,6 +8,7 @@ from pipelex.base_exceptions import PipelexConfigErrorfrom pipelex.cogt.extract.extract_job import ExtractJobfrom pipelex.cogt.img_gen.img_gen_job import ImgGenJobfrom pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract+from pipelex.cogt.judgment.judgment_job import JudgmentJobfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.search.search_job import SearchJobfrom pipelex.config import get_config@@ -158,6 +159,15 @@ class ReportingManager(ReportingProtocol):self._emit_usage_event(search_job, tokens_usage=search_tokens_usage)+ def _report_judgment_job(self, *, judgment_job: JudgmentJob):+ judgment_tokens_usage = judgment_job.job_report.judgment_tokens_usage++ if not judgment_tokens_usage:+ log.warning("Judgment job has no judgment_tokens_usage")+ return++ self._emit_usage_event(judgment_job, tokens_usage=judgment_tokens_usage)+def _emit_usage_event(self, inference_job: InferenceJobAbstract, *, tokens_usage: AnyTokensUsage) -> None:"""Emit a UsageReportEvent for this job.@@ -318,5 +328,7 @@ class ReportingManager(ReportingProtocol):self._report_extract_job(extract_job=inference_job)elif isinstance(inference_job, SearchJob):self._report_search_job(search_job=inference_job)+ elif isinstance(inference_job, JudgmentJob):+ self._report_judgment_job(judgment_job=inference_job)else:log.warning(f"ReportingManager does not support reporting for inference job type: {type(inference_job).__name__}")diff --git a/pipelex/reporting/reporting_types.py b/pipelex/reporting/reporting_types.pyindex ef5c47e23..4c0d89131 100644--- a/pipelex/reporting/reporting_types.py+++ b/pipelex/reporting/reporting_types.py@@ -6,11 +6,12 @@ from pydantic import Fieldfrom pipelex.cogt.extract.extract_report import ExtractTokensUsagefrom pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage+from pipelex.cogt.judgment.judgment_report import JudgmentTokensUsagefrom pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.search.search_report import SearchTokensUsageAnyTokensUsage = Annotated[- LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage,+ LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage | JudgmentTokensUsage,Field(discriminator="model_type"),]TokensUsage = AnyTokensUsagediff --git a/pipelex/system/job_metadata.py b/pipelex/system/job_metadata.pyindex a3071884f..b1fadb47e 100644--- a/pipelex/system/job_metadata.py+++ b/pipelex/system/job_metadata.py@@ -57,6 +57,7 @@ class JobCategory(StrEnum):JINJA2_JOB = "jinja2_job"EXTRACT_JOB = "extract_job"SEARCH_JOB = "search_job"+ JUDGMENT_JOB = "judgment_job"class UnitJobId(StrEnum):@@ -66,6 +67,7 @@ class UnitJobId(StrEnum):EXTRACT_PAGES = "extract_pages"SEARCH_SOURCED_ANSWER = "search_sourced_answer"SEARCH_STRUCTURED = "search_structured"+ JUDGMENT_ANSWER = "judgment_answer"@propertydef model_kind(self) -> str:@@ -78,6 +80,8 @@ class UnitJobId(StrEnum):return "Extract"case UnitJobId.SEARCH_SOURCED_ANSWER | UnitJobId.SEARCH_STRUCTURED:return "Search"+ case UnitJobId.JUDGMENT_ANSWER:+ return "Judgment"class RunMetadata(BaseModel):diff --git a/pyproject.toml b/pyproject.tomlindex 35a8caba9..6979d31ab 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -297,9 +297,9 @@ norecursedirs = []addopts = ["--import-mode=importlib",- "-ra", # Show all test outcomes (including skips)+ "-ra", # Show all test outcomes (including skips)"-m",- "not (inference or llm or img_gen or extract or search or pipelex_api)",+ "not (inference or llm or img_gen or extract or search or judgment or pipelex_api)",]asyncio_default_fixture_loop_scope = "session"xfail_strict = true@@ -317,6 +317,7 @@ markers = ["img_gen: slow and costly due to image generation inference calls","extract: slow and costly due to doc extraction inference calls","search: slow and costly due to search inference calls",+ "judgment: slow and costly due to judgment inference calls","gha_disabled: tests that should not run in GitHub Actions","codex_disabled: tests that should not run in Codex","dry_runnable: tests that can be run in dry-run mode",diff --git a/subject_grants.toml b/subject_grants.tomlindex 857b7e3aa..6a9f9d427 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -1276,6 +1276,10 @@ rationale = "Factory from source: builds the result from the object_class; singlparam = "output_class"rationale = "Factory from source: builds the result from the output_class; single operand."+["pipelex/cogt/content_generation/content_generator_protocol.py::ContentGeneratorProtocol.make_judgment_answers"]+param = "judgment_assignment"+rationale = "Worker verb-object: processes the judgment_assignment; single operand, mirroring make_search_sourced_answer."+["pipelex/cogt/content_generation/content_generator_protocol.py::ContentGeneratorProtocol.make_search_sourced_answer"]param = "search_assignment"rationale = "Factory from source: builds the result from the search_assignment; single operand."@@ -1324,6 +1328,10 @@ rationale = "Consumes the extract_assignment; single operand."param = "img_gen_assignment"rationale = "Consumes the img_gen_assignment; single operand."+["pipelex/cogt/content_generation/dry_mock.py::dry_judgment_gen_answers"]+param = "judgment_assignment"+rationale = "Worker verb-object: processes the judgment_assignment; single operand."+["pipelex/cogt/content_generation/dry_mock.py::dry_llm_gen_object"]param = "object_assignment"rationale = "Consumes the object_assignment; single operand."@@ -1464,6 +1472,18 @@ rationale = "Processes the img-gen assignment; single operand."param = "img_gen_assignment"rationale = "Processes the img-gen assignment; single operand."+["pipelex/cogt/content_generation/judgment_generate.py::_make_judgment_job"]+param = "judgment_assignment"+rationale = "Factory from source: builds the job from the judgment_assignment; single operand."++["pipelex/cogt/content_generation/judgment_generate.py::_make_judgment_worker"]+param = "judgment_assignment"+rationale = "Factory from source: builds the worker from the judgment_assignment; single operand."++["pipelex/cogt/content_generation/judgment_generate.py::judgment_gen_answers"]+param = "judgment_assignment"+rationale = "Worker verb-object: processes the judgment_assignment; single operand."+["pipelex/cogt/content_generation/llm_generate.py::llm_gen_object"]param = "object_assignment"rationale = "Processes the object assignment; single operand."@@ -2000,6 +2020,26 @@ rationale = "Positional-Callable tenacity wait strategy: retry_state is the singparam = "send_request"rationale = "Runs the send_request callable under transport retry; single operand."+["pipelex/cogt/judgment/judgment_report.py::JudgmentTokenCostReportField.report_field_for_cost_by_category"]+param = "token_category"+rationale = "Formatter verb-object: renders the field name for the token_category; single operand."++["pipelex/cogt/judgment/judgment_report.py::JudgmentTokenCostReportField.report_field_for_nb_tokens_by_category"]+param = "token_category"+rationale = "Formatter verb-object: renders the field name for the token_category; single operand."++["pipelex/cogt/judgment/judgment_worker_abstract.py::JudgmentWorkerAbstract._judge"]+param = "judgment_job"+rationale = "Worker verb-object: processes the judgment_job; single operand."++["pipelex/cogt/judgment/judgment_worker_abstract.py::JudgmentWorkerAbstract.judge"]+param = "judgment_job"+rationale = "Worker verb-object: processes the judgment_job; single operand."++["pipelex/cogt/judgment/judgment_worker_factory.py::JudgmentWorkerFactory.make_judgment_worker"]+param = "inference_model"+rationale = "Factory from source: builds the worker from the inference_model; single operand."+["pipelex/cogt/llm/llm_job.py::LLMJob.llm_job_before_start"]param = "inference_model"rationale = "Prepares the job against the inference_model; single operand."@@ -2220,6 +2260,10 @@ rationale = "Verb–object: validates/checks the name; single operand."param = "name"rationale = "Verb–object: validates/checks the name; single operand."+["pipelex/cogt/models/model_deck.py::ModelDeck._warn_if_ambiguous_judgment"]+param = "name"+rationale = "Predicate verb-object: warns about the name; single operand, mirroring the sibling per-family warners."+["pipelex/cogt/models/model_deck.py::ModelDeck._warn_if_ambiguous_llm"]param = "name"rationale = "Verb–object: validates/checks the name; single operand."@@ -2244,6 +2288,10 @@ rationale = "Keyed accessor by the extract_choice; single operand."param = "img_gen_choice"rationale = "Keyed accessor by the img_gen_choice; single operand."+["pipelex/cogt/models/model_deck.py::ModelDeck.get_judgment_setting"]+param = "judgment_choice"+rationale = "Keyed accessor by the judgment_choice; single operand."+["pipelex/cogt/models/model_deck.py::ModelDeck.get_llm_setting"]param = "llm_choice"rationale = "Keyed accessor by the llm_choice; single operand."@@ -2280,6 +2328,10 @@ rationale = "Verb–object: validates/checks the extract_choice; single operand.param = "img_gen_choice"rationale = "Verb–object: validates/checks the img_gen_choice; single operand."+["pipelex/cogt/models/model_deck_check.py::check_judgment_choice_with_deck"]+param = "judgment_choice"+rationale = "Verb-object: validates the judgment_choice; single operand."+["pipelex/cogt/models/model_deck_check.py::check_llm_choice_with_deck"]param = "llm_choice"rationale = "Verb–object: validates/checks the llm_choice; single operand."diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txtindex ee8763104..7b8e81c5f 100644--- a/tests/data/errors/error_identity.txt+++ b/tests/data/errors/error_identity.txt@@ -149,6 +149,10 @@ Jinja2TemplateSecurityError | Jinja 2 template security | https://docs.pipelex.cJinja2TemplateSyntaxError | Jinja 2 template syntax | https://docs.pipelex.com/latest/errors/jinja2-template-syntax-error/JobMetadataError | Job metadata | https://docs.pipelex.com/latest/errors/job-metadata-error/JsonTypeError | Json type | https://docs.pipelex.com/latest/errors/json-type-error/+JudgmentAnswerMismatchError | Judgment answer mismatch | https://docs.pipelex.com/latest/errors/judgment-answer-mismatch-error/+JudgmentHandleNotFoundError | Judgment handle not found | https://docs.pipelex.com/latest/errors/judgment-handle-not-found-error/+JudgmentJobFailureError | Judgment job failure | https://docs.pipelex.com/latest/errors/judgment-job-failure-error/+JudgmentModelNotFoundError | Judgment model not found | https://docs.pipelex.com/latest/errors/judgment-model-not-found-error/KitError | Kit | https://docs.pipelex.com/latest/errors/kit-error/KitIndexLoadingError | Kit index loading | https://docs.pipelex.com/latest/errors/kit-index-loading-error/LLMAssignmentError | LLM assignment | https://docs.pipelex.com/latest/errors/llm-assignment-error/diff --git a/tests/integration/pipelex/cogt/test_setup_inference_workers.py b/tests/integration/pipelex/cogt/test_setup_inference_workers.pyindex 5d98543f8..3d83353c8 100644--- a/tests/integration/pipelex/cogt/test_setup_inference_workers.py+++ b/tests/integration/pipelex/cogt/test_setup_inference_workers.py@@ -15,7 +15,7 @@ class TestSetupInferenceWorkers:_ = inference_manager.get_extract_worker(extract_handle=model_handle)case ModelType.IMG_GEN:_ = inference_manager.get_img_gen_worker(img_gen_handle=model_handle)- case ModelType.SEARCH:+ case ModelType.SEARCH | ModelType.JUDGMENT:passcase ModelType.DOC_GEN:# Every document engine the deck serves has a worker: the kit declares only the built-in one, and adiff --git a/tests/integration/pipelex/conftest.py b/tests/integration/pipelex/conftest.pyindex fc2be53ce..80bd41230 100644--- a/tests/integration/pipelex/conftest.py+++ b/tests/integration/pipelex/conftest.py@@ -7,6 +7,7 @@ This conftest.py imports fixtures from organized modules, making them availablefrom .fixtures.combo_fixtures import (extract_combo,img_gen_combo,+ judgment_combo,llm_combo,search_combo,)@@ -35,6 +36,7 @@ __all__ = ["img_gen_combo","extract_combo","search_combo",+ "judgment_combo",# Model handle fixtures"model_handle_for_openai","model_handle_for_anthropic",diff --git a/tests/integration/pipelex/fixtures/combo_fixtures.py b/tests/integration/pipelex/fixtures/combo_fixtures.pyindex 8816171f5..50590ea1c 100644--- a/tests/integration/pipelex/fixtures/combo_fixtures.py+++ b/tests/integration/pipelex/fixtures/combo_fixtures.py@@ -33,6 +33,7 @@ from tests.integration.pipelex.fixtures.model_selection import (ModelCombo,get_extract_combos,get_img_gen_combos,+ get_judgment_combos,get_llm_combos,get_search_combos,)@@ -168,3 +169,18 @@ def search_combo(request: pytest.FixtureRequest):yield combo_cleanup_routing(monkeypatch, temp_dir)++[email protected](scope="module", params=get_judgment_combos())+def judgment_combo(request: pytest.FixtureRequest):+ """Provides a valid ModelCombo(handle, backend) for judgment with routing configured.++ Yields:+ ModelCombo(handle, backend) for judgment.+ """+ combo: ModelCombo = request.param+ monkeypatch, temp_dir = _setup_routing_for_backend(combo.backend)++ yield combo++ _cleanup_routing(monkeypatch, temp_dir)diff --git a/tests/integration/pipelex/fixtures/model_selection.py b/tests/integration/pipelex/fixtures/model_selection.pyindex ea973274c..2c1548e6d 100644--- a/tests/integration/pipelex/fixtures/model_selection.py+++ b/tests/integration/pipelex/fixtures/model_selection.py@@ -6,8 +6,10 @@ from the generated fixture file.from __future__ import annotations+import importlibfrom functools import cachefrom pathlib import Path+from typing import castimport pytest@@ -22,6 +24,13 @@ MISSING_FIXTURES_ERROR_MSG = ("Please run 'make regenerate-test-models' or 'make install' to generate it.")+GENERATED_FIXTURES_MODULE = "tests.integration.pipelex.fixtures._generated_model_sets"++STALE_FIXTURES_ERROR_MSG = (+ "Generated test fixtures file at {path} has no '{name}': it predates the model family that set belongs to.\n"+ "Please run 'make regenerate-test-models' or 'make install' to regenerate it."+)+def _or_skip(combos: list[ModelCombo], label: str) -> list[ModelCombo]:"""Return combos as-is, or a single skip-marked param if the list is empty."""@@ -40,6 +49,24 @@ def _ensure_generated_fixtures_exist() -> None:raise FileNotFoundError(MISSING_FIXTURES_ERROR_MSG)+def _load_generated_combos(*, name: str) -> list[ModelCombo]:+ """Read one combo set from the generated fixtures file, naming the cure when the file is stale.++ The file is gitignored and only the targets the message names regenerate it, so a checkout that+ gains a model family keeps a file without that family's set. The conftest reads these sets at+ import time, where a bare ImportError interrupts the whole session with nothing pointing at the fix.++ Raises:+ FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file has no set of that name.+ """+ _ensure_generated_fixtures_exist()+ generated_module = importlib.import_module(GENERATED_FIXTURES_MODULE)+ if not hasattr(generated_module, name):+ raise ImportError(STALE_FIXTURES_ERROR_MSG.format(path=GENERATED_FIXTURES_PATH, name=name))+ return list(cast("list[ModelCombo]", getattr(generated_module, name)))++@cachedef get_llm_combos() -> list[ModelCombo]:"""Get the list of valid (llm_model, backend) combinations.@@ -49,13 +76,9 @@ def get_llm_combos() -> list[ModelCombo]:Raises:FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file predates this set."""- _ensure_generated_fixtures_exist()- from tests.integration.pipelex.fixtures._generated_model_sets import ( # ruff: ignore[import-outside-top-level]- LLM_COMBOS,- )-- return _or_skip(list(LLM_COMBOS), "LLM")+ return _or_skip(_load_generated_combos(name="LLM_COMBOS"), "LLM")@cache@@ -67,13 +90,9 @@ def get_img_gen_combos() -> list[ModelCombo]:Raises:FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file predates this set."""- _ensure_generated_fixtures_exist()- from tests.integration.pipelex.fixtures._generated_model_sets import ( # ruff: ignore[import-outside-top-level]- IMG_GEN_COMBOS,- )-- return _or_skip(list(IMG_GEN_COMBOS), "image generation")+ return _or_skip(_load_generated_combos(name="IMG_GEN_COMBOS"), "image generation")@cache@@ -85,13 +104,23 @@ def get_extract_combos() -> list[ModelCombo]:Raises:FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file predates this set."""- _ensure_generated_fixtures_exist()- from tests.integration.pipelex.fixtures._generated_model_sets import ( # ruff: ignore[import-outside-top-level]- EXTRACT_COMBOS,- )+ return _or_skip(_load_generated_combos(name="EXTRACT_COMBOS"), "extraction")- return _or_skip(list(EXTRACT_COMBOS), "extraction")++@cache+def get_judgment_combos() -> list[ModelCombo]:+ """Get the list of valid (judgment_model, backend) combinations.++ Returns:+ List of ModelCombo(handle, backend).++ Raises:+ FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file predates this set.+ """+ return _or_skip(_load_generated_combos(name="JUDGMENT_COMBOS"), "judgment")@cache@@ -103,10 +132,6 @@ def get_search_combos() -> list[ModelCombo]:Raises:FileNotFoundError: If the generated fixtures file does not exist.+ ImportError: If the generated fixtures file predates this set."""- _ensure_generated_fixtures_exist()- from tests.integration.pipelex.fixtures._generated_model_sets import ( # ruff: ignore[import-outside-top-level]- SEARCH_COMBOS,- )-- return _or_skip(list(SEARCH_COMBOS), "search")+ return _or_skip(_load_generated_combos(name="SEARCH_COMBOS"), "search")diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.pyindex 3f6aa5eb0..783cb5fef 100644--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py@@ -127,7 +127,7 @@ def make_doc_gen_backends(*, registrar: PluginRegistrar) -> dict[str, MakeWorkermatch family:case InferenceFamily.DOC_GEN:backends[sdk] = make_worker- case InferenceFamily.LLM | InferenceFamily.IMG_GEN | InferenceFamily.EXTRACT | InferenceFamily.SEARCH:+ case InferenceFamily.LLM | InferenceFamily.IMG_GEN | InferenceFamily.EXTRACT | InferenceFamily.SEARCH | InferenceFamily.JUDGMENT:passreturn backendsdiff --git a/tests/unit/pipelex/cli/test_agent_models_cmd.py b/tests/unit/pipelex/cli/test_agent_models_cmd.pyindex 2b6ef5a98..91f45c2a8 100644--- a/tests/unit/pipelex/cli/test_agent_models_cmd.py+++ b/tests/unit/pipelex/cli/test_agent_models_cmd.py@@ -72,16 +72,19 @@ def _make_fake_model_deck() -> Any:extract_presets = TestData.EXTRACT_PRESETSimg_gen_presets = TestData.IMG_GEN_PRESETSsearch_presets: ClassVar[dict[str, Any]] = {}+ judgment_presets: ClassVar[dict[str, Any]] = {}llm_aliases = TestData.LLM_ALIASESextract_aliases = TestData.EXTRACT_ALIASESimg_gen_aliases = TestData.IMG_GEN_ALIASESsearch_aliases: ClassVar[dict[str, str]] = {}+ judgment_aliases: ClassVar[dict[str, str]] = {}llm_waterfalls = TestData.LLM_WATERFALLSextract_waterfalls = TestData.EXTRACT_WATERFALLSimg_gen_waterfalls = TestData.IMG_GEN_WATERFALLSsearch_waterfalls: ClassVar[dict[str, list[str]]] = {}+ judgment_waterfalls: ClassVar[dict[str, list[str]]] = {}def get_optional_inference_model(self, model_handle: str, model_type: ModelType) -> _FakeInferenceModelSpec | None:entry = TestData.INFERENCE_MAP.get(model_handle)diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.pyindex c71a3012b..9e1a186f0 100644--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py@@ -20,6 +20,7 @@ from pipelex.cogt.content_generation import assignment_modelsfrom pipelex.cogt.content_generation.assignment_models import (ExtractAssignment,ImgGenAssignment,+ JudgmentAssignment,LLMAssignment,ObjectAssignment,RenderDocumentAssignment,@@ -62,6 +63,7 @@ URL_FIELD_INVENTORY: dict[type[BaseModel], set[str]] = {RenderDocumentAssignment: {"composition.layout.blocks.url", "composition.layout.blocks.blocks.url"},SearchAssignment: set(),SearchObjectAssignment: set(),+ JudgmentAssignment: set(),}diff --git a/tests/unit/pipelex/cogt/content_generation/test_judgment_generate_dry_branch.py b/tests/unit/pipelex/cogt/content_generation/test_judgment_generate_dry_branch.pynew file mode 100644index 000000000..3f499a5de--- /dev/null+++ b/tests/unit/pipelex/cogt/content_generation/test_judgment_generate_dry_branch.py@@ -0,0 +1,92 @@+"""The ``run_mode == DRY`` branch in the judgment leaf.++Contract: under DRY the leaf mints a deterministic verdict per question without resolving any worker+or touching the model deck, and — the part that matters most for this family — it reports no+uncertainty at all. LIVE keeps the real path.+"""++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams+from pipelex.cogt.content_generation.judgment_generate import judgment_gen_answers+from pipelex.cogt.judgment.judgment_models import (+ ChoiceAnswer,+ ChoiceQuestion,+ RatingAnswer,+ RatingQuestion,+ YesNoAnswer,+ YesNoQuestion,+)+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+++class TestJudgmentGenerateDryBranch:+ def _assignment(self, *, run_mode: PipeRunMode) -> JudgmentAssignment:+ return JudgmentAssignment(+ job_metadata=JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_judgment_dry")+ ),+ cogt_run_params=CogtRunParams(run_mode=run_mode),+ state={"message": "the roof is on fire"},+ questions={+ "is_urgent": YesNoQuestion(instructions="Does this need an answer today?"),+ "topic": ChoiceQuestion(instructions="What is this about?", options={"fire": None, "flood": "water damage"}),+ "severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"]),+ },+ judgment_setting=JudgmentSetting(model="mock-judgment-handle"),+ )++ @pytest.mark.asyncio+ async def test_dry_mints_answers_without_worker(self, mocker: MockerFixture) -> None:+ """DRY: one deterministic answer per question; the model deck is never resolved."""+ deck_spy = mocker.patch("pipelex.cogt.content_generation.judgment_generate.get_model_deck")++ answers = await judgment_gen_answers(self._assignment(run_mode=PipeRunMode.DRY))++ deck_spy.assert_not_called()+ assert set(answers) == {"is_urgent", "topic", "severity"}+ yes_no = answers["is_urgent"]+ choice = answers["topic"]+ rating = answers["severity"]+ assert isinstance(yes_no, YesNoAnswer)+ assert isinstance(choice, ChoiceAnswer)+ assert isinstance(rating, RatingAnswer)+ assert yes_no.yes_no is True+ assert choice.choice == "fire"+ assert rating.level == 0++ @pytest.mark.asyncio+ async def test_dry_reports_no_uncertainty(self) -> None:+ """Nothing measured anything, so every uncertainty member is absent rather than invented."""+ answers = await judgment_gen_answers(self._assignment(run_mode=PipeRunMode.DRY))++ yes_no = answers["is_urgent"]+ choice = answers["topic"]+ rating = answers["severity"]+ assert isinstance(yes_no, YesNoAnswer)+ assert isinstance(choice, ChoiceAnswer)+ assert isinstance(rating, RatingAnswer)+ assert yes_no.probability is None+ assert choice.confidence is None+ assert choice.probabilities is None+ assert rating.position is None+ assert rating.confidence is None+ assert rating.probabilities is None++ @pytest.mark.asyncio+ async def test_live_resolves_worker(self, mocker: MockerFixture) -> None:+ """LIVE keeps the real path: the worker is resolved from the deck and runs the judgment."""+ sentinel = mocker.MagicMock()+ worker = mocker.MagicMock()+ worker.judge = mocker.AsyncMock(return_value=sentinel)+ mocker.patch("pipelex.cogt.content_generation.judgment_generate._make_judgment_worker", return_value=worker)+ mocker.patch("pipelex.cogt.content_generation.judgment_generate._make_judgment_job", return_value=mocker.MagicMock())++ result = await judgment_gen_answers(self._assignment(run_mode=PipeRunMode.LIVE))++ worker.judge.assert_awaited_once()+ assert result is sentineldiff --git a/tests/unit/pipelex/cogt/judgment/test_data.py b/tests/unit/pipelex/cogt/judgment/test_data.pynew file mode 100644index 000000000..6c7efec4b--- /dev/null+++ b/tests/unit/pipelex/cogt/judgment/test_data.py@@ -0,0 +1,26 @@+from typing import ClassVar++from pipelex.cogt.judgment.judgment_models import (+ ChoiceAnswer,+ ChoiceQuestion,+ JudgmentAnswer,+ JudgmentQuestion,+ RatingAnswer,+ RatingQuestion,+ YesNoAnswer,+ YesNoQuestion,+)+++class JudgmentTestCases:+ THREE_QUESTIONS: ClassVar[dict[str, JudgmentQuestion]] = {+ "is_urgent": YesNoQuestion(instructions="Does this message need an answer today?"),+ "topic": ChoiceQuestion(instructions="What is this message about?", options={"fire": None, "flood": "water damage"}),+ "severity": RatingQuestion(instructions="How severe is the situation?", levels=["mild", "bad", "critical"]),+ }++ THREE_ANSWERS: ClassVar[dict[str, JudgmentAnswer]] = {+ "is_urgent": YesNoAnswer(probability=0.93),+ "topic": ChoiceAnswer(choice="fire", confidence=0.88, probabilities={"fire": 0.88, "flood": 0.12}),+ "severity": RatingAnswer(level=2, position=1.82, confidence=0.7, probabilities={0: 0.02, 1: 0.14, 2: 0.84}),+ }diff --git a/tests/unit/pipelex/cogt/judgment/test_judgment_models.py b/tests/unit/pipelex/cogt/judgment/test_judgment_models.pynew file mode 100644index 000000000..bc62e98d0--- /dev/null+++ b/tests/unit/pipelex/cogt/judgment/test_judgment_models.py@@ -0,0 +1,124 @@+"""What the question and answer models refuse.++The one rule that is not a type is ``YesNoAnswer``'s: an answer carrying neither a probability nor a+boolean is not an answer at all. Everything else here pins the optionality that lets a backend which+measured nothing stay honest.+"""++import pytest+from pydantic import TypeAdapter, ValidationError++from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment+from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams+from pipelex.cogt.judgment.judgment_models import (+ ChoiceAnswer,+ ChoiceQuestion,+ JudgmentAnswer,+ JudgmentKind,+ RatingAnswer,+ RatingQuestion,+ YesNoAnswer,+ YesNoQuestion,+)+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting+from pipelex.system.job_metadata import JobMetadata, RunMetadata+from pipelex.system.pipe_run_mode import PipeRunMode+++class TestJudgmentModels:+ def test_a_yes_no_answer_needs_at_least_one_verdict(self) -> None:+ with pytest.raises(ValidationError):+ YesNoAnswer()++ @pytest.mark.parametrize(+ "answer",+ [+ YesNoAnswer(probability=0.9),+ YesNoAnswer(yes_no=True),+ YesNoAnswer(probability=0.9, yes_no=True),+ ],+ ids=["probability_only", "boolean_only", "both"],+ )+ def test_a_yes_no_answer_accepts_either_form(self, answer: YesNoAnswer) -> None:+ assert answer.kind == JudgmentKind.YES_NO++ def test_a_probability_outside_zero_to_one_is_refused(self) -> None:+ with pytest.raises(ValidationError):+ YesNoAnswer(probability=1.5)++ def test_a_choice_question_needs_an_option(self) -> None:+ with pytest.raises(ValidationError):+ ChoiceQuestion(instructions="Which one?", options={})++ def test_a_rating_question_needs_a_level(self) -> None:+ with pytest.raises(ValidationError):+ RatingQuestion(instructions="How much?", levels=[])++ def test_an_option_may_be_declared_without_a_description(self) -> None:+ question = ChoiceQuestion(instructions="Which one?", options={"fire": None, "flood": "water damage"})+ assert question.options["fire"] is None++ def test_a_choice_answer_may_carry_only_its_verdict(self) -> None:+ answer = ChoiceAnswer(choice="fire")+ assert answer.confidence is None+ assert answer.probabilities is None++ def test_a_rating_answer_keys_its_distribution_by_level_index(self) -> None:+ answer = RatingAnswer(level=2, probabilities={0: 0.02, 1: 0.14, 2: 0.84})+ assert answer.probabilities is not None+ assert sorted(answer.probabilities) == [0, 1, 2]++ def test_a_rating_answer_coerces_the_wire_string_keys_it_is_handed(self) -> None:+ """The vendor's wire form keys a distribution by strings; the model holds integers."""+ answer = RatingAnswer.model_validate({"level": 2, "probabilities": {"0": 0.02, "2": 0.98}})+ assert answer.probabilities == {0: 0.02, 2: 0.98}++ def test_every_question_carries_its_kind(self) -> None:+ assert YesNoQuestion(instructions="?").kind == JudgmentKind.YES_NO+ assert ChoiceQuestion(instructions="?", options={"a": None}).kind == JudgmentKind.CHOICE+ assert RatingQuestion(instructions="?", levels=["low"]).kind == JudgmentKind.RATING++ def test_an_assignment_survives_the_round_trip_a_distributed_boundary_makes_it_take(self) -> None:+ """Questions and answers are the whole wire payload, so the discriminator must work both ways.++ Nothing else guards this: under a distributed orchestrator the assignment is dumped on the+ submitter and validated on a worker, and a discriminated union that dumps but does not+ re-validate would fail there and nowhere else.+ """+ assignment = JudgmentAssignment(+ job_metadata=JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_judgment_wire")+ ),+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),+ state={"message": "the roof is on fire", "read_count": 3, "starred": True, "labels": ["home", "urgent"]},+ questions={+ "is_urgent": YesNoQuestion(instructions="Does this need an answer today?"),+ "topic": ChoiceQuestion(instructions="What is this about?", options={"fire": None, "flood": "water damage"}),+ "severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"]),+ },+ judgment_setting=JudgmentSetting(model="some-judgment-handle"),+ )++ restored = JudgmentAssignment.model_validate(assignment.model_dump(mode="json"))++ assert restored.state == assignment.state+ assert isinstance(restored.questions["is_urgent"], YesNoQuestion)+ assert isinstance(restored.questions["topic"], ChoiceQuestion)+ assert isinstance(restored.questions["severity"], RatingQuestion)+ assert restored.judgment_handle == "some-judgment-handle"++ def test_answers_survive_the_same_round_trip(self) -> None:+ answers: dict[str, JudgmentAnswer] = {+ "is_urgent": YesNoAnswer(probability=0.93),+ "topic": ChoiceAnswer(choice="fire", confidence=0.88, probabilities={"fire": 0.88, "flood": 0.12}),+ "severity": RatingAnswer(level=2, position=1.82, confidence=0.7, probabilities={0: 0.02, 1: 0.14, 2: 0.84}),+ }+ adapter: TypeAdapter[dict[str, JudgmentAnswer]] = TypeAdapter(dict[str, JudgmentAnswer])++ restored = adapter.validate_python(adapter.dump_python(answers, mode="json"))++ assert isinstance(restored["is_urgent"], YesNoAnswer)+ assert isinstance(restored["topic"], ChoiceAnswer)+ severity = restored["severity"]+ assert isinstance(severity, RatingAnswer)+ assert severity.probabilities == {0: 0.02, 1: 0.14, 2: 0.84}diff --git a/tests/unit/pipelex/cogt/judgment/test_judgment_worker_contract.py b/tests/unit/pipelex/cogt/judgment/test_judgment_worker_contract.pynew file mode 100644index 000000000..bfd39b3f2--- /dev/null+++ b/tests/unit/pipelex/cogt/judgment/test_judgment_worker_contract.py@@ -0,0 +1,246 @@+"""The judgment worker's template method: what it stamps, what it checks, and when it reports.++The guards here are the reason a backend cannot quietly answer the wrong thing. Each one is+mutation-tested in the sense the repo means: break the guard in ``judgment_worker_abstract`` and+exactly one of these goes red.+"""++import pytest+from typing_extensions import override++from pipelex.cogt.exceptions import CogtError, JudgmentAnswerMismatchError+from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract+from pipelex.cogt.judgment.judgment_job import JudgmentJob+from pipelex.cogt.judgment.judgment_job_factory import JudgmentJobFactory+from pipelex.cogt.judgment.judgment_models import (+ ChoiceAnswer,+ ChoiceQuestion,+ JudgmentAnswer,+ JudgmentQuestion,+ RatingAnswer,+ RatingQuestion,+ YesNoAnswer,+ YesNoQuestion,+)+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting+from pipelex.cogt.judgment.judgment_worker_abstract import JudgmentWorkerAbstract+from pipelex.cogt.llm.thinking_mode import ThinkingMode+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.usage.cost_category import CostCategory+from pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocol+from pipelex.system.job_metadata import JobCategory, JobMetadata, RunMetadata, UnitJobId+from tests.unit.pipelex.cogt.judgment.test_data import JudgmentTestCases+++class _FakeJudgmentWorker(JudgmentWorkerAbstract):+ """A worker that answers whatever it was told to, including the wrong thing."""++ def __init__(+ self,+ inference_model: InferenceModelSpec,+ *,+ answers: dict[str, JudgmentAnswer] | None = None,+ raises: CogtError | None = None,+ reporting_delegate: ReportingProtocol | None = None,+ ) -> None:+ JudgmentWorkerAbstract.__init__(self, inference_model, reporting_delegate=reporting_delegate)+ self._answers = answers or {}+ self._raises = raises++ @override+ async def _judge(self, judgment_job: JudgmentJob) -> dict[str, JudgmentAnswer]:+ if self._raises is not None:+ raise self._raises+ return self._answers+++class _RecordingDelegate(ReportingNoOp):+ """A reporting delegate that keeps what it was handed, so the `finally` can be asserted on."""++ def __init__(self) -> None:+ self.reported: list[InferenceJobAbstract] = []++ @override+ def report_inference_job(self, inference_job: InferenceJobAbstract) -> None:+ self.reported.append(inference_job)+++class TestJudgmentWorkerContract:+ def _job(self, questions: dict[str, JudgmentQuestion]) -> JudgmentJob:+ return JudgmentJobFactory.make_judgment_job(+ state={"message": "the roof is on fire"},+ questions=questions,+ judgment_setting=JudgmentSetting(model="fake-judgment-handle"),+ job_metadata=JobMetadata(+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="u", pipeline_run_id="run_judgment")+ ),+ )++ def _model(self) -> InferenceModelSpec:+ return InferenceModelSpec(+ backend_name="fake_backend",+ name="fake-judgment-handle",+ sdk="fake_judgment_sdk",+ model_type=ModelType.JUDGMENT,+ model_id="fake-judgment-1.0",+ inputs=["text"],+ outputs=["judgments"],+ costs={CostCategory.INPUT: 0.042, CostCategory.OUTPUT: 0},+ thinking_mode=ThinkingMode.NONE,+ max_tokens=None,+ max_prompt_images=None,+ )++ async def test_it_answers_every_question_and_stamps_the_job(self) -> None:+ job = self._job(JudgmentTestCases.THREE_QUESTIONS)+ worker = _FakeJudgmentWorker(self._model(), answers=JudgmentTestCases.THREE_ANSWERS)++ answers = await worker.judge(job)++ assert set(answers) == set(JudgmentTestCases.THREE_QUESTIONS)+ assert job.job_metadata.job_category == JobCategory.JUDGMENT_JOB+ assert job.job_metadata.unit_job_id == UnitJobId.JUDGMENT_ANSWER+ assert job.job_metadata.started_at is not None+ assert job.job_metadata.completed_at is not None++ async def test_it_refuses_an_answer_to_a_question_nobody_asked(self) -> None:+ job = self._job({"is_urgent": YesNoQuestion(instructions="Is it urgent?")})+ worker = _FakeJudgmentWorker(+ self._model(),+ answers={"is_urgent": YesNoAnswer(probability=0.9), "is_spam": YesNoAnswer(probability=0.1)},+ )++ with pytest.raises(JudgmentAnswerMismatchError) as exc_info:+ await worker.judge(job)++ assert "is_spam" in str(exc_info.value)++ async def test_it_refuses_a_dropped_answer(self) -> None:+ job = self._job(JudgmentTestCases.THREE_QUESTIONS)+ worker = _FakeJudgmentWorker(self._model(), answers={"is_urgent": YesNoAnswer(probability=0.9)})++ with pytest.raises(JudgmentAnswerMismatchError) as exc_info:+ await worker.judge(job)++ assert "severity" in str(exc_info.value)++ async def test_it_refuses_an_answer_of_the_wrong_kind(self) -> None:+ job = self._job({"severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"])})+ worker = _FakeJudgmentWorker(self._model(), answers={"severity": ChoiceAnswer(choice="mild")})++ with pytest.raises(JudgmentAnswerMismatchError) as exc_info:+ await worker.judge(job)++ assert "severity" in str(exc_info.value)+ assert "rating" in str(exc_info.value)++ @pytest.mark.parametrize(+ ("answer", "named"),+ [+ pytest.param(ChoiceAnswer(choice="storm"), "storm", id="choice"),+ pytest.param(ChoiceAnswer(choice="fire", probabilities={"fire": 0.7, "storm": 0.3}), "storm", id="choice_distribution"),+ ],+ )+ async def test_it_refuses_an_option_the_question_does_not_offer(self, answer: ChoiceAnswer, named: str) -> None:+ job = self._job({"topic": ChoiceQuestion(instructions="Which topic?", options={"fire": None, "flood": "water damage"})})+ worker = _FakeJudgmentWorker(self._model(), answers={"topic": answer})++ with pytest.raises(JudgmentAnswerMismatchError) as exc_info:+ await worker.judge(job)++ assert "topic" in str(exc_info.value)+ assert named in str(exc_info.value)++ @pytest.mark.parametrize(+ ("answer", "named"),+ [+ pytest.param(RatingAnswer(level=3), "3", id="level_one_past_the_top"),+ pytest.param(RatingAnswer(level=1, probabilities={0: 0.2, 1: 0.5, 3: 0.3}), "3", id="level_distribution"),+ pytest.param(RatingAnswer(level=1, probabilities={-1: 0.2, 1: 0.8}), "-1", id="negative_level_distribution"),+ ],+ )+ async def test_it_refuses_a_level_beyond_the_scale(self, answer: RatingAnswer, named: str) -> None:+ job = self._job({"severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"])})+ worker = _FakeJudgmentWorker(self._model(), answers={"severity": answer})++ with pytest.raises(JudgmentAnswerMismatchError) as exc_info:+ await worker.judge(job)++ assert "severity" in str(exc_info.value)+ assert f"[{named}]" in str(exc_info.value)++ async def test_it_accepts_the_top_level_and_a_full_distribution(self) -> None:+ job = self._job(+ {+ "severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"]),+ "topic": ChoiceQuestion(instructions="Which topic?", options={"fire": None, "flood": "water damage"}),+ }+ )+ worker = _FakeJudgmentWorker(+ self._model(),+ answers={+ "severity": RatingAnswer(level=2, probabilities={0: 0.1, 1: 0.2, 2: 0.7}),+ "topic": ChoiceAnswer(choice="flood", probabilities={"fire": 0.4, "flood": 0.6}),+ },+ )++ answers = await worker.judge(job)++ assert set(answers) == {"severity", "topic"}++ async def test_it_reports_on_the_way_out_even_when_the_answers_are_rejected(self) -> None:+ """A rejected answer set was still paid for: the provider answered before the guard ran."""+ job = self._job({"is_urgent": YesNoQuestion(instructions="Is it urgent?")})+ delegate = _RecordingDelegate()+ worker = _FakeJudgmentWorker(+ self._model(),+ answers={"something_else": YesNoAnswer(probability=0.9)},+ reporting_delegate=delegate,+ )++ with pytest.raises(JudgmentAnswerMismatchError):+ await worker.judge(job)++ assert delegate.reported == [job]+ assert job.job_metadata.completed_at is not None++ async def test_it_reports_on_the_happy_path_too(self) -> None:+ job = self._job({"is_urgent": YesNoQuestion(instructions="Is it urgent?")})+ delegate = _RecordingDelegate()+ worker = _FakeJudgmentWorker(self._model(), answers={"is_urgent": YesNoAnswer(probability=0.9)}, reporting_delegate=delegate)++ await worker.judge(job)++ assert delegate.reported == [job]++ async def test_it_names_the_model_and_the_backend_on_a_provider_failure(self) -> None:+ job = self._job({"is_urgent": YesNoQuestion(instructions="Is it urgent?")})+ worker = _FakeJudgmentWorker(self._model(), raises=CogtError("the provider said no"))++ with pytest.raises(CogtError) as exc_info:+ await worker.judge(job)++ assert exc_info.value.model_handle == "fake-judgment-handle"+ assert exc_info.value.backend_name == "fake_backend"++ async def test_a_choice_answer_carries_the_option_it_names(self) -> None:+ job = self._job({"topic": ChoiceQuestion(instructions="Which topic?", options={"fire": None, "flood": "water damage"})})+ worker = _FakeJudgmentWorker(self._model(), answers={"topic": ChoiceAnswer(choice="fire", confidence=0.8)})++ answers = await worker.judge(job)++ answer = answers["topic"]+ assert isinstance(answer, ChoiceAnswer)+ assert answer.choice == "fire"++ async def test_a_rating_answer_carries_its_level(self) -> None:+ job = self._job({"severity": RatingQuestion(instructions="How severe?", levels=["mild", "bad", "critical"])})+ worker = _FakeJudgmentWorker(self._model(), answers={"severity": RatingAnswer(level=2, position=1.8)})++ answers = await worker.judge(job)++ answer = answers["severity"]+ assert isinstance(answer, RatingAnswer)+ assert answer.level == 2diff --git a/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py b/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.pyindex 6620e81a4..4f337102e 100644--- a/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py+++ b/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py@@ -15,6 +15,7 @@ from pipelex.cogt.config_cogt import ModelDeckConfigfrom pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSettingfrom pipelex.cogt.img_gen.img_gen_job_components import Qualityfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting+from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice, JudgmentSettingfrom pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSetting, LLMSettingChoicesDefaultsfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck import ModelDeck@@ -40,6 +41,8 @@ def _make_deck(img_gen_choice_default: ImgGenModelChoice = "img-gen-default-handle",search_presets: dict[str, SearchSetting] | None = None,search_choice_default: SearchModelChoice = "search-default-handle",+ judgment_presets: dict[str, JudgmentSetting] | None = None,+ judgment_choice_default: JudgmentModelChoice | None = None,llm_aliases: dict[str, str] | None = None,llm_waterfalls: dict[str, list[str]] | None = None,) -> ModelDeck:@@ -72,6 +75,11 @@ def _make_deck(search_waterfalls={},search_presets=search_presets or {},search_choice_default=search_choice_default,+ # Judgment — its choice default is optional, so the deck's own absent default is the base case here+ judgment_aliases={},+ judgment_waterfalls={},+ judgment_presets=judgment_presets or {},+ judgment_choice_default=judgment_choice_default,model_deck_config=ModelDeckConfig(is_model_fallback_enabled=True, missing_presets_reaction=ProblemReaction.NONE),)@@ -87,11 +95,12 @@ class TestCollectDeckReferencedHandles:(ExtractSetting(model="extract-model"), "extract-model"),(ImgGenSetting(model="img-gen-model"), "img-gen-model"),(SearchSetting(model="search-model"), "search-model"),+ (JudgmentSetting(model="judgment-model"), "judgment-model"),],)def test_extract_choice_handle(self,- choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | ModelReference | str | None,+ choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | JudgmentSetting | ModelReference | str | None,expected: str | None,) -> None:"""_extract_choice_handle normalises every choice shape: None, raw str, ModelReference, typed setting."""@@ -112,6 +121,8 @@ class TestCollectDeckReferencedHandles:img_gen_choice_default="img-gen-default-handle",search_presets={"web": SearchSetting(model="search-preset-web")},search_choice_default="search-default-handle",+ judgment_presets={"verdict": JudgmentSetting(model="judgment-preset-verdict")},+ judgment_choice_default="judgment-default-handle",)collected = _collect_deck_referenced_handles(deck)@@ -127,6 +138,8 @@ class TestCollectDeckReferencedHandles:("img-gen-default-handle", ModelType.IMG_GEN),("search-preset-web", ModelType.SEARCH),("search-default-handle", ModelType.SEARCH),+ ("judgment-preset-verdict", ModelType.JUDGMENT),+ ("judgment-default-handle", ModelType.JUDGMENT),}def test_aliases_and_waterfalls_are_not_enumerated_directly(self) -> None:@@ -142,3 +155,16 @@ class TestCollectDeckReferencedHandles:assert "alias-target-handle" not in handlesassert "my-waterfall" not in handlesassert "waterfall-target-handle" not in handles++ def test_an_absent_judgment_default_contributes_no_reference(self) -> None:+ """The judgment family alone may have no choice default, and then it names no handle to check.++ This is what keeps a deck with no judgment backend installed bootable: the gateway's+ membership check walks exactly these references, and a default naming a model no enabled+ backend serves would fail every boot, judgment or not.+ """+ deck = _make_deck(judgment_choice_default=None)++ collected = _collect_deck_referenced_handles(deck)++ assert not [pair for pair in collected if pair[1] == ModelType.JUDGMENT]diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_check.py b/tests/unit/pipelex/cogt/models/test_model_deck_check.pyindex f694d84c2..420fb2ddf 100644--- a/tests/unit/pipelex/cogt/models/test_model_deck_check.py+++ b/tests/unit/pipelex/cogt/models/test_model_deck_check.py@@ -8,6 +8,7 @@ from pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.cogt.extract.extract_setting import ExtractSettingfrom pipelex.cogt.img_gen.img_gen_job_components import Qualityfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenSetting+from pipelex.cogt.judgment.judgment_setting import JudgmentSettingfrom pipelex.cogt.llm.llm_setting import LLMSetting, LLMSettingChoicesDefaultsfrom pipelex.cogt.llm.thinking_mode import ThinkingModefrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec@@ -16,6 +17,7 @@ from pipelex.cogt.models.model_deck import ModelDeckfrom pipelex.cogt.models.model_deck_check import (check_extract_choice_with_deck,check_img_gen_choice_with_deck,+ check_judgment_choice_with_deck,check_llm_choice_with_deck,check_search_choice_with_deck,)@@ -51,6 +53,7 @@ class TestModelDeckCheck:"extract-engine": self._create_model_spec("extract-engine", ModelType.TEXT_EXTRACTOR),"img-painter": self._create_model_spec("img-painter", ModelType.IMG_GEN),"web-searcher": self._create_model_spec("web-searcher", ModelType.SEARCH),+ "verdict-giver": self._create_model_spec("verdict-giver", ModelType.JUDGMENT),},# LLM-specificllm_default_temperature=0.7,@@ -78,6 +81,10 @@ class TestModelDeckCheck:search_waterfalls={"fallback-search": ["web-searcher"]},search_presets={"cheap-search": SearchSetting(model="web-searcher")},search_choice_default="@default-search",+ # Judgment-specific+ judgment_aliases={"best-judgment": "verdict-giver"},+ judgment_waterfalls={"fallback-judgment": ["verdict-giver"]},+ judgment_presets={"cheap-judgment": JudgmentSetting(model="verdict-giver")},model_deck_config=ModelDeckConfig(is_model_fallback_enabled=False, missing_presets_reaction=ProblemReaction.NONE),)@@ -88,6 +95,7 @@ class TestModelDeckCheck:pytest.param(check_extract_choice_with_deck, ExtractSetting(model="extract-engine"), id="extract"),pytest.param(check_search_choice_with_deck, SearchSetting(model="web-searcher"), id="search"),pytest.param(check_img_gen_choice_with_deck, ImgGenSetting(model="img-painter"), id="img_gen"),+ pytest.param(check_judgment_choice_with_deck, JudgmentSetting(model="verdict-giver"), id="judgment"),],)def test_setting_instance_short_circuits_without_deck_lookup(@@ -122,6 +130,10 @@ class TestModelDeckCheck:pytest.param(check_img_gen_choice_with_deck, "@best-img", id="img_gen-alias"),pytest.param(check_img_gen_choice_with_deck, "~fallback-img", id="img_gen-waterfall"),pytest.param(check_img_gen_choice_with_deck, "img-painter", id="img_gen-handle"),+ pytest.param(check_judgment_choice_with_deck, "$cheap-judgment", id="judgment-preset"),+ pytest.param(check_judgment_choice_with_deck, "@best-judgment", id="judgment-alias"),+ pytest.param(check_judgment_choice_with_deck, "~fallback-judgment", id="judgment-waterfall"),+ pytest.param(check_judgment_choice_with_deck, "verdict-giver", id="judgment-handle"),],)def test_found_choice_returns_none(@@ -231,6 +243,38 @@ class TestModelDeckCheck:"inference_models",id="img_gen-handle",),+ pytest.param(+ check_judgment_choice_with_deck,+ "$missing-preset",+ ModelReferenceKind.PRESET,+ ModelType.JUDGMENT,+ "judgment_presets",+ id="judgment-preset",+ ),+ pytest.param(+ check_judgment_choice_with_deck,+ "@missing-alias",+ ModelReferenceKind.ALIAS,+ ModelType.JUDGMENT,+ "judgment_aliases",+ id="judgment-alias",+ ),+ pytest.param(+ check_judgment_choice_with_deck,+ "~missing-waterfall",+ ModelReferenceKind.WATERFALL,+ ModelType.JUDGMENT,+ "judgment_waterfalls",+ id="judgment-waterfall",+ ),+ pytest.param(+ check_judgment_choice_with_deck,+ "missing-handle",+ ModelReferenceKind.HANDLE,+ ModelType.JUDGMENT,+ "inference_models",+ id="judgment-handle",+ ),],)def test_not_found_choice_raises_with_context(diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_loader_judgment.py b/tests/unit/pipelex/cogt/models/test_model_deck_loader_judgment.pynew file mode 100644index 000000000..bae486cc5--- /dev/null+++ b/tests/unit/pipelex/cogt/models/test_model_deck_loader_judgment.py@@ -0,0 +1,31 @@+"""A deck installed before the judgment family existed still loads.++``pipelex update`` is the only thing that installs a new deck file, and nothing runs it at boot,+so every project and global deck predating ``6_judgment_deck.toml`` reaches the loader without a+``judgment`` section. That absence has to read as "no judgment model", which is what the kit's own+empty section says, rather than fail every boot.+"""++from pathlib import Path++from pipelex.cogt.models.model_deck_loader import load_model_deck_blueprint+from pipelex.kit.paths import get_kit_configs_dir++JUDGMENT_DECK_FILE_NAME = "6_judgment_deck.toml"+++class TestModelDeckLoaderJudgment:+ def _kit_deck_paths(self) -> list[str]:+ deck_dir = Path(str(get_kit_configs_dir())) / "inference" / "deck"+ return sorted(str(path) for path in deck_dir.glob("*.toml"))++ def test_a_deck_without_the_judgment_file_loads_with_no_judgment_model(self) -> None:+ deck_paths = [path for path in self._kit_deck_paths() if not path.endswith(JUDGMENT_DECK_FILE_NAME)]+ assert len(deck_paths) < len(self._kit_deck_paths()), "the kit deck no longer ships the judgment file this test removes"++ blueprint = load_model_deck_blueprint(model_deck_paths=deck_paths)++ assert blueprint.judgment.aliases == {}+ assert blueprint.judgment.presets == {}+ assert blueprint.judgment.waterfalls == {}+ assert blueprint.judgment.choice_default is Nonediff --git a/tests/unit/pipelex/cogt/models/test_model_deck_references.py b/tests/unit/pipelex/cogt/models/test_model_deck_references.pyindex 9e2ef72ea..46bc8e4bf 100644--- a/tests/unit/pipelex/cogt/models/test_model_deck_references.py+++ b/tests/unit/pipelex/cogt/models/test_model_deck_references.py@@ -4,6 +4,7 @@ This test suite systematically validates that model deck references point to valpreventing configuration errors from being discovered at runtime."""+from collections.abc import Mappingfrom typing import castimport pytest@@ -11,6 +12,7 @@ import pytestfrom pipelex.cogt.doc_gen.doc_gen_setting import DocGenSettingfrom pipelex.cogt.extract.extract_setting import ExtractSettingfrom pipelex.cogt.img_gen.img_gen_setting import ImgGenSetting+from pipelex.cogt.judgment.judgment_setting import JudgmentSettingfrom pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck import (@@ -115,7 +117,7 @@ class TestModelDeckReferences:def _find_invalid_preset_references(self,- presets: dict[str, LLMSetting] | dict[str, ExtractSetting] | dict[str, ImgGenSetting] | dict[str, SearchSetting] | dict[str, DocGenSetting],+ presets: Mapping[str, LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | DocGenSetting | JudgmentSetting],all_aliases: dict[str, str],all_waterfalls: dict[str, list[str]],known_model_handles: dict[str, ModelType],@@ -180,6 +182,8 @@ class TestModelDeckReferences:return model_deck_blueprint.searchcase ModelType.DOC_GEN:return model_deck_blueprint.doc_gen+ case ModelType.JUDGMENT:+ return model_deck_blueprint.judgment@pytest.mark.parametrize(("model_type", "deck_name"),@@ -189,6 +193,7 @@ class TestModelDeckReferences:(ModelType.IMG_GEN, "ImgGen"),(ModelType.SEARCH, "Search"),(ModelType.DOC_GEN, "DocGen"),+ (ModelType.JUDGMENT, "Judgment"),],)def test_aliases_reference_valid_targets(@@ -223,6 +228,7 @@ class TestModelDeckReferences:(ModelType.IMG_GEN, "ImgGen"),(ModelType.SEARCH, "Search"),(ModelType.DOC_GEN, "DocGen"),+ (ModelType.JUDGMENT, "Judgment"),],)def test_presets_reference_valid_models(@@ -257,6 +263,7 @@ class TestModelDeckReferences:(ModelType.IMG_GEN, "ImgGen"),(ModelType.SEARCH, "Search"),(ModelType.DOC_GEN, "DocGen"),+ (ModelType.JUDGMENT, "Judgment"),],)def test_waterfalls_contain_valid_models(@@ -290,6 +297,7 @@ class TestModelDeckReferences:(ModelType.IMG_GEN, "ImgGen"),(ModelType.SEARCH, "Search"),(ModelType.DOC_GEN, "DocGen"),+ (ModelType.JUDGMENT, "Judgment"),],)def test_aliases_no_circular_references(diff --git a/tests/unit/pipelex/cogt/usage/test_cost_registry.py b/tests/unit/pipelex/cogt/usage/test_cost_registry.pyindex 380e75b52..29fd3c3fc 100644--- a/tests/unit/pipelex/cogt/usage/test_cost_registry.py+++ b/tests/unit/pipelex/cogt/usage/test_cost_registry.py@@ -3,14 +3,21 @@ from pathlib import Pathfrom typing import Anyimport pytest+from pydantic import BaseModelfrom pytest_mock import MockerFixture-from pipelex.cogt.llm.llm_report import LLMTokenCostReportField, LLMTokensUsage+from pipelex.cogt.extract.extract_report import ExtractTokenCostReport+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokenCostReport+from pipelex.cogt.judgment.judgment_report import JudgmentTokenCostReport, JudgmentTokenCostReportField+from pipelex.cogt.llm.llm_report import LLMTokenCostReport, LLMTokenCostReportField, LLMTokensUsage+from pipelex.cogt.search.search_report import SearchTokenCostReportfrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.cost_registry import CostRegistryfrom pipelex.cogt.usage.token_category import TokenCategory+from pipelex.reporting.reporting_types import AnyTokensUsagefrom pipelex.system.exceptions import MissingDependencyErrorfrom pipelex.system.job_metadata import JobMetadata+from tests.unit.pipelex.cogt.usage.test_data import UsageFixturesclass TestCostRegistry:@@ -632,3 +639,35 @@ class TestCostRegistry:)assert CostRegistry.build_cost_summary([dry]) is None++ @pytest.mark.parametrize(+ ("tokens_usage", "expected_report_class"),+ [+ (UsageFixtures.llm_usage(), LLMTokenCostReport),+ (UsageFixtures.img_gen_usage(), ImgGenTokenCostReport),+ (UsageFixtures.extract_usage(), ExtractTokenCostReport),+ (UsageFixtures.search_usage(), SearchTokenCostReport),+ (UsageFixtures.judgment_usage(), JudgmentTokenCostReport),+ ],+ ids=["llm", "img_gen", "extract", "search", "judgment"],+ )+ def test_every_family_gets_its_own_cost_report(self, tokens_usage: AnyTokensUsage, expected_report_class: type[BaseModel]):+ """No family falls through to an extract report.++ The dispatch used to end in a bare extract return, so any usage it did not recognise billed+ as an extraction — silently, and with the wrong model-type label on the wire. This is the+ runtime half of the guarantee; the static half is the ``assert_never`` that closes the match.+ """+ cost_report = CostRegistry.compute_cost_report(tokens_usage=tokens_usage)++ assert isinstance(cost_report, expected_report_class)+ assert cost_report.model_type == tokens_usage.model_type++ def test_a_judgment_cost_report_names_its_model_in_the_flat_record(self):+ """The grouping key reads the family's own name field, so a judgment is not grouped as 'unknown'."""+ cost_report = CostRegistry.complete_cost_report(tokens_usage=UsageFixtures.judgment_usage())+ aggregated = CostRegistry.aggregate_costs(tokens_usages=[UsageFixtures.judgment_usage()])++ assert cost_report.as_flat_dictionary()[JudgmentTokenCostReportField.JUDGMENT_NAME] == "jev"+ assert "jev" in aggregated.grouped_by_model+ assert aggregated.model_types["jev"] == "judgment"diff --git a/tests/unit/pipelex/cogt/usage/test_data.py b/tests/unit/pipelex/cogt/usage/test_data.pyindex 7998f17f2..ae8be299b 100644--- a/tests/unit/pipelex/cogt/usage/test_data.py+++ b/tests/unit/pipelex/cogt/usage/test_data.py@@ -2,6 +2,7 @@ from datetime import datetimefrom pipelex.cogt.extract.extract_report import ExtractTokensUsagefrom pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage+from pipelex.cogt.judgment.judgment_report import JudgmentTokensUsagefrom pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.search.search_report import SearchTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategory, CostsByCategoryDict@@ -89,9 +90,19 @@ class UsageFixtures:unit_costs=dict(RATED_UNIT_COSTS),)+ @classmethod+ def judgment_usage(cls) -> JudgmentTokensUsage:+ return JudgmentTokensUsage(+ job_metadata=cls.full_job_metadata(unit_job_id=UnitJobId.JUDGMENT_ANSWER, job_category=JobCategory.JUDGMENT_JOB),+ inference_model_name="jev",+ inference_model_id="jev-1.13.0",+ nb_tokens_by_category=dict(RATED_NB_TOKENS),+ unit_costs=dict(RATED_UNIT_COSTS),+ )+@classmethoddef all_variants(cls) -> list[AnyTokensUsage]:- return [cls.llm_usage(), cls.img_gen_usage(), cls.extract_usage(), cls.search_usage()]+ return [cls.llm_usage(), cls.img_gen_usage(), cls.extract_usage(), cls.search_usage(), cls.judgment_usage()]@classmethoddef unrated_usage(cls) -> LLMTokensUsage:diff --git a/tests/unit/pipelex/cogt/usage/test_usage_cost.py b/tests/unit/pipelex/cogt/usage/test_usage_cost.pyindex cc5f07fdf..337021529 100644--- a/tests/unit/pipelex/cogt/usage/test_usage_cost.py+++ b/tests/unit/pipelex/cogt/usage/test_usage_cost.py@@ -22,7 +22,7 @@ class TestComputeTokensUsageCost:@pytest.mark.parametrize("tokens_usage",UsageFixtures.all_variants(),- ids=["llm", "img_gen", "extract", "search"],+ ids=["llm", "img_gen", "extract", "search", "judgment"],)def test_parity_with_cost_registry_per_record(self, tokens_usage: AnyTokensUsage):"""The wire cost equals the canonical CostRegistry total for the same record — one cost engine, no drift."""@@ -43,4 +43,9 @@ class TestComputeTokensUsageCost:assert record_cost is not Nonewire_total += record_costaggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)- assert wire_total == aggregated.total_cost+ # Compared to within floating-point noise rather than bit-for-bit: the two totals are the same+ # sum of the same per-record costs, but they add them up in different orders and in different+ # groupings, so once there are enough records the last bit diverges. A real drift between the+ # two — a category one side excludes and the other does not — is many orders of magnitude+ # larger than this tolerance and still fails here.+ assert abs(wire_total - aggregated.total_cost) < 1e-12diff --git a/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py b/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.pyindex cf9148e55..e914afb6c 100644--- a/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py+++ b/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py@@ -98,6 +98,27 @@ class TestRunnerProtocolSurfaces:assert deck.waterfalls == {"llm": {"main_chain": ["smart_llm", "cheap_llm"]}, "extract": {}}list_models_mock.assert_called_once_with(categories=None)+ async def test_models_skips_a_category_the_protocol_has_no_word_for(self, mocker: MockerFixture) -> None:+ """A judgment preset must not take the whole deck down.++ `judgment` is a category this runtime serves and the standard's flat model list cannot yet+ name, so its presets are left out of `models` rather than raising on the enum; the routing+ extensions, which are keyed by plain strings, still carry it. Delete this test once the+ protocol's `ModelCategory` learns the member.+ """+ payload: dict[str, Any] = {+ "presets": {"llm": [{"name": "smart_llm"}], "judgment": [{"name": "strict_verdict"}]},+ "aliases": {"llm": {"best": "smart_llm"}, "judgment": {"default-judgment": "strict_verdict"}},+ "waterfalls": {"llm": {}, "judgment": {}},+ }+ mocker.patch("pipelex.pipeline.runner.list_models", return_value=payload)+ runner = PipelexMTHDSProtocol()++ deck = await runner.models()++ assert [(model_info.name, model_info.type) for model_info in deck.models] == [("smart_llm", MthdsModelCategory.LLM)]+ assert deck.aliases["judgment"] == {"default-judgment": "strict_verdict"}+async def test_models_category_filter_translates_to_builder_enum(self, mocker: MockerFixture) -> None:"""A protocol-level category filter reaches list_models as the builder's enum."""llm_only_payload: dict[str, Any] = {diff --git a/tests/unit/pipelex/reporting/test_usage_records.py b/tests/unit/pipelex/reporting/test_usage_records.pyindex e84fb90d8..a94fd84e2 100644--- a/tests/unit/pipelex/reporting/test_usage_records.py+++ b/tests/unit/pipelex/reporting/test_usage_records.py@@ -48,8 +48,9 @@ class TestTokensUsageRecords:(UsageFixtures.img_gen_usage(), "img_gen", "img_gen_job", "img_gen_text_to_image"),(UsageFixtures.extract_usage(), "extract", "extract_job", "extract_pages"),(UsageFixtures.search_usage(), "search", "search_job", "search_sourced_answer"),+ (UsageFixtures.judgment_usage(), "judgment", "judgment_job", "judgment_answer"),],- ids=["llm", "img_gen", "extract", "search"],+ ids=["llm", "img_gen", "extract", "search", "judgment"],)def test_conversion_per_variant(self,@@ -78,7 +79,7 @@ class TestTokensUsageRecords:@pytest.mark.parametrize("tokens_usage",[*UsageFixtures.all_variants(), UsageFixtures.unrated_usage()],- ids=["llm", "img_gen", "extract", "search", "unrated"],+ ids=["llm", "img_gen", "extract", "search", "judgment", "unrated"],)def test_leak_regression(self, tokens_usage: AnyTokensUsage):"""The dumped wire record carries the contract fields exactly — none of the MUST-NOTdiff --git a/wip/judgment-family/design.md b/wip/judgment-family/design.mdindex 42a16432a..8fc39556c 100644--- a/wip/judgment-family/design.md+++ b/wip/judgment-family/design.md@@ -137,9 +137,11 @@ The answers follow the same rule as the natives: `choice` and `level` are the on**Job.** `JudgmentJob(InferenceJobAbstract)` carries `state: JudgmentState` (a JSON-compatible value), `questions: dict[str, JudgmentQuestion]` (non-empty), `job_params: JudgmentJobParams` holding the `JudgmentSetting`, and `job_report` holding `JudgmentTokensUsage`. `JudgmentJobFactory.make_judgment_job(...)` stamps `JobCategory.JUDGMENT_JOB`.-**Worker.** `JudgmentWorkerAbstract(InferenceWorkerAbstract)` exposes one template method, `async def judge(self, judgment_job: JudgmentJob) -> dict[str, JudgmentAnswer]`, and one abstract hook, `_judge`. The template method copies `SearchWorkerAbstract.search_sourced_answer` step for step — validate, stamp `UnitJobId.JUDGMENT_ANSWER`, `judgment_job_before_start`, call the hook, `fill_model_and_provider` on a `CogtError`, and complete and report in `finally` — including the commented reason the reporting sits in `finally` (`search_worker_abstract.py:47`). After the hook returns, the template method checks that the answers' keys are exactly the questions' keys and that each answer's kind matches its question's, so no backend can return a verdict for a question nobody asked.+**Worker.** `JudgmentWorkerAbstract(InferenceWorkerAbstract)` exposes one template method, `async def judge(self, judgment_job: JudgmentJob) -> dict[str, JudgmentAnswer]`, and one abstract hook, `_judge`. The template method copies `SearchWorkerAbstract.search_sourced_answer` step for step — validate, stamp `UnitJobId.JUDGMENT_ANSWER`, `judgment_job_before_start`, call the hook, `fill_model_and_provider` on a `CogtError`, and complete and report in `finally` — including the commented reason the reporting sits in `finally` (`search_worker_abstract.py:47`). After the hook returns, the template method checks that the answers' keys are exactly the questions' keys, that each answer's kind matches its question's, and that every option and level an answer names — its verdict and the keys of its distribution — is one its question offered, so no backend can return a verdict for a question nobody asked or a verdict the question never offered.-**Setting and deck.** `JudgmentSetting(ConfigModel)` has `model: str` and `description: str | None` and nothing else; `JudgmentModelChoice` is the usual union with a parsed model reference. `JudgmentDeckBlueprint` has `aliases`, `waterfalls`, `presets` and `choice_default`, flattened onto `ModelDeck` with `get_judgment_setting`, `validate_judgment_presets` and `check_judgment_choice_with_deck` as for search. The deck file is `pipelex/kit/configs/inference/deck/5_judgment_deck.toml` — the numeric prefix is what makes it kit-managed (`pipelex/cogt/models/deck_manifest.py:98`) — with `choice_default = "@default-judgment"` and the alias pointing at a versioned handle, never at a `latest` alias: a verdict model that changes under a method silently changes what every threshold in that method means.+**Setting and deck.** `JudgmentSetting(ConfigModel)` has `model: str` and `description: str | None` and nothing else; `JudgmentModelChoice` is the usual union with a parsed model reference. `JudgmentDeckBlueprint` has `aliases`, `waterfalls`, `presets` and `choice_default`, flattened onto `ModelDeck` with `get_judgment_setting`, `validate_judgment_presets` and `check_judgment_choice_with_deck` as for search. The deck file is `pipelex/kit/configs/inference/deck/5_judgment_deck.toml` — the numeric prefix is what makes it kit-managed (`pipelex/cogt/models/deck_manifest.py:98`) — and an alias in it points at a versioned handle, never at a `latest` alias: a verdict model that changes under a method silently changes what every threshold in that method means.++**`choice_default` is optional for this family alone, and phase 2 discovered why.** Every other family's default names a handle the Pipelex Gateway carries, so it always resolves. A judgment backend is brought by the user, and `ModelManager._enforce_gateway_model_membership` walks every handle a deck's presets and choice defaults name: a default resolving to `jev-1.13.0`, which the active routing profile sends to the gateway by default and which the gateway does not serve, raises `GatewayUnknownModelError` at **every boot**, for every user, whether or not they ever ask for a judgment. So `JudgmentDeckBlueprint.choice_default` is `JudgmentModelChoice | None = None`, the kit deck ships no default, and a judgment names its own model. The consequence for Part 1 is real and belongs to the "hosted serving" open question: `@default-judgment` is not a usable default for `PipeJudge` until the hosted plane serves the family, so phase 3's operator must require an explicit model or fail with a message saying no judgment model is configured.**The family enums.** There are several, and all must move together: `InferenceFamily` (`inference_backend_registry.py:11`), `ModelType` (`pipelex/cogt/model_backends/model_type.py:4`), `InferenceErrorFamily` with its failure and not-found class tables (`pipelex/cogt/inference/error_render.py:35`), and `ModelCategory` in the builder (`pipelex/builder/operations/models_ops.py`). The match statements over them are exhaustive and free of `case _`, so the type checker walks to most arms.diff --git a/wip/judgment-family/plan.md b/wip/judgment-family/plan.mdindex c82738230..93e11cfa4 100644--- a/wip/judgment-family/plan.md+++ b/wip/judgment-family/plan.md@@ -79,19 +79,48 @@ Both carry the same item; 2a is `Advances` and 2b is `Closes`. Whether 2b stacks- **The rating answer's keys change type across three layers**, and each conversion is somebody's job: strings on the wire, integers after the SDK parses, `dict[int, float]` on `RatingAnswer` per part 3 — and then text keys again on the `Rating` native per part 2, because the language's dict keys are text. The worker owns the first hop and the kernel the last; phase 3 inherits the last one.- **The deck's versioned handle is `jev-1.13.0`**, and `GET /v1/models` will not confirm it: the endpoint lists only `jev-latest` and `jev-preview`. The deck TOML carries a comment saying a newer id is found on the vendor's models page, so the next person does not go looking for an API that has it.-- [ ] **Family identity.** `JUDGMENT` in `InferenceFamily`, `ModelType` and `InferenceErrorFamily`, with the failure and not-found class tables; `JudgmentJobFailureError`, `JudgmentModelNotFoundError` and `JudgmentHandleNotFoundError` in `pipelex/cogt/exceptions.py`; `JobCategory.JUDGMENT_JOB`, `UnitJobId.JUDGMENT_ANSWER` and its display arm. Follow the type checker through every exhaustive match it flags.-- [ ] **The package** `pipelex/cogt/judgment/`: question and answer models (the `YesNoAnswer` at-least-one validator included), setting and model choice, usage and cost report, job, job factory, worker contract, worker factory. Unit tests drive the worker's template method through a fake worker: the answers' keys must equal the questions' keys, each answer's kind must match its question's, and reporting happens in `finally` on success and on failure. Mutation-test each guard by breaking it and watching the test go red.-- [ ] **Deck.** `JudgmentDeckBlueprint`, the flat fields on `ModelDeck`, `get_judgment_setting`, `validate_judgment_presets`, `check_judgment_choice_with_deck`, the suggestion arms, the model manager's handle collection and flattening, and `pipelex/kit/configs/inference/deck/5_judgment_deck.toml` with the default alias on a versioned handle.-- [ ] **Reporting and cost.** `JudgmentTokensUsage` and its report in both unions, the `ReportingManager` dispatch arm, and in `CostRegistry.compute_cost_report` a judgment arm, an explicit extract arm, and `assert_never` to close. Real token counts, priced per million like an LLM; update the `model_type` comment on `ModelUsageSpec`.-- [ ] **The leaf.** `JudgmentAssignment`, `pipelex/cogt/content_generation/judgment_generate.py` with the dry branch first, `dry_judgment_gen_answers` (deterministic, uncertainty absent), and `make_judgment_answers` on the protocol and the generator. A unit test pins the dry branch.+- [x] **Family identity.** `JUDGMENT` in `InferenceFamily`, `ModelType` and `InferenceErrorFamily`, with the failure and not-found class tables; `JudgmentJobFailureError`, `JudgmentModelNotFoundError` and `JudgmentHandleNotFoundError` in `pipelex/cogt/exceptions.py`; `JobCategory.JUDGMENT_JOB`, `UnitJobId.JUDGMENT_ANSWER` and its display arm. Follow the type checker through every exhaustive match it flags.+- [x] **The package** `pipelex/cogt/judgment/`: question and answer models (the `YesNoAnswer` at-least-one validator included), setting and model choice, usage and cost report, job, job factory, worker contract, worker factory. Unit tests drive the worker's template method through a fake worker: the answers' keys must equal the questions' keys, each answer's kind must match its question's, and reporting happens in `finally` on success and on failure. Mutation-test each guard by breaking it and watching the test go red.+- [x] **Deck.** `JudgmentDeckBlueprint`, the flat fields on `ModelDeck`, `get_judgment_setting`, `validate_judgment_presets`, `check_judgment_choice_with_deck`, the suggestion arms, the model manager's handle collection and flattening, and `pipelex/kit/configs/inference/deck/5_judgment_deck.toml` with the default alias on a versioned handle.+- [x] **Reporting and cost.** `JudgmentTokensUsage` and its report in both unions, the `ReportingManager` dispatch arm, and in `CostRegistry.compute_cost_report` a judgment arm, an explicit extract arm, and `assert_never` to close. Real token counts, priced per million like an LLM; update the `model_type` comment on `ModelUsageSpec`.+- [x] **The leaf.** `JudgmentAssignment`, `pipelex/cogt/content_generation/judgment_generate.py` with the dry branch first, `dry_judgment_gen_answers` (deterministic, uncertainty absent), and `make_judgment_answers` on the protocol and the generator. A unit test pins the dry branch.- [ ] **The backend.** `pipelex/providers/typesafe/`: plugin, worker, translation both ways, exceptions and error classification, written against the four findings above and unit-tested by replaying the recorded responses in `spike/responses/` — the successes are wire bodies and the refusals are `describe_exception` envelopes with the wire body under `body`, which is what the classification tests read. The vendor's ten-level cap and its option-count limits are enforced in this translation, not in the blueprint. The plugin joins `KERNEL_BUILTIN_PLUGINS`; the client is cached through the `SdkClientRegistry`; retry and timeout are passed explicitly. Backend record in `backends.toml`, `backends/typesafe.toml`, a routing profile, and the `.pipelex/inference/` mirror with its kit manifest. Check that a boot with the backend enabled and the key unset behaves as it does for Linkup, and add the arm to the keyless boot test.- [ ] **Plugin surface tests.** The expected-backends table in `tests/unit/pipelex/plugins/test_inference_backend_coverage.py`, and a missing-extra guard test mirroring Linkup's.-- [ ] **Model listing.** `ModelCategory.JUDGMENT` in `pipelex/builder/operations/models_ops.py` with its arms, so `pipelex-agent models` and `check-model` know the family.-- [ ] **Test infrastructure.** The `judgment` marker and its term in the default deselect expression in `pyproject.toml`; every family table in `preprocess_test_models_cmd.py`; a `[collections.judgment]` and a `judgment_models` key on every profile in `.pipelex-dev/test_profiles.toml`, empty everywhere except a new `[profiles.typesafe]` whose `backends` is `["typesafe"]`; `get_judgment_combos` with its skip, the `judgment_combo` fixture and its re-export; regenerate `_generated_model_sets.py`.+- [x] **Model listing.** `ModelCategory.JUDGMENT` in `pipelex/builder/operations/models_ops.py` with its arms, so `pipelex-agent models` and `check-model` know the family.+- [x] **Test infrastructure.** The `judgment` marker and its term in the default deselect expression in `pyproject.toml`; every family table in `preprocess_test_models_cmd.py`; a `[collections.judgment]` and a `judgment_models` key on every profile in `.pipelex-dev/test_profiles.toml`, empty everywhere except a new `[profiles.typesafe]` whose `backends` is `["typesafe"]`; `get_judgment_combos` with its skip, the `judgment_combo` fixture and its re-export; regenerate `_generated_model_sets.py`.- [ ] **Live tests**, marked `inference` and `judgment`, in `tests/integration/pipelex/cogt/test_judgment.py` and `config_coverage/test_cc_judgment.py`: each question kind on an unambiguous case, several questions in one job, usage tokens present, a versioned model id reported, and an error case (an unknown model handle). Verdicts are asserted exactly and probabilities within ±0.05, per checkpoint 1; the exact instruction strings are pinned, since rewording moved one case from 0.11 to 0.30 while repetition moved it 0.01.- [ ] **Docs.** `docs/under-the-hood/inference-backend-plugins.md` (the family comment, the "serves all four" sentence, the worker-contract table), the distributed content generation, dry-run and per-node usage pages, a new `docs/contribute/inference-family-surface.md` listing every site a family touches, and its `mkdocs.yml` entry. Regenerate the error pages and the identity snapshot (`make gep`, `make gei`). Changelog entry.- [ ] `make agent-check`, resolving any drift contract it opens; `make agent-test`; `make ti PROF=typesafe TEST=judgment`; then `make rtm` to put the generated model sets back on the default profile before committing, since the `PROF=` run rewrites that tracked file.+### What 2a changed against the design++Two things, both recorded in `design.md` in the same change.++**The judgment deck has no `choice_default`, and the blueprint makes it optional.** Shipping one broke every boot: `ModelManager._enforce_gateway_model_membership` walks every handle a deck's presets and choice defaults name, the default routing profile sends an unmatched handle to the Pipelex Gateway, and the gateway does not serve Jev — so `GatewayUnknownModelError` fired in `tests/unit/pipelex/test_hub_lifecycle.py` before any judgment was ever asked for. `JudgmentDeckBlueprint.choice_default` is therefore `JudgmentModelChoice | None = None`, alone among the families. **This is phase 3's to carry**: `@default-judgment` cannot be `PipeJudge`'s default until the hosted plane serves the family, so the operator must require an explicit model, and the design's "hosted serving" open question now has a consequence attached to it.++**`CostRegistry.compute_cost_report` became a `match` rather than an `if`-chain.** With an explicit extract arm the trailing `isinstance` was provably true, which pyright refuses as `reportUnnecessaryIsInstance`, so the dispatch is a match over the usage classes closed by `case _: assert_never(...)`. The guarantee the design asked for is unchanged and now has a runtime test beside the static one (`test_every_family_gets_its_own_cost_report`).++**The MTHDS protocol has no word for the family, and `PipelexMTHDSProtocol.models()` would have raised on it.** `mthds.protocol.models.ModelCategory` carries `llm`, `extract`, `img_gen` and `search`; the runner builds one `ModelInfo` per preset and types it by calling that enum on our own category key, so the first judgment preset in a deck turns `models()` into a `ValueError` for every caller, not just for judgment. The runner now skips a category the protocol cannot name, leaving the judgment aliases and waterfalls on the string-keyed routing extensions, and [L-260919-afb6c0](http://localhost:4747/i/L-260919-afb6c0) asks `mthds` for the member. Nothing is broken today only because the kit deck ships no judgment preset — **2b's typesafe deck is what makes this live**, so that item is effectively a gate on 2b rather than a nicety.++**One pre-existing test was loosened, and it is worth knowing why.** `test_parity_with_aggregate_run_total` compared two float sums for bit-for-bit equality; they are the same per-record costs added in different orders, and a fifth record made the last bit diverge. It now compares to within `1e-12`, which is still many orders of magnitude tighter than any real drift between the two cost paths.++### Checkpoint 2a — the family, handed off++**Where the work is.** Worktree `_pipelex--judgment-cogt-family`, branch `feature/Judgment-cogt-family`, which is **stacked on `feature/Judgment-family`** (phase 1, open as PR #1214). Its first three commits are phase 1's; everything after them is 2a. The pull request for 2a therefore targets `feature/Judgment-family`, not `dev`, until phase 1 merges — per the workspace's stacked-PR rule — and its body carries `Advances L-260919-502f36`, because 2b is what closes the item.++**What is done.** Every box in the 2a half of the list above, and nothing from 2b. Concretely: the family enums and their failure/not-found tables, `pipelex/cogt/judgment/` entire, the deck (blueprint, flat fields, `get_judgment_setting`, `validate_judgment_presets`, `check_judgment_choice_with_deck`, the ambiguity warner, the model manager's collection and flattening, `5_judgment_deck.toml` and its `.pipelex/` mirror), reporting and cost, the leaf with its dry branch and its protocol method, `ModelCategory.JUDGMENT`, and the test infrastructure. `make agent-check` passes, including the `cli-docs` drift contract, which this change opened and which is acked. No `typesafe` name appears anywhere outside `pyproject.toml`'s already-landed extra.++**What is left, and it is all 2b**: `pipelex/providers/typesafe/` with its worker, translation and error classification written against the four findings above; the backend records and the `.pipelex/inference/` mirror; the plugin surface tests; the `[profiles.typesafe]` arm and a `[collections.judgment]` naming `jev-1.13.0` (deliberately removed from 2a, since it names the vendor); the live tests; and the docs for both halves, including the new `docs/contribute/inference-family-surface.md`. The changelog entry under `## [Unreleased]` describes 2a's family and is 2b's to extend with the backend rather than to duplicate.++**The commands a cold session needs**, all from the worktree root:++- `make agent-check` after every code change, with changes staged first so the drift digest sees them.+- `make agent-test` to close, or `.venv/bin/pytest tests/unit/pipelex/cogt/judgment/ tests/unit/pipelex/cogt/content_generation/test_judgment_generate_dry_branch.py tests/unit/pipelex/cogt/models/ tests/unit/pipelex/cogt/usage/ -q` for the fast loop over what 2a touched.+- `make rtm` after any `make ti PROF=…` run. In this repo `tests/integration/pipelex/fixtures/_generated_model_sets.py` is **gitignored**, so the plan's warning about it being a tracked file does not apply here — running `rtm` is still right for local consistency.+- ⚠ `make agent-check` runs the keyword-only auto-fixer, and it **silently rewrote four of this phase's signatures** before their grants existed: `get_judgment_setting`, `_warn_if_ambiguous_judgment` and both `JudgmentTokenCostReportField.report_field_for_*` staticmethods. Record the grant with `make sgr FUNC=… RATIONALE=…` *before* running the check, and do not put backticks in a rationale.++**Open questions, none blocking 2b's code.** The design's own list still stands unratified (the standard's native names, how much optional surface to pin, `question` against `prompt`, hosted serving), and phase 3's gate waits on it. What 2a added to that list is the consequence recorded above: with no `choice_default`, `PipeJudge` cannot default a model, so "hosted serving" is now a question with a user-visible cost attached rather than a purely internal one.+### Checkpoint 2Record here: what the live tests cover and the command that ran them; any contract change against the design, with the design corrected; the measured cost of a test run. File the follow-up for `pipelex-server/temporal/` (an activity and a queue for the new leaf), which can be written as soon as the leaf exists. Confirm the gate below before starting phase 3.