Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Multi-question-judge · L-261009-0a6ed9 (#1412)

Louis Choquel46e228cb69e2
diff --git a/.drift/acks/pipelex-kernel-docs.toml b/.drift/acks/pipelex-kernel-docs.toml
index 38e142c76..3b94fed28 100644
--- a/.drift/acks/pipelex-kernel-docs.toml
+++ b/.drift/acks/pipelex-kernel-docs.toml
@@ -1,14 +1,14 @@
contract = "pipelex-kernel-docs"
-digest = "sha256:2432a4877f71dbe3676b5a174163343ddb2804f4a7987e9a52f8cac4522fb1f4"
+digest = "sha256:f6449898b2c6db663c5a11f287652c487620dd24dec35654a0883fb62c2fd91d"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-10-10T18:04:24Z"
-rationale = "real-catch: the shared assembly now skips a file reference whose root input is declared optional and was not given, so it takes no number and hands over no file, while ImageReference and DocumentReference gained is_optional and the judgment_ops docstring stopped calling absence the template's business alone. pipelex-kernel.md still said nothing was left out on the kernel's side; the judgment paragraph now says the assembly skips an absent optional input's files, and the prompt-assembly paragraph describes the skip, the is_optional mark the shared factory analysis sets and ContextProviderAbstract.is_variable_present."
+reviewed_at = "2026-10-10T20:46:23Z"
+rationale = "real-catch: run_multi_judgment now hands the verdicts over by field name, so an aliased field keeps its verdict, and decides a refusal by class_field_may_hold_nothing, the predicate the load's concept walk reads a class field by, storing None outright when the field's type admits it. pipelex-kernel.md said a refused field was left absent when neither required nor defaulted; the judgment paragraph now states the shared rule and the by-name handover, and gains the PipeJudge execution-data paragraph moved out of the building-methods page."
[trigger_files]
"pipelex/kernel/__init__.py" = "blob:c4f1b0fe2ccc17db9110733f71b612f4ac1ba4ce"
"pipelex/kernel/compose_ops.py" = "blob:ccd8ba141246380eb5e9d55e693fa6fedf38233a"
"pipelex/kernel/compose_results.py" = "blob:a837ae9680c7d4d0eb4dee37d4657c410e5b5cea"
-"pipelex/kernel/exceptions.py" = "blob:45e9ad60c2699f7b57cad0a46c13f0b07b4cfbc9"
+"pipelex/kernel/exceptions.py" = "blob:e6c9fc0ddae10e533f8aaf2fd0f8afcb46881461"
"pipelex/kernel/extract_ops.py" = "blob:19d556f0254017d7d52c30f4f1a46e2235289042"
"pipelex/kernel/extract_results.py" = "blob:79405a56760f3d4b483660d76441fff95f58c64d"
"pipelex/kernel/func_ops.py" = "blob:97c68c977ddc260c9b36b7eda5f4c7964e956a27"
@@ -16,8 +16,8 @@ rationale = "real-catch: the shared assembly now skips a file reference whose ro
"pipelex/kernel/img_gen_ops.py" = "blob:b85360f6c118e93be53bf8d1a04504ccb7552469"
"pipelex/kernel/img_gen_prompt.py" = "blob:c653190d40e4323f763b9da511bd55a4a1646888"
"pipelex/kernel/img_gen_results.py" = "blob:46b4ca11c70d906ae6d58ce66461f9b3414f1bfa"
-"pipelex/kernel/judgment_ops.py" = "blob:46baec716f5d3a6f2a969afd68c9ba7b2533af2c"
-"pipelex/kernel/judgment_results.py" = "blob:ad759849054a74aaf1fedcbf7bec31d2f753a13c"
+"pipelex/kernel/judgment_ops.py" = "blob:68316783e27e9a55b95a66929d661c1ed5d6d5a3"
+"pipelex/kernel/judgment_results.py" = "blob:4dbbcb204309265999de59a179f4420a0f83be93"
"pipelex/kernel/llm_ops.py" = "blob:d37a9c3718a37f5e7d1550d1512dddbf79032cae"
"pipelex/kernel/llm_prompt_content.py" = "blob:f996cead6e7789494f01f69c128ae71d6dcf8466"
"pipelex/kernel/llm_results.py" = "blob:6e3b8a2878b5bb284ece5fac10262fb59ac2029d"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a4cbe77ed..40450849e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,11 +6,12 @@
- **`Rating` carries the label of its level**: the `Rating` native has an optional `label`, which the MTHDS standard's native set pinned at `5.0.0` defines as the name of the selected level when the scale declares labels, held in `RatingContent.label` and stated by the input-form descriptor, the I/O contracts and the generated Python and TypeScript types. A `PipeJudge` whose levels carry labels fills it with the label of the level it chose, while a `PipeLLM` whose output is `Rating`, or any other producer, may fill it like any optional member; either way a `Rating` renders in a prompt as its label when it has one and as its level otherwise, so `$severity` reads `Workaround available` for a rating carrying that label and `1` for one carrying none, and its JSON rendering carries the label only when present. Since a crate materializes the pinned definition, a crate whose methods use `Rating` gets a new normalized fingerprint, and types generated before this release gain the field when `pipelex codegen types` regenerates them.
- **A log line's advice in `user_action`**: a log call puts what its reader should do in a `user_action` field, one imperative sentence, and the `console` sink prints it whole on a line of its own under the record, after `→ `, in cyan, while every other sink writes it as a field like any other. Pipelex's own warnings carry their advice there, such as `Run pipelex migrate to update it` on a stale configuration file, so their message says only what happened.
+- **A `PipeJudge` asks several questions in one request**: a `PipeJudge` may set `questions` instead of `question`, a table of questions keyed by name, each holding its `question` and its own `options`, `levels`, `criteria` and `threshold`, checked under the single form's rules with a refusal naming the question, and the judging model answers them all in one request over the same evidence; the verdicts fill the output's structure, whose fields must be exactly the question names, each holding its question's `YesNo`, `Choice` or `Rating` or a concept refining it, which the load checks field by field (`inadequate_output_concept`). A question the model declines leaves its field holding nothing when the field may hold nothing, being neither `required` nor given a `default_value`, or, in a structure class, typed to admit `None`, with a warning naming it under `judgment_question` and its refusal in the step's execution data, and fails the step with `JudgmentRefusedError` naming the question otherwise, since a default would read as a verdict the model never gave. The kernel's `run_multi_judgment` does the same for a programmatic caller, taking the questions as `AskedQuestion`s and the output's content class, and returning a `MultiJudgmentResult` with each question's outcome; a class whose fields are not the questions' names is refused with `JudgmentOutputFieldsError` before the model is called.
- **`package_prefix` chooses which console lines name their package**: a setting under `[runtime.log.rich_log]` decides which lines the `console` sink starts with their logger's top-level package name, dimmed. `libraries`, the default, keeps the rule that names another library's package and leaves a line from `pipelex` or a `pipelex_` package bare; `all` names every line's package, so `pipelex: `, `pipelex_api: ` and a plugin's name tell apart which component spoke; `none` names none. A line logged on the root logger never carries one.
### Changed
-- **MTHDS standard 5.0.0 (Breaking)**: pipelex now requires `mthds==0.21.0` and `pipelex-sdk==0.38.0`, the releases that report MTHDS standard `5.0.0`, so everyone downstream inherits those exact versions and a project that pins either package itself has to move with them. `pipelex --version` prints `mthds-standard 5.0.0`, every crate pipelex normalizes is stamped with `mthds_version` `5.0.0` and materializes the native set the standard pinned at `5.0.0`, and a package whose `METHODS.toml` constrains `mthds_version` to `^4.0.0`, `~4.0.0`, `4.*` or exactly `4.0.0` now logs a warning when it loads, which `>=4.0.0` or `^5.0.0` does not. `PipeJudge` judges the evidence of its `prompt` with two-sided criteria and labelled levels, as the standard's `5.0.0` PipeJudge defines, but does not yet take the several `questions` it defines.
+- **MTHDS standard 5.0.0 (Breaking)**: pipelex now requires `mthds==0.21.0` and `pipelex-sdk==0.38.0`, the releases that report MTHDS standard `5.0.0`, so everyone downstream inherits those exact versions and a project that pins either package itself has to move with them. `pipelex --version` prints `mthds-standard 5.0.0`, every crate pipelex normalizes is stamped with `mthds_version` `5.0.0` and materializes the native set the standard pinned at `5.0.0`, and a package whose `METHODS.toml` constrains `mthds_version` to `^4.0.0`, `~4.0.0`, `4.*` or exactly `4.0.0` now logs a warning when it loads, which `>=4.0.0` or `^5.0.0` does not. `PipeJudge` judges the evidence of its `prompt` with two-sided criteria and labelled levels, and asks several `questions` in one request, as the standard's `5.0.0` PipeJudge defines.
- **`PipeJudge` judges the evidence of its `prompt` (Breaking)**: `prompt` is now required and holds the evidence, a template rendered exactly as a `PipeLLM`'s prompt is with its images and documents presented to the judging model as files, beside a required `question`: `prompt` is no longer a synonym of `question`, a pipe writing its question there is refused with a message saying where each goes, and a pipe is rewritten as `prompt = "@message"` beside its question. The model sees an input only as the two templates render it: every declared input must be read by the prompt or the question (`extraneous_input_variable`), an image or a document read by the question is refused (`input_stuff_spec_mismatch`), the model's file capability is checked against what the prompt reads, an absent optional input renders as its guard says, such as `@?note`, and a `Dynamic` input renders as text. `criteria` takes both `yes` and `no`, a one-sided or empty table being refused; a level of `levels` may be a `{ label, description }` table, labels all or none and distinct, and the `Rating` verdict then carries the label of the level chosen; a model that declines the question fails the step with the new `JudgmentRefusedError`, a content error naming the pipe and the model, where no verdict is ever read off a refusal.
- **The judgment worker contract carries a rendered prompt (Breaking)**: `JudgmentJob` and `JudgmentAssignment` carry a `JudgmentPrompt`, the rendered text with its images and documents, in place of the `state`, `images` and `documents` keyed by input name, and `JudgmentState` is gone; `YesNoQuestion` carries one optional `criteria` of type `YesNoCriteria` with both sides in place of `yes_criterion` and `no_criterion`; `RatingQuestion.levels` is a list of `RatingLevel`, each with a `label`, a `description` or both; and `JudgmentWorkerAbstract.judge` returns a `JudgmentOutcome` per question, an answer or a `JudgmentRefusal`. The judgment contract a backend plugin implements, with the family's errors a worker raises, is published in the Inference SPI table, and the kernel's `run_judgment` takes the evidence as a `UserPromptContent`, `build_judgment_material` and `material_value` being gone.
- **Console lines without the emoji prefix or the path column (Breaking)**: a line logged under the `pipelex` package or a `pipelex_` one, the runtime's, the API server's or a plugin's named that way, and a line logged on the root logger start with their message, and a line another library logged starts with that library's top-level package name, dimmed, such as `httpx: `, where every line used to start with an emoji or a bracketed logger name. The column naming the source file and line of each record is off by default; set `is_show_path = true` under `[runtime.log.rich_log]` to show it, `is_link_path_enabled` still making it clickable. The `pipelex.tools.log.log_formatter` module is gone with its `EmojiLogFormatter`, so code that formatted records with it uses the stdlib's `logging.Formatter`.
diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yaml
index e1dfa9a8c..37ddefba4 100644
--- a/docs/api-server/openapi/pipelex-api.openapi.yaml
+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml
@@ -3164,6 +3164,51 @@ components:
has no effect on this arm (unlike `rendered_markdown`, which rides both arms because failure
text is exactly what a human surface wants).'
+ JudgeQuestionBlueprint:
+ properties:
+ question:
+ type: string
+ title: Question
+ options:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Options
+ levels:
+ anyOf:
+ - items:
+ anyOf:
+ - type: string
+ - $ref: '#/components/schemas/JudgeRatingLevel'
+ type: array
+ - type: 'null'
+ title: Levels
+ criteria:
+ anyOf:
+ - $ref: '#/components/schemas/JudgeYesNoCriteria'
+ - type: 'null'
+ threshold:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Threshold
+ additionalProperties: false
+ type: object
+ required:
+ - question
+ title: JudgeQuestionBlueprint
+ description: 'One question of a PipeJudge asking several: what it asks, and the fields the single form sets on the pipe.
+
+
+ Closed, as the standard requires: `question` is the instruction, a plain text template, and the kind
+
+ is decided by which of `options` and `levels` the question declares, under the single form''s rules,
+
+ which `PipeJudgeBlueprint` checks, since a refusal names the question and only its key in `questions`
+
+ does. That key also names the output field holding its verdict.'
JudgeRatingLevel:
properties:
label:
@@ -5226,8 +5271,17 @@ components:
type: string
title: Prompt
question:
- type: string
+ anyOf:
+ - type: string
+ - type: 'null'
title: Question
+ questions:
+ anyOf:
+ - additionalProperties:
+ $ref: '#/components/schemas/JudgeQuestionBlueprint'
+ type: object
+ - type: 'null'
+ title: Questions
model:
anyOf:
- $ref: '#/components/schemas/JudgmentSetting'
@@ -5266,19 +5320,23 @@ components:
- description
- output
- prompt
- - question
title: PipeJudgeBlueprint
- description: 'Asks a judging model one closed question about the evidence its prompt presents: yes/no, a choice, or
- a rating.
+ description: 'Asks a judging model closed questions about the evidence its prompt presents: yes/no, a choice, or a rating.
`prompt` is the evidence template, rendered and checked as a PipeLLM''s prompt is, the images and
- documents it reads presented to the model as files. `question` is the instruction, a plain text
+ documents it reads presented to the model as files. The single form asks one `question`, a plain
+
+ text template, and produces its verdict native; the multi form asks the several `questions` of a
+
+ table keyed by name over the same evidence in one request, and fills a structure whose fields are
+
+ those names with one verdict each. A question''s kind is decided by which of `options` and `levels`
- template. The kind is decided by which of `options` and `levels` the pipe declares, never by a
+ it declares, never by a field of its own, and the single form declares them on the pipe. Every
- field of its own. Every declared input is read by the prompt or by the question.'
+ declared input is read by the prompt or by a question.'
PipeLLMBlueprint:
properties:
source:
diff --git a/docs/building-methods/pipes/pipe-operators/PipeJudge.md b/docs/building-methods/pipes/pipe-operators/PipeJudge.md
index dc41aa401..be08330ee 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeJudge.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeJudge.md
@@ -1,10 +1,10 @@
---
-description: "PipeJudge asks a judgment model one closed question about the evidence its prompt presents, and returns a verdict with a measure of how sure it is: yes or no, one option out of a set, or a level on a scale."
+description: "PipeJudge asks a judgment model closed questions about the evidence its prompt presents, and returns a verdict per question with a measure of how sure it is: yes or no, one option out of a set, or a level on a scale."
---
# PipeJudge
-The `PipeJudge` operator asks a judgment model one closed question about the evidence its prompt presents, and stores the verdict it answers with: a `YesNo`, a `Choice` or a `Rating`, each carrying the probabilities the model measured. A judgment model is built for this kind of question: it answers with a verdict and the probabilities it measured, where a language model asked the same question writes its answer out as text.
+The `PipeJudge` operator asks a judgment model a closed question about the evidence its prompt presents, and stores the verdict it answers with: a `YesNo`, a `Choice` or a `Rating`, each carrying the probabilities the model measured. It may also ask several questions about the same evidence in one request, and fill a structure with one verdict per question (see [Several questions in one request](#several-questions-in-one-request)). A judgment model is built for this kind of question: it answers with a verdict and the probabilities it measured, where a language model asked the same question writes its answer out as text.
## How it works
@@ -36,10 +36,58 @@ The output must be that native or a concept refining it, and `Dynamic` is neithe
**The verdict.** A yes/no question's verdict follows from its probability: yes when the probability of yes is at least the `threshold`, which defaults to `0.5`. A model that reports no probability gives its own verdict, and a declared threshold then cannot apply, which the run logs as a warning. A choice's verdict is the key of the option chosen, with the model's `confidence` and the `probabilities` of every option when it reports them. A rating's verdict is the index of the level chosen, `0` being the first level declared, with the model's `confidence`, the `probabilities` of every level keyed by its index written as text, and a continuous `position` on the scale, when it reports them. When the scale's levels carry labels, the verdict's `label` is the label of the level chosen, copied from the pipe's declaration.
-**A refusal fails the step.** A judgment model may decline to answer a question, and a refusal carries no verdict: it is never read as a no, as the first option or as the lowest level. The step fails with a [`JudgmentRefusedError`](../../../errors/judgment-refused-error.md) naming the pipe and the model; reword the question so it can be answered from the evidence, or give the step different evidence.
+**A refusal fails the step.** A judgment model may decline to answer a question, and a refusal carries no verdict: it is never read as a no, as the first option or as the lowest level. A pipe asking one question has nowhere to leave its verdict absent, so the step fails with a [`JudgmentRefusedError`](../../../errors/judgment-refused-error.md) naming the pipe and the model; reword the question so it can be answered from the evidence, or give the step different evidence. A pipe asking several questions may leave a verdict absent instead, as the next section explains.
The [native concepts page](../../concepts/native-concepts.md) describes the fields of the three verdicts. A downstream pipe reads them like any other fields, for example a [`PipeCondition`](../pipe-controllers/PipeCondition.md) routing on `routing.choice`, or one gating on `approved.probability`.
+## Several questions in one request
+
+A `PipeJudge` may ask several questions about the same evidence, written in `questions`, one table per question, each keyed by a name. The model answers every question in one request, each independently of the others, and the verdicts fill the output's structure, each in the field of its question's name:
+
+```toml
+[concept.MessageTriage]
+description = "What a triage of an inbox message found"
+
+[concept.MessageTriage.structure]
+urgent = { type = "concept", concept_ref = "YesNo", description = "Whether the message needs an answer today", required = true }
+team = { type = "concept", concept_ref = "Choice", description = "The team that should handle it", required = true }
+severity = { type = "concept", concept_ref = "Rating", description = "How severe the reported problem is" }
+
+[pipe.triage_message]
+type = "PipeJudge"
+description = "Triages a message: its urgency, its team and its severity"
+inputs = { message = "Text" }
+output = "MessageTriage"
+model = "@default-judgment"
+prompt = """
+A message from someone writing to the inbox:
+@message
+"""
+
+[pipe.triage_message.questions.urgent]
+question = "Does this message need an answer today?"
+criteria = { yes = "Something is broken or blocked for the sender right now", no = "The sender can wait for a reply" }
+threshold = 0.7
+
+[pipe.triage_message.questions.team]
+question = "Which team should handle this message?"
+options = { billing = "Charges, invoices, refunds and subscriptions", technical = "Errors, outages and features that do not work", other = "" }
+
+[pipe.triage_message.questions.severity]
+question = "How severe is the problem this message reports?"
+levels = [{ label = "Minor" }, { label = "Major" }, { label = "Blocking" }]
+```
+
+**Each question carries its own kind.** A question's table holds its `question` and the fields that decide its kind, `options`, `levels`, `criteria` and `threshold`, under the same rules as a pipe asking one question; no other field is allowed in it. The pipe sets exactly one of `question` and `questions`, and a pipe asking several sets none of the kind fields itself: they belong to the question they qualify. A question's name must be a valid field name: a Python identifier that is not a keyword, does not start with an underscore and is not a reserved name.
+
+**The output is a structure holding one verdict per question.** The output must be a concept with a structure, or a concept refining one, whose fields are exactly the question names: a question with no field to hold its verdict is refused when the method loads, and so is a field no question answers, since nothing could ever fill it. Each field is a `concept` field holding its question's verdict native, `YesNo`, `Choice` or `Rating`, or a concept refining it, and holds one verdict: a list field, a plain value such as a `boolean` and a concept of another kind are refused, each with a message naming the field and the question. The output itself carries no multiplicity; to judge each item of a list, map the `PipeJudge` over it with a [`PipeBatch`](../pipe-controllers/PipeBatch.md).
+
+**The questions share the evidence.** The prompt renders the evidence once, and every question is asked about it. A question may read a short parameter of its own, such as `$topic`, and every declared input must be read by the prompt or by at least one question.
+
+**A refusal leaves an optional field holding nothing.** When the model declines one question, the step leaves that question's field holding nothing if the field is optional, being neither `required` nor given a `default_value`, and logs a warning naming the question. When the field is `required` or carries a `default_value`, the step fails with a [`JudgmentRefusedError`](../../../errors/judgment-refused-error.md) naming the question, since a default filled in for it would read as a verdict the model never gave: reword the question, give the step different evidence, or make the field optional with no `default_value` so that a refusal leaves it holding nothing. Leaving a field optional with no default is how a method says that a refusal on that question is acceptable.
+
+**A question depending on another's verdict is a later step.** The questions of one `PipeJudge` are answered independently over the same evidence, so none of them can read another's verdict. When one question only makes sense once another is answered, such as rating a problem's severity only when the message reports a problem at all, ask the second in a later `PipeJudge` of a [`PipeSequence`](../pipe-controllers/PipeSequence.md), behind a [`PipeCondition`](../pipe-controllers/PipeCondition.md) routing on the first verdict when the second should not always be asked.
+
## Configuration
### The judgment model
@@ -52,16 +100,19 @@ No judgment model is served out of the box: the model deck names no default for
| ------------- | ---------- | ----------- | -------- |
| `type` | string | The type of the pipe: `PipeJudge`. | Yes |
| `description` | string | A description of the judgment. | Yes |
-| `inputs` | dictionary | The inputs, as a dictionary mapping input names to concept codes. Every input must be read by `prompt` or by `question`. | No |
-| `output` | string | `YesNo`, `Choice` or `Rating`, according to the kind of question, or a concept refining it. | Yes |
+| `inputs` | dictionary | The inputs, as a dictionary mapping input names to concept codes. Every input must be read by `prompt` or by a question. | No |
+| `output` | string | Asking one question: `YesNo`, `Choice` or `Rating`, according to the kind of question, or a concept refining it. Asking several: a concept with a structure whose fields are exactly the question names, each holding its question's verdict native or a concept refining it. | Yes |
| `model` | string | The judgment model, alias or preset, such as `"@default-judgment"`. Required unless your model deck names a default for judgments. | No |
| `prompt` | string | The evidence template: what the model judges. Rendered as a `PipeLLM` prompt, with the `@variable` and `$variable` shorthand and Jinja2, and presents the images and documents it reads as files. Must not be empty. | Yes |
-| `question` | string | The one closed question to ask about the evidence. A plain text template, which may interpolate a short parameter with the `$` prefix and reads no image or document. Must not be empty. | Yes |
+| `question` | string | The one closed question to ask about the evidence. A plain text template, which may interpolate a short parameter with the `$` prefix and reads no image or document. Must not be empty. | One of `question` or `questions` |
+| `questions` | dictionary | Several closed questions to ask about the evidence in one request, each keyed by the name of the output field holding its verdict. Each is a table holding its `question` and, for its kind, `options`, `levels`, `criteria` and `threshold`, as described below for a pipe asking one question. At least one question. | One of `question` or `questions` |
| `options` | dictionary | For a choice question: each key is an option, each value describes it. An empty string leaves an option undescribed. At least two options. | No |
| `levels` | list | For a rating question: the levels from lowest to highest, at least two. Each is a string describing the level, or a table carrying a `label`, a `description` or both. Either every level carries a label or none does, and labels are distinct. | No |
| `criteria` | dictionary | For a yes/no question: what a `yes` and what a `no` mean, both required. | No |
| `threshold` | number | For a yes/no question: the probability of yes at or above which the verdict is yes, strictly between 0 and 1. Defaults to `0.5`. | No |
+`options`, `levels`, `criteria` and `threshold` are set on the pipe when it asks one `question`, and in each question's table when it asks several `questions`, never on the pipe.
+
### Example: A yes/no question with a threshold
```toml
@@ -147,7 +198,7 @@ The model sees the rendered prompt, `The total of an invoice, in euros: 7420.0`,
## Writing good judgments
-- **Ask one narrow question per pipe.** A question that bundles two judgments ("is it urgent and about billing?") gets one verdict for both. Split it into two `PipeJudge` steps, and combine their verdicts downstream.
+- **Ask one narrow thing per question.** A question that bundles two judgments ("is it urgent and about billing?") gets one verdict for both. Split it into two questions of one `PipeJudge` when they share the evidence and neither depends on the other's verdict, and into two `PipeJudge` steps of a `PipeSequence` when one does.
- **Keep the evidence focused.** The judgment vendors advise that a large evidence the question does not need degrades the answers, so render in the prompt what the question is about and leave the rest out: a dotted reference such as `$invoice.total` reads one field, where `@invoice` sets out the whole value.
- **Write criteria for both answers.** `criteria` declares what a `yes` and what a `no` mean, and both are required, so a table with one side or none is refused: when one side is the plain opposite of the other, write it as its complement, "The amount is five thousand euros or less" beside "The amount is above five thousand euros", and leave `criteria` out when the question needs none.
- **Make levels describe situations, not degrees.** "Users cannot do their work at all" gives the model something to recognize, where "Very severe" only restates the scale; a label names the level in a few words, and the description says what it stands for. A long scale of thin levels buys resolution the model cannot supply, and a scale holds at most ten levels: TypeSafe's model refuses more, and a longer scale fails the step before the model is called.
diff --git a/docs/building-methods/pipes/pipe-operators/index.md b/docs/building-methods/pipes/pipe-operators/index.md
index fdda61c25..c6af9811c 100644
--- a/docs/building-methods/pipes/pipe-operators/index.md
+++ b/docs/building-methods/pipes/pipe-operators/index.md
@@ -20,7 +20,7 @@ Here are the primary pipe operators available in Pipelex:
- [**`PipeExtract`**](./PipeExtract.md): Performs Optical Character Recognition (OCR) on images and PDF documents to extract text and embedded images, and fetches and extracts content from web pages.
- [**`PipeImgGen`**](./PipeImgGen.md): Generates images from a text prompt using models like GPT Image, Flux, or other image generation models.
- [**`PipeSearch`**](./PipeSearch.md): Searches the web using a configurable search provider and returns structured results with an answer and source citations.
-- [**`PipeJudge`**](./PipeJudge.md): Asks a judgment model one closed question about the evidence its prompt presents, and returns a `YesNo`, a `Choice` or a `Rating` verdict with the probabilities the model measured.
+- [**`PipeJudge`**](./PipeJudge.md): Asks a judgment model a closed question about the evidence its prompt presents, and returns a `YesNo`, a `Choice` or a `Rating` verdict with the probabilities the model measured, or asks several in one request and fills a structure with their verdicts.
- [**`PipeFunc`**](./PipeFunc.md): An escape hatch that allows you to execute any custom Python function, giving you maximum flexibility.
- [**`PipeCompose`**](./PipeCompose.md): Composes outputs deterministically from working memory — renders Jinja2 templates for formatted reports or complex prompts, or constructs structured objects by mapping fields from inputs, without an LLM.
- [**`PipeDocGen`**](./PipeDocGen.md): Generates a document file from its inputs, calling no AI model: a PDF laid out from a structure or formatted from a Markdown report, and with the Pipelex document generation plugin a PDF from an HTML template, an Excel workbook, a Word document or a PowerPoint deck.
@@ -121,11 +121,12 @@ Searches the web and returns structured results with sources.
## PipeJudge
-Asks a judgment model one closed question about the evidence its prompt presents.
+Asks a judgment model closed questions about the evidence its prompt presents.
### Key Features
- Yes/no, choice and rating questions, the kind read from the fields the pipe declares
+- Several questions in one request, their verdicts filling the fields of a structure named after them
- Verdicts carrying the probabilities the model measured, and a threshold for a yes/no
- The evidence written as a prompt template, rendered as a `PipeLLM`'s prompt is, its images and documents sent as files to a model that reads them
- Two-sided criteria for a yes/no, and rating levels that may carry labels, which the verdict reports
diff --git a/docs/contribute/inference-family-surface.md b/docs/contribute/inference-family-surface.md
index 44257b317..03167d2ad 100644
--- a/docs/contribute/inference-family-surface.md
+++ b/docs/contribute/inference-family-surface.md
@@ -34,7 +34,7 @@ Most of the list is enforced by the type checker: the matches over the family en
12. **Files, when the family reads them.** The job and the assignment carry them inside the prompt that numbers them, as `JudgmentJob` and `JudgmentAssignment` carry a `JudgmentPrompt` whose `images` and `documents` are the files its `text` marks with `[Image N]` and `[Document N]` tokens, and the assignment's `referenced_uris()` returns each one, or a scoped run's read authorization never sees them. A family whose operator renders a template shapes its contract around the rendered prompt rather than the step's inputs: the operator assembles it through `pipelex/kernel/prompt_assembly.py`, which PipeLLM, PipeImgGen and PipeJudge share, so the worker receives one text and its ordered files and never reads an input by name. The worker's template method checks that the model reads them before its hook runs, raising the family's capability error (`JudgmentWorkerAbstract._check_can_read_files`), and runs the per-file format checks in `pipelex/cogt/inference/prompt_file_checks.py`, which the LLM family shares. What a model reads is the `inputs` of its spec in the backend's model file, so the capability is configuration, and an operator that knows its model at load time refuses a file input there too, as `PipeJudge` does.
-13. **Outcomes the worker may return besides an answer.** A model may decline a question, and the worker reports that as an outcome rather than as an error or a made-up answer: `judge` returns a `JudgmentOutcome` per question, an answer of the question's kind or a `JudgmentRefusal`, and the base checks an answer's key, kind and offered option while accepting a refusal for any question it was asked. What a refusal means is the caller's policy, never the worker's: the kernel's single-question judgment raises `JudgmentRefusedError`, a content error naming the step and the model, and never reads a refusal as a no, a first option or a lowest level.
+13. **Outcomes the worker may return besides an answer.** A model may decline a question, and the worker reports that as an outcome rather than as an error or a made-up answer: `judge` returns a `JudgmentOutcome` per question, an answer of the question's kind or a `JudgmentRefusal`, and the base checks an answer's key, kind and offered option while accepting a refusal for any question it was asked. What a refusal means is the caller's policy, never the worker's: the kernel's single-question judgment raises `JudgmentRefusedError`, a content error naming the step and the model, its several-question judgment leaves an optional field absent and raises the same error naming the question behind a required one, and neither reads a refusal as a no, a first option or a lowest level.
14. **Silent: a default nothing serves boots cleanly and fails at run time.** A boot checks only that each alias a deck names exists, not that an enabled backend serves the model it resolves to, so a `default-*` alias pointing at a model no backend declares loads without complaint and fails the first pipe that relies on it with a `ModelNotFoundError`, for everyone running on their own keys. `tests/unit/pipelex/kit/test_shipped_deck_defaults.py` is the guard: every `default-*` alias of the shipped deck must reach, through aliases, presets and every entry of a waterfall, a model some backend file of the kit declares. Give a new family's defaults such a model, or ship no default, as `judgment` ships no `choice_default`.
diff --git a/docs/errors/judgment-refused-error.md b/docs/errors/judgment-refused-error.md
index 35a3a16ac..37d0369a3 100644
--- a/docs/errors/judgment-refused-error.md
+++ b/docs/errors/judgment-refused-error.md
@@ -7,7 +7,7 @@ description: "Reference for the `JudgmentRefusedError` Pipelex error class."
# Judgment refused
-The judging model declined to answer a step's question.
+The judging model declined to answer a step's question, whose output cannot hold nothing.
| Field | Value |
|---|---|
diff --git a/docs/features/judgment.md b/docs/features/judgment.md
index b84d4b7c8..2537f8e5b 100644
--- a/docs/features/judgment.md
+++ b/docs/features/judgment.md
@@ -16,10 +16,11 @@ Many steps of a method come down to a closed question: is this message urgent, w
- **Three kinds of question** — A yes/no question with criteria for both answers, a choice among declared options, or a rating on a scale of declared levels, each with a label, a description or both, the kind read from the fields the pipe declares
- **Measured probabilities** — Each verdict carries the probabilities the model measured, so a method can act on how sure it is as well as on what it decided
- **Thresholds** — A yes/no question can declare the probability at or above which its verdict is yes
+- **Several questions in one request** — A step can ask several questions about the same evidence, each of its own kind, and fill a structure with one verdict per question, each in the field of its question's name
- **The evidence is a prompt** — The model judges what the step's prompt renders and nothing else, so the author decides what it sees, down to one field of a larger value; the images and documents the prompt reads are sent as files to a model that reads them
- **Labels on a rating** — When the scale's levels carry labels, the verdict carries the label of the level chosen, copied from the declaration
-- **No verdict from a refusal** — A model that declines to answer fails the step with an error naming the pipe and the model, rather than a verdict read off its silence
-- **Checked when the method loads** — A step with no model, an input neither its prompt nor its question reads, an output that disagrees with its kind of question, or a file its prompt reads that its model does not is refused before any run spends anything
+- **No verdict from a refusal** — A model that declines to answer fails the step with an error naming the pipe and the model, rather than a verdict read off its silence; a step asking several questions leaves the declined question's field holding nothing instead when that field is optional with no default, and fails when it is required or carries a `default_value`
+- **Checked when the method loads** — A step with no model, an input neither its prompt nor its questions read, an output that disagrees with its kind of question or a structure whose fields are not exactly its questions, or a file its prompt reads that its model does not is refused before any run spends anything
## Usage in Pipelines
diff --git a/docs/features/pipe-operators.md b/docs/features/pipe-operators.md
index ff5f0ab1f..1da7fb6fd 100644
--- a/docs/features/pipe-operators.md
+++ b/docs/features/pipe-operators.md
@@ -43,7 +43,7 @@ See [PipeSearch reference](../building-methods/pipes/pipe-operators/PipeSearch.m
## PipeJudge
-One closed question asked of a judgment model about the evidence the step's prompt renders: yes or no, one option out of a set, or a level on a scale. The verdict comes back as a typed `YesNo`, `Choice` or `Rating` carrying the probabilities the model measured, ready to route a `PipeCondition` or gate a step on a threshold.
+A closed question asked of a judgment model about the evidence the step's prompt renders: yes or no, one option out of a set, or a level on a scale. The verdict comes back as a typed `YesNo`, `Choice` or `Rating` carrying the probabilities the model measured, ready to route a `PipeCondition` or gate a step on a threshold. A step may also ask several such questions about the same evidence in one request, and fill a structure with one verdict per question.
See [PipeJudge reference](../building-methods/pipes/pipe-operators/PipeJudge.md).
diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.md
index 2605e099c..cf8aea344 100644
--- a/docs/features/validation-dry-run.md
+++ b/docs/features/validation-dry-run.md
@@ -31,7 +31,7 @@ A pipe that references a sub-pipe from a package that isn't loaded is reported a
Execute pipelines with mocked LLM responses to test pipeline logic, data flow, and orchestration without making API calls.
- **Mock generation** — Format-compliant mock values for constrained fields, including structured outputs
-- **Judgments rendered, not answered** — A dry `PipeJudge` renders its evidence prompt and its question as a live run does, then answers with a mock verdict of the right kind carrying no probability; on a scale whose levels carry labels, the mock level carries its declared label
+- **Judgments rendered, not answered** — A dry `PipeJudge` renders its evidence prompt and its questions as a live run does, then answers each with a mock verdict of the right kind carrying no probability; on a scale whose levels carry labels, the mock level carries its declared label
- **Configurable mock behavior** — Control mock list sizes, template handling, and response formats
- **Full pipeline execution** — Working memory, controllers, and data flow all work as in production
- **No credentials needed** — A dry run boots without inference credentials, on both `pipelex run --dry-run` and `pipelex-agent run --dry-run`, and so does `pipelex validate`. Such a boot loads every enabled backend with all its models but resolves none of their keys, so validation and dry runs give the same verdict on a machine that holds no key at all as on one that holds them all: presets, the default models and a bare handle such as `model = "gpt-4o-mini"` resolve alike
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index 5a3644101..3e5a1ddd6 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -483,6 +483,7 @@ What varies goes in `fields`, named by the [naming convention](#naming-conventio
| `item_count` | integer | The number of items a `PipeBatch` fans out over |
| `max_concurrency` | integer | The bound on the branches a `PipeBatch` runs at once; absent when the configuration's `"unbounded"` sets none |
| `threshold` | number | The probability threshold a judgment declares, between `0` and `1` |
+| `judgment_question` | string | A question of a PipeJudge asking several, by the name its `questions` table gives it: `urgent` |
| `template_file` | string | A PipeDocGen's template file, as the pipe declares it |
| `finding_message`, `finding_location` | string | The text of a PipeDocGen template finding, and where in the template it was found |
| `function_name` | string | A function PipeFunc registers or calls, by its name |
diff --git a/docs/under-the-hood/pipelex-kernel.md b/docs/under-the-hood/pipelex-kernel.md
index e3d3de6b7..f4ac015dd 100644
--- a/docs/under-the-hood/pipelex-kernel.md
+++ b/docs/under-the-hood/pipelex-kernel.md
@@ -76,7 +76,7 @@ Both façade calls take the concept and the output class the caller wants, defau
| `pipelex.kernel.extract_ops` | `resolve_extract_setting`, `build_extract_job_params`, `run_extract` |
| `pipelex.kernel.img_gen_ops` | `resolve_img_gen_setting`, `resolve_default_size`, `build_img_gen_job_params`, `run_img_gen` |
| `pipelex.kernel.search_ops` | `resolve_search_setting`, `run_search` |
-| `pipelex.kernel.judgment_ops` | `judgment_setting_of_choice`, `served_judgment_model`, `resolve_judgment_setting`, `make_verdict_content`, `run_judgment` |
+| `pipelex.kernel.judgment_ops` | `judgment_setting_of_choice`, `served_judgment_model`, `resolve_judgment_setting`, `make_verdict_content`, `run_judgment`, `AskedQuestion`, `run_multi_judgment` |
| `pipelex.kernel.compose_ops` | `build_compose_context`, `build_composed_content`, `run_compose_template` |
| `pipelex.kernel.func_ops` | `call_registered_function`, `run_func` |
| `pipelex.kernel.memory_ops` | `shape_inputs`, `store_result`, `extract_main_content` / `extract_named_content`, `extract_main_content_as_list` / `extract_named_content_as_list` |
@@ -90,9 +90,11 @@ The two operator-specific `assemble_*` functions are there because `run_llm_text
`served_llm_model` gives the spec of the LLM a handle resolves to on this boot, an alias to its target and a waterfall to its first served member as a run resolves them, or `None` when no backend serves one. `check_llm_setting_with_served_model` refuses an LLM setting the model it resolves to refuses, before anything is spent: it runs the request check the model's backend registered beside its worker factory, which is the check the worker itself runs before every call, against the job params the setting gives once the model's constraints apply, for a text or a structured output. For a built-in backend it raises an `LLMSettingRefusedError`, an `LLMCapabilityError` carrying the worker's reason with the model named by its deck handle rather than by the SDK, backend and provider model id the worker's own refusal names, so its message is shown to a caller even under strict error disclosure. An external plugin's refusal is raised as the plugin wrote it, and its message reaches a caller only when the plugin raised it as caller-facing copy. A backend that registered no check is held to the rule every worker shares, that a model whose spec declares `thinking_mode = "none"` takes no reasoning setting, and a model no backend serves, or whose backend's SDK is not installed, is left to the run. The interpreter's `PipeLLM` and `PipeStructure` call it when a method loads; a programmatic caller calls it before `run_llm_text` or `run_llm_object` to get the same refusal without a call.
-`run_judgment` asks one question about the evidence of a prompt. It takes the step's evidence template as a `UserPromptContent`, the same carrier a PipeLLM's user prompt travels in, and assembles it through `assemble_user_prompt`, so the evidence is rendered, and its images and documents numbered into `[Image N]` and `[Document N]` tokens, exactly as a PipeLLM's prompt is; the text and the ordered files become the `JudgmentPrompt` the worker receives. The question arrives as the cogt question with its instructions still a template, rendered against memory as a `BASIC` template, and is sent as the job's only question, under one fixed key. An optional input that holds no value renders as the template's guard says, through `@?name` or a Jinja2 test, and the shared assembly skips its files, as it does for every operator. A refusal is a worker outcome like an answer, and `run_judgment`'s policy for it is to fail: one question has nowhere to leave its verdict absent, so it raises `JudgmentRefusedError`, a content error naming the step and the model. The `JudgmentResult` carries the prompt sent and the rendered question beside the verdict, which is what the operator records as its execution data. `make_verdict_content` turns the answer into the verdict native: a yes/no answer's probability decides the verdict against the step's threshold, inclusive, or against 0.5 when none is declared; an answer with no probability keeps the model's own verdict, and when a threshold was declared a live run logs a warning and its `JudgmentResult` reports `threshold_applied` as false. A rating takes the label of the level the answer names off the scale the question declared, since no backend reports one, a dry run included, and its distribution is keyed by the level index written as text. `judgment_setting_of_choice` gives the setting of the step's model, else of the deck's default, and raises `JudgmentModelMissingError` when neither names one, since the deck serves none by default. `served_judgment_model` gives the spec of the model a handle resolves to on this boot, or `None` when no backend serves one, a waterfall none of whose models is served included. `resolve_judgment_setting` then pins the setting to the handle the model resolves to; on a dry run, a model no backend serves on this boot keeps the deck's handle, since no judging worker is called.
+`run_judgment` asks one question about the evidence of a prompt. It takes the step's evidence template as a `UserPromptContent`, the same carrier a PipeLLM's user prompt travels in, and assembles it through `assemble_user_prompt`, so the evidence is rendered, and its images and documents numbered into `[Image N]` and `[Document N]` tokens, exactly as a PipeLLM's prompt is; the text and the ordered files become the `JudgmentPrompt` the worker receives. The question arrives as the cogt question with its instructions still a template, rendered against memory as a `BASIC` template, and is sent as the job's only question, under one fixed key. An optional input that holds no value renders as the template's guard says, through `@?name` or a Jinja2 test, and the shared assembly skips its files, as it does for every operator. A refusal is a worker outcome like an answer, and `run_judgment`'s policy for it is to fail: one question has nowhere to leave its verdict absent, so it raises `JudgmentRefusedError`, a content error naming the step and the model. The `JudgmentResult` carries the prompt sent and the rendered question beside the verdict, which is what the operator records as its execution data. `run_multi_judgment` asks several questions about one evidence, in one request, and fills a structure with their verdicts. The evidence is assembled and each question rendered as `run_judgment` does, and the questions travel as `AskedQuestion`s, each a cogt question beside the threshold its yes/no verdict reads its probability against, keyed by name in one job. A question's name is also the name of the field holding its verdict in `output_class`, the output's content class, which the caller resolves and hands down, as `run_llm_object` takes its output class, since the kernel does not read the concept library; a class whose fields are not exactly the questions' names is refused with `JudgmentOutputFieldsError`, a `ValueError` raised before the judging model is called. Each answer becomes its verdict native under `make_verdict_content`'s rules, and the verdicts are handed to the class as plain data keyed by field name, whatever alias a field carries, so a field typed by a concept refining the native is built as that concept's class. A refusal stores nothing in its question's field when that field may hold nothing, by `class_field_may_hold_nothing` (`pipelex.core.stuffs.non_null_any`), the rule the load's concept walk reads a class field by too: its type admits `None`, whether it is required or has a default, or it is neither required nor given a default. A field whose type admits `None` is given it outright, so a required one validates and a defaulted one does not fill in its default. The refusal is logged as a warning naming the question and recorded in the result; behind any other field it raises `JudgmentRefusedError`, which then names the question too, since leaving a defaulted field out would fill in its default, a verdict nobody gave. The `MultiJudgmentResult` carries the prompt sent and, per question, a `QuestionJudgment`: the rendered question, the raw outcome, a refusal included, and whether a threshold decided the verdict. A dry run answers every question and never refuses. `make_verdict_content` turns the answer into the verdict native: a yes/no answer's probability decides the verdict against the step's threshold, inclusive, or against 0.5 when none is declared; an answer with no probability keeps the model's own verdict, and when a threshold was declared a live run logs a warning and its `JudgmentResult` reports `threshold_applied` as false. A rating takes the label of the level the answer names off the scale the question declared, since no backend reports one, a dry run included, and its distribution is keyed by the level index written as text. `judgment_setting_of_choice` gives the setting of the step's model, else of the deck's default, and raises `JudgmentModelMissingError` when neither names one, since the deck serves none by default. `served_judgment_model` gives the spec of the model a handle resolves to on this boot, or `None` when no backend serves one, a waterfall none of whose models is served included. `resolve_judgment_setting` then pins the setting to the handle the model resolves to; on a dry run, a model no backend serves on this boot keeps the deck's handle, since no judging worker is called.
-**Every entry point that renders a template takes a required `templating_style`** — `run_llm_text`, `run_llm_object`, `derive_structure_prompt`, `assemble_user_prompt`, `assemble_llm_prompt`, `assemble_img_gen_prompt`, `run_search`, `run_judgment`, `run_compose_template`. It is not nullable, and that is the contract: the Jinja2 filters that tag and format a value have no default of their own, so a call that omitted the style would render a prompt in a shape nobody chose. `resolve_templating_style(authored=...)` is what produces one — pass what the caller authored, or `None` to take the runtime default.
+A PipeJudge records these results as its execution data, kept per question. A step asking one question records its rendered question, its kind as text, its threshold, whether the threshold decided the verdict and the model's raw answer at the top level, beside the rendered prompt and the resolved model. A step asking several records the rendered prompt and the model there too, and nests the rest under `questions`, keyed by each question's name, as `rendered_question`, `judgment_kind`, `threshold`, `threshold_applied` and `outcome`, the outcome being the raw answer or the refusal, since a field holding nothing shows nothing of what the model said.
+
+**Every entry point that renders a template takes a required `templating_style`** — `run_llm_text`, `run_llm_object`, `derive_structure_prompt`, `assemble_user_prompt`, `assemble_llm_prompt`, `assemble_img_gen_prompt`, `run_search`, `run_judgment`, `run_multi_judgment`, `run_compose_template`. It is not nullable, and that is the contract: the Jinja2 filters that tag and format a value have no default of their own, so a call that omitted the style would render a prompt in a shape nobody chose. `resolve_templating_style(authored=...)` is what produces one — pass what the caller authored, or `None` to take the runtime default.
There are **no re-exports**: `pipelex/kernel/__init__.py` holds doctrine and nothing else, and every symbol is imported from the module that defines it. For this package that is a layering property rather than a style one — a module that re-exports across layers is a layer boundary with the sign filed off.
@@ -173,7 +175,7 @@ Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`,
A record emitted inside a `log.context` block carries the run's identifiers — `request_id`, `pipeline_run_id` and `pipe_run_id` — as fields a sink can filter on ([Logging](../tools/logging.md#the-run-scoped-context)). The interpreter binds them around a run and around each pipe; a program driving the kernel has no interpreter around it, so the kernel binds them itself, split the same way as the usage lifecycle above.
-- **Each step binds itself.** Every kernel function that takes a `job_metadata` — `run_llm_text`, `run_llm_object`, `generate_object_content`, `run_extract`, `run_search`, `run_judgment` and `run_img_gen` — opens its body with `with job_metadata.log_context():`, so every line it emits, from prompt assembly to the memory write-back and any failure raised on the way, names the run and the step. A unit test sweeps `pipelex/kernel/` and fails on a function that takes a `job_metadata` without opening its body that way, so the rule holds for the next function added as well.
+- **Each step binds itself.** Every kernel function that takes a `job_metadata` — `run_llm_text`, `run_llm_object`, `generate_object_content`, `run_extract`, `run_search`, `run_judgment`, `run_multi_judgment` and `run_img_gen` — opens its body with `with job_metadata.log_context():`, so every line it emits, from prompt assembly to the memory write-back and any failure raised on the way, names the run and the step. A unit test sweeps `pipelex/kernel/` and fails on a function that takes a `job_metadata` without opening its body that way, so the rule holds for the next function added as well.
- **The host binds the run.** `PipelexKernel.log_context()` binds `request_id` and `pipeline_run_id` and no step, and a host wraps its run in it. The kernel cannot do this for the host, for the reason it cannot own the usage lifecycle: a kernel call has no run boundary to hang the binding on. Inside the host's block, each step's binding merges over the run's for the length of the call, so a line the host emits between two calls names the run, and a line emitted during one names the step as well.
```python
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index 200b3be7b..aa9fa1184 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -506,30 +506,48 @@ class JudgmentModelNotFoundError(ModelNotFoundError):
class JudgmentRefusedError(CogtError):
- """The judging model declined to answer a step's question.
+ """The judging model declined to answer a step's question, whose output cannot hold nothing.
A refusal is a worker's outcome, never its error: this is the operator's policy for a question
- whose verdict has nowhere to be left absent. It is a content error, in the input domain, because
- what a model declines to judge is the question asked over the evidence given, and the remedy is
- the author's or the caller's: a reworded question or different evidence. The message names only
- the step and the model's deck handle, so it is kept verbatim for the caller.
+ whose output cannot hold nothing, the one question of a single-question step or a question of
+ several whose output field is required or defaulted. It is a content error, in the input domain,
+ because what a model declines to judge is the question asked over the evidence given, and the
+ remedy is the author's or the caller's: a reworded question, different evidence, or, for a question
+ of several, an optional field with no default, which holds nothing on a refusal. The message names only the step, the
+ model's deck handle and the question's name, which the author wrote, so it is kept verbatim for the
+ caller.
"""
error_category = InferenceErrorCategory.CONTENT
_authors_caller_facing_message = True
- def __init__(self, *, pipe_code: str | None, model_handle: str):
+ def __init__(self, *, pipe_code: str | None, model_handle: str, question_name: str | None = None):
step = f"PipeJudge '{pipe_code}'" if pipe_code else "This judgment"
- message = (
- f"{step}: the judgment model '{model_handle}' declined to answer its question. "
- "Try to reword the question so it can be answered from the evidence, or give the step different evidence."
- )
+ message: str
+ user_action_detail: str
+ if question_name is None:
+ message = (
+ f"{step}: the judgment model '{model_handle}' declined to answer its question. "
+ "Try to reword the question so it can be answered from the evidence, or give the step different evidence."
+ )
+ user_action_detail = "Reword the question, or give the step different evidence."
+ else:
+ message = (
+ f"{step}: the judgment model '{model_handle}' declined to answer the question '{question_name}', and its output field "
+ "cannot hold nothing, being required or given a default. Try to reword the question so it can be answered from the "
+ "evidence, give the step different evidence, or make its output field optional with no default, so that it holds "
+ "nothing on a refusal."
+ )
+ user_action_detail = (
+ "Reword the question, give the step different evidence, or make the question's output field optional with no default."
+ )
super().__init__(
message,
- user_action=UserAction(kind=UserActionKind.CHANGE_INPUT, detail="Reword the question, or give the step different evidence."),
+ user_action=UserAction(kind=UserActionKind.CHANGE_INPUT, detail=user_action_detail),
)
self.pipe_code = pipe_code
self.model_handle = model_handle
+ self.question_name = question_name
class JudgmentModelMissingError(PipelexError):
diff --git a/pipelex/core/concepts/concept_blueprint.py b/pipelex/core/concepts/concept_blueprint.py
index 707fbbc79..58fe61141 100644
--- a/pipelex/core/concepts/concept_blueprint.py
+++ b/pipelex/core/concepts/concept_blueprint.py
@@ -1,9 +1,13 @@
-import keyword
from typing import Any, cast
from pydantic import BaseModel, ConfigDict, SerializerFunctionWrapHandler, field_validator, model_serializer, model_validator
-from pipelex.core.concepts.concept_structure_blueprint import RESERVED_FIELD_NAMES, ConceptStructureBlueprint
+from pipelex.core.concepts.concept_structure_blueprint import (
+ RESERVED_FIELD_NAMES,
+ ConceptStructureBlueprint,
+ FieldNameRefusal,
+ structure_field_name_refusal,
+)
from pipelex.core.concepts.validation import is_concept_ref_or_code_valid
ConceptStructureBlueprintType = str | ConceptStructureBlueprint
@@ -77,7 +81,10 @@ class ConceptBlueprint(BaseModel):
cls, structure: str | dict[str, ConceptStructureBlueprintType] | None
) -> str | dict[str, ConceptStructureBlueprintType] | None:
if isinstance(structure, dict):
- invalid_identifier_fields = [field_name for field_name in structure if not field_name.isidentifier() or keyword.iskeyword(field_name)]
+ # One rule for a field name, shared with every name that stands for a field: each name is refused by
+ # the first rule it breaks, and each rule keeps its own message.
+ refusals = {field_name: structure_field_name_refusal(field_name=field_name) for field_name in structure}
+ invalid_identifier_fields = [field_name for field_name, refusal in refusals.items() if refusal is FieldNameRefusal.NOT_AN_IDENTIFIER]
if invalid_identifier_fields:
invalid_list = ", ".join(f"'{name}'" for name in sorted(invalid_identifier_fields))
msg = (
@@ -87,7 +94,7 @@ class ConceptBlueprint(BaseModel):
raise ValueError(msg)
# Check for reserved field names
- reserved_fields_used = [field_name for field_name in structure if field_name in RESERVED_FIELD_NAMES]
+ reserved_fields_used = [field_name for field_name, refusal in refusals.items() if refusal is FieldNameRefusal.RESERVED]
if reserved_fields_used:
fields_word = "field" if len(reserved_fields_used) == 1 else "fields"
reserved_fields_used_list = ", ".join(f"'{name}'" for name in sorted(reserved_fields_used))
@@ -100,7 +107,7 @@ class ConceptBlueprint(BaseModel):
raise ValueError(msg)
# Check for field names starting with underscore (reserved for internal use)
- underscore_fields = [field_name for field_name in structure if field_name.startswith("_")]
+ underscore_fields = [field_name for field_name, refusal in refusals.items() if refusal is FieldNameRefusal.UNDERSCORE]
if underscore_fields:
fields_word = "field" if len(underscore_fields) == 1 else "fields"
underscore_list = ", ".join(f"'{name}'" for name in sorted(underscore_fields))
diff --git a/pipelex/core/concepts/concept_structure_blueprint.py b/pipelex/core/concepts/concept_structure_blueprint.py
index 7cb647fef..031eb90c1 100644
--- a/pipelex/core/concepts/concept_structure_blueprint.py
+++ b/pipelex/core/concepts/concept_structure_blueprint.py
@@ -1,3 +1,4 @@
+import keyword
from datetime import date, datetime, time
from enum import StrEnum
from typing import Any, Self
@@ -47,6 +48,40 @@ class ConceptStructureBlueprintFieldType(StrEnum):
CONCEPT = "concept"
+class FieldNameRefusal(StrEnum):
+ """Why a concept structure refuses a name for a field, by the first rule it breaks."""
+
+ # Not a Python identifier, or a Python keyword.
+ NOT_AN_IDENTIFIER = "not_an_identifier"
+ # One of `RESERVED_FIELD_NAMES`, an underscore-led pipe run parameter key included.
+ RESERVED = "reserved"
+ # Starts with an underscore, which is reserved for internal use.
+ UNDERSCORE = "underscore"
+
+
+def structure_field_name_refusal(*, field_name: str) -> FieldNameRefusal | None:
+ """Why a concept structure refuses this name for a field, or `None` when it admits it.
+
+ The one rule for a field name, which `ConceptBlueprint` applies to its `structure` keys: a Python
+ identifier that is not a Python keyword, is none of `RESERVED_FIELD_NAMES` and does not start with
+ an underscore. The rules are checked in that order, so a reserved name that starts with an
+ underscore is refused as reserved. A name that elsewhere stands for a field, such as the key of a
+ PipeJudge's `questions`, is held to the same rule.
+ """
+ if not field_name.isidentifier() or keyword.iskeyword(field_name):
+ return FieldNameRefusal.NOT_AN_IDENTIFIER
+ if field_name in RESERVED_FIELD_NAMES:
+ return FieldNameRefusal.RESERVED
+ if field_name.startswith("_"):
+ return FieldNameRefusal.UNDERSCORE
+ return None
+
+
+def is_admitted_structure_field_name(*, field_name: str) -> bool:
+ """Whether a concept structure admits this name for a field, by `structure_field_name_refusal`'s rule."""
+ return structure_field_name_refusal(field_name=field_name) is None
+
+
def field_may_hold_nothing(*, required: bool, default_value: Any) -> bool:
"""Whether a structure field's value may hold nothing: by the standard's rule, only when it is neither `required` nor defaulted."""
return not required and default_value is None
diff --git a/pipelex/core/stuffs/non_null_any.py b/pipelex/core/stuffs/non_null_any.py
index 4d0646f07..2897c860f 100644
--- a/pipelex/core/stuffs/non_null_any.py
+++ b/pipelex/core/stuffs/non_null_any.py
@@ -39,3 +39,17 @@ def field_admits_none(*, field_info: FieldInfo) -> bool:
"""Whether a class field admits `None`, read off its annotation, `NonNullAny` being the one `Any` that does not."""
is_non_null_any = any(isinstance(constraint, AfterValidator) and constraint.func is _refuse_none for constraint in field_info.metadata)
return annotation_admits_none(annotation=field_info.annotation) and not is_non_null_any
+
+
+def class_field_may_hold_nothing(*, field_info: FieldInfo) -> bool:
+ """Whether a structure class's field may hold nothing: its annotation admits `None`, or it is neither required nor defaulted.
+
+ The standard's rule, `field_may_hold_nothing`, read off a class rather than a structure table: a class states
+ what a field may hold through its annotation, so one admitting `None` may hold nothing whether it is required
+ or has a default, and so may one left unrequired with no default. A generated class spells a field that may
+ not hold nothing so that this reads the same as its table: a required or defaulted field refuses `None`.
+
+ An unrequired field with no default is the only kind whose default is `None`: a required field has no
+ default at all, and a field given a default factory has none of its own either.
+ """
+ return field_admits_none(field_info=field_info) or field_info.default is None
diff --git a/pipelex/kernel/exceptions.py b/pipelex/kernel/exceptions.py
index 45e9ad60c..e6c9fc0dd 100644
--- a/pipelex/kernel/exceptions.py
+++ b/pipelex/kernel/exceptions.py
@@ -5,3 +5,14 @@ class PromptContentError(ValueError):
raised before it moved into the kernel: callers that already handle it as a value error keep
working, and nothing in the error taxonomy changes.
"""
+
+
+class JudgmentOutputFieldsError(ValueError):
+ """A judgment asking several questions was handed an output class whose fields are not exactly its questions' names.
+
+ Each question's verdict fills the field of its name, so a question with no field would be answered for
+ nothing and a field no question answers could never be filled. A `PipeJudge` has its output checked
+ against its questions when its method loads, so this is a programmatic caller's contract broken, and
+ it is raised before the judging model is called. A `ValueError`, as `PromptContentError` is: the
+ caller handed the kernel a value it cannot work with.
+ """
diff --git a/pipelex/kernel/judgment_ops.py b/pipelex/kernel/judgment_ops.py
index 46baec716..68316783e 100644
--- a/pipelex/kernel/judgment_ops.py
+++ b/pipelex/kernel/judgment_ops.py
@@ -10,8 +10,16 @@ user-prompt assembly (`pipelex.kernel.prompt_assembly`). What the evidence holds
business, so an input reaches the judging model only when the prompt reads it. An optional input the run
was not given renders as the template's guard says, and the assembly skips its files, numbering and
handing over none of them.
+
+A step asks one question, `run_judgment`, whose verdict native is the result, or several,
+`run_multi_judgment`, whose verdicts fill a structure field by field. Either way it is one request to the
+judging model, every question answered independently over the same evidence.
"""
+from typing import NamedTuple
+
+from pydantic import BaseModel, ConfigDict
+
from pipelex import log
from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment
from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams
@@ -19,6 +27,7 @@ from pipelex.cogt.exceptions import JudgmentAnswerMismatchError, JudgmentModelMi
from pipelex.cogt.judgment.judgment_models import (
ChoiceAnswer,
JudgmentAnswer,
+ JudgmentOutcome,
JudgmentPrompt,
JudgmentQuestion,
JudgmentRefusal,
@@ -34,10 +43,12 @@ from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.concepts.concept import Concept
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.non_null_any import class_field_may_hold_nothing, field_admits_none
from pipelex.core.stuffs.rating_content import RatingContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.yes_no_content import YesNoContent
-from pipelex.kernel.judgment_results import JudgmentResult
+from pipelex.kernel.exceptions import JudgmentOutputFieldsError
+from pipelex.kernel.judgment_results import JudgmentResult, MultiJudgmentResult, QuestionJudgment
from pipelex.kernel.memory_ops import store_result
from pipelex.kernel.prompt_assembly import UserPromptContent, assemble_user_prompt
from pipelex.runtime_hub import get_content_generator, get_model_deck
@@ -53,6 +64,28 @@ JUDGMENT_QUESTION_KEY = "question"
DEFAULT_YES_NO_THRESHOLD = 0.5
+class AskedQuestion(BaseModel):
+ """One question of a judgment asking several, with the threshold its yes/no verdict reads its probability against.
+
+ The question's instructions are still the authored template, rendered against memory when the step
+ runs. The threshold is the operator's policy, so it travels beside the question rather than in it,
+ and a question of another kind never carries one.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ question: JudgmentQuestion
+ threshold: float | None = None
+
+
+class _AskedJudgment(NamedTuple):
+ """What one request to the judging model was and returned: the evidence, the questions as asked, and their outcomes by key."""
+
+ prompt: JudgmentPrompt
+ questions: dict[str, JudgmentQuestion]
+ outcomes: dict[str, JudgmentOutcome]
+
+
def judgment_setting_of_choice(*, judgment_choice: JudgmentModelChoice | None = None, pipe_code: str | None = None) -> JudgmentSetting:
"""The deck chain for a judgment: the step's own model, else the deck's default, as a setting.
@@ -116,7 +149,7 @@ async def run_judgment(
result_name: str | None = None,
result_code: str | None = None,
) -> JudgmentResult:
- """A whole judgment step: assemble the evidence, render the question, judge, read the verdict, store.
+ """A whole judgment step asking one question: assemble the evidence, render the question, judge, read the verdict, store.
The evidence is assembled as a PipeLLM's user prompt is, its images and documents numbered into
`[Image N]` and `[Document N]` tokens. The question arrives as the cogt question with its
@@ -133,24 +166,17 @@ async def run_judgment(
refining concept as every operator stores a refinement.
"""
with job_metadata.log_context():
- assembled_prompt = await assemble_user_prompt(prompt_content=prompt_content, context_provider=memory, templating_style=templating_style)
- judgment_prompt = JudgmentPrompt(text=assembled_prompt.text, images=assembled_prompt.images, documents=assembled_prompt.documents)
- rendered_question = await render_template(
- template=question.instructions,
- category=TemplateCategory.BASIC,
- context=memory.generate_context(),
- templating_style=templating_style,
- )
- asked_question = question.model_copy(update={"instructions": rendered_question})
- judgment_assignment = JudgmentAssignment(
+ asked_judgment = await _ask_judgment(
+ memory=memory,
+ prompt_content=prompt_content,
+ questions={JUDGMENT_QUESTION_KEY: question},
+ judgment_setting=judgment_setting,
job_metadata=job_metadata,
cogt_run_params=cogt_run_params,
- prompt=judgment_prompt,
- questions={JUDGMENT_QUESTION_KEY: asked_question},
- judgment_setting=judgment_setting,
+ templating_style=templating_style,
)
- outcomes = await get_content_generator().make_judgment_answers(judgment_assignment=judgment_assignment)
- outcome = outcomes[JUDGMENT_QUESTION_KEY]
+ asked_question = asked_judgment.questions[JUDGMENT_QUESTION_KEY]
+ outcome = asked_judgment.outcomes[JUDGMENT_QUESTION_KEY]
if isinstance(outcome, JudgmentRefusal):
raise JudgmentRefusedError(pipe_code=job_metadata.pipe_code, model_handle=judgment_setting.model)
content, threshold_applied = make_verdict_content(
@@ -164,14 +190,158 @@ async def run_judgment(
return JudgmentResult(
memory=store_result(memory=memory, concept=concept, content=content, result_name=result_name, result_code=result_code),
content=content,
- prompt=judgment_prompt,
- rendered_question=rendered_question,
+ prompt=asked_judgment.prompt,
+ rendered_question=asked_question.instructions,
judgment_setting=judgment_setting,
answer=outcome,
threshold_applied=threshold_applied,
)
+async def run_multi_judgment(
+ *,
+ memory: WorkingMemory,
+ prompt_content: UserPromptContent,
+ questions: dict[str, AskedQuestion],
+ judgment_setting: JudgmentSetting,
+ concept: Concept,
+ output_class: type[StuffContent],
+ job_metadata: JobMetadata,
+ cogt_run_params: CogtRunParams,
+ templating_style: TemplatingStyle,
+ result_name: str | None = None,
+ result_code: str | None = None,
+) -> MultiJudgmentResult:
+ """A whole judgment step asking several questions over one evidence, in one request, and filling a structure with their verdicts.
+
+ The evidence is assembled and each question rendered as `run_judgment` does, and every question goes
+ in one job keyed by its name, which is also the name of the field of `output_class` holding its
+ verdict. The caller resolves `output_class`, the structure class of the output concept, and hands it
+ down, as PipeLLM's object path does, since the kernel does not read the concept library: its fields
+ must be exactly the questions' names, which `JudgmentOutputFieldsError` refuses before the judging
+ model is called. Each answer becomes its verdict native under the single form's rules, the question's
+ own threshold deciding a yes/no verdict.
+
+ A refusal stores nothing in its question's field when that field may hold nothing, by
+ `class_field_may_hold_nothing`, the rule the load's concept walk reads a class by: its type admits
+ `None`, or it is neither required nor given a default. The refusal is logged and recorded in the
+ result's `judgments`. Any other refused question raises `JudgmentRefusedError`, a content error naming
+ the step, the model and the question: leaving a defaulted field out would fill in its default, a
+ verdict nobody gave. The verdicts are handed to `output_class` by field name, whatever alias a field
+ carries. A dry run answers every question and never refuses.
+ """
+ with job_metadata.log_context():
+ _check_questions_are_the_output_fields(question_names=list(questions), output_class=output_class)
+ asked_judgment = await _ask_judgment(
+ memory=memory,
+ prompt_content=prompt_content,
+ questions={question_name: asked_question.question for question_name, asked_question in questions.items()},
+ judgment_setting=judgment_setting,
+ job_metadata=job_metadata,
+ cogt_run_params=cogt_run_params,
+ templating_style=templating_style,
+ )
+ field_values: dict[str, object] = {}
+ judgments: dict[str, QuestionJudgment] = {}
+ for question_name, asked_question in questions.items():
+ question = asked_judgment.questions[question_name]
+ outcome = asked_judgment.outcomes[question_name]
+ if isinstance(outcome, JudgmentRefusal):
+ field_info = output_class.model_fields[question_name]
+ if not class_field_may_hold_nothing(field_info=field_info):
+ raise JudgmentRefusedError(pipe_code=job_metadata.pipe_code, model_handle=judgment_setting.model, question_name=question_name)
+ log.warning(
+ "A judgment's model declined a question, and its field holds nothing",
+ fields={"pipe_code": job_metadata.pipe_code, "model_handle": judgment_setting.model, "judgment_question": question_name},
+ )
+ # A field whose type admits nothing is given it outright, as a required one must be and as one carrying a
+ # default must be lest the default fill in; any other field that may hold nothing has `None` for default.
+ if field_admits_none(field_info=field_info):
+ field_values[question_name] = None
+ judgments[question_name] = QuestionJudgment(rendered_question=question.instructions, outcome=outcome)
+ continue
+ verdict, threshold_applied = make_verdict_content(
+ question=question, answer=outcome, threshold=asked_question.threshold, is_dry=cogt_run_params.run_mode.is_dry
+ )
+ if threshold_applied is False:
+ log.warning(
+ "A judgment's model reported no probability, and its own verdict stands",
+ fields={
+ "pipe_code": job_metadata.pipe_code,
+ "threshold": asked_question.threshold,
+ "model_handle": judgment_setting.model,
+ "judgment_question": question_name,
+ },
+ )
+ # Handed over as plain data, so a field typed by a concept refining the verdict native, whose class
+ # subclasses the native's, is built as that class rather than refusing the native's instance.
+ field_values[question_name] = verdict.model_dump()
+ judgments[question_name] = QuestionJudgment(rendered_question=question.instructions, outcome=outcome, threshold_applied=threshold_applied)
+ # Keyed by field name, the name each question goes by, so a field carrying an alias still receives its verdict.
+ content = output_class.model_validate(field_values, by_alias=False, by_name=True)
+ return MultiJudgmentResult(
+ memory=store_result(memory=memory, concept=concept, content=content, result_name=result_name, result_code=result_code),
+ content=content,
+ prompt=asked_judgment.prompt,
+ judgment_setting=judgment_setting,
+ judgments=judgments,
+ )
+
+
+def _check_questions_are_the_output_fields(*, question_names: list[str], output_class: type[StuffContent]) -> None:
+ """Refuse an output class whose fields are not exactly the questions' names, naming every name on either side."""
+ field_names = list(output_class.model_fields)
+ unasked_fields = [field_name for field_name in field_names if field_name not in question_names]
+ homeless_questions = [question_name for question_name in question_names if question_name not in field_names]
+ if not unasked_fields and not homeless_questions:
+ return
+ complaints: list[str] = []
+ if unasked_fields:
+ complaints.append(f"no question answers its fields {', '.join(repr(field_name) for field_name in unasked_fields)}")
+ if homeless_questions:
+ complaints.append(f"it has no field for the questions {', '.join(repr(question_name) for question_name in homeless_questions)}")
+ msg = (
+ f"A judgment asking several questions fills the field of each question's name, and the output class '{output_class.__name__}' "
+ f"does not hold exactly the questions' fields: {'; '.join(complaints)}."
+ )
+ raise JudgmentOutputFieldsError(msg)
+
+
+async def _ask_judgment(
+ *,
+ memory: WorkingMemory,
+ prompt_content: UserPromptContent,
+ questions: dict[str, JudgmentQuestion],
+ judgment_setting: JudgmentSetting,
+ job_metadata: JobMetadata,
+ cogt_run_params: CogtRunParams,
+ templating_style: TemplatingStyle,
+) -> _AskedJudgment:
+ """One request to the judging model: the evidence assembled, every question rendered as a `BASIC` template, all sent in one job."""
+ with job_metadata.log_context():
+ assembled_prompt = await assemble_user_prompt(prompt_content=prompt_content, context_provider=memory, templating_style=templating_style)
+ judgment_prompt = JudgmentPrompt(text=assembled_prompt.text, images=assembled_prompt.images, documents=assembled_prompt.documents)
+ context = memory.generate_context()
+ asked_questions: dict[str, JudgmentQuestion] = {}
+ for question_key, question in questions.items():
+ rendered_question = await render_template(
+ template=question.instructions,
+ category=TemplateCategory.BASIC,
+ context=context,
+ templating_style=templating_style,
+ )
+ asked_questions[question_key] = question.model_copy(update={"instructions": rendered_question})
+ judgment_assignment = JudgmentAssignment(
+ job_metadata=job_metadata,
+ cogt_run_params=cogt_run_params,
+ prompt=judgment_prompt,
+ questions=asked_questions,
+ judgment_setting=judgment_setting,
+ )
+ outcomes = await get_content_generator().make_judgment_answers(judgment_assignment=judgment_assignment)
+ return _AskedJudgment(prompt=judgment_prompt, questions=asked_questions, outcomes=outcomes)
+
+
def make_verdict_content(
*, question: JudgmentQuestion, answer: JudgmentAnswer, threshold: float | None, is_dry: bool
) -> tuple[StuffContent, bool | None]:
diff --git a/pipelex/kernel/judgment_results.py b/pipelex/kernel/judgment_results.py
index ad7598490..4dbbcb204 100644
--- a/pipelex/kernel/judgment_results.py
+++ b/pipelex/kernel/judgment_results.py
@@ -1,4 +1,4 @@
-"""What a kernel judgment call hands back.
+"""What a kernel judgment call hands back, for a judgment asking one question and for one asking several.
Four intermediates ride along because the kernel is where they are produced and the interpreter's
execution-graph tracer records them: the assembled evidence prompt, the rendered question, the
@@ -12,7 +12,7 @@ the base `StuffContent`, so use `kajson` rather than a plain `model_dump()`.
from pydantic import BaseModel, ConfigDict
-from pipelex.cogt.judgment.judgment_models import JudgmentAnswer, JudgmentPrompt
+from pipelex.cogt.judgment.judgment_models import JudgmentAnswer, JudgmentOutcome, JudgmentPrompt
from pipelex.cogt.judgment.judgment_setting import JudgmentSetting
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -38,3 +38,37 @@ class JudgmentResult(BaseModel):
judgment_setting: JudgmentSetting
answer: JudgmentAnswer
threshold_applied: bool | None = None
+
+
+class QuestionJudgment(BaseModel):
+ """What became of one question of a judgment asking several.
+
+ `rendered_question` is the question as the model was asked it. `outcome` is the judging model's raw
+ outcome: an answer, which carries a distribution the verdict native's rendering does not show, or a
+ refusal, which left the question's field absent. `threshold_applied` reads as `JudgmentResult`'s does
+ for this question, and is `None` for a refusal.
+ """
+
+ model_config = ConfigDict(frozen=True)
+
+ rendered_question: str
+ outcome: JudgmentOutcome
+ threshold_applied: bool | None = None
+
+
+class MultiJudgmentResult(BaseModel):
+ """The outcome of a kernel judgment call asking several questions over one evidence.
+
+ `content` is the output structure, holding the verdict native of each question in the field of its
+ name, a refused question's field holding nothing. `prompt` is the evidence as the model was
+ given it, and `judgments` says, by question name, what each question was asked as and what the model
+ answered, refusals included.
+ """
+
+ model_config = ConfigDict(frozen=True)
+
+ memory: WorkingMemory
+ content: StuffContent
+ prompt: JudgmentPrompt
+ judgment_setting: JudgmentSetting
+ judgments: dict[str, QuestionJudgment]
diff --git a/pipelex/language/mthds_schema_generator.py b/pipelex/language/mthds_schema_generator.py
index c02df715f..708e83069 100644
--- a/pipelex/language/mthds_schema_generator.py
+++ b/pipelex/language/mthds_schema_generator.py
@@ -21,7 +21,7 @@ from pipelex.pipe_machinery.validation import (
SEQUENCE_STEP_BATCH_OVER_PATTERN,
STORED_NAME_PATTERN,
)
-from pipelex.pipe_operators.judge.pipe_judge_blueprint import JudgeRatingLevel, PipeJudgeBlueprint
+from pipelex.pipe_operators.judge.pipe_judge_blueprint import QUESTION_KIND_FIELDS, JudgeQuestionBlueprint, JudgeRatingLevel, PipeJudgeBlueprint
from pipelex.pipe_signature.pipe_signature_blueprint import PipeSignatureBlueprint
from pipelex.tools.misc.package_utils import get_package_version
@@ -80,6 +80,7 @@ def generate_mthds_schema() -> dict[str, Any]:
schema = _constrain_stored_names(schema)
schema = _constrain_batch_over(schema)
schema = _constrain_rating_levels(schema)
+ schema = _constrain_judge_questions(schema)
return _add_taplo_metadata(schema)
@@ -396,7 +397,7 @@ def _set_batch_over_grammar(*, step_schema: dict[str, Any], pattern: str, descri
def _constrain_rating_levels(schema: dict[str, Any]) -> dict[str, Any]:
- """Make a PipeJudge's `levels` label every level or none, as its load does.
+ """Make the `levels` of a PipeJudge, and of each of its `questions`, label every level or none, as its load does.
A level is a string, which is its description, or a closed `{label, description}` table, and the load
refuses a scale labelling some of its levels only, since a `Rating` verdict reports the label of its
@@ -405,9 +406,16 @@ def _constrain_rating_levels(schema: dict[str, Any]) -> dict[str, Any]:
"""
schema = copy.deepcopy(schema)
definitions = schema.get("definitions", {})
- levels_schema = definitions.get(PipeJudgeBlueprint.__name__, {}).get("properties", {}).get("levels")
- if levels_schema is None or JudgeRatingLevel.__name__ not in definitions:
+ if JudgeRatingLevel.__name__ not in definitions:
return schema
+ for definition_name in (PipeJudgeBlueprint.__name__, JudgeQuestionBlueprint.__name__):
+ levels_schema = definitions.get(definition_name, {}).get("properties", {}).get("levels")
+ if levels_schema is not None:
+ _split_levels_scale(levels_schema=levels_schema, definition_name=definition_name)
+ return schema
+
+
+def _split_levels_scale(*, levels_schema: dict[str, Any], definition_name: str) -> None:
level_ref = {"$ref": f"#/definitions/{JudgeRatingLevel.__name__}"}
unlabelled_level = {"allOf": [level_ref, {"properties": {"label": {"type": "null"}}}]}
labelled_level = {"allOf": [copy.deepcopy(level_ref), {"required": ["label"], "properties": {"label": {"type": "string"}}}]}
@@ -415,7 +423,7 @@ def _constrain_rating_levels(schema: dict[str, Any]) -> dict[str, Any]:
arms: list[dict[str, Any]] = levels_schema.get("anyOf", [])
array_arms = [arm for arm in arms if arm.get("type") == "array"]
if len(array_arms) != 1:
- msg = f"Expected one array arm on {PipeJudgeBlueprint.__name__}.levels, found {len(array_arms)}"
+ msg = f"Expected one array arm on {definition_name}.levels, found {len(array_arms)}"
raise RuntimeError(msg)
array_arm = array_arms[0]
scale_arms = [
@@ -423,6 +431,28 @@ def _constrain_rating_levels(schema: dict[str, Any]) -> dict[str, Any]:
{**copy.deepcopy(array_arm), "items": labelled_level},
]
levels_schema["anyOf"] = [scale_arm for arm in arms for scale_arm in (scale_arms if arm is array_arm else [arm])]
+
+
+def _constrain_judge_questions(schema: dict[str, Any]) -> dict[str, Any]:
+ """Make a PipeJudge ask one `question` or several `questions`, and keep the kind fields off one asking several, as its load does.
+
+ Exactly one of `question` and `questions` is set, which Draft 4 says with a `oneOf` of the two
+ `required`. A PipeJudge asking several sets `options`, `levels`, `criteria` and `threshold` on each
+ question rather than on the pipe, which a `not` refuses, and its `questions` holds at least one
+ question. The grammar of a question's key, a field name of the output's structure, is left to the
+ load, since a Python identifier admits letters no portable pattern spells.
+ """
+ schema = copy.deepcopy(schema)
+ judge_schema = schema.get("definitions", {}).get(PipeJudgeBlueprint.__name__)
+ if judge_schema is None:
+ return schema
+ judge_schema["oneOf"] = [{"required": ["question"]}, {"required": ["questions"]}]
+ judge_schema["not"] = {"required": ["questions"], "anyOf": [{"required": [kind_field]} for kind_field in QUESTION_KIND_FIELDS]}
+ questions_schema = judge_schema.get("properties", {}).get("questions")
+ if questions_schema is not None:
+ for arm in questions_schema.get("anyOf", [questions_schema]):
+ if arm.get("type") == "object":
+ arm["minProperties"] = 1
return schema
diff --git a/pipelex/pipe_controllers/binding/binding_concept_resolvers.py b/pipelex/pipe_controllers/binding/binding_concept_resolvers.py
index 774e42f38..c7014971f 100644
--- a/pipelex/pipe_controllers/binding/binding_concept_resolvers.py
+++ b/pipelex/pipe_controllers/binding/binding_concept_resolvers.py
@@ -36,7 +36,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.native.pinned_blueprints import make_pinned_native_blueprint
from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.qualified_ref import QualifiedRef
-from pipelex.core.stuffs.non_null_any import field_admits_none
+from pipelex.core.stuffs.non_null_any import class_field_may_hold_nothing
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstract
from pipelex.libraries.concept.exceptions import ConceptLibraryError
@@ -306,11 +306,7 @@ class LibraryConceptWalkResolver(ConceptWalkResolver):
return WalkableConcept(concept_ref=concept_key, shape=ConceptShape.STRUCTURE, fields=walkable_fields)
def _walkable_field_from_class_field(self, *, name: str, field_info: FieldInfo, package_alias: str | None) -> WalkableField:
- # A class states what a field may hold through its annotation: one that admits `None` may hold nothing, whether it
- # is required or has a default, and so does one left unrequired with no default. `NonNullAny` is the one `Any` that
- # refuses `None`, as a generated class spells a required open field.
- is_nullable = field_admits_none(field_info=field_info)
- may_hold_nothing = is_nullable or (not field_info.is_required() and field_info.default is None and field_info.default_factory is None)
+ may_hold_nothing = class_field_may_hold_nothing(field_info=field_info)
annotation, _ = _strip_optional(annotation=field_info.annotation)
is_list = get_origin(annotation) is list
if is_list:
diff --git a/pipelex/pipe_machinery/validation.py b/pipelex/pipe_machinery/validation.py
index c9ff365c2..0610a2145 100644
--- a/pipelex/pipe_machinery/validation.py
+++ b/pipelex/pipe_machinery/validation.py
@@ -103,7 +103,7 @@ def check_inputs_match_variables(
"""Refuse a variable the templates read that no input declares, then a declared input no template reads.
The rule shared by every operator that reads its inputs through templates: PipeLLM, PipeCompose,
- PipeSearch, PipeImgGen and PipeJudge, whose prompt and question are read together. It is the
+ PipeSearch, PipeImgGen and PipeJudge, whose prompt and questions are read together. It is the
composition of `check_variables_are_declared` and `check_inputs_are_read`, in that order.
Raises:
diff --git a/pipelex/pipe_operators/judge/pipe_judge.py b/pipelex/pipe_operators/judge/pipe_judge.py
index b2d48a032..804fe61ed 100644
--- a/pipelex/pipe_operators/judge/pipe_judge.py
+++ b/pipelex/pipe_operators/judge/pipe_judge.py
@@ -1,5 +1,6 @@
-from typing import Any, Literal
+from typing import Any, Literal, Self
+from pydantic import BaseModel, ConfigDict, model_validator
from typing_extensions import override
from pipelex.cogt.judgment.judgment_models import (
@@ -14,16 +15,26 @@ from pipelex.cogt.judgment.judgment_models import (
from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice
from pipelex.cogt.models.model_deck_check import check_judgment_choice_with_deck
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
+from pipelex.core.concepts.concept import Concept
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.interpreter_hub import get_concept_library, get_native_concept
-from pipelex.kernel.judgment_ops import judgment_setting_of_choice, resolve_judgment_setting, run_judgment, served_judgment_model
+from pipelex.kernel.judgment_ops import (
+ AskedQuestion,
+ judgment_setting_of_choice,
+ resolve_judgment_setting,
+ run_judgment,
+ run_multi_judgment,
+ served_judgment_model,
+)
from pipelex.kernel.prompt_assembly import UserPromptContent
from pipelex.kernel.prompt_references import DocumentReference, ImageReference
from pipelex.kernel.templating_style_ops import resolve_templating_style
+from pipelex.pipe_controllers.binding.binding_concept_resolvers import LibraryConceptWalkResolver, library_concept_key
+from pipelex.pipe_controllers.binding.binding_derivation import BindingValueKind, ConceptShape, WalkableField
from pipelex.pipe_machinery.template_guard_lint import lint_authored_template
from pipelex.pipe_operators.judge.exceptions import PipeJudgeInputCapabilityError
from pipelex.pipe_operators.pipe_operator import PipeOperator
@@ -59,39 +70,104 @@ def _kind_declaration(*, judgment_kind: JudgmentKind) -> str:
return "declares `levels`, so it asks a rating question"
-class PipeJudge(PipeOperator[PipeJudgeOutput]):
- """Asks a judging model one closed question about the evidence its prompt presents, and stores the verdict native it answers with.
+def _kind_question(*, judgment_kind: JudgmentKind) -> str:
+ match judgment_kind:
+ case JudgmentKind.YES_NO:
+ return "a yes/no question"
+ case JudgmentKind.CHOICE:
+ return "a choice question"
+ case JudgmentKind.RATING:
+ return "a rating question"
- `prompt_content` is the evidence: the prompt's template and the images and documents it references,
- assembled when the step runs exactly as a PipeLLM's user prompt is. `judgment_question` is the cogt
- question with its instructions still the authored template, so the kernel renders it against memory
- when the step runs; its kind decides the verdict native. The question presents no file, so the
- image and document references the factory found in it are held only for the load to refuse them.
+
+def _holds_verdict(*, concept: Concept, verdict_native: NativeConceptCode) -> bool:
+ """Whether a concept holds the verdict native: the native itself or a concept refining it, strictly compatible.
+
+ `Dynamic` is compatible with every concept, but a judgment's verdict is always its kind's native, so a
+ `Dynamic` concept never holds one.
"""
+ if NativeConceptCode.is_dynamic_concept(concept_code=concept.code):
+ return False
+ return get_concept_library().is_compatible(tested_concept=concept, wanted_concept=get_native_concept(native_concept=verdict_native), strict=True)
+
+
+class PipeJudgeQuestion(BaseModel):
+ """One question a PipeJudge asks, as the operator holds it: the cogt question and the policy the operator reads its verdict with.
+
+ `judgment_question` is the cogt question with its instructions still the authored template, so the
+ kernel renders it against memory when the step runs; its kind decides the verdict native. A yes/no
+ question's `threshold` is the operator's policy, so it travels beside the question. The question
+ presents no file, so the image and document references the factory found in it are held only for the
+ load to refuse them.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
- type: Literal["PipeJudge"] = "PipeJudge"
- judgment_choice: JudgmentModelChoice | None
- prompt_content: UserPromptContent
judgment_question: JudgmentQuestion
- question_image_references: list[ImageReference] | None = None
- question_document_references: list[DocumentReference] | None = None
threshold: float | None = None
+ image_references: list[ImageReference] | None = None
+ document_references: list[DocumentReference] | None = None
@property
def judgment_kind(self) -> JudgmentKind:
return self.judgment_question.kind
@property
- def question_blueprint(self) -> TemplateBlueprint:
+ def template_blueprint(self) -> TemplateBlueprint:
return TemplateBlueprint(template=self.judgment_question.instructions, category=TemplateCategory.BASIC)
+ @property
+ def file_variable_paths(self) -> list[str]:
+ """The variables the question reads that hold an image or a document, which it cannot present."""
+ return [
+ *(image_reference.variable_path for image_reference in self.image_references or []),
+ *(document_reference.variable_path for document_reference in self.document_references or []),
+ ]
+
+
+class PipeJudge(PipeOperator[PipeJudgeOutput]):
+ """Asks a judging model closed questions about the evidence its prompt presents, and stores the verdicts they answer with.
+
+ `prompt_content` is the evidence: the prompt's template and the images and documents it references,
+ assembled when the step runs exactly as a PipeLLM's user prompt is. A step asks exactly one of two
+ ways: one `question`, whose verdict native is the output, or several `questions`, keyed by name, whose
+ verdicts fill the output's structure, each in the field of its question's name.
+ """
+
+ type: Literal["PipeJudge"] = "PipeJudge"
+ judgment_choice: JudgmentModelChoice | None
+ prompt_content: UserPromptContent
+ question: PipeJudgeQuestion | None = None
+ questions: dict[str, PipeJudgeQuestion] | None = None
+
+ @model_validator(mode="after")
+ def _asks_one_question_or_several(self) -> Self:
+ if self.question is not None and self.questions is not None:
+ msg = f"PipeJudge '{self.code}' asks one question or several, and it was given both a single question and several."
+ raise ValueError(msg)
+ if self.question is None and not self.questions:
+ msg = f"PipeJudge '{self.code}' asks one question or several, and it was given none."
+ raise ValueError(msg)
+ return self
+
+ @property
+ def labelled_questions(self) -> list[tuple[str, PipeJudgeQuestion]]:
+ """Every question the step asks, with the words a message names it by: "question" alone, or "question 'name'" among several."""
+ if self.questions is not None:
+ return [(f"question '{question_name}'", question) for question_name, question in self.questions.items()]
+ if self.question is not None:
+ return [("question", self.question)]
+ return []
+
@override
def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
return self.inputs
@override
def required_variables(self) -> set[str]:
- full_paths = self.prompt_content.template.required_variables() | self.question_blueprint.required_variables()
+ full_paths = self.prompt_content.template.required_variables()
+ for _, question in self.labelled_questions:
+ full_paths |= question.template_blueprint.required_variables()
return {get_root_from_dotted_path(path) for path in full_paths if not path.startswith("_")}
@override
@@ -100,18 +176,17 @@ class PipeJudge(PipeOperator[PipeJudgeOutput]):
with self.locating_model_choice(field_name="model"):
check_judgment_choice_with_deck(judgment_choice=self.judgment_choice)
- # Only the prompt presents files: the question is rendered against the raw context with no token
+ # Only the prompt presents files: a question is rendered against the raw context with no token
# parameters, so an image or a document it read would be written into the question sent to the
# vendor as the content's string form, such as its storage URL, rather than presented as a file.
- question_file_paths = [
- *(image_reference.variable_path for image_reference in self.question_image_references or []),
- *(document_reference.variable_path for document_reference in self.question_document_references or []),
- ]
- if question_file_paths:
+ for question_label, question in self.labelled_questions:
+ question_file_paths = question.file_variable_paths
+ if not question_file_paths:
+ continue
quoted_paths = ", ".join(f"'{variable_path}'" for variable_path in question_file_paths)
msg = (
- f"PipeJudge '{self.code}' reads {quoted_paths} in its question, which is an image or a document, and only the prompt "
- "presents files to the judging model: read it in `prompt`, and ask about it in `question`."
+ f"PipeJudge '{self.code}' reads {quoted_paths} in its {question_label}, which is an image or a document, and only the "
+ "prompt presents files to the judging model: read it in `prompt`, and ask about it in the question."
)
raise PipeValidationError(
message=msg,
@@ -122,8 +197,9 @@ class PipeJudge(PipeOperator[PipeJudgeOutput]):
)
# Template lints: no private names, and every reference to a declared-optional input guarded (D7).
- question_blueprint = self.question_blueprint
- for template_blueprint, template_label in [(self.prompt_content.template, "prompt"), (question_blueprint, "question")]:
+ labelled_templates = [(self.prompt_content.template, "prompt")]
+ labelled_templates.extend((question.template_blueprint, question_label) for question_label, question in self.labelled_questions)
+ for template_blueprint, template_label in labelled_templates:
lint_authored_template(
pipe_code=self.code,
domain_code=self.domain_code,
@@ -176,29 +252,130 @@ class PipeJudge(PipeOperator[PipeJudgeOutput]):
@override
def validate_output_with_library(self):
- """The output agrees with the question's kind: its verdict native, or a concept refining it.
+ """The output agrees with what the step asks: one question's verdict native, or a structure holding every question's verdict.
- `Dynamic` is compatible with every concept, but a judgment's verdict is always its kind's native,
- so a `Dynamic` output is refused like any other that is not that native.
+ Asking one question, the output is its kind's verdict native or a concept refining it. Asking
+ several, the output is a concept with a structure whose fields are exactly the question names,
+ each holding one of its question's verdict native, or of a concept refining it.
"""
- verdict_native = verdict_native_for_kind(judgment_kind=self.judgment_kind)
- if not NativeConceptCode.is_dynamic_concept(concept_code=self.output.concept.code) and get_concept_library().is_compatible(
- tested_concept=self.output.concept,
- wanted_concept=get_native_concept(native_concept=verdict_native),
- strict=True,
- ):
+ if self.questions is not None:
+ self._validate_output_holds_every_verdict(questions=self.questions)
+ return
+ if self.question is None:
+ return
+ judgment_kind = self.question.judgment_kind
+ verdict_native = verdict_native_for_kind(judgment_kind=judgment_kind)
+ if _holds_verdict(concept=self.output.concept, verdict_native=verdict_native):
return
msg = (
- f"This PipeJudge {_kind_declaration(judgment_kind=self.judgment_kind)}: its output must be `{verdict_native}` "
+ f"This PipeJudge {_kind_declaration(judgment_kind=judgment_kind)}: its output must be `{verdict_native}` "
f"or a concept refining it, and it declares `{self.output.concept.concept_ref}`."
)
- raise PipeValidationError(
- message=msg,
+ raise self._inadequate_output_error(message=msg, required_concept_codes=[verdict_native.concept_ref])
+
+ def _validate_output_holds_every_verdict(self, *, questions: dict[str, PipeJudgeQuestion]) -> None:
+ """Refuse an output that is no structure, a question with no field, a field no question answers, and a field that cannot hold its verdict.
+
+ The output's structure is read as the binding walk reads it, so a concept refining a structured
+ concept offers the fields it inherits, and a structure that exists only as a Python class offers
+ its class's fields.
+ """
+ concept_library = get_concept_library()
+ output_ref = self.output.concept.concept_ref
+ walkable_output = LibraryConceptWalkResolver(concept_library=concept_library).resolve_walkable_concept(
+ concept_ref=library_concept_key(concept_library=concept_library, concept=self.output.concept)
+ )
+ match walkable_output.shape:
+ case ConceptShape.STRUCTURE:
+ pass
+ case ConceptShape.VALUE | ConceptShape.NO_STRUCTURE:
+ msg = (
+ "This PipeJudge asks several questions and fills a structure with one verdict per question, so its output must be a "
+ f"concept with a structure, and `{output_ref}` {walkable_output.shape_reason}."
+ )
+ raise self._inadequate_output_error(message=msg)
+ for question_name in questions:
+ if walkable_output.get_field(name=question_name) is None:
+ msg = (
+ f"The question '{question_name}' has no field of `{output_ref}` to hold its verdict: a PipeJudge asking several "
+ f"questions fills the field named after each question, so add a field '{question_name}' to `{output_ref}`, or "
+ "name the question after one of its fields."
+ )
+ raise self._inadequate_output_error(message=msg)
+ unasked_field_names = [field_name for field_name in walkable_output.field_names if field_name not in questions]
+ if unasked_field_names:
+ quoted_names = ", ".join(f"'{field_name}'" for field_name in unasked_field_names)
+ field_words = "the field" if len(unasked_field_names) == 1 else "the fields"
+ msg = (
+ f"A PipeJudge asking several questions fills only the fields named after them, and `{output_ref}` has {field_words} "
+ f"{quoted_names}, which no question answers and so could never be filled: ask a question of that name, or remove the "
+ "field from the structure."
+ )
+ raise self._inadequate_output_error(message=msg)
+ for question_name, question in questions.items():
+ walkable_field = walkable_output.get_field(name=question_name)
+ if walkable_field is not None:
+ self._validate_field_holds_its_verdict(
+ output_ref=output_ref, question_name=question_name, question=question, walkable_field=walkable_field
+ )
+
+ def _validate_field_holds_its_verdict(
+ self, *, output_ref: str, question_name: str, question: PipeJudgeQuestion, walkable_field: WalkableField
+ ) -> None:
+ """Refuse a field that cannot hold one verdict of its question's kind: a list, a plain value, or a concept other than the native's line."""
+ verdict_native = verdict_native_for_kind(judgment_kind=question.judgment_kind)
+ asks = _kind_question(judgment_kind=question.judgment_kind)
+ remedy = f"declare it a field of type `concept` whose `concept_ref` names `{verdict_native.concept_ref}` or a concept refining it"
+ held_concept: str
+ if walkable_field.is_list:
+ msg = (
+ f"The field '{question_name}' of `{output_ref}` holds a list, and the question '{question_name}' produces one verdict: "
+ f"{remedy}. To judge each item of a list, map the PipeJudge over it with a PipeBatch."
+ )
+ raise self._inadequate_output_error(message=msg, required_concept_codes=[verdict_native.concept_ref])
+ match walkable_field.value_kind:
+ case BindingValueKind.CONCEPT:
+ field_concept_ref = walkable_field.concept_ref
+ if field_concept_ref is not None and _holds_verdict(
+ concept=get_concept_library().get_required_concept(concept_ref=field_concept_ref), verdict_native=verdict_native
+ ):
+ return
+ held_concept = f"`{field_concept_ref}`" if field_concept_ref is not None else "no concept"
+ case (
+ BindingValueKind.TEXT
+ | BindingValueKind.NUMBER
+ | BindingValueKind.YES_NO
+ | BindingValueKind.DATE
+ | BindingValueKind.DATETIME
+ | BindingValueKind.TIME
+ | BindingValueKind.JSON
+ | BindingValueKind.ANYTHING
+ ):
+ msg = (
+ f"The field '{question_name}' of `{output_ref}` holds a plain value rather than a concept, and the question "
+ f"'{question_name}' asks {asks}, whose verdict is a `{verdict_native.concept_ref}`: {remedy}."
+ )
+ raise self._inadequate_output_error(message=msg, required_concept_codes=[verdict_native.concept_ref])
+ case BindingValueKind.UNDERIVABLE:
+ msg = (
+ f"The field '{question_name}' of `{output_ref}` holds a value no concept describes, since "
+ f"{walkable_field.underivable_reason}, and the question '{question_name}' asks {asks}: {remedy}."
+ )
+ raise self._inadequate_output_error(message=msg, required_concept_codes=[verdict_native.concept_ref])
+ msg = (
+ f"The question '{question_name}' asks {asks}, so its field must hold `{verdict_native.concept_ref}` or a concept refining "
+ f"it, and it holds {held_concept}."
+ )
+ raise self._inadequate_output_error(message=msg, required_concept_codes=[verdict_native.concept_ref])
+
+ def _inadequate_output_error(self, *, message: str, required_concept_codes: list[str] | None = None) -> PipeValidationError:
+ return PipeValidationError(
+ message=message,
error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,
domain_code=self.domain_code,
pipe_code=self.code,
provided_concept_code=self.output.concept.concept_ref,
- required_concept_codes=[verdict_native.concept_ref],
+ required_concept_codes=required_concept_codes,
)
@override
@@ -213,41 +390,91 @@ class PipeJudge(PipeOperator[PipeJudgeOutput]):
# Resolved per run into a local and never cached onto `self`, for the reason `pipe_llm.py`
# states about its own settings.
judgment_setting = resolve_judgment_setting(judgment_choice=self.judgment_choice, pipe_code=self.code, is_dry=pipe_run_params.run_mode.is_dry)
- judgment_result = await run_judgment(
- memory=working_memory,
- prompt_content=self.prompt_content,
- question=self.judgment_question,
- judgment_setting=judgment_setting,
- concept=self.output.concept,
- job_metadata=job_metadata,
- cogt_run_params=pipe_run_params.cogt_run_params,
- # No authored style on this operator: its prompt and question templates take the runtime
- # default, the same one an LLM pipe that declares nothing gets.
- templating_style=resolve_templating_style(authored=None),
- threshold=self.threshold,
- result_name=output_name,
- )
-
- # Capture execution data for the graph tracer. The raw answer rides along because it carries
- # what the verdict native's rendering does not show, such as a choice's whole distribution.
- judgment_prompt = judgment_result.prompt
- execution_data_dict: dict[str, Any] = {
- "rendered_prompt": judgment_prompt.text,
- "nb_prompt_images": len(judgment_prompt.images),
- "nb_prompt_documents": len(judgment_prompt.documents),
- "rendered_question": judgment_result.rendered_question,
- "resolved_model": judgment_result.judgment_setting.model,
- "judgment_kind": self.judgment_kind.value,
- "answer": judgment_result.answer.model_dump(mode="json", exclude_none=True),
- }
- if self.threshold is not None:
- execution_data_dict["threshold"] = self.threshold
- if judgment_result.threshold_applied is not None:
- execution_data_dict["threshold_applied"] = judgment_result.threshold_applied
-
+ # No authored style on this operator: its prompt and question templates take the runtime default,
+ # the same one an LLM pipe that declares nothing gets.
+ templating_style = resolve_templating_style(authored=None)
+ execution_data_dict: dict[str, Any]
+ result_memory: WorkingMemory
+ if self.questions is not None:
+ multi_result = await run_multi_judgment(
+ memory=working_memory,
+ prompt_content=self.prompt_content,
+ questions={
+ question_name: AskedQuestion(question=question.judgment_question, threshold=question.threshold)
+ for question_name, question in self.questions.items()
+ },
+ judgment_setting=judgment_setting,
+ concept=self.output.concept,
+ # The kernel does not read the concept library, so the output's content class is resolved here and
+ # handed down, as PipeLLM's object path does.
+ output_class=get_concept_library().get_structure_class(concept=self.output.concept),
+ job_metadata=job_metadata,
+ cogt_run_params=pipe_run_params.cogt_run_params,
+ templating_style=templating_style,
+ result_name=output_name,
+ )
+ question_records: dict[str, dict[str, Any]] = {}
+ for question_name, question in self.questions.items():
+ judgment = multi_result.judgments[question_name]
+ question_record: dict[str, Any] = {
+ "rendered_question": judgment.rendered_question,
+ # The plain string: the execution data is serialized, and the enum would be written out with its class.
+ "judgment_kind": question.judgment_kind.value,
+ "outcome": judgment.outcome.model_dump(mode="json", exclude_none=True),
+ }
+ if question.threshold is not None:
+ question_record["threshold"] = question.threshold
+ if judgment.threshold_applied is not None:
+ question_record["threshold_applied"] = judgment.threshold_applied
+ question_records[question_name] = question_record
+ execution_data_dict = {
+ "rendered_prompt": multi_result.prompt.text,
+ "nb_prompt_images": len(multi_result.prompt.images),
+ "nb_prompt_documents": len(multi_result.prompt.documents),
+ "resolved_model": multi_result.judgment_setting.model,
+ # Each question's raw outcome rides along, a refusal included, since a field holding nothing shows nothing.
+ "questions": question_records,
+ }
+ result_memory = multi_result.memory
+ elif self.question is not None:
+ judgment_result = await run_judgment(
+ memory=working_memory,
+ prompt_content=self.prompt_content,
+ question=self.question.judgment_question,
+ judgment_setting=judgment_setting,
+ concept=self.output.concept,
+ job_metadata=job_metadata,
+ cogt_run_params=pipe_run_params.cogt_run_params,
+ templating_style=templating_style,
+ threshold=self.question.threshold,
+ result_name=output_name,
+ )
+ # The raw answer rides along because it carries what the verdict native's rendering does not
+ # show, such as a choice's whole distribution.
+ judgment_prompt = judgment_result.prompt
+ execution_data_dict = {
+ "rendered_prompt": judgment_prompt.text,
+ "nb_prompt_images": len(judgment_prompt.images),
+ "nb_prompt_documents": len(judgment_prompt.documents),
+ "rendered_question": judgment_result.rendered_question,
+ "resolved_model": judgment_result.judgment_setting.model,
+ # The plain string: the execution data is serialized, and the enum would be written out with its class.
+ "judgment_kind": self.question.judgment_kind.value,
+ "answer": judgment_result.answer.model_dump(mode="json", exclude_none=True),
+ }
+ if self.question.threshold is not None:
+ execution_data_dict["threshold"] = self.question.threshold
+ if judgment_result.threshold_applied is not None:
+ execution_data_dict["threshold_applied"] = judgment_result.threshold_applied
+ result_memory = judgment_result.memory
+ else:
+ msg = f"PipeJudge '{self.code}' asks no question, which its construction refuses."
+ raise PipeValidationError(message=msg, domain_code=self.domain_code, pipe_code=self.code)
+
+ # Capture execution data for the graph tracer.
self._register_execution_data(job_metadata=job_metadata, execution_data=execution_data_dict)
return PipeJudgeOutput(
- working_memory=judgment_result.memory,
+ working_memory=result_memory,
pipeline_run_id=job_metadata.run_metadata.pipeline_run_id,
)
diff --git a/pipelex/pipe_operators/judge/pipe_judge_blueprint.py b/pipelex/pipe_operators/judge/pipe_judge_blueprint.py
index 5c623e93b..1063b623c 100644
--- a/pipelex/pipe_operators/judge/pipe_judge_blueprint.py
+++ b/pipelex/pipe_operators/judge/pipe_judge_blueprint.py
@@ -7,6 +7,7 @@ from pipelex.cogt.judgment.judgment_models import JudgmentKind, RatingLevel
from pipelex.cogt.judgment.judgment_setting import JudgmentModelChoice
from pipelex.cogt.templating.exceptions import TemplateSigilSyntaxError
from pipelex.cogt.templating.template_preprocessor import preprocess_template
+from pipelex.core.concepts.concept_structure_blueprint import is_admitted_structure_field_name
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.pipe_machinery.pipe_blueprint import PipeBlueprint
from pipelex.pipe_machinery.validation import check_inputs_match_variables
@@ -72,19 +73,70 @@ class JudgeRatingLevel(BaseModel):
description: str | None = None
+# The fields that decide a question's kind and its verdict, which the single form sets on the pipe and the
+# multi form on each of its questions.
+QUESTION_KIND_FIELDS = ("options", "levels", "criteria", "threshold")
+
+
+class JudgeQuestionBlueprint(BaseModel):
+ """One question of a PipeJudge asking several: what it asks, and the fields the single form sets on the pipe.
+
+ Closed, as the standard requires: `question` is the instruction, a plain text template, and the kind
+ is decided by which of `options` and `levels` the question declares, under the single form's rules,
+ which `PipeJudgeBlueprint` checks, since a refusal names the question and only its key in `questions`
+ does. That key also names the output field holding its verdict.
+ """
+
+ model_config = ConfigDict(extra="forbid")
+
+ question: str
+ options: dict[str, str] | None = None
+ levels: list[str | JudgeRatingLevel] | None = None
+ criteria: JudgeYesNoCriteria | None = None
+ threshold: float | None = None
+
+ @model_validator(mode="before")
+ @classmethod
+ def refuse_a_missing_or_empty_question(cls, values: Any) -> Any:
+ if not isinstance(values, dict):
+ return values
+ raw_table = cast("dict[str, Any]", values)
+ if raw_table.get("question") is None:
+ msg = "Each question of `questions` writes what it asks in `question`, and this one sets none."
+ raise ValueError(msg)
+ question = raw_table["question"]
+ if isinstance(question, str) and not question.strip():
+ msg = "Each question of `questions` asks something, so its `question` cannot be empty."
+ raise ValueError(msg)
+ return raw_table
+
+ @property
+ def judgment_kind(self) -> JudgmentKind:
+ return _judgment_kind_of(options=self.options, levels=self.levels)
+
+ @property
+ def rating_levels(self) -> list[RatingLevel] | None:
+ """The scale as the judgment family carries it: a string level is its description."""
+ return _rating_levels_of(levels=self.levels)
+
+
class PipeJudgeBlueprint(PipeBlueprint):
- """Asks a judging model one closed question about the evidence its prompt presents: yes/no, a choice, or a rating.
+ """Asks a judging model closed questions about the evidence its prompt presents: yes/no, a choice, or a rating.
`prompt` is the evidence template, rendered and checked as a PipeLLM's prompt is, the images and
- documents it reads presented to the model as files. `question` is the instruction, a plain text
- template. The kind is decided by which of `options` and `levels` the pipe declares, never by a
- field of its own. Every declared input is read by the prompt or by the question.
+ documents it reads presented to the model as files. The single form asks one `question`, a plain
+ text template, and produces its verdict native; the multi form asks the several `questions` of a
+ table keyed by name over the same evidence in one request, and fills a structure whose fields are
+ those names with one verdict each. A question's kind is decided by which of `options` and `levels`
+ it declares, never by a field of its own, and the single form declares them on the pipe. Every
+ declared input is read by the prompt or by a question.
"""
type: Literal["PipeJudge"] = "PipeJudge"
pipe_category: Literal["PipeOperator"] = "PipeOperator"
prompt: str
- question: str
+ question: str | None = None
+ questions: dict[str, JudgeQuestionBlueprint] | None = None
model: JudgmentModelChoice | None = None
options: dict[str, str] | None = None
levels: list[str | JudgeRatingLevel] | None = None
@@ -97,18 +149,29 @@ class PipeJudgeBlueprint(PipeBlueprint):
"""Name the field a bundle written for the old operator gets wrong, where pydantic would only say one is missing.
Before this version `prompt` was a synonym of `question`, so a table setting `prompt` alone most
- likely holds a question. An empty prompt or question is refused here too.
+ likely holds a question. A pipe asks one `question` or several `questions`, never both. A field
+ written as null holds no value, as a dump writes the form not taken, so the form is read off the
+ values rather than the keys. An empty prompt or question is refused here too.
"""
if not isinstance(values, dict):
return values
raw_table = cast("dict[str, Any]", values)
- if "question" not in raw_table:
+ sets_question = raw_table.get("question") is not None
+ sets_questions = raw_table.get("questions") is not None
+ if not sets_question and not sets_questions:
msg = (
"A PipeJudge asks a question about the evidence of its prompt, and this one sets no `question`: "
- "`prompt` holds the evidence the question is asked over, and the question is written in `question`."
+ "`prompt` holds the evidence the question is asked over, and the question is written in `question`, "
+ "or several in `questions`."
+ )
+ raise ValueError(msg)
+ if sets_question and sets_questions:
+ msg = (
+ "A PipeJudge asks one `question` or several `questions`, and this one sets both `question` and `questions`: "
+ "keep `question` for a single verdict, or move it into `questions`."
)
raise ValueError(msg)
- if "prompt" not in raw_table:
+ if raw_table.get("prompt") is None:
msg = (
"A PipeJudge judges the evidence its `prompt` presents, and this one sets no `prompt`: write the evidence "
'as a template that reads the inputs, such as `prompt = "@message"`.'
@@ -120,7 +183,7 @@ class PipeJudgeBlueprint(PipeBlueprint):
if isinstance(prompt, str) and not prompt.strip():
msg = "A PipeJudge judges the evidence its prompt presents, so its `prompt` cannot be empty."
raise ValueError(msg)
- question = raw_table["question"]
+ question = raw_table.get("question")
if isinstance(question, str) and not question.strip():
msg = "A PipeJudge asks one question, so its `question` cannot be empty."
raise ValueError(msg)
@@ -128,104 +191,181 @@ class PipeJudgeBlueprint(PipeBlueprint):
@model_validator(mode="after")
def validate_question_kind(self) -> Self:
- if self.options is not None and self.levels is not None:
- msg = "A PipeJudge declares `options` for a choice question or `levels` for a rating question, not both."
+ if self.questions is None:
+ _validate_question_kind(
+ options=self.options, levels=self.levels, criteria=self.criteria, threshold=self.threshold, subject="this PipeJudge"
+ )
+ return self
+ if not self.questions:
+ msg = "A PipeJudge's `questions` holds at least one question, and this one holds none: add a question, or ask one in `question`."
raise ValueError(msg)
- if self.options is not None or self.levels is not None:
- kind_field = "options" if self.options is not None else "levels"
- for yes_no_field in ("criteria", "threshold"):
- if getattr(self, yes_no_field) is not None:
- msg = (
- f"`{yes_no_field}` applies to a yes/no question only, and this PipeJudge declares `{kind_field}`, "
- f"which makes it a {self.judgment_kind} question. Remove `{yes_no_field}`."
- )
- raise ValueError(msg)
- if self.options is not None:
- if len(self.options) < 2:
- msg = f"A choice question needs at least two `options` to choose between, and this one declares {len(self.options)}."
+ for question_name in self.questions:
+ if not is_admitted_structure_field_name(field_name=question_name):
+ msg = (
+ "Each key of `questions` names the output field holding its question's verdict, so it must be a name a concept "
+ f"structure admits for a field, and '{question_name}' is not: a field name is a Python identifier that is not a "
+ "Python keyword, does not start with an underscore and is not a reserved name."
+ )
raise ValueError(msg)
- if any(not option.strip() for option in self.options):
- msg = "Every key of `options` names an option, so none may be empty."
+ for kind_field in QUESTION_KIND_FIELDS:
+ if getattr(self, kind_field) is not None:
+ msg = (
+ f"A PipeJudge asking several `questions` sets `{kind_field}` on each question rather than on the pipe: "
+ f"move `{kind_field}` into the question it belongs to."
+ )
raise ValueError(msg)
- if self.levels is not None:
- _validate_levels(levels=self.levels)
- if self.threshold is not None and not 0 < self.threshold < 1:
- msg = f"`threshold` is a probability of yes strictly between 0 and 1, and this PipeJudge declares {self.threshold}."
- raise ValueError(msg)
+ for question_name, question in self.questions.items():
+ _validate_question_kind(
+ options=question.options,
+ levels=question.levels,
+ criteria=question.criteria,
+ threshold=question.threshold,
+ subject=f"the question '{question_name}'",
+ )
return self
@property
- def judgment_kind(self) -> JudgmentKind:
- if self.options is not None:
- return JudgmentKind.CHOICE
- if self.levels is not None:
- return JudgmentKind.RATING
- return JudgmentKind.YES_NO
+ def judgment_kind(self) -> JudgmentKind | None:
+ """The single form's kind, read off the pipe's `options` and `levels`; `None` in the multi form, whose questions each have their own."""
+ if self.questions is not None:
+ return None
+ return _judgment_kind_of(options=self.options, levels=self.levels)
@property
def rating_levels(self) -> list[RatingLevel] | None:
- """The scale as the judgment family carries it: a string level is its description."""
- if self.levels is None:
- return None
- rating_levels: list[RatingLevel] = []
- for level in self.levels:
- if isinstance(level, str):
- rating_levels.append(RatingLevel(description=level))
- else:
- rating_levels.append(RatingLevel(label=level.label, description=level.description))
- return rating_levels
+ """The single form's scale as the judgment family carries it: a string level is its description."""
+ return _rating_levels_of(levels=self.levels)
@override
def validate_inputs(self):
- """The two-way check every operator that reads its inputs through templates applies, over the prompt and the question together."""
+ """The two-way check every operator that reads its inputs through templates applies, over the prompt and every question together."""
declared_inputs: set[str] = set(self.inputs.keys()) if self.inputs else set()
variable_paths = _template_variable_paths(
template_source=self.prompt, template_category=TemplateCategory.LLM_PROMPT, declared_inputs=declared_inputs, field_name="prompt"
)
- variable_paths |= _template_variable_paths(
- template_source=self.question, template_category=TemplateCategory.BASIC, declared_inputs=declared_inputs, field_name="question"
- )
- check_inputs_match_variables(declared_inputs=declared_inputs, variable_paths=variable_paths, reader="prompt or question")
+ if self.questions is None:
+ # The before-validator guarantees the single form's question.
+ question = self.question or ""
+ variable_paths |= _template_variable_paths(
+ template_source=question, template_category=TemplateCategory.BASIC, declared_inputs=declared_inputs, field_name="question"
+ )
+ check_inputs_match_variables(declared_inputs=declared_inputs, variable_paths=variable_paths, reader="prompt or question")
+ return
+ for question_name, question_blueprint in self.questions.items():
+ variable_paths |= _template_variable_paths(
+ template_source=question_blueprint.question,
+ template_category=TemplateCategory.BASIC,
+ declared_inputs=declared_inputs,
+ field_name=f"question '{question_name}'",
+ )
+ check_inputs_match_variables(declared_inputs=declared_inputs, variable_paths=variable_paths, reader="prompt or questions")
@override
def validate_output(self):
parsed_output = parse_concept_with_multiplicity(concept_ref_or_code=self.output)
- if parsed_output.multiplicity:
+ if not parsed_output.multiplicity:
+ return
+ if self.questions is not None:
msg = (
- f"A PipeJudge produces one verdict, so its output carries no multiplicity, and this one declares '{self.output}'. "
- "To judge each item of a list, map the PipeJudge over it with a PipeBatch."
+ f"A PipeJudge asking several questions fills one structure with their verdicts, so its output carries no multiplicity, "
+ f"and this one declares '{self.output}'. To judge each item of a list, map the PipeJudge over it with a PipeBatch."
)
raise ValueError(msg)
+ msg = (
+ f"A PipeJudge produces one verdict, so its output carries no multiplicity, and this one declares '{self.output}'. "
+ "To judge each item of a list, map the PipeJudge over it with a PipeBatch."
+ )
+ raise ValueError(msg)
+
+
+def _judgment_kind_of(*, options: dict[str, str] | None, levels: list[str | JudgeRatingLevel] | None) -> JudgmentKind:
+ """A question's kind, read off which of `options` and `levels` it declares."""
+ if options is not None:
+ return JudgmentKind.CHOICE
+ if levels is not None:
+ return JudgmentKind.RATING
+ return JudgmentKind.YES_NO
+
+
+def _rating_levels_of(*, levels: list[str | JudgeRatingLevel] | None) -> list[RatingLevel] | None:
+ """A declared scale as the judgment family carries it: a string level is its description."""
+ if levels is None:
+ return None
+ rating_levels: list[RatingLevel] = []
+ for level in levels:
+ if isinstance(level, str):
+ rating_levels.append(RatingLevel(description=level))
+ else:
+ rating_levels.append(RatingLevel(label=level.label, description=level.description))
+ return rating_levels
-def _validate_levels(*, levels: list[str | JudgeRatingLevel]) -> None:
+def _validate_question_kind(
+ *,
+ options: dict[str, str] | None,
+ levels: list[str | JudgeRatingLevel] | None,
+ criteria: JudgeYesNoCriteria | None,
+ threshold: float | None,
+ subject: str,
+) -> None:
+ """The single form's rules on the fields deciding a question's kind, for a pipe or for one of its questions, as `subject` names it."""
+ if options is not None and levels is not None:
+ msg = f"A question declares `options` for a choice question or `levels` for a rating question, not both, and {subject} declares both."
+ raise ValueError(msg)
+ if options is not None or levels is not None:
+ kind_field = "options" if options is not None else "levels"
+ judgment_kind = _judgment_kind_of(options=options, levels=levels)
+ for yes_no_field, yes_no_value in (("criteria", criteria), ("threshold", threshold)):
+ if yes_no_value is not None:
+ msg = (
+ f"`{yes_no_field}` applies to a yes/no question only, and {subject} declares `{kind_field}`, "
+ f"which makes it a {judgment_kind} question. Remove `{yes_no_field}`."
+ )
+ raise ValueError(msg)
+ if options is not None:
+ if len(options) < 2:
+ msg = f"A choice question needs at least two `options` to choose between, and {subject} declares {len(options)}."
+ raise ValueError(msg)
+ if any(not option.strip() for option in options):
+ msg = f"Every key of `options` names an option, so none may be empty, and {subject} declares an empty one."
+ raise ValueError(msg)
+ if levels is not None:
+ _validate_levels(levels=levels, subject=subject)
+ if threshold is not None and not 0 < threshold < 1:
+ msg = f"`threshold` is a probability of yes strictly between 0 and 1, and {subject} declares {threshold}."
+ raise ValueError(msg)
+
+
+def _validate_levels(*, levels: list[str | JudgeRatingLevel], subject: str) -> None:
"""A scale of at least two levels, none empty, its labels all or none and distinct, since a label is what the verdict reports."""
if len(levels) < 2:
- msg = f"A rating question needs at least two `levels` on its scale, and this one declares {len(levels)}."
+ msg = f"A rating question needs at least two `levels` on its scale, and {subject} declares {len(levels)}."
raise ValueError(msg)
labels: list[str] = []
for index_level, level in enumerate(levels):
if isinstance(level, str):
if not level.strip():
- msg = f"Every level of a rating question describes a situation, so none may be empty: level {index_level} is."
+ msg = f"Every level of a rating question describes a situation, so none may be empty, and level {index_level} of {subject} is."
raise ValueError(msg)
continue
if level.label is None and level.description is None:
- msg = f"A rating level written as a table carries a `label`, a `description` or both, and level {index_level} carries neither."
+ msg = (
+ f"A rating level written as a table carries a `label`, a `description` or both, and level {index_level} of {subject} carries neither."
+ )
raise ValueError(msg)
for field_name, field_value in (("label", level.label), ("description", level.description)):
if field_value is not None and not field_value.strip():
- msg = f"A rating level's `{field_name}` cannot be empty, and level {index_level}'s is."
+ msg = f"A rating level's `{field_name}` cannot be empty, and that of level {index_level} of {subject} is."
raise ValueError(msg)
if level.label is not None:
labels.append(level.label)
if labels and len(labels) != len(levels):
- msg = "On a rating scale every level carries a `label` or none does, and this one labels some of its levels only."
+ msg = f"On a rating scale every level carries a `label` or none does, and {subject} labels some of its levels only."
raise ValueError(msg)
duplicated_labels = sorted({label for label in labels if labels.count(label) > 1})
if duplicated_labels:
repeated = ", ".join(f"'{label}'" for label in duplicated_labels)
- msg = f"A rating verdict reports the label of its level, so the labels of one scale are distinct, and {repeated} is repeated."
+ msg = f"A rating verdict reports the label of its level, so the labels of one scale are distinct, and {subject} repeats {repeated}."
raise ValueError(msg)
diff --git a/pipelex/pipe_operators/judge/pipe_judge_factory.py b/pipelex/pipe_operators/judge/pipe_judge_factory.py
index 9beaa24ec..388c9c4dc 100644
--- a/pipelex/pipe_operators/judge/pipe_judge_factory.py
+++ b/pipelex/pipe_operators/judge/pipe_judge_factory.py
@@ -2,14 +2,14 @@ from typing import Any
from typing_extensions import override
-from pipelex.cogt.judgment.judgment_models import YesNoCriteria
+from pipelex.cogt.judgment.judgment_models import RatingLevel, YesNoCriteria
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
from pipelex.kernel.prompt_assembly import UserPromptContent
from pipelex.pipe_machinery.pipe_factory import PipeFactoryProtocol
-from pipelex.pipe_operators.judge.pipe_judge import PipeJudge, make_judgment_question
-from pipelex.pipe_operators.judge.pipe_judge_blueprint import PipeJudgeBlueprint
+from pipelex.pipe_operators.judge.pipe_judge import PipeJudge, PipeJudgeQuestion, make_judgment_question
+from pipelex.pipe_operators.judge.pipe_judge_blueprint import JudgeYesNoCriteria, PipeJudgeBlueprint
from pipelex.pipe_operators.shared.template_file_references import analyze_template_file_references
from pipelex.tools.jinja2.template_category import TemplateCategory
@@ -38,19 +38,31 @@ class PipeJudgeFactory(PipeFactoryProtocol[PipeJudgeBlueprint, PipeJudge]):
image_references=prompt_files.image_references or None,
document_references=prompt_files.document_references or None,
)
- # The question presents no file: whatever image or document it reads is held for the load to refuse.
- question_files = analyze_template_file_references(
- template_source=blueprint.question, input_specs=input_specs, domain_code=domain_code, template_category=TemplateCategory.BASIC
- )
- question_image_references = question_files.image_references or None
- question_document_references = question_files.document_references or None
- criteria = blueprint.criteria
- judgment_question = make_judgment_question(
- question_template=blueprint.question,
- options=blueprint.options,
- levels=blueprint.rating_levels,
- criteria=YesNoCriteria(yes=criteria.yes, no=criteria.no) if criteria is not None else None,
- )
+ question: PipeJudgeQuestion | None = None
+ if blueprint.question is not None:
+ question = _make_pipe_judge_question(
+ question_template=blueprint.question,
+ options=blueprint.options,
+ levels=blueprint.rating_levels,
+ criteria=blueprint.criteria,
+ threshold=blueprint.threshold,
+ input_specs=input_specs,
+ domain_code=domain_code,
+ )
+ questions: dict[str, PipeJudgeQuestion] | None = None
+ if blueprint.questions is not None:
+ questions = {
+ question_name: _make_pipe_judge_question(
+ question_template=question_blueprint.question,
+ options=question_blueprint.options,
+ levels=question_blueprint.rating_levels,
+ criteria=question_blueprint.criteria,
+ threshold=question_blueprint.threshold,
+ input_specs=input_specs,
+ domain_code=domain_code,
+ )
+ for question_name, question_blueprint in blueprint.questions.items()
+ }
return PipeJudge(
domain_code=domain_code,
code=pipe_code,
@@ -59,8 +71,37 @@ class PipeJudgeFactory(PipeFactoryProtocol[PipeJudgeBlueprint, PipeJudge]):
inputs=inputs,
judgment_choice=blueprint.model,
prompt_content=prompt_content,
- judgment_question=judgment_question,
- question_image_references=question_image_references,
- question_document_references=question_document_references,
- threshold=blueprint.threshold,
+ question=question,
+ questions=questions,
)
+
+
+def _make_pipe_judge_question(
+ *,
+ question_template: str,
+ options: dict[str, str] | None,
+ levels: list[RatingLevel] | None,
+ criteria: JudgeYesNoCriteria | None,
+ threshold: float | None,
+ input_specs: dict[str, str],
+ domain_code: str,
+) -> PipeJudgeQuestion:
+ """One question as the operator holds it, built alike whether the step asks it alone or among several.
+
+ The question presents no file: whatever image or document it reads is held for the load to refuse.
+ """
+ question_files = analyze_template_file_references(
+ template_source=question_template, input_specs=input_specs, domain_code=domain_code, template_category=TemplateCategory.BASIC
+ )
+ judgment_question = make_judgment_question(
+ question_template=question_template,
+ options=options,
+ levels=levels,
+ criteria=YesNoCriteria(yes=criteria.yes, no=criteria.no) if criteria is not None else None,
+ )
+ return PipeJudgeQuestion(
+ judgment_question=judgment_question,
+ threshold=threshold,
+ image_references=question_files.image_references or None,
+ document_references=question_files.document_references or None,
+ )
diff --git a/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/bundle.mthds
new file mode 100644
index 000000000..a0192e0fa
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/bundle.mthds
@@ -0,0 +1,43 @@
+domain = "message_triage"
+description = "Triaging an inbox message with several questions asked in one judgment"
+main_pipe = "triage_message"
+
+[concept.Team]
+description = "The team that handles a message"
+refines = "Choice"
+
+[concept.MessageTriage]
+description = "What a triage of an inbox message found"
+
+[concept.MessageTriage.structure]
+urgent = { type = "concept", concept_ref = "YesNo", description = "Whether the message needs an answer today", required = true }
+team = { type = "concept", concept_ref = "Team", description = "The team that should handle the message", required = true }
+severity = { type = "concept", concept_ref = "Rating", description = "How severe the problem the message reports is" }
+
+[pipe.triage_message]
+type = "PipeJudge"
+description = "Triages a message: whether it is urgent, which team handles it, and how severe its problem is"
+inputs = { message = "Text" }
+output = "MessageTriage"
+model = "@default-judgment"
+prompt = """
+A message from someone writing to the inbox:
+@message
+"""
+
+[pipe.triage_message.questions.urgent]
+question = "Does this message need an answer today?"
+criteria = { yes = "Something is broken or blocked for the sender right now", no = "The sender can wait for a reply" }
+threshold = 0.7
+
+[pipe.triage_message.questions.team]
+question = "Which team should handle this message?"
+options = { billing = "Charges, invoices, refunds and subscriptions", technical = "Errors, outages and features that do not work", other = "" }
+
+[pipe.triage_message.questions.severity]
+question = "How severe is the problem this message reports?"
+levels = [
+ { label = "Minor", description = "Nothing stops working" },
+ { label = "Major", description = "A workaround exists" },
+ { label = "Blocking", description = "The sender cannot do their work at all" },
+]
diff --git a/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/entry.toml b/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/entry.toml
new file mode 100644
index 000000000..f9bb03194
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/operator_judge_message_triage/entry.toml
@@ -0,0 +1,11 @@
+name = "operator_judge_message_triage"
+description = "Several questions about one inbox message asked in one judgment, their verdicts filling the fields of a triage named after them"
+
+validity = "valid"
+tier = "inference"
+granularity = "focused"
+
+covers = [
+ "operator.judge",
+ "feature.structured_output",
+]
diff --git a/subject_grants.toml b/subject_grants.toml
index 4b8a11c71..4e25b479f 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -3544,6 +3544,10 @@ rationale = "One post-processing pass of the schema pipeline, called as a verb o
param = "schema"
rationale = "Schema-pipeline stage transforming the schema dict; single operand."
+["pipelex/language/mthds_schema_generator.py::_constrain_judge_questions"]
+param = "schema"
+rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."
+
["pipelex/language/mthds_schema_generator.py::_constrain_rating_levels"]
param = "schema"
rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/pipe_judge.mthds b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/pipe_judge.mthds
index e6a121357..54d49d0d9 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/pipe_judge.mthds
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/pipe_judge.mthds
@@ -1,5 +1,17 @@
domain = "pipe_judge_e2e"
-description = "Test PipeJudge with one question of each kind, each asked about the evidence its prompt presents"
+description = "Test PipeJudge with one question of each kind, each asked about the evidence its prompt presents, and with several asked at once"
+
+[concept.Team]
+description = "The team that handles a message"
+refines = "Choice"
+
+[concept.MessageTriage]
+description = "What a triage of an inbox message found"
+
+[concept.MessageTriage.structure]
+urgent = { type = "concept", concept_ref = "YesNo", description = "Whether the message needs an answer today", required = true }
+team = { type = "concept", concept_ref = "Team", description = "The team that should handle the message", required = true }
+severity = { type = "concept", concept_ref = "Rating", description = "How severe the problem the message reports is" }
[pipe.judge_urgent_e2e]
type = "PipeJudge"
@@ -84,3 +96,31 @@ levels = [
{ label = "Degraded", description = "a workaround exists" },
{ label = "Blocking", description = "users cannot do their work at all" },
]
+
+[pipe.triage_message_e2e]
+type = "PipeJudge"
+description = "Triage a message: whether it is urgent, which team handles it, and how severe its problem is"
+inputs = { message = "Text" }
+output = "MessageTriage"
+model = "@default-judgment"
+prompt = """
+A message from someone writing to the inbox:
+@message
+"""
+
+[pipe.triage_message_e2e.questions.urgent]
+question = "Does this message need an answer today?"
+criteria = { yes = "Something is broken or blocked for the sender right now", no = "The sender can wait for a reply" }
+threshold = 0.7
+
+[pipe.triage_message_e2e.questions.team]
+question = "Which team should handle this message?"
+options = { billing = "Charges, invoices, refunds and subscriptions", technical = "Errors, outages and features that do not work", other = "" }
+
+[pipe.triage_message_e2e.questions.severity]
+question = "How severe is the problem this message reports?"
+levels = [
+ { label = "Cosmetic", description = "nothing stops working" },
+ { label = "Degraded", description = "a workaround exists" },
+ { label = "Blocking", description = "users cannot do their work at all" },
+]
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_data.py b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_data.py
index 7c58f178d..1ad5070d9 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_data.py
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_data.py
@@ -20,6 +20,13 @@ class PipeJudgeTestCases:
SEVERITY_LABELS: ClassVar[list[str]] = ["Cosmetic", "Degraded", "Blocking"]
+ # The questions the triage asks about the urgent message, by name, exactly as sent.
+ TRIAGE_QUESTIONS: ClassVar[dict[str, str]] = {
+ "urgent": "Does this message need an answer today?",
+ "team": "Which team should handle this message?",
+ "severity": "How severe is the problem this message reports?",
+ }
+
# The verdicts a live judgment returns, measured on the deck's default judgment model.
EXPECTED_URGENT_PROBABILITY: ClassVar[float] = 0.98
EXPECTED_BILLING_PROBABILITY: ClassVar[float] = 1.0
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_pipe_judge.py b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_pipe_judge.py
index 6805fd3a3..3d0da5dcf 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_pipe_judge.py
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_judge/test_pipe_judge.py
@@ -1,10 +1,13 @@
-"""E2E tests for PipeJudge: one question of each kind, asked about the evidence its prompt presents, on the default judgment model."""
+"""E2E tests for PipeJudge: one question of each kind, and several at once, asked about the evidence its prompt presents."""
import pytest
from pipelex import pretty_print
from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.rating_content import RatingContent
from pipelex.core.stuffs.text_content import TextContent
+from pipelex.core.stuffs.yes_no_content import YesNoContent
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.system.pipe_run_mode import PipeRunMode
from tests.e2e.pipelex.pipes.pipe_operators.pipe_judge.test_data import PipeJudgeTestCases
@@ -116,3 +119,40 @@ class TestPipeJudge:
assert set(verdict.probabilities) == {"0", "1", "2"}
assert verdict.position is not None
assert abs(verdict.position - PipeJudgeTestCases.EXPECTED_BLOCKING_POSITION) <= PROBABILITY_TOLERANCE * 2
+
+ async def test_several_questions_fill_a_structure(self, pipe_run_mode: PipeRunMode) -> None:
+ """One request asks every question about the same evidence, and each verdict fills the field of its question's name."""
+ runner = PipelexMTHDSProtocol(library_dirs=LIBRARY_DIRS, pipe_run_mode=pipe_run_mode)
+ pipeline_response = await runner.execute(
+ pipe_code="triage_message_e2e",
+ inputs={"message": TextContent(text=PipeJudgeTestCases.URGENT_MESSAGE)},
+ )
+
+ pipe_output = pipeline_response.pipe_output
+ assert pipe_output is not None
+ graph_spec = pipe_output.graph_spec
+ assert graph_spec is not None
+ (judge_node,) = [node for node in graph_spec.nodes if node.pipe_code == "triage_message_e2e"]
+ execution_data = judge_node.execution_data
+ assert execution_data["rendered_prompt"] == PipeJudgeTestCases.URGENT_EVIDENCE
+ assert {name: question["rendered_question"] for name, question in execution_data["questions"].items()} == PipeJudgeTestCases.TRIAGE_QUESTIONS
+ assert {name: question["judgment_kind"] for name, question in execution_data["questions"].items()} == {
+ "urgent": "yes_no",
+ "team": "choice",
+ "severity": "rating",
+ }
+ # Read field by field off the structure's plain data, each verdict typed by its native's class.
+ triage = pipe_output.main_stuff.content.model_dump()
+ if pipe_run_mode.is_live:
+ pretty_print(triage, title="Several judgments")
+ pretty_print(execution_data["questions"], title="Their outcomes")
+ urgent = YesNoContent.model_validate(triage["urgent"])
+ # The declared threshold of 0.7 decides the urgency from the probability.
+ assert urgent.yes_no is True
+ assert urgent.probability is not None
+ assert abs(urgent.probability - PipeJudgeTestCases.EXPECTED_URGENT_PROBABILITY) <= PROBABILITY_TOLERANCE
+ assert ChoiceContent.model_validate(triage["team"]).choice == "technical"
+ assert triage["severity"] is not None
+ severity = RatingContent.model_validate(triage["severity"])
+ assert severity.level == PipeJudgeTestCases.EXPECTED_BLOCKING_LEVEL
+ assert severity.label == "Blocking"
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py b/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
index dd93a96d1..17ec10ffc 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/conftest.py
@@ -3,8 +3,8 @@ from collections.abc import Iterator
import pytest
from pipelex.cogt.model_backends.model_type import ModelType
-from pipelex.runtime_hub import get_model_deck
-from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData
+from pipelex.runtime_hub import get_class_registry, get_model_deck
+from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import ClassBackedOpenTriage, ClassBackedTriage, PipeJudgeLoadTestData
@pytest.fixture
@@ -40,3 +40,16 @@ def unserved_judgment_waterfall() -> Iterator[str]:
yield "~unserved_judges"
finally:
del model_deck.judgment_waterfalls["unserved_judges"]
+
+
+def class_backed_triages() -> Iterator[None]:
+ """The hand-written triage structure classes, registered for one test so a concept may name them as its structure."""
+ class_registry = get_class_registry()
+ class_registry.register_class(ClassBackedTriage)
+ class_registry.register_class(ClassBackedOpenTriage)
+ try:
+ yield
+ finally:
+ class_registry.unregister_class(ClassBackedTriage)
+ class_registry.unregister_class(ClassBackedOpenTriage)
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_data.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_data.py
index 845fa2ce3..f625457bd 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/test_data.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_data.py
@@ -1,5 +1,12 @@
import json
-from typing import ClassVar
+from typing import Any, ClassVar
+
+from pydantic import Field
+
+from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.rating_content import RatingContent
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.yes_no_content import YesNoContent
class PipeJudgeLoadTestData:
@@ -42,6 +49,112 @@ output = "{output}"
"""
+class PipeJudgeSeveralQuestionsTestData:
+ """A bundle with one PipeJudge step asking several questions, built from its output's structure and its question tables."""
+
+ PIPE_CODE = "judge_it"
+ MESSAGE = "Every payment page has shown an error since this morning, and we cannot take any order."
+
+ # The output's fields: a required yes/no, a required choice refined by `Team`, and an optional rating.
+ STRUCTURE: ClassVar[dict[str, str]] = {
+ "urgent": '{ type = "concept", concept_ref = "YesNo", description = "Whether the message is urgent", required = true }',
+ "team": '{ type = "concept", concept_ref = "Team", description = "The team that handles it", required = true }',
+ "severity": '{ type = "concept", concept_ref = "Rating", description = "How severe the reported issue is" }',
+ }
+
+ QUESTIONS: ClassVar[dict[str, str]] = {
+ "urgent": 'question = "Is the message urgent?"\nthreshold = 0.7\ncriteria = { yes = "It cannot wait", no = "It can wait" }',
+ "team": 'question = "Which team handles it?"\noptions = { billing = "Charges and invoices", technical = "Errors and outages" }',
+ "severity": 'question = "How severe is the reported issue?"\nlevels = [{ label = "Minor" }, { label = "Major" }]',
+ }
+
+ @classmethod
+ def bundle(
+ cls,
+ *,
+ structure: dict[str, str] | None = None,
+ questions: dict[str, str] | None = None,
+ output: str = "Triage",
+ inputs: str = '{ message = "Text" }',
+ prompt: str = "@message",
+ extra_concepts: str = "",
+ ) -> str:
+ """The step's output structure and question tables as written, each field and question a TOML line or table body by name."""
+ structure_lines = "\n".join(f"{name} = {field}" for name, field in (structure if structure is not None else cls.STRUCTURE).items())
+ question_tables = "\n\n".join(
+ f"[pipe.judge_it.questions.{name}]\n{table}" for name, table in (questions if questions is not None else cls.QUESTIONS).items()
+ )
+ return f"""
+domain = "judge_several"
+description = "Loading a PipeJudge step asking several questions"
+
+[concept]
+Team = {{ description = "The team a ticket goes to", refines = "Choice" }}
+Note = "A note"
+{extra_concepts}
+
+[concept.Triage]
+description = "What a triage of a message found"
+
+[concept.Triage.structure]
+{structure_lines}
+
+[pipe.judge_it]
+type = "PipeJudge"
+description = "Judge the evidence"
+inputs = {inputs}
+output = "{output}"
+model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"
+prompt = {json.dumps(prompt)}
+
+{question_tables}
+"""
+
+
+class ClassBackedTriage(StructuredContent):
+ """A hand-written structure class holding one verdict per question, the rating left optional."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(default=None, description="How severe the reported issue is")
+
+
+class ClassBackedOpenTriage(StructuredContent):
+ """A hand-written structure class whose yes/no field is typed `Any`, which maps to no concept."""
+
+ urgent: Any = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(default=None, description="How severe the reported issue is")
+
+
+class PipeJudgeClassBackedTestData:
+ """A PipeJudge asking several questions into a concept whose structure is a registered Python class."""
+
+ @classmethod
+ def bundle(cls, *, structure_class_name: str) -> str:
+ question_tables = "\n\n".join(
+ f"[pipe.judge_it.questions.{name}]\n{table}" for name, table in PipeJudgeSeveralQuestionsTestData.QUESTIONS.items()
+ )
+ return f"""
+domain = "judge_class_backed"
+description = "A PipeJudge filling a class-backed structure"
+
+[concept.Triage]
+description = "What a triage of a message found"
+structure = "{structure_class_name}"
+
+[pipe.judge_it]
+type = "PipeJudge"
+description = "Judge the evidence"
+inputs = {{ message = "Text" }}
+output = "Triage"
+model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"
+prompt = "@message"
+
+{question_tables}
+"""
+
+
class PipeJudgeOptionalFileTestData:
"""A PipeJudge whose evidence prompt reads an optional file, run with that file absent or present."""
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_execution_data.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_execution_data.py
new file mode 100644
index 000000000..eda8999bd
--- /dev/null
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_execution_data.py
@@ -0,0 +1,68 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment
+from pipelex.cogt.content_generation.content_generator import ContentGenerator
+from pipelex.cogt.judgment.judgment_models import ChoiceAnswer, JudgmentOutcome, RatingAnswer, YesNoAnswer
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.pipe_operators.judge.pipe_judge import PipeJudge
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.pipe_run_mode import PipeRunMode
+from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData, PipeJudgeSeveralQuestionsTestData
+
+_ANSWERS: dict[str, JudgmentOutcome] = {
+ "urgent": YesNoAnswer(probability=0.9),
+ "team": ChoiceAnswer(choice="technical", confidence=0.8),
+ "severity": RatingAnswer(level=1, confidence=0.7),
+}
+
+
+def _answer(*, judgment_assignment: JudgmentAssignment) -> dict[str, JudgmentOutcome]:
+ """Answer each question by its name, the single form's one question as a yes/no."""
+ return {name: _ANSWERS.get(name, YesNoAnswer(probability=0.9)) for name in judgment_assignment.questions}
+
+
[email protected](loop_scope="class")
+class TestPipeJudgeExecutionData:
+ @pytest.mark.parametrize(
+ ("topic", "bundle", "expected_kinds"),
+ [
+ pytest.param(
+ "one question",
+ PipeJudgeLoadTestData.bundle(step_fields=f'model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"'),
+ {"": "yes_no"},
+ id="one_question",
+ ),
+ pytest.param(
+ "several questions",
+ PipeJudgeSeveralQuestionsTestData.bundle(),
+ {"urgent": "yes_no", "team": "choice", "severity": "rating"},
+ id="several_questions",
+ ),
+ ],
+ )
+ async def test_a_question_kind_is_recorded_as_plain_text(
+ self,
+ mocker: MockerFixture,
+ topic: str, # ruff: ignore[unused-method-argument]
+ bundle: str,
+ expected_kinds: dict[str, str],
+ ) -> None:
+ """The kind is stored as a plain string, never as the enum, which a serializer would write out with its class."""
+ mocker.patch.object(ContentGenerator, "make_judgment_answers", side_effect=_answer)
+ execution_data_spy = mocker.spy(PipeJudge, "_register_execution_data")
+
+ await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(
+ mthds_contents=[bundle],
+ pipe_code="judge_it",
+ inputs={"message": TextContent(text=PipeJudgeSeveralQuestionsTestData.MESSAGE)},
+ )
+
+ execution_data = execution_data_spy.call_args.kwargs["execution_data"]
+ recorded_kinds = (
+ {name: record["judgment_kind"] for name, record in execution_data["questions"].items()}
+ if "questions" in execution_data
+ else {"": execution_data["judgment_kind"]}
+ )
+ assert recorded_kinds == expected_kinds
+ assert all(type(recorded_kind) is str for recorded_kind in recorded_kinds.values())
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_acceptances.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_acceptances.py
index 79d724674..65d83da89 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_acceptances.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_acceptances.py
@@ -3,7 +3,12 @@
import pytest
from pipelex.pipeline.validate_bundle import validate_bundle
-from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData
+from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import (
+ ClassBackedTriage,
+ PipeJudgeClassBackedTestData,
+ PipeJudgeLoadTestData,
+ PipeJudgeSeveralQuestionsTestData,
+)
_MODEL = f'model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"'
@@ -54,3 +59,35 @@ class TestPipeJudgeLoadAcceptances:
inputs='{ photo = "Image" }', prompt="Inspect this photo: $photo", step_fields=f'model = "{unserved_judgment_waterfall}"'
)
await validate_bundle(mthds_contents=[bundle])
+
+ async def test_several_questions_filling_a_structure_are_admitted(self) -> None:
+ """Each field holds its question's verdict native or a concept refining it, and an optional field may hold nothing."""
+ result = await validate_bundle(mthds_contents=[PipeJudgeSeveralQuestionsTestData.bundle()])
+ assert "judge_several.judge_it" in {pipe.pipe_ref for pipe in result.pipes}
+
+ async def test_an_output_refining_the_structured_concept_is_admitted(self) -> None:
+ """A refinement inherits the structure it refines, so its fields are the question names all the same."""
+ bundle = PipeJudgeSeveralQuestionsTestData.bundle(
+ output="UrgentTriage", extra_concepts='UrgentTriage = { description = "The triage of an urgent message", refines = "Triage" }'
+ )
+ await validate_bundle(mthds_contents=[bundle])
+
+ async def test_a_field_refining_its_verdict_native_is_admitted(self) -> None:
+ structure = {
+ **PipeJudgeSeveralQuestionsTestData.STRUCTURE,
+ "severity": '{ type = "concept", concept_ref = "Severity", description = "How severe it is" }',
+ }
+ bundle = PipeJudgeSeveralQuestionsTestData.bundle(
+ structure=structure, extra_concepts='Severity = { description = "How severe an issue is", refines = "Rating" }'
+ )
+ await validate_bundle(mthds_contents=[bundle])
+
+ async def test_an_input_read_by_one_of_several_questions_alone_is_admitted(self) -> None:
+ questions = {**PipeJudgeSeveralQuestionsTestData.QUESTIONS, "urgent": 'question = "Is the message about $topic and urgent?"'}
+ bundle = PipeJudgeSeveralQuestionsTestData.bundle(inputs='{ message = "Text", topic = "Text" }', questions=questions)
+ await validate_bundle(mthds_contents=[bundle])
+
+ @pytest.mark.usefixtures("class_backed_triages")
+ async def test_several_questions_filling_a_class_backed_structure_are_admitted(self) -> None:
+ """A structure that exists only as a Python class offers its class's fields, each typed by its verdict native's class."""
+ await validate_bundle(mthds_contents=[PipeJudgeClassBackedTestData.bundle(structure_class_name=ClassBackedTriage.__name__)])
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
index 5e52157ed..acdaaadcf 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_load_refusals.py
@@ -6,9 +6,16 @@ from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.runtime_hub import get_model_deck
-from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeLoadTestData
+from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import (
+ ClassBackedOpenTriage,
+ PipeJudgeClassBackedTestData,
+ PipeJudgeLoadTestData,
+ PipeJudgeSeveralQuestionsTestData,
+)
_MODEL = f'model = "{PipeJudgeLoadTestData.JUDGMENT_MODEL}"'
+_STRUCTURE = PipeJudgeSeveralQuestionsTestData.STRUCTURE
+_QUESTIONS = PipeJudgeSeveralQuestionsTestData.QUESTIONS
async def _refusal_report(bundle: str) -> str:
@@ -116,3 +123,86 @@ class TestPipeJudgeLoadRefusals:
report = await _refusal_report(PipeJudgeLoadTestData.bundle(output=output, step_fields=f"{_MODEL}\n{step_fields}"))
assert asks in report
assert f"declares `native.{output}`" in report
+
+ @pytest.mark.parametrize(
+ ("output", "shape"),
+ [
+ pytest.param("Text", "holds its value in a single field", id="a_single_field_native"),
+ pytest.param("Note", "is declared with neither a structure nor refines", id="a_concept_described_only"),
+ pytest.param("Dynamic", "is structureless by definition", id="dynamic"),
+ ],
+ )
+ async def test_several_questions_into_an_output_without_a_structure_are_refused(self, output: str, shape: str) -> None:
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(output=output))
+ assert "fills a structure with one verdict per question, so its output must be a concept with a structure" in report
+ assert shape in report
+ assert "'inadequate_output_concept'" in report
+
+ async def test_a_field_no_question_answers_is_refused(self) -> None:
+ """Nothing could ever fill such a field, so the load refuses it, required or not."""
+ structure = {**_STRUCTURE, "notes": '{ type = "text", description = "Notes" }'}
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(structure=structure))
+ assert "the field 'notes', which no question answers" in report
+ assert "`judge_several.Triage`" in report
+
+ async def test_a_question_with_no_field_is_refused(self) -> None:
+ structure = {name: field for name, field in _STRUCTURE.items() if name != "severity"}
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(structure=structure))
+ assert "The question 'severity' has no field of `judge_several.Triage` to hold its verdict" in report
+
+ @pytest.mark.parametrize(
+ ("field", "complaint"),
+ [
+ pytest.param(
+ '{ type = "list", item_type = "concept", item_concept_ref = "YesNo", description = "Verdicts", required = true }',
+ "holds a list, and the question 'urgent' produces one verdict",
+ id="a_list_of_verdicts",
+ ),
+ pytest.param(
+ '{ type = "boolean", description = "Whether it is urgent", required = true }',
+ "holds a plain value rather than a concept, and the question 'urgent' asks a yes/no question",
+ id="a_plain_boolean",
+ ),
+ pytest.param(
+ '{ type = "concept", concept_ref = "Choice", description = "A choice", required = true }',
+ "The question 'urgent' asks a yes/no question, so its field must hold `native.YesNo` or a concept refining it, "
+ "and it holds `native.Choice`",
+ id="the_verdict_of_another_kind",
+ ),
+ pytest.param(
+ '{ type = "concept", concept_ref = "Dynamic", description = "Anything at all", required = true }',
+ "so its field must hold `native.YesNo` or a concept refining it, and it holds `native.Dynamic`",
+ id="dynamic",
+ ),
+ pytest.param(
+ '{ type = "concept", concept_ref = "Anything", description = "Any value at all", required = true }',
+ "so its field must hold `native.YesNo` or a concept refining it, and it holds `native.Anything`",
+ id="anything",
+ ),
+ ],
+ )
+ async def test_a_field_that_cannot_hold_its_question_verdict_is_refused(self, field: str, complaint: str) -> None:
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(structure={**_STRUCTURE, "urgent": field}))
+ assert complaint in report
+ assert "'inadequate_output_concept'" in report
+
+ @pytest.mark.usefixtures("class_backed_triages")
+ async def test_a_class_field_no_concept_describes_is_refused(self) -> None:
+ report = await _refusal_report(PipeJudgeClassBackedTestData.bundle(structure_class_name=ClassBackedOpenTriage.__name__))
+ assert "holds a value no concept describes, since its Python type" in report
+ assert "maps to no concept, and the question 'urgent' asks a yes/no question" in report
+ assert "'inadequate_output_concept'" in report
+
+ @pytest.mark.usefixtures("judgment_model_reading_files")
+ async def test_a_file_read_by_one_of_several_questions_is_refused_naming_it(self) -> None:
+ questions = {**_QUESTIONS, "urgent": 'question = "Is $photo showing an emergency?"'}
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(inputs='{ message = "Text", photo = "Image" }', questions=questions))
+ assert "reads 'photo' in its question 'urgent'" in report
+ assert "only the prompt presents files" in report
+ assert "'input_stuff_spec_mismatch'" in report
+
+ async def test_an_unguarded_optional_input_in_one_of_several_questions_is_refused(self) -> None:
+ questions = {**_QUESTIONS, "team": 'question = "Which team handles it, given $note?"\noptions = { billing = "", technical = "" }'}
+ report = await _refusal_report(PipeJudgeSeveralQuestionsTestData.bundle(inputs='{ message = "Text", note = "Text?" }', questions=questions))
+ assert "'optional_input_unguarded'" in report
+ assert "note" in report
diff --git a/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_several_questions_run.py b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_several_questions_run.py
new file mode 100644
index 000000000..ecea3cfad
--- /dev/null
+++ b/tests/integration/pipelex/pipes/operator/pipe_judge/test_pipe_judge_several_questions_run.py
@@ -0,0 +1,117 @@
+"""A PipeJudge asking several questions runs as one judgment and fills its output's structure, a refusal included."""
+
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.assignment_models import JudgmentAssignment
+from pipelex.cogt.content_generation.content_generator import ContentGenerator
+from pipelex.cogt.judgment.judgment_models import (
+ ChoiceAnswer,
+ ChoiceQuestion,
+ JudgmentOutcome,
+ JudgmentRefusal,
+ RatingAnswer,
+ RatingLevel,
+ RatingQuestion,
+ YesNoAnswer,
+ YesNoCriteria,
+ YesNoQuestion,
+)
+from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.rating_content import RatingContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.core.stuffs.yes_no_content import YesNoContent
+from pipelex.pipe_operators.judge.pipe_judge import PipeJudge
+from pipelex.pipeline.exceptions import PipelineExecutionError
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.pipe_run_mode import PipeRunMode
+from tests.integration.pipelex.pipes.operator.pipe_judge.test_data import PipeJudgeSeveralQuestionsTestData
+
+_ANSWERS: dict[str, JudgmentOutcome] = {
+ "urgent": YesNoAnswer(probability=0.6),
+ "team": ChoiceAnswer(choice="technical", confidence=0.8),
+ "severity": RatingAnswer(level=1, confidence=0.7),
+}
+
+
+async def _run(*, pipe_run_mode: PipeRunMode) -> Any:
+ response = await PipelexMTHDSProtocol(pipe_run_mode=pipe_run_mode).execute(
+ mthds_contents=[PipeJudgeSeveralQuestionsTestData.bundle()],
+ pipe_code=PipeJudgeSeveralQuestionsTestData.PIPE_CODE,
+ inputs={"message": TextContent(text=PipeJudgeSeveralQuestionsTestData.MESSAGE)},
+ )
+ assert response.pipe_output is not None
+ return response.pipe_output.main_stuff.content
+
+
[email protected](loop_scope="class")
+class TestPipeJudgeSeveralQuestionsRun:
+ async def test_every_verdict_fills_its_field_and_a_refusal_leaves_an_optional_one_absent(self, mocker: MockerFixture) -> None:
+ """One job asks every question; the optional field a refusal leaves absent is recorded in the execution data."""
+ asked_jobs: list[JudgmentAssignment] = []
+
+ def _answer(*, judgment_assignment: JudgmentAssignment) -> dict[str, JudgmentOutcome]:
+ asked_jobs.append(judgment_assignment)
+ return {**_ANSWERS, "severity": JudgmentRefusal()}
+
+ mocker.patch.object(ContentGenerator, "make_judgment_answers", side_effect=_answer)
+ execution_data_spy = mocker.spy(PipeJudge, "_register_execution_data")
+
+ triage = await _run(pipe_run_mode=PipeRunMode.LIVE)
+
+ (job,) = asked_jobs
+ assert list(job.questions) == ["urgent", "team", "severity"]
+ assert job.questions["urgent"] == YesNoQuestion(
+ instructions="Is the message urgent?", criteria=YesNoCriteria(yes="It cannot wait", no="It can wait")
+ )
+ assert job.questions["team"] == ChoiceQuestion(
+ instructions="Which team handles it?", options={"billing": "Charges and invoices", "technical": "Errors and outages"}
+ )
+ assert job.questions["severity"] == RatingQuestion(
+ instructions="How severe is the reported issue?", levels=[RatingLevel(label="Minor"), RatingLevel(label="Major")]
+ )
+ assert job.prompt.text == f"<message>\n{PipeJudgeSeveralQuestionsTestData.MESSAGE}\n</message>"
+ # The threshold of 0.7 turns the model's 0.6 into a no, and `Team` refines `Choice`, so its field holds a choice.
+ assert triage.urgent == YesNoContent(yes_no=False, probability=0.6)
+ assert isinstance(triage.team, ChoiceContent)
+ assert triage.team.choice == "technical"
+ assert triage.severity is None
+ execution_data = execution_data_spy.call_args.kwargs["execution_data"]
+ assert execution_data["rendered_prompt"] == job.prompt.text
+ assert execution_data["resolved_model"] == job.judgment_setting.model
+ assert execution_data["questions"] == {
+ "urgent": {
+ "rendered_question": "Is the message urgent?",
+ "judgment_kind": "yes_no",
+ "threshold": 0.7,
+ "threshold_applied": True,
+ "outcome": {"kind": "yes_no", "probability": 0.6},
+ },
+ "team": {
+ "rendered_question": "Which team handles it?",
+ "judgment_kind": "choice",
+ "outcome": {"kind": "choice", "choice": "technical", "confidence": 0.8},
+ },
+ "severity": {
+ "rendered_question": "How severe is the reported issue?",
+ "judgment_kind": "rating",
+ "outcome": {"kind": "refusal"},
+ },
+ }
+
+ async def test_a_refusal_behind_a_required_field_fails_the_step_naming_the_question(self, mocker: MockerFixture) -> None:
+ mocker.patch.object(ContentGenerator, "make_judgment_answers", return_value={**_ANSWERS, "team": JudgmentRefusal()})
+
+ with pytest.raises(PipelineExecutionError) as exc_info:
+ await _run(pipe_run_mode=PipeRunMode.LIVE)
+
+ assert "declined to answer the question 'team', and its output field cannot hold nothing" in str(exc_info.value)
+
+ async def test_a_dry_run_answers_every_question(self) -> None:
+ triage = await _run(pipe_run_mode=PipeRunMode.DRY)
+
+ assert isinstance(triage.urgent, YesNoContent)
+ assert triage.team.choice == "billing"
+ assert triage.severity == RatingContent(level=0, label="Minor")
diff --git a/tests/unit/pipelex/cogt/judgment/fake_judgment_worker.py b/tests/unit/pipelex/cogt/judgment/fake_judgment_worker.py
index a57ef4005..49cb8fdaf 100644
--- a/tests/unit/pipelex/cogt/judgment/fake_judgment_worker.py
+++ b/tests/unit/pipelex/cogt/judgment/fake_judgment_worker.py
@@ -19,7 +19,7 @@ from pipelex.system.job_metadata import JobMetadata, RunMetadata
class FakeJudgmentWorker(JudgmentWorkerAbstract):
- """A worker that answers whatever it was told to, including the wrong thing, and records whether it was reached."""
+ """A worker that answers whatever it was told to, including the wrong thing, and records the jobs it was given."""
def __init__(
self,
@@ -33,10 +33,12 @@ class FakeJudgmentWorker(JudgmentWorkerAbstract):
self._answers = answers or {}
self._raises = raises
self.was_called = False
+ self.judged_jobs: list[JudgmentJob] = []
@override
async def _judge(self, judgment_job: JudgmentJob) -> dict[str, JudgmentOutcome]:
self.was_called = True
+ self.judged_jobs.append(judgment_job)
if self._raises is not None:
raise self._raises
return self._answers
diff --git a/tests/unit/pipelex/core/concepts/concept_blueprint/test_concept_blueprint.py b/tests/unit/pipelex/core/concepts/concept_blueprint/test_concept_blueprint.py
index f709d9ef7..0d6eb7818 100644
--- a/tests/unit/pipelex/core/concepts/concept_blueprint/test_concept_blueprint.py
+++ b/tests/unit/pipelex/core/concepts/concept_blueprint/test_concept_blueprint.py
@@ -173,6 +173,21 @@ class TestConceptStructureBlueprintValidation:
with pytest.raises(ValidationError, match="valid Python identifiers"):
ConceptBlueprint(description="Invalid field name", structure={field_name: "A value"})
+ @pytest.mark.parametrize(
+ ("field_name", "message_fragment"),
+ [
+ pytest.param("model_config", "Cannot use reserved fields in concept structure. Problematic fields: ''model_config''", id="reserved"),
+ pytest.param("_dynamic_output_concept", "Cannot use reserved fields in concept structure", id="reserved_before_underscore"),
+ pytest.param("_internal", "Cannot use 'field' starting with underscore in concept structure", id="underscore"),
+ pytest.param("_not-valid", "valid Python identifiers", id="identifier_before_underscore"),
+ ],
+ )
+ def test_concept_rejects_reserved_and_private_field_names(self, field_name: str, message_fragment: str):
+ """Each rule keeps its own message, checked in order: identifier, then reserved, then underscore."""
+ with pytest.raises(ValidationError) as exc_info:
+ ConceptBlueprint(description="Invalid field name", structure={field_name: "A value"})
+ assert message_fragment in str(exc_info.value)
+
def test_edge_cases(self):
"""Test edge cases for validation."""
# Valid field without default_value
diff --git a/tests/unit/pipelex/kernel/test_data.py b/tests/unit/pipelex/kernel/test_data.py
new file mode 100644
index 000000000..5d0f988f9
--- /dev/null
+++ b/tests/unit/pipelex/kernel/test_data.py
@@ -0,0 +1,122 @@
+from typing import ClassVar
+
+from pydantic import Field, create_model
+
+from pipelex.cogt.judgment.judgment_models import (
+ ChoiceAnswer,
+ ChoiceQuestion,
+ JudgmentOutcome,
+ RatingAnswer,
+ RatingLevel,
+ RatingQuestion,
+ YesNoAnswer,
+ YesNoQuestion,
+)
+from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.rating_content import RatingContent
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.yes_no_content import YesNoContent
+from pipelex.kernel.judgment_ops import AskedQuestion
+
+
+class JudgedTriage(StructuredContent):
+ """The structure a triage judgment fills: two required verdicts and an optional one."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(default=None, description="How severe the reported issue is")
+
+
+class DefaultedTriage(StructuredContent):
+ """A triage whose rating, left unrequired, carries a default verdict, as a structure field with a `default_value` is generated.
+
+ The field never holds nothing, so a refusal can neither empty it nor fill in its default, a verdict nobody gave.
+ """
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent = Field(default=RatingContent(level=0, label="Minor"), description="How severe the reported issue is")
+
+
+class NullableRequiredTriage(StructuredContent):
+ """A triage whose rating is required yet admits nothing, so a refusal stores nothing there."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(description="How severe the reported issue is")
+
+
+class NullableDefaultedTriage(StructuredContent):
+ """A triage whose rating carries a default yet admits nothing, so a refusal stores nothing there rather than the default."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(default=RatingContent(level=0, label="Minor"), description="How severe the reported issue is")
+
+
+# A hand-written class may give a field `None` for default while typing it without `None`: the field may hold
+# nothing, by its default, though `None` given outright is refused. Built dynamically, since a type checker
+# rightly refuses that default written in a class body.
+LooseDefaultTriage: type[StructuredContent] = create_model(
+ "LooseDefaultTriage",
+ __base__=StructuredContent,
+ urgent=(YesNoContent, Field(description="Whether the message is urgent")),
+ team=(ChoiceContent, Field(description="The team that handles it")),
+ severity=(RatingContent, Field(default=None, description="How severe the reported issue is")),
+)
+
+
+class AliasedRequiredTriage(StructuredContent):
+ """A triage whose required yes/no field goes by an alias: its answered verdict must land in it all the same."""
+
+ urgent: YesNoContent = Field(alias="isUrgent", description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent | None = Field(default=None, description="How severe the reported issue is")
+
+
+class AliasedOptionalTriage(StructuredContent):
+ """A triage whose optional choice field goes by an alias: its answered verdict must not be dropped for nothing."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent | None = Field(default=None, alias="handlingTeam", description="The team that handles it")
+ severity: RatingContent | None = Field(default=None, description="How severe the reported issue is")
+
+
+class AliasedDefaultedTriage(StructuredContent):
+ """A triage whose defaulted rating field goes by an alias: its answered verdict must not be swapped for the default."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent = Field(default=RatingContent(level=0, label="Minor"), alias="howSevere", description="How severe the reported issue is")
+
+
+def _blocking_rating() -> RatingContent:
+ return RatingContent(level=1, label="Major")
+
+
+class FactoryDefaultedTriage(StructuredContent):
+ """A triage whose rating, left unrequired, gets a default verdict from a factory: a refusal must never fill it in."""
+
+ urgent: YesNoContent = Field(description="Whether the message is urgent")
+ team: ChoiceContent = Field(description="The team that handles it")
+ severity: RatingContent = Field(default_factory=_blocking_rating, description="How severe the reported issue is")
+
+
+class MultiJudgmentTestCases:
+ """A triage asking three questions of a message, one of each kind, and the answers a worker gives them."""
+
+ MESSAGE: ClassVar[str] = "The invoice page crashes for every customer."
+
+ QUESTIONS: ClassVar[dict[str, AskedQuestion]] = {
+ "urgent": AskedQuestion(question=YesNoQuestion(instructions="Is this urgent, given that it says '{{ message }}'?"), threshold=0.8),
+ "team": AskedQuestion(question=ChoiceQuestion(instructions="Which team handles it?", options={"billing": None, "technical": "Errors"})),
+ "severity": AskedQuestion(
+ question=RatingQuestion(instructions="How severe is it?", levels=[RatingLevel(label="Minor"), RatingLevel(label="Major")])
+ ),
+ }
+
+ ANSWERS: ClassVar[dict[str, JudgmentOutcome]] = {
+ "urgent": YesNoAnswer(probability=0.9),
+ "team": ChoiceAnswer(choice="technical", confidence=0.7),
+ "severity": RatingAnswer(level=1, confidence=0.6),
+ }
diff --git a/tests/unit/pipelex/kernel/test_run_multi_judgment.py b/tests/unit/pipelex/kernel/test_run_multi_judgment.py
new file mode 100644
index 000000000..2caeaf38e
--- /dev/null
+++ b/tests/unit/pipelex/kernel/test_run_multi_judgment.py
@@ -0,0 +1,234 @@
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import JudgmentRefusedError
+from pipelex.cogt.judgment.judgment_models import (
+ JudgmentOutcome,
+ JudgmentRefusal,
+ YesNoAnswer,
+ YesNoQuestion,
+)
+from pipelex.cogt.judgment.judgment_setting import JudgmentSetting
+from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.stuffs.choice_content import ChoiceContent
+from pipelex.core.stuffs.rating_content import RatingContent
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.core.stuffs.yes_no_content import YesNoContent
+from pipelex.kernel.exceptions import JudgmentOutputFieldsError
+from pipelex.kernel.judgment_ops import AskedQuestion, run_multi_judgment
+from pipelex.kernel.judgment_results import MultiJudgmentResult
+from pipelex.kernel.pipelex_kernel import PipelexKernel
+from pipelex.kernel.prompt_assembly import UserPromptContent
+from pipelex.kernel.templating_style_ops import resolve_templating_style
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.tools.jinja2.template_category import TemplateCategory
+from tests.unit.pipelex.cogt.judgment.fake_judgment_worker import FakeJudgmentWorker, make_fake_judgment_model
+from tests.unit.pipelex.kernel.test_data import (
+ AliasedDefaultedTriage,
+ AliasedOptionalTriage,
+ AliasedRequiredTriage,
+ DefaultedTriage,
+ FactoryDefaultedTriage,
+ JudgedTriage,
+ LooseDefaultTriage,
+ MultiJudgmentTestCases,
+ NullableDefaultedTriage,
+ NullableRequiredTriage,
+)
+
+
[email protected](loop_scope="class")
+class TestRunMultiJudgment:
+ async def _run(
+ self,
+ mocker: MockerFixture,
+ *,
+ answers: dict[str, JudgmentOutcome],
+ run_mode: PipeRunMode = PipeRunMode.LIVE,
+ questions: dict[str, AskedQuestion] | None = None,
+ worker: FakeJudgmentWorker | None = None,
+ output_class: type[StructuredContent] = JudgedTriage,
+ ) -> tuple[FakeJudgmentWorker, MultiJudgmentResult]:
+ """Run the judgment through the content generator, which reaches the fake worker as it would reach a backend's."""
+ worker = worker or FakeJudgmentWorker(make_fake_judgment_model(), answers=answers)
+ mocker.patch("pipelex.cogt.content_generation.judgment_generate._make_judgment_worker", return_value=worker)
+ kernel = PipelexKernel.make(storage_scope="test/scope", read_scope=None, run_mode=run_mode, user_id="judgment-ops")
+ anything = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.ANYTHING)
+ memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(concept=anything, content=TextContent(text=MultiJudgmentTestCases.MESSAGE), name="message")
+ )
+ result = await run_multi_judgment(
+ memory=memory,
+ prompt_content=UserPromptContent(
+ template=TemplateBlueprint(template="A message from a customer: {{ message }}", category=TemplateCategory.LLM_PROMPT)
+ ),
+ questions=questions or MultiJudgmentTestCases.QUESTIONS,
+ judgment_setting=JudgmentSetting(model="fake-judgment-handle"),
+ concept=ConceptFactory.make(
+ concept_code="Triage", domain_code="judge_kernel", description="A triage", structure_class_name=output_class.__name__
+ ),
+ output_class=output_class,
+ job_metadata=kernel.make_step_metadata(pipe_code="triage_message"),
+ cogt_run_params=kernel.cogt_run_params,
+ templating_style=resolve_templating_style(authored=None),
+ result_name="triage",
+ )
+ return worker, result
+
+ async def test_it_asks_every_question_in_one_job_and_fills_the_structure(self, mocker: MockerFixture) -> None:
+ worker, result = await self._run(mocker, answers=MultiJudgmentTestCases.ANSWERS)
+
+ (job,) = worker.judged_jobs
+ assert list(job.questions) == ["urgent", "team", "severity"]
+ assert job.questions["urgent"].instructions == f"Is this urgent, given that it says '{MultiJudgmentTestCases.MESSAGE}'?"
+ assert job.prompt.text == f"A message from a customer: {MultiJudgmentTestCases.MESSAGE}"
+ expected = JudgedTriage(
+ urgent=YesNoContent(yes_no=True, probability=0.9),
+ team=ChoiceContent(choice="technical", confidence=0.7),
+ severity=RatingContent(level=1, label="Major", confidence=0.6),
+ )
+ assert result.content == expected
+ assert result.memory.get_stuff("triage").content == expected
+ assert result.prompt == job.prompt
+ assert {name: judgment.rendered_question for name, judgment in result.judgments.items()} == {
+ "urgent": f"Is this urgent, given that it says '{MultiJudgmentTestCases.MESSAGE}'?",
+ "team": "Which team handles it?",
+ "severity": "How severe is it?",
+ }
+ assert {name: judgment.outcome for name, judgment in result.judgments.items()} == MultiJudgmentTestCases.ANSWERS
+ assert {name: judgment.threshold_applied for name, judgment in result.judgments.items()} == {
+ "urgent": True,
+ "team": None,
+ "severity": None,
+ }
+
+ async def test_each_question_reads_its_own_threshold(self, mocker: MockerFixture) -> None:
+ questions = {
+ **MultiJudgmentTestCases.QUESTIONS,
+ "urgent": AskedQuestion(question=YesNoQuestion(instructions="Is this urgent?"), threshold=0.95),
+ }
+ _, result = await self._run(mocker, answers=MultiJudgmentTestCases.ANSWERS, questions=questions)
+
+ assert isinstance(result.content, JudgedTriage)
+ assert result.content.urgent == YesNoContent(yes_no=False, probability=0.9)
+
+ async def test_a_refused_question_behind_an_optional_field_leaves_it_absent(self, mocker: MockerFixture) -> None:
+ """The refusal is recorded beside the verdicts and logged, and no verdict is read off it."""
+ warning = mocker.patch("pipelex.kernel.judgment_ops.log.warning")
+
+ _, result = await self._run(mocker, answers={**MultiJudgmentTestCases.ANSWERS, "severity": JudgmentRefusal()})
+
+ assert result.content == JudgedTriage(
+ urgent=YesNoContent(yes_no=True, probability=0.9), team=ChoiceContent(choice="technical", confidence=0.7), severity=None
+ )
+ assert result.judgments["severity"].outcome == JudgmentRefusal()
+ assert result.judgments["severity"].threshold_applied is None
+ warning.assert_called_once()
+ assert warning.call_args.kwargs["fields"] == {
+ "pipe_code": "triage_message",
+ "model_handle": "fake-judgment-handle",
+ "judgment_question": "severity",
+ }
+
+ @pytest.mark.parametrize("output_class", [NullableRequiredTriage, NullableDefaultedTriage, LooseDefaultTriage])
+ async def test_a_refused_question_behind_a_field_that_may_hold_nothing_stores_nothing(
+ self, mocker: MockerFixture, output_class: type[StructuredContent]
+ ) -> None:
+ """A field whose type admits nothing holds nothing on a refusal, never its default, and so does one whose default is nothing."""
+ _, result = await self._run(mocker, answers={**MultiJudgmentTestCases.ANSWERS, "severity": JudgmentRefusal()}, output_class=output_class)
+
+ assert isinstance(result.content, output_class)
+ assert result.content.model_dump() == {
+ "urgent": YesNoContent(yes_no=True, probability=0.9).model_dump(),
+ "team": ChoiceContent(choice="technical", confidence=0.7).model_dump(),
+ "severity": None,
+ }
+ assert result.judgments["severity"].outcome == JudgmentRefusal()
+
+ async def test_a_refused_question_behind_a_required_field_raises_naming_it(self, mocker: MockerFixture) -> None:
+ with pytest.raises(JudgmentRefusedError) as exc_info:
+ await self._run(mocker, answers={**MultiJudgmentTestCases.ANSWERS, "team": JudgmentRefusal()})
+
+ message = str(exc_info.value)
+ assert "PipeJudge 'triage_message'" in message
+ assert "'fake-judgment-handle' declined to answer the question 'team'" in message
+ assert "make its output field optional" in message
+ assert exc_info.value.question_name == "team"
+ assert exc_info.value.error_category == "content"
+
+ @pytest.mark.parametrize("output_class", [DefaultedTriage, FactoryDefaultedTriage])
+ async def test_a_refused_question_behind_a_defaulted_field_raises_rather_than_filling_the_default(
+ self, mocker: MockerFixture, output_class: type[StructuredContent]
+ ) -> None:
+ """A default would read as a verdict nobody gave, so a field that may not hold nothing fails the step."""
+ with pytest.raises(JudgmentRefusedError) as exc_info:
+ await self._run(mocker, answers={**MultiJudgmentTestCases.ANSWERS, "severity": JudgmentRefusal()}, output_class=output_class)
+
+ assert "declined to answer the question 'severity'" in str(exc_info.value)
+ assert "make its output field optional with no default" in str(exc_info.value)
+ assert exc_info.value.question_name == "severity"
+
+ @pytest.mark.parametrize("output_class", [AliasedRequiredTriage, AliasedOptionalTriage, AliasedDefaultedTriage])
+ async def test_an_aliased_field_keeps_its_answered_verdict(self, mocker: MockerFixture, output_class: type[StructuredContent]) -> None:
+ """The verdicts are handed over by field name, the name each question goes by, whatever alias a field carries."""
+ _, result = await self._run(mocker, answers=MultiJudgmentTestCases.ANSWERS, output_class=output_class)
+
+ assert isinstance(result.content, output_class)
+ assert result.content.model_dump() == {
+ "urgent": YesNoContent(yes_no=True, probability=0.9).model_dump(),
+ "team": ChoiceContent(choice="technical", confidence=0.7).model_dump(),
+ "severity": RatingContent(level=1, label="Major", confidence=0.6).model_dump(),
+ }
+
+ async def test_a_dry_run_answers_every_question(self, mocker: MockerFixture) -> None:
+ worker, result = await self._run(mocker, answers={}, run_mode=PipeRunMode.DRY)
+
+ assert not worker.was_called
+ assert result.content == JudgedTriage(
+ urgent=YesNoContent(yes_no=True), team=ChoiceContent(choice="billing"), severity=RatingContent(level=0, label="Minor")
+ )
+
+ async def test_a_live_threshold_without_a_probability_warns_naming_the_question(self, mocker: MockerFixture) -> None:
+ warning = mocker.patch("pipelex.kernel.judgment_ops.log.warning")
+
+ _, result = await self._run(mocker, answers={**MultiJudgmentTestCases.ANSWERS, "urgent": YesNoAnswer(yes_no=True)})
+
+ assert result.judgments["urgent"].threshold_applied is False
+ warning.assert_called_once()
+ assert warning.call_args.kwargs["fields"] == {
+ "pipe_code": "triage_message",
+ "threshold": 0.8,
+ "model_handle": "fake-judgment-handle",
+ "judgment_question": "urgent",
+ }
+
+ @pytest.mark.parametrize(
+ ("questions", "named"),
+ [
+ pytest.param(
+ {name: MultiJudgmentTestCases.QUESTIONS[name] for name in ("urgent", "team")}, "'severity'", id="a_field_no_question_answers"
+ ),
+ pytest.param(
+ {**MultiJudgmentTestCases.QUESTIONS, "notes": AskedQuestion(question=YesNoQuestion(instructions="Any notes?"))},
+ "'notes'",
+ id="a_question_with_no_field",
+ ),
+ ],
+ )
+ async def test_questions_that_are_not_the_output_fields_are_refused_before_any_call(
+ self, mocker: MockerFixture, questions: dict[str, Any], named: str
+ ) -> None:
+ worker = FakeJudgmentWorker(make_fake_judgment_model(), answers=MultiJudgmentTestCases.ANSWERS)
+ with pytest.raises(JudgmentOutputFieldsError) as exc_info:
+ await self._run(mocker, answers=MultiJudgmentTestCases.ANSWERS, questions=questions, worker=worker)
+
+ assert named in str(exc_info.value)
+ assert JudgedTriage.__name__ in str(exc_info.value)
+ assert not worker.was_called
diff --git a/tests/unit/pipelex/language/test_mthds_schema.py b/tests/unit/pipelex/language/test_mthds_schema.py
index a1cc402ba..2eba1d62c 100644
--- a/tests/unit/pipelex/language/test_mthds_schema.py
+++ b/tests/unit/pipelex/language/test_mthds_schema.py
@@ -292,12 +292,50 @@ class TestMthdsSchemaGeneration:
pytest.param({"question": "is it?"}, False, id="question-alone"),
pytest.param({"prompt": "@message"}, False, id="prompt-alone"),
pytest.param({}, False, id="neither"),
+ pytest.param(
+ {
+ "prompt": "@message",
+ "questions": {"urgent": {"question": "is it?", "threshold": 0.7}, "team": {"question": "who?", "options": {"a": ""}}},
+ },
+ True,
+ id="prompt-and-questions",
+ ),
+ pytest.param(
+ {"prompt": "@message", "question": "is it?", "questions": {"urgent": {"question": "is it?"}}}, False, id="question-and-questions"
+ ),
+ pytest.param({"prompt": "@message", "questions": {}}, False, id="no-question-in-questions"),
+ pytest.param({"prompt": "@message", "questions": {"urgent": {"threshold": 0.7}}}, False, id="question-table-without-question"),
+ pytest.param(
+ {"prompt": "@message", "questions": {"urgent": {"question": "is it?", "prompt": "@message"}}}, False, id="unknown-key-in-a-question"
+ ),
+ pytest.param(
+ {"prompt": "@message", "questions": {"urgent": {"question": "is it?"}}, "threshold": 0.7}, False, id="threshold-on-the-pipe"
+ ),
+ pytest.param(
+ {"prompt": "@message", "questions": {"urgent": {"question": "is it?"}}, "options": {"a": ""}}, False, id="options-on-the-pipe"
+ ),
+ pytest.param(
+ {"prompt": "@message", "questions": {"urgent": {"question": "is it?"}}, "levels": ["a", "b"]}, False, id="levels-on-the-pipe"
+ ),
+ pytest.param(
+ {"prompt": "@message", "questions": {"urgent": {"question": "is it?"}}, "criteria": {"yes": "y", "no": "n"}},
+ False,
+ id="criteria-on-the-pipe",
+ ),
+ pytest.param(
+ {"prompt": "@message", "questions": {"severity": {"question": "how bad?", "levels": [{"label": "Low"}, "High"]}}},
+ False,
+ id="a-question-labelling-some-levels-only",
+ ),
],
)
def test_pipe_judge_takes_its_evidence_prompt_and_its_question(
self, schema: dict[str, Any], judge_fields: dict[str, Any], should_validate: bool
) -> None:
- """A PipeJudge writes the evidence in `prompt` and asks about it in `question`, both required, as its blueprint reads them."""
+ """A PipeJudge writes the evidence in `prompt` and asks about it in one `question` or several `questions`, as its blueprint reads them.
+
+ Asking several, the kind fields go on each question, never on the pipe, and `questions` holds at least one question.
+ """
validator = _pipe_union_oneof_validator(schema)
table = {"type": "PipeJudge", "description": "A judge", "output": "YesNo", **judge_fields}
assert validator.is_valid(table) is should_validate, f"{sorted(judge_fields)} should {'' if should_validate else 'not '}validate"
diff --git a/tests/unit/pipelex/mthds_parsing/test_parser_preliminary_text.py b/tests/unit/pipelex/mthds_parsing/test_parser_preliminary_text.py
index 6d97dfe96..9e4781797 100644
--- a/tests/unit/pipelex/mthds_parsing/test_parser_preliminary_text.py
+++ b/tests/unit/pipelex/mthds_parsing/test_parser_preliminary_text.py
@@ -1,6 +1,9 @@
+import pytest
+
from pipelex.mthds_parsing.parser import MthdsParser
from pipelex.mthds_parsing.pipelex_bundle_blueprint import StepRole
from pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint
+from pipelex.pipe_operators.judge.pipe_judge_blueprint import PipeJudgeBlueprint
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
from pipelex.pipe_operators.structure.pipe_structure_blueprint import PipeStructureBlueprint
@@ -67,3 +70,46 @@ prompt = "Talk about $topic"
assert set(bundle.pipe.keys()) == {"make_foo"}
assert isinstance(bundle.pipe["make_foo"], PipeLLMBlueprint)
assert bundle.elaboration_metadata is None
+
+ @pytest.mark.parametrize(
+ "judge_fields",
+ [
+ pytest.param('question = "Is it urgent?"', id="one_question"),
+ pytest.param('[pipe.judge_it.questions.urgent]\nquestion = "Is it urgent?"', id="several_questions"),
+ ],
+ )
+ def test_a_bundle_holding_a_pipe_judge_elaborates(self, judge_fields: str) -> None:
+ """The elaboration revalidates the whole bundle from its dump, the PipeJudge's form not taken included."""
+ mthds_content = f"""domain = "test_pipes"
+description = "A preliminary_text PipeLLM beside a PipeJudge"
+
+[concept]
+Foo = "A foo concept"
+
+[concept.Triage]
+description = "A triage"
+
+[concept.Triage.structure]
+urgent = {{ type = "concept", concept_ref = "YesNo", description = "Whether it is urgent", required = true }}
+
+[pipe.make_foo]
+type = "PipeLLM"
+description = "Make a Foo via preliminary text"
+inputs = {{ topic = "Text" }}
+output = "Foo"
+prompt = "Talk about $topic"
+structuring_method = "preliminary_text"
+
+[pipe.judge_it]
+type = "PipeJudge"
+description = "Judge a message"
+inputs = {{ message = "Text" }}
+output = "{"YesNo" if judge_fields.startswith("question") else "Triage"}"
+prompt = "@message"
+{judge_fields}
+"""
+ bundle = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content)
+
+ assert bundle.pipe is not None
+ assert set(bundle.pipe.keys()) == {"make_foo", "make_foo__draft_text", "make_foo__structure", "judge_it"}
+ assert isinstance(bundle.pipe["judge_it"], PipeJudgeBlueprint)
diff --git a/tests/unit/pipelex/pipe_operators/pipe_judge/data.py b/tests/unit/pipelex/pipe_operators/pipe_judge/data.py
index 644f0968a..f17bcde1b 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_judge/data.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_judge/data.py
@@ -19,6 +19,38 @@ def _judge(**fields: Any) -> dict[str, Any]:
return {key: value for key, value in blueprint.items() if value is not None}
+def _questions(**overrides: Any) -> dict[str, Any]:
+ """Three questions of each kind about one message, with the given question tables on top: a table given as `None` is dropped."""
+ questions: dict[str, Any] = {
+ "urgent": {"question": "Is it urgent?", "threshold": 0.7, "criteria": {"yes": "It cannot wait", "no": "It can wait"}},
+ "team": {"question": "Which team handles it?", "options": {"billing": "Charges and invoices", "technical": ""}},
+ "severity": {"question": "How severe is it?", "levels": [{"label": "Low"}, {"label": "High"}]},
+ }
+ questions.update(overrides)
+ return {key: value for key, value in questions.items() if value is not None}
+
+
+def _multi(**fields: Any) -> dict[str, Any]:
+ """A PipeJudge blueprint asking several questions over the evidence of one message, with the given fields on top.
+
+ A field given as `None` is left out, so `_multi(question=…)` adds a single-form question beside the several.
+ """
+ blueprint: dict[str, Any] = {
+ "description": "d",
+ "inputs": {"message": "Text"},
+ "output": "MessageTriage",
+ "prompt": "A message from a customer:\n@message",
+ "questions": _questions(),
+ }
+ blueprint.update(fields)
+ return {key: value for key, value in blueprint.items() if value is not None}
+
+
+def _one_question(**question_fields: Any) -> dict[str, Any]:
+ """A PipeJudge blueprint asking one question through `questions`, the question table being the given fields."""
+ return _multi(questions={"verdict": question_fields})
+
+
class PipeJudgeBlueprintTestCases:
"""Each refusal is (test_id, blueprint fields, a fragment of its message); each acceptance is (test_id, fields, the kind it resolves to)."""
@@ -45,7 +77,7 @@ class PipeJudgeBlueprintTestCases:
("one_option", _judge(output="Choice", options={"only": "the one"}), "at least two `options`"),
("empty_option_key", _judge(output="Choice", options={"": "nothing", "b": ""}), "none may be empty"),
("one_level", _judge(output="Rating", levels=["only"]), "at least two `levels`"),
- ("empty_level", _judge(output="Rating", levels=["low", " "]), "level 1 is"),
+ ("empty_level", _judge(output="Rating", levels=["low", " "]), "level 1 of this PipeJudge is"),
("empty_level_table", _judge(output="Rating", levels=["low", {}]), "carries a `label`, a `description` or both"),
("blank_level_label", _judge(output="Rating", levels=[{"label": "Low"}, {"label": " ", "description": "High"}]), "level 1"),
("unknown_level_key", _judge(output="Rating", levels=[{"label": "Low"}, {"label": "High", "score": 3}]), "score"),
@@ -57,7 +89,7 @@ class PipeJudgeBlueprintTestCases:
(
"duplicated_labels",
_judge(output="Rating", levels=[{"label": "Low"}, {"label": "High"}, {"label": "Low"}]),
- "the labels of one scale are distinct, and 'Low' is repeated",
+ "the labels of one scale are distinct, and this PipeJudge repeats 'Low'",
),
("threshold_zero", _judge(threshold=0), "strictly between 0 and 1"),
("threshold_one", _judge(threshold=1), "strictly between 0 and 1"),
@@ -111,3 +143,122 @@ class PipeJudgeBlueprintTestCases:
),
("rating_unlabelled_tables_and_strings", _judge(output="Rating", levels=[{"description": "Barely"}, "Severe"]), JudgmentKind.RATING),
]
+
+ # Several questions: (test_id, blueprint fields, a fragment of its message).
+ REFUSED_SEVERAL: ClassVar[list[tuple[str, dict[str, Any], str]]] = [
+ ("question_and_questions", _multi(question="Is it urgent?"), "sets both `question` and `questions`"),
+ ("neither_question_nor_questions", _multi(questions=None), "or several in `questions`"),
+ ("no_question_in_questions", _multi(questions={}), "`questions` holds at least one question, and this one holds none"),
+ ("key_not_an_identifier", _multi(questions=_questions(**{"is-urgent": {"question": "Urgent?"}})), "'is-urgent' is not"),
+ ("key_a_python_keyword", _multi(questions=_questions(**{"class": {"question": "Urgent?"}})), "'class' is not"),
+ ("key_starting_with_an_underscore", _multi(questions=_questions(_urgent={"question": "Urgent?"})), "'_urgent' is not"),
+ ("key_a_reserved_name", _multi(questions=_questions(model_config={"question": "Urgent?"})), "'model_config' is not"),
+ ("options_on_the_pipe", _multi(options={"a": "", "b": ""}), "sets `options` on each question rather than on the pipe"),
+ ("levels_on_the_pipe", _multi(levels=["low", "high"]), "sets `levels` on each question rather than on the pipe"),
+ ("criteria_on_the_pipe", _multi(criteria={"yes": "y", "no": "n"}), "sets `criteria` on each question rather than on the pipe"),
+ ("threshold_on_the_pipe", _multi(threshold=0.7), "sets `threshold` on each question rather than on the pipe"),
+ ("question_table_without_question", _one_question(threshold=0.7), "writes what it asks in `question`, and this one sets none"),
+ ("empty_question_in_a_table", _one_question(question=""), "its `question` cannot be empty"),
+ ("blank_question_in_a_table", _one_question(question=" \n "), "its `question` cannot be empty"),
+ ("prompt_in_a_question_table", _one_question(question="Is it urgent?", prompt="@message"), "prompt"),
+ # A question's kind refusals name the question, which the rule's own words would not tell apart from another.
+ (
+ "question_with_options_and_levels",
+ _one_question(question="Q?", options={"a": "", "b": ""}, levels=["low", "high"]),
+ "not both, and the question 'verdict' declares both",
+ ),
+ (
+ "criteria_beside_options_in_a_question",
+ _one_question(question="Q?", options={"a": "", "b": ""}, criteria={"yes": "y", "no": "n"}),
+ "`criteria` applies to a yes/no question only, and the question 'verdict' declares `options`",
+ ),
+ (
+ "question_threshold_out_of_range",
+ _one_question(question="Q?", threshold=1.5),
+ "strictly between 0 and 1, and the question 'verdict' declares 1.5",
+ ),
+ (
+ "question_with_one_option",
+ _one_question(question="Q?", options={"only": ""}),
+ "at least two `options` to choose between, and the question 'verdict' declares 1",
+ ),
+ (
+ "question_with_an_empty_option",
+ _one_question(question="Q?", options={"": "nothing", "b": ""}),
+ "none may be empty, and the question 'verdict' declares an empty one",
+ ),
+ (
+ "question_with_one_level",
+ _one_question(question="Q?", levels=["only"]),
+ "at least two `levels` on its scale, and the question 'verdict' declares 1",
+ ),
+ ("question_with_an_empty_level", _one_question(question="Q?", levels=["low", " "]), "level 1 of the question 'verdict' is"),
+ (
+ "question_with_an_empty_level_table",
+ _one_question(question="Q?", levels=["low", {}]),
+ "level 1 of the question 'verdict' carries neither",
+ ),
+ (
+ "question_with_a_blank_level_label",
+ _one_question(question="Q?", levels=[{"label": "Low"}, {"label": " ", "description": "High"}]),
+ "`label` cannot be empty, and that of level 1 of the question 'verdict' is",
+ ),
+ (
+ "question_with_mixed_labels",
+ _one_question(question="Q?", levels=[{"label": "Low"}, "High"]),
+ "every level carries a `label` or none does, and the question 'verdict' labels some of its levels only",
+ ),
+ (
+ "question_with_duplicated_labels",
+ _one_question(question="Q?", levels=[{"label": "Low"}, {"label": "High"}, {"label": "Low"}]),
+ "the labels of one scale are distinct, and the question 'verdict' repeats 'Low'",
+ ),
+ ("question_with_a_lone_criterion", _one_question(question="Q?", criteria={"yes": "y"}), "these declare `yes` without `no`"),
+ (
+ "undeclared_variable_in_a_question",
+ _multi(questions=_questions(urgent={"question": "Is it about $topic?"})),
+ "Variable 'topic' is read by the prompt or questions",
+ ),
+ (
+ "input_no_template_reads",
+ _multi(inputs={"message": "Text", "history": "Text"}),
+ "Input 'history' is declared but never read by the prompt or questions",
+ ),
+ ("output_with_brackets", _multi(output="MessageTriage[]"), "fills one structure with their verdicts"),
+ ]
+
+ # A field written as null holds no value, so it neither sets a form nor clashes with the other one.
+ REFUSED_NULLS: ClassVar[list[tuple[str, dict[str, Any], str]]] = [
+ ("null_question_alone", {**_judge(), "question": None}, "the question is written in `question`, or several in `questions`"),
+ ("null_question_and_null_questions", {**_judge(), "question": None, "questions": None}, "or several in `questions`"),
+ ("null_question_in_a_question_table", _one_question(question=None), "writes what it asks in `question`, and this one sets none"),
+ ("null_prompt", {**_judge(), "prompt": None}, "judges the evidence its `prompt` presents, and this one sets no `prompt`"),
+ ]
+
+ ACCEPTED_NULLS: ClassVar[list[tuple[str, dict[str, Any]]]] = [
+ ("question_beside_null_questions", {**_judge(), "questions": None}),
+ ("questions_beside_null_question", {**_multi(), "question": None}),
+ ]
+
+ # Both forms, with every field a blueprint may carry, to round-trip through a dump.
+ ROUND_TRIPS: ClassVar[list[tuple[str, dict[str, Any]]]] = [
+ ("single_yes_no", _judge(model="@default-judgment", threshold=0.8, criteria={"yes": "Today", "no": "Later"})),
+ ("single_rating", _judge(output="Rating", levels=[{"label": "Low", "description": "Barely"}, {"label": "High"}])),
+ ("single_choice", _judge(output="Choice", options={"billing": "", "technical": "Errors"})),
+ ("several", _multi(model="@default-judgment")),
+ ]
+
+ # Several questions: (test_id, blueprint fields, the kind each question resolves to).
+ ACCEPTED_SEVERAL: ClassVar[list[tuple[str, dict[str, Any], dict[str, JudgmentKind]]]] = [
+ (
+ "one_question_of_each_kind",
+ _multi(),
+ {"urgent": JudgmentKind.YES_NO, "team": JudgmentKind.CHOICE, "severity": JudgmentKind.RATING},
+ ),
+ ("a_single_question_in_questions", _one_question(question="Is it urgent?"), {"verdict": JudgmentKind.YES_NO}),
+ (
+ "an_input_read_by_one_question_alone",
+ _multi(inputs={"message": "Text", "topic": "Text"}, questions=_questions(urgent={"question": "Is it about $topic?"})),
+ {"urgent": JudgmentKind.YES_NO, "team": JudgmentKind.CHOICE, "severity": JudgmentKind.RATING},
+ ),
+ ]
diff --git a/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge.py b/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge.py
new file mode 100644
index 000000000..da36ad2a6
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge.py
@@ -0,0 +1,58 @@
+from typing import Any
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.cogt.judgment.judgment_models import ChoiceQuestion, YesNoQuestion
+from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
+from pipelex.kernel.prompt_assembly import UserPromptContent
+from pipelex.pipe_operators.judge.pipe_judge import PipeJudge, PipeJudgeQuestion
+from pipelex.tools.jinja2.template_category import TemplateCategory
+
+_URGENT = PipeJudgeQuestion(judgment_question=YesNoQuestion(instructions="Is it urgent?"), threshold=0.7)
+_TEAM = PipeJudgeQuestion(judgment_question=ChoiceQuestion(instructions="Which team handles $topic?", options={"billing": None, "technical": None}))
+
+
+def _pipe_judge(**question_fields: Any) -> PipeJudge:
+ return PipeJudge(
+ code="judge_it",
+ domain_code="judge_unit",
+ description="Judge the evidence",
+ output=StuffSpec(concept=ConceptFactory.make_native_concept(NativeConceptCode.YES_NO)),
+ judgment_choice=None,
+ prompt_content=UserPromptContent(template=TemplateBlueprint(template="$message", category=TemplateCategory.LLM_PROMPT)),
+ **question_fields,
+ )
+
+
+class TestPipeJudge:
+ @pytest.mark.parametrize(
+ ("question_fields", "message_fragment"),
+ [
+ pytest.param({"question": _URGENT, "questions": {"team": _TEAM}}, "it was given both a single question and several", id="both"),
+ pytest.param({}, "it was given none", id="neither"),
+ pytest.param({"questions": {}}, "it was given none", id="no_question_among_several"),
+ ],
+ )
+ def test_a_step_asks_one_question_or_several_never_both_nor_none(self, question_fields: dict[str, Any], message_fragment: str):
+ with pytest.raises(ValidationError) as exc_info:
+ _pipe_judge(**question_fields)
+ assert message_fragment in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ ("question_fields", "expected_labels"),
+ [
+ pytest.param({"question": _URGENT}, ["question"], id="one"),
+ pytest.param({"questions": {"urgent": _URGENT, "team": _TEAM}}, ["question 'urgent'", "question 'team'"], id="several"),
+ ],
+ )
+ def test_each_question_is_labelled_for_messages(self, question_fields: dict[str, Any], expected_labels: list[str]):
+ pipe_judge = _pipe_judge(**question_fields)
+ assert [question_label for question_label, _ in pipe_judge.labelled_questions] == expected_labels
+
+ def test_the_variables_of_every_question_are_required(self):
+ pipe_judge = _pipe_judge(questions={"urgent": _URGENT, "team": _TEAM})
+ assert pipe_judge.required_variables() == {"message", "topic"}
diff --git a/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge_blueprint.py
index 77a34bb82..9d7da5e89 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_judge/test_pipe_judge_blueprint.py
@@ -3,8 +3,8 @@ from typing import Any
import pytest
from pydantic import ValidationError
-from pipelex.cogt.judgment.judgment_models import JudgmentKind
-from pipelex.pipe_operators.judge.pipe_judge_blueprint import JudgeRatingLevel, PipeJudgeBlueprint
+from pipelex.cogt.judgment.judgment_models import JudgmentKind, RatingLevel
+from pipelex.pipe_operators.judge.pipe_judge_blueprint import JudgeRatingLevel, JudgeYesNoCriteria, PipeJudgeBlueprint
from tests.unit.pipelex.pipe_operators.pipe_judge.data import PipeJudgeBlueprintTestCases
@@ -20,6 +20,62 @@ class TestPipeJudgeBlueprint:
blueprint = PipeJudgeBlueprint.model_validate(blueprint_fields)
assert blueprint.judgment_kind == expected_kind, test_id
+ @pytest.mark.parametrize(("test_id", "blueprint_fields", "message_fragment"), PipeJudgeBlueprintTestCases.REFUSED_SEVERAL)
+ def test_several_questions_refused(self, test_id: str, blueprint_fields: dict[str, Any], message_fragment: str):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeJudgeBlueprint.model_validate(blueprint_fields)
+ assert message_fragment in str(exc_info.value), test_id
+
+ @pytest.mark.parametrize(("test_id", "blueprint_fields", "expected_kinds"), PipeJudgeBlueprintTestCases.ACCEPTED_SEVERAL)
+ def test_several_questions_accepted(self, test_id: str, blueprint_fields: dict[str, Any], expected_kinds: dict[str, JudgmentKind]):
+ """Each question's kind is read off its own `options` and `levels`, and the pipe itself has no single kind."""
+ blueprint = PipeJudgeBlueprint.model_validate(blueprint_fields)
+ assert blueprint.questions is not None, test_id
+ assert {name: question.judgment_kind for name, question in blueprint.questions.items()} == expected_kinds, test_id
+ assert blueprint.question is None, test_id
+ assert blueprint.judgment_kind is None, test_id
+
+ @pytest.mark.parametrize(("test_id", "blueprint_fields", "message_fragment"), PipeJudgeBlueprintTestCases.REFUSED_NULLS)
+ def test_a_null_question_sets_no_form(self, test_id: str, blueprint_fields: dict[str, Any], message_fragment: str):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeJudgeBlueprint.model_validate(blueprint_fields)
+ assert message_fragment in str(exc_info.value), test_id
+
+ @pytest.mark.parametrize(("test_id", "blueprint_fields"), PipeJudgeBlueprintTestCases.ACCEPTED_NULLS)
+ def test_a_null_beside_the_other_form_is_no_clash(self, test_id: str, blueprint_fields: dict[str, Any]):
+ blueprint = PipeJudgeBlueprint.model_validate(blueprint_fields)
+ assert (blueprint.question is None) != (blueprint.questions is None), test_id
+
+ @pytest.mark.parametrize("by_alias", [False, True])
+ @pytest.mark.parametrize(("test_id", "blueprint_fields"), PipeJudgeBlueprintTestCases.ROUND_TRIPS)
+ def test_a_blueprint_validates_from_its_own_dump(self, test_id: str, blueprint_fields: dict[str, Any], by_alias: bool):
+ """A dump writes every field, the form not taken as null, and validating it gives back the same blueprint."""
+ blueprint = PipeJudgeBlueprint.model_validate(blueprint_fields)
+ assert PipeJudgeBlueprint.model_validate(blueprint.model_dump(by_alias=by_alias)) == blueprint, test_id
+
+ def test_a_question_of_several_carries_its_own_fields(self):
+ blueprint = PipeJudgeBlueprint.model_validate(
+ {
+ "description": "d",
+ "inputs": {"message": "Text"},
+ "output": "MessageTriage",
+ "prompt": "@message",
+ "questions": {
+ "urgent": {"question": "Is it urgent?", "threshold": 0.7, "criteria": {"yes": "It cannot wait", "no": "It can wait"}},
+ "severity": {"question": "How severe?", "levels": ["Barely", {"description": "Nothing works"}]},
+ },
+ }
+ )
+ assert blueprint.questions is not None
+ urgent = blueprint.questions["urgent"]
+ assert urgent.question == "Is it urgent?"
+ assert urgent.threshold == 0.7
+ assert urgent.criteria == JudgeYesNoCriteria(yes="It cannot wait", no="It can wait")
+ assert urgent.rating_levels is None
+ severity = blueprint.questions["severity"]
+ assert severity.levels == ["Barely", JudgeRatingLevel(description="Nothing works")]
+ assert severity.rating_levels == [RatingLevel(description="Barely"), RatingLevel(description="Nothing works")]
+
def test_the_prompt_and_the_question_are_two_fields(self):
"""`prompt` is the evidence and `question` the instruction, so neither is read as the other."""
blueprint = PipeJudgeBlueprint.model_validate(