Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Pipellm-returns-empty-text · L-260903-2b8290 (#1414)

Louis Choquel4b6e1419ee4a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 128c8df04..810c573da 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,20 @@
# Changelog
+## [Unreleased]
+
+### Added
+
+- **`LLMCompletionTruncatedError` and `LLMCompletionRefusedError`**: two subclasses of `LLMCompletionError` report a text completion the model did not finish, the first one cut at its output limit or context window, the second one refused by the model or stopped by a safety filter. Both are content errors in the `input` domain, not retryable, answered with HTTP 422 and caller-facing under STRICT disclosure, with a `change_input` next step; the message is one sentence naming the model, the pipe, the provider's stop value and, for a truncation, the output tokens used and the `max_tokens` sent, and never carries the partial text. A truncation carries the limit it hit as `truncation_limit`, `max_tokens` or `context_window`, a stop at the context window advises a shorter input, a lower reasoning effort or a model with a larger window rather than a higher `max_tokens`, and a cut when the request sent no `max_tokens` advises setting the pipe's up to the model's limit.
+
+### Changed
+
+- **The Bedrock client's `chat` returns a `BedrockChatResult` (Breaking)**: `BedrockClientProtocol.chat` returns a named result holding the text, the token usage and Converse's `stopReason`, in place of a `(text, usage)` tuple, so a custom Bedrock client returns `read_converse_response(response=…)` or builds the result itself.
+- **A lowered Anthropic structured-output limit is said**: when a structured generation on an Anthropic model lowers `max_tokens` to fit its structured-output timeout, the call logs it, at warning level when the pipe set the limit and at debug level when it is the model's default, and an error the lowered limit can explain, a structured output that failed validation or came back incomplete, or a timeout, ends with a sentence saying the limit was lowered and to what, never a rate limit, an authentication failure, an overload, a server error or a lost connection.
+
+### Fixed
+
+- **A truncated, refused or filtered text no longer passes for a result (Breaking)**: every LLM worker, OpenAI chat completions with a gateway, Portkey or OpenRouter in front of it, the OpenAI Responses API, Anthropic, Google, Bedrock and Mistral, now reads its response's stop value before it returns a text, so a text cut at `max_tokens` or refused fails its pipe with `LLMCompletionTruncatedError` or `LLMCompletionRefusedError` where it used to come back as a successful output, empty or partial, its billed usage still reported and recorded on its span. Each provider's vocabulary is read, including a gateway's pass-through of another provider's value such as `max_tokens` or `content_filtered`, an unknown value being logged at warning level and taken as normal, except that an `incomplete` Responses API answer is a truncation whatever its reason; a Gemini prompt blocked before any candidate fails with `LLMCompletionRefusedError` naming its block reason, and structured generation keeps its configured re-asks. A Bedrock answer holding no text block fails with an `LLMCompletionError` instead of an `IndexError`.
+
## [v0.82.0] - 2026-10-10
### Added
diff --git a/docs/building-methods/pipes/pipe-operators/PipeLLM.md b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
index 8ada4ed13..e61d35ff5 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeLLM.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeLLM.md
@@ -257,6 +257,8 @@ When `model` is specified as a table (inline LLM setting), it accepts the follow
A setting the model it resolves to cannot take is refused when the method loads, before a run spends anything: validation runs the check the model's worker runs before every call, so a reasoning setting on a model that does not reason, a reasoning budget on a model taking only an effort, or a budget its `max_tokens` cannot hold is an `llm_setting_refused_by_model` error on the `model` or `model_to_structure` field holding the setting. The setting checked is the one the output is generated with: `model` for a single text, the structuring setting for anything else.
+A text the model cannot finish within `max_tokens`, or within its context window, fails the pipe with [`LLMCompletionTruncatedError`](../../../errors/llm-completion-truncated-error.md) rather than passing the partial text on as the pipe's output; a text the model refuses or a safety filter stops fails it with [`LLMCompletionRefusedError`](../../../errors/llm-completion-refused-error.md). The first names the output tokens used and the limit sent, so raising `max_tokens` or shortening the input is the fix; a model that reasons spends its reasoning from the same limit.
+
!!! tip "More on Reasoning"
For provider-specific behavior and model-specific examples, see [Reasoning Controls](../../../under-the-hood/reasoning-controls.md).
diff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.md
index 073383659..c9a6c1882 100644
--- a/docs/errors/inference-and-providers.md
+++ b/docs/errors/inference-and-providers.md
@@ -52,6 +52,8 @@ own page. Classes are grouped by subsystem.
- [`LLMAssignmentError`](llm-assignment-error.md) — LLM assignment
- [`LLMCapabilityError`](llm-capability-error.md) — LLM capability
- [`LLMCompletionError`](llm-completion-error.md) — LLM completion
+- [`LLMCompletionRefusedError`](llm-completion-refused-error.md) — LLM completion refused
+- [`LLMCompletionTruncatedError`](llm-completion-truncated-error.md) — LLM completion truncated
- [`LLMConfigError`](llm-config-error.md) — LLM config
- [`LLMHandleNotFoundError`](llm-handle-not-found-error.md) — LLM handle not found
- [`LLMModelNotFoundError`](llm-model-not-found-error.md) — LLM model not found
diff --git a/docs/errors/llm-completion-error.md b/docs/errors/llm-completion-error.md
index a8b23dfcf..0e4561477 100644
--- a/docs/errors/llm-completion-error.md
+++ b/docs/errors/llm-completion-error.md
@@ -3,10 +3,23 @@ title: "LLM completion"
description: "Reference for the `LLMCompletionError` Pipelex error class."
---
-<!-- pipelex:generated -->
+<!-- pipelex:authored -->
# LLM completion
+An LLM call answered, but its answer cannot be used as the pipe's output: the provider returned no choice, no text or a content type the worker does not read, or a structured generation failed for a reason no provider classified. Its category, and so whether it is retried, depends on the case, and its `user_action` gives the next step.
+
+Two subclasses report the answers a provider returns as a success although the model did not finish its text. Every LLM worker reads its response's stop signal before it returns any text, so a text cut at its limit, refused or filtered fails its pipe instead of passing for a result:
+
+- [`LLMCompletionTruncatedError`](llm-completion-truncated-error.md): the model was cut off before it finished, at its output limit (`max_tokens`) or at its context window, which the error tells apart. At the output limit, raise the pipe's `max_tokens` or shorten its input; at the context window, which no higher `max_tokens` lifts, shorten the input, lower the reasoning effort or choose a model with a larger window.
+- [`LLMCompletionRefusedError`](llm-completion-refused-error.md): the model declined to write the text, or a provider's safety filter stopped it. Revise the pipe's prompt or its input.
+
+Both are content errors in the `input` domain: they are not retryable, since the same request stops the same way, an HTTP surface answers them with 422, and their message survives STRICT disclosure. The message is one sentence naming the model by its deck handle, the pipe when the job knows it, the provider's stop value and, for a truncation, the output tokens used and the limit sent, then the next step; it never carries the partial text or the provider's body. An agent branches on the report's fields rather than the prose: `error_type`, `error_category` (`content`), `error_domain` (`input`), `retryable` (`false`) and `user_action`, whose kind is `change_input`.
+
+The stop values each worker reads are those of its provider's SDK, read as one vocabulary so that a gateway passing each provider's own value through the OpenAI completions shape is read too: OpenAI's `finish_reason`, Anthropic's `stop_reason`, Bedrock Converse's `stopReason`, Gemini's `finish_reason`, Mistral's `finish_reason`, and the Responses API's `incomplete` status with its reason. A stop value the classifier does not know is logged at warning level with the model and taken as normal, so a value a provider adds later fails no call. Two signals are never normal, whatever reason comes with them: an `incomplete` Responses API answer is a truncation when its reason is missing or unknown, and a Gemini answer with no candidate whose prompt feedback names a block reason is a refusal. Structured generation is not read this way: its configured re-asks are left as they are.
+
+A structured generation on an Anthropic model holds `max_tokens` to what its structured-output timeout lets the model produce. When that lowers the limit, an error the lowered limit can explain, a structured output that failed validation or came back incomplete, or a timeout, ends with a sentence saying the limit was lowered and to what, while a rate limit, an authentication failure, an overload, a server error or a lost connection carries no such sentence; a `max_tokens` the pipe set and the call lowers is also logged at warning level.
+
| Field | Value |
|---|---|
| `error_type` | `LLMCompletionError` |
diff --git a/docs/errors/llm-completion-refused-error.md b/docs/errors/llm-completion-refused-error.md
new file mode 100644
index 000000000..dfa86d91b
--- /dev/null
+++ b/docs/errors/llm-completion-refused-error.md
@@ -0,0 +1,35 @@
+---
+title: "LLM completion refused"
+description: "Reference for the `LLMCompletionRefusedError` Pipelex error class."
+---
+
+<!-- pipelex:authored -->
+
+# LLM completion refused
+
+A text completion the model declined to write, or that a provider's safety filter stopped.
+
+A provider answers such a completion as a success, often with no text at all and sometimes with the beginning of one, and a stop value saying why it ended: `content_filter` on OpenAI chat completions and on an `incomplete` Responses API answer, `refusal` on Anthropic, `content_filtered` and `guardrail_intervened` on Bedrock, and on Gemini `SAFETY`, `RECITATION`, `PROHIBITED_CONTENT`, `BLOCKLIST`, `SPII` and their kin. A Gemini prompt blocked before the model wrote anything comes back with no candidate at all and its reason in the prompt feedback, `prompt_feedback.block_reason`, such as `SAFETY` or `PROHIBITED_CONTENT`: it is a refusal too, whatever the reason, and the error names that reason as its stop value. The worker raises this error instead of handing the text back, so the pipe fails rather than passing an empty or cut text for a result.
+
+The message names the model by its deck handle, the pipe and the stop value, then the next step:
+
+```text
+The model 'claude-5-sonnet' declined to finish the text of pipe 'ui_designer', or a content filter stopped it (stop reason 'content_filtered'), so the text cannot be used. Revise the pipe's prompt or its input.
+```
+
+A run reports it located at the failing pipe, `Pipe 'ui_designer' failed (flow → ui_designer): The model …`. The partial text and the provider's body never appear in it.
+
+**What to do.** Revise the pipe's prompt or its input: rephrase what triggered the refusal, or remove the content the filter stopped on. Another model, whose policy or filter differs, may also answer.
+
+**For an agent.** Branch on the report's fields: `error_type` is `LLMCompletionRefusedError`, `error_category` is `content`, `error_domain` is `input` (an HTTP surface answers 422), `retryable` is `false`, since the same request is refused the same way, and `user_action.kind` is `change_input`. The message is caller-facing, so it survives STRICT disclosure.
+
+| Field | Value |
+|---|---|
+| `error_type` | `LLMCompletionRefusedError` |
+| `title` | LLM completion refused |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/llm-completion-refused-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.cogt.exceptions` |
+| Parent class | [`LLMCompletionError`](llm-completion-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/llm-completion-truncated-error.md b/docs/errors/llm-completion-truncated-error.md
new file mode 100644
index 000000000..95e7eea00
--- /dev/null
+++ b/docs/errors/llm-completion-truncated-error.md
@@ -0,0 +1,48 @@
+---
+title: "LLM completion truncated"
+description: "Reference for the `LLMCompletionTruncatedError` Pipelex error class."
+---
+
+<!-- pipelex:authored -->
+
+# LLM completion truncated
+
+A text completion stopped before the model finished it: it hit its output limit, or its context window.
+
+A provider answers such a completion as a success, with whatever text the model had written, empty or partial, and a stop value saying why it ended. The error tells the two limits apart, since they are lifted differently, and carries the one hit as its `truncation_limit`:
+
+- **The output limit** (`max_tokens`): `length` on OpenAI and Mistral, `max_tokens` on Anthropic and Bedrock, `MAX_TOKENS` on Gemini, and an `incomplete` Responses API answer whose reason is `max_output_tokens`, missing or unknown (the status alone says the text is unfinished; the error names the reason as its stop value when the answer gives one, and `incomplete` when it gives none).
+- **The context window** (`context_window`), which the input and the output filled together: `model_context_window_exceeded` on Anthropic and Bedrock, and `model_length` on Mistral.
+
+The worker raises this error instead of handing the text back, so the pipe fails rather than passing a truncated contract, summary or JSON for a result.
+
+The message names the model by its deck handle, the pipe, the stop value, the output tokens used and the `max_tokens` the request sent, or that it sent none, then the next step, which depends on the limit hit:
+
+```text
+The model 'claude-5-sonnet' was cut off before it finished the text of pipe 'ui_designer' (stop reason 'max_tokens', 4096 output tokens used, max_tokens set to 4096), so the text is incomplete. Raise the pipe's max_tokens, or shorten its input.
+```
+
+```text
+The model 'gpt-5' was cut off before it finished the text of pipe 'ui_designer' (stop reason 'length', 16384 output tokens used, no max_tokens sent), so the text is incomplete. Set the pipe's max_tokens up to the model's limit, lower its reasoning effort, or shorten its input.
+```
+
+```text
+The model 'claude-5-sonnet' filled its context window before it finished the text of pipe 'ui_designer' (stop reason 'model_context_window_exceeded', 812 output tokens used, max_tokens set to 4096), so the text is incomplete. Shorten the pipe's input, lower its reasoning effort, or choose a model with a larger context window.
+```
+
+A run reports it located at the failing pipe, `Pipe 'ui_designer' failed (flow → ui_designer): The model …`. The partial text and the provider's body never appear in it.
+
+**What to do.** At the output limit, raise the pipe's `max_tokens`, for instance `model = { model = "claude-5-sonnet", max_tokens = 16384 }`, or shorten its input. When the request sent no `max_tokens`, the pipe's setting being left to the provider, the provider's own default cut the text, so there is no limit of the pipe's to raise: set the pipe's `max_tokens`, up to the model's limit, lower its reasoning effort, or shorten its input. When the model thinks before it writes, its thinking counts against the same limit, so a lower reasoning effort also leaves more room for the text. At the context window, a higher `max_tokens` cannot help, since the input and the output already filled the window: shorten the pipe's input, lower its reasoning effort, or choose a model with a larger context window.
+
+**For an agent.** Branch on the report's fields: `error_type` is `LLMCompletionTruncatedError`, `error_category` is `content`, `error_domain` is `input` (an HTTP surface answers 422), `retryable` is `false`, since the same request stops at the same limit, and `user_action.kind` is `change_input`, its `detail` the next step for the limit hit. The message is caller-facing, so it survives STRICT disclosure.
+
+| Field | Value |
+|---|---|
+| `error_type` | `LLMCompletionTruncatedError` |
+| `title` | LLM completion truncated |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/llm-completion-truncated-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.cogt.exceptions` |
+| Parent class | [`LLMCompletionError`](llm-completion-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/tools/logging.md b/docs/tools/logging.md
index 3e5a1ddd6..0282aac36 100644
--- a/docs/tools/logging.md
+++ b/docs/tools/logging.md
@@ -363,7 +363,7 @@ Every inference call ends with `Inference call ends`, on the logger `pipelex.cog
| `cost_usd` | number, in US dollars | What the call cost, when it reported usage and the model has rates |
| `duration_ms`, `outcome`, `error.type` | | As above |
-A handle names one model per model type, so a query groups by `gen_ai.operation.name` and `model_handle` together. The model keys are read when the call ends, so a worker that names its model in the checks before the call is named on the event as on the call's span. Every worker writes under `gen_ai.response.model` the provider's id the deck pins for the handle, the one the call is made with, and none writes the id a provider answers with: a provider that says another model answered is named on a warning of its own, the TypeSafe judgment worker's, under `answered_model_id`. A print that ended before its handle was resolved, refused by the read scope, or whose handle resolves to no model served here, names the model by its handle alone, under `model_handle` and `gen_ai.request.model`, and leaves off `backend_name`, `sdk` and `gen_ai.response.model`, which only a served model has. The usage is the one the call recorded on its job's report, the object the base hands to `ReportingProtocol.report_inference_job` when it reports the job and the reporting manager turns into a usage event, and the event prices it with `compute_tokens_usage_cost`, the cost engine the usage records and the run's cost report read, so a usage the cost report counts has the same price there as on the event. The event carries what the job recorded, though, and the cost report only what was reported: the search and judgment bases report a job whichever way it ends, while the LLM, image-generation and extraction bases report it only when it succeeds, so a failure of theirs the provider billed carries a cost on the event that the run's cost report does not count. A token count or a cost the call did not record is left off, never written as zero: a call that failed before the provider answered, or that the checks before it refused, recorded no usage, a model with no rates has no price, and a document engine reports no usage at all, so a print carries neither. A failure the provider billed, a search or a judgment whose answer a guard refused, carries the usage it recorded.
+A handle names one model per model type, so a query groups by `gen_ai.operation.name` and `model_handle` together. The model keys are read when the call ends, so a worker that names its model in the checks before the call is named on the event as on the call's span. Every worker writes under `gen_ai.response.model` the provider's id the deck pins for the handle, the one the call is made with, and none writes the id a provider answers with: a provider that says another model answered is named on a warning of its own, the TypeSafe judgment worker's, under `answered_model_id`. A print that ended before its handle was resolved, refused by the read scope, or whose handle resolves to no model served here, names the model by its handle alone, under `model_handle` and `gen_ai.request.model`, and leaves off `backend_name`, `sdk` and `gen_ai.response.model`, which only a served model has. The usage is the one the call recorded on its job's report, the object the base hands to `ReportingProtocol.report_inference_job` when it reports the job and the reporting manager turns into a usage event, and the event prices it with `compute_tokens_usage_cost`, the cost engine the usage records and the run's cost report read, so a usage the cost report counts has the same price there as on the event. The event carries what the job recorded, though, and the cost report only what was reported: the search and judgment bases report a job whichever way it ends, while the LLM, image-generation and extraction bases report it only when it succeeds, so a failure of theirs the provider billed carries a cost on the event that the run's cost report does not count. One failure is reported all the same: a text the LLM base's stop check refuses, cut at its limit or refused by the model or a filter, which the provider answered and billed, so its usage is reported once and recorded on its span before the error propagates. A token count or a cost the call did not record is left off, never written as zero: a call that failed before the provider answered, or that the checks before it refused, recorded no usage, a model with no rates has no price, and a document engine reports no usage at all, so a print carries neither. A failure the provider billed, a search or a judgment whose answer a guard refused, carries the usage it recorded.
Rates are per million tokens, so a call billed by the request or by the page records each unit as a million tokens in and out, and the rate table prices one unit: a Linkup search or fetch is one request, and an extraction whose provider reports no usage is priced by its pages. Its usage says so in its `pricing_unit`, `request` or `page` rather than `token`, and the event carries the call's cost and no token counts, since a request or a page is not a token. A worker records such a usage with `record_unit_priced_usage` (`pipelex.cogt.usage.usage_cost`), a plugin's worker as much as a provider's, and never writes the counts itself.
@@ -523,6 +523,11 @@ What varies goes in `fields`, named by the [naming convention](#naming-conventio
| `sdk` | string | The SDK a backend reaches a model through, as the backends configuration names it: `openai`, `bedrock_anthropic` |
| `answered_model_id` | string | The provider's id of the model a provider says answered a call, where it is not the pinned id the call was made with, which `gen_ai.response.model` names |
| `fixed_temperature` | number | The one temperature a model accepts, as its constraints declare it, used in place of the one requested |
+| `stop_reason` | string | Why a provider's answer says its text ended, in that provider's own vocabulary: `max_tokens`, `content_filtered`, `MAX_TOKENS`; for a Responses API answer, its status: `incomplete` |
+| `incomplete_reason` | string or `null` | The reason an `incomplete` Responses API answer gives in its details, `null` when it gives none |
+| `block_reason` | string | The reason a Gemini answer holding no candidate gives in its prompt feedback for blocking the prompt: `SAFETY` |
+| `requested_max_tokens`, `effective_max_tokens` | integer | The output limit a call was given, the pipe's setting or the model's default, and the lower one it sends instead |
+| `timeout_seconds` | integer | A call's timeout, in seconds, as the configuration sets it: a configured limit, not a measured duration, which `duration_ms` names |
| `inference_job_type` | string | An inference job's class: `LLMJob`, `ImgGenJob` |
| `image_size` | string | An image size as a generation request spells it, width by height: `2560x1440` |
| `nb_steps` | integer | The number of inference steps an image generation job requests, spelled as its parameters spell it |
diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.md
index 544bb37e7..47a8ff012 100644
--- a/docs/under-the-hood/error-model.md
+++ b/docs/under-the-hood/error-model.md
@@ -568,6 +568,10 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT
| Malformed prompt image / bad prompt parameter | `CONTENT` class-level (`PromptImageFormatError`, `LLMPromptParameterError`, …) → `error_domain = INPUT` → **HTTP 422** |
| Any other provider **HTTP 400** | `CONTENT` → `error_domain = INPUT` → **HTTP 422**. This is the widest reach of the derivation: a 400 covers a context-length overflow and a parameter the model rejects alike, and an engine-side request-construction fault lands here too — reported as the caller's to fix, and absent from the 5xx rate |
| Local file extractor raises a builtin (docling, pypdfium2) | `ValueError` / `RuntimeError` / `FileNotFoundError` → `CONTENT` → `error_domain = INPUT` → **HTTP 422**; `OSError` → `TRANSIENT` (see `_LOCAL_EXTRACT_BY_TYPE_NAME`) |
+| A text completion stops at its output limit or context window (`length`, `max_tokens`, `MAX_TOKENS`, an `incomplete` Responses answer at `max_output_tokens`, `model_context_window_exceeded`, `model_length`), with partial or empty text | Every LLM worker reads the stop before returning text and raises `LLMCompletionTruncatedError` → `CONTENT` / `CHANGE_INPUT`, non-retryable; `error_domain = INPUT` → **HTTP 422**, caller-facing under STRICT. The message names the model, the pipe, the stop value, the output tokens used and the `max_tokens` sent, never the partial text. The error's `truncation_limit` tells the output limit from the context window (`model_context_window_exceeded`, `model_length`), and so does its next step: a higher `max_tokens` or a shorter input for the first, a shorter input, a lower reasoning effort or a larger model for the second, never a higher `max_tokens`. A gateway in non-strict mode passing each provider's own value through the completions shape is read the same way, since the classifier reads the union of the provider vocabularies |
+| A text completion refused or stopped by a safety filter (`content_filter`, `refusal`, `content_filtered`, `guardrail_intervened`, Gemini's `SAFETY` and its kin), or a Gemini prompt blocked before any candidate (its `prompt_feedback.block_reason`, whatever its value) | `LLMCompletionRefusedError` → `CONTENT` / `CHANGE_INPUT`, non-retryable; `error_domain = INPUT` → **HTTP 422**, caller-facing under STRICT |
+| A stop value no provider vocabulary lists | Logged at warning level with the model and the value, and taken as normal: the text is returned. An `incomplete` Responses answer whose reason is missing or unknown is the exception: logged at warning level and taken as a truncation, since the status alone says the text is unfinished |
+| An Anthropic structured generation whose `max_tokens` its structured-output timeout lowers | Sent with the lowered limit; logged at warning level when the pipe set the limit, at debug level when it is the model's default; an error the lowered limit can explain, a structured output that failed validation or came back incomplete once the re-asks are spent, or a timeout, keeps its class and category and ends with a sentence saying the limit was lowered and to what; a rate limit, an authentication failure, an overload, a server error or a lost connection carries no such sentence |
| LLM returns schema-mismatched JSON | `instructor` re-asks; if exhausted → the last `pydantic.ValidationError` → `CONTENT` / `CHANGE_INPUT` → `error_domain = INPUT` → **HTTP 422** |
| The gateway does not allow the model for this account (`model_not_allowed_error`, HTTP 412) | `CONFIGURATION` / `CHANGE_MODEL`, never retried; the advice names the model handle; `error_domain = CONFIG` → HTTP 500 |
| Connection dropped mid-request | `AMBIGUOUS` → non-retryable (outcome unknown); `error_domain = RUNTIME` |
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index aa9fa1184..fdffc0d47 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -396,6 +396,110 @@ class LLMCompletionError(CogtError):
pass
+class CompletionTruncationLimit(StrEnum):
+ """The limit a truncated text completion hit, which decides what can lift it."""
+
+ # The output limit the request sent: a higher max_tokens, or a shorter input, leaves room for the text
+ MAX_TOKENS = "max_tokens"
+ # The model's context window, which the input and the output filled together: a higher max_tokens cannot help
+ CONTEXT_WINDOW = "context_window"
+
+
+class LLMCompletionTruncatedError(LLMCompletionError):
+ """A text completion stopped before the model finished it: it hit its output limit, or its context window.
+
+ A worker raises it when its provider's stop signal says the text was cut (see
+ `pipelex.cogt.llm.completion_stop`), instead of handing back the partial or empty text as a result. It
+ is a content error, in the input domain, and not retryable: the same request stops at the same limit,
+ and the remedy is the author's or the caller's. Which remedy depends on the limit hit, carried as
+ `truncation_limit`: at the output limit, a higher `max_tokens` on the pipe or a shorter input, or, when the
+ request sent no `max_tokens` and the provider's default cut the text, a `max_tokens` set on the pipe up to
+ the model's limit, a lower reasoning effort or a shorter input; at the context window, which the input and
+ the output filled together, a shorter input, less thinking or a model with a larger window, never a higher
+ `max_tokens`. The message names only the pipe, the model's deck handle, the provider's stop value and the
+ token counts, never the partial text or the provider's body, so it is kept verbatim for the caller.
+ """
+
+ error_category = InferenceErrorCategory.CONTENT
+ _authors_caller_facing_message = True
+
+ def __init__(
+ self,
+ *,
+ model_handle: str,
+ stop_reason: str,
+ truncation_limit: CompletionTruncationLimit,
+ pipe_code: str | None = None,
+ max_tokens: int | None = None,
+ output_tokens: int | None = None,
+ ):
+ subject = f"the text of pipe '{pipe_code}'" if pipe_code else "its text"
+ details = [f"stop reason '{stop_reason}'"]
+ if output_tokens is not None:
+ details.append(f"{output_tokens} output tokens used")
+ if max_tokens is not None:
+ details.append(f"max_tokens set to {max_tokens}")
+ what_happened: str
+ next_step: str
+ match truncation_limit:
+ case CompletionTruncationLimit.MAX_TOKENS:
+ what_happened = "was cut off"
+ if max_tokens is not None:
+ next_step = "Raise the pipe's max_tokens, or shorten its input." if pipe_code else "Raise max_tokens, or shorten the input."
+ else:
+ # The request left the limit to the provider's default, so there is no max_tokens of the pipe's to raise
+ details.append("no max_tokens sent")
+ next_step = (
+ "Set the pipe's max_tokens up to the model's limit, lower its reasoning effort, or shorten its input."
+ if pipe_code
+ else "Set max_tokens up to the model's limit, lower the reasoning effort, or shorten the input."
+ )
+ case CompletionTruncationLimit.CONTEXT_WINDOW:
+ what_happened = "filled its context window"
+ next_step = (
+ "Shorten the pipe's input, lower its reasoning effort, or choose a model with a larger context window."
+ if pipe_code
+ else "Shorten the input, lower the reasoning effort, or choose a model with a larger context window."
+ )
+ message = (
+ f"The model '{model_handle}' {what_happened} before it finished {subject} ({', '.join(details)}), so the text is incomplete. {next_step}"
+ )
+ super().__init__(message, user_action=UserAction(kind=UserActionKind.CHANGE_INPUT, detail=next_step))
+ self.model_handle = model_handle
+ self.stop_reason = stop_reason
+ self.truncation_limit = truncation_limit
+ self.pipe_code = pipe_code
+ self.max_tokens = max_tokens
+ self.output_tokens = output_tokens
+
+
+class LLMCompletionRefusedError(LLMCompletionError):
+ """A text completion the model declined to write, or that a provider's safety filter stopped.
+
+ A worker raises it when its provider's stop signal says the model refused or a filter cut the text (see
+ `pipelex.cogt.llm.completion_stop`), instead of handing back the partial or empty text as a result. It
+ is a content error, in the input domain, and not retryable: what a model declines or a filter blocks is
+ the prompt and the input it was given, so the remedy is to revise them. The message names only the pipe,
+ the model's deck handle and the provider's stop value, never the partial text or the provider's body, so
+ it is kept verbatim for the caller.
+ """
+
+ error_category = InferenceErrorCategory.CONTENT
+ _authors_caller_facing_message = True
+
+ def __init__(self, *, model_handle: str, stop_reason: str, pipe_code: str | None = None):
+ subject = f"the text of pipe '{pipe_code}'" if pipe_code else "its text"
+ next_step = "Revise the pipe's prompt or its input." if pipe_code else "Revise the prompt or the input."
+ message = (
+ f"The model '{model_handle}' declined to finish {subject}, or a content filter stopped it (stop reason '{stop_reason}'), "
+ f"so the text cannot be used. {next_step}"
+ )
+ super().__init__(message, user_action=UserAction(kind=UserActionKind.CHANGE_INPUT, detail=next_step))
+ self.model_handle = model_handle
+ self.stop_reason = stop_reason
+ self.pipe_code = pipe_code
+
+
class LLMAssignmentError(CogtError):
pass
diff --git a/pipelex/cogt/llm/completion_stop.py b/pipelex/cogt/llm/completion_stop.py
new file mode 100644
index 000000000..4c6353a12
--- /dev/null
+++ b/pipelex/cogt/llm/completion_stop.py
@@ -0,0 +1,303 @@
+"""How a text completion stopped: normally, cut short, or refused, read from the provider's own stop signal.
+
+A provider answers a completion that did not end normally with a success all the same: a text cut at the
+output limit, or refused, or stopped by a safety filter, comes back as a 200 carrying whatever text the model
+had written, empty or partial, and a stop value saying why it ended. Every LLM worker therefore passes its
+response's stop value through `classify_stop_reason` before it returns any text, and raises on an outcome that
+is not normal (see `LLMWorkerAbstract._check_completion_stop`), so a text the model could not finish fails its
+pipe instead of passing for a result.
+
+The classifier knows every vocabulary a worker can receive, each checked against the provider SDK's own type,
+and reads them as one: a gateway in non-strict mode passes each provider's own value through the OpenAI
+completions shape, so that worker can receive any of them, and the values agree wherever two vocabularies share
+one. A value the classifier does not know is logged at warning level and taken as normal, so a value a provider
+adds later cannot fail every call. A missing value is taken as normal without a word, since some
+OpenAI-compatible servers send none. Two signals say on their own that the text is unusable, whatever reason comes
+with them: a Responses API answer whose status is `incomplete` is never normal, an unknown or missing reason being
+taken as a truncation, and a Gemini answer with no candidate whose prompt feedback names a block reason is a refusal.
+
+The classifier only turns a stop that was silently a success into an error. Structured generation is not read
+here: its stops are the structuring library's business, and its configured re-asks are left as they are.
+"""
+
+from collections.abc import Mapping
+from enum import StrEnum
+from types import MappingProxyType
+
+from pipelex import log
+from pipelex.cogt.exceptions import CompletionTruncationLimit, LLMCompletionRefusedError, LLMCompletionTruncatedError
+
+
+class CompletionStopOutcome(StrEnum):
+ """What a completion's stop value says about its text."""
+
+ NORMAL = "normal"
+ # Cut at the output limit the request sent
+ TRUNCATED = "truncated"
+ # Cut when the input and the output together filled the model's context window, which no higher output limit lifts
+ CONTEXT_WINDOW_EXCEEDED = "context_window_exceeded"
+ REFUSED = "refused"
+
+
+_NORMAL = CompletionStopOutcome.NORMAL
+_TRUNCATED = CompletionStopOutcome.TRUNCATED
+_CONTEXT_WINDOW_EXCEEDED = CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED
+_REFUSED = CompletionStopOutcome.REFUSED
+
+# `openai.types.chat.chat_completion.Choice.finish_reason`, plus `refusal`, the value a gateway's strict OpenAI
+# compliance maps Anthropic's refusal to.
+OPENAI_CHAT_FINISH_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "stop": _NORMAL,
+ "tool_calls": _NORMAL,
+ "function_call": _NORMAL,
+ "length": _TRUNCATED,
+ "content_filter": _REFUSED,
+ "refusal": _REFUSED,
+ }
+)
+
+# `anthropic.types.StopReason`, plus `model_context_window_exceeded`, which the Messages API returns when the
+# context window ends the text and a gateway passes through.
+ANTHROPIC_STOP_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "end_turn": _NORMAL,
+ "stop_sequence": _NORMAL,
+ "tool_use": _NORMAL,
+ "pause_turn": _NORMAL,
+ "max_tokens": _TRUNCATED,
+ "model_context_window_exceeded": _CONTEXT_WINDOW_EXCEEDED,
+ "refusal": _REFUSED,
+ }
+)
+
+# `types_aiobotocore_bedrock_runtime.literals.StopReasonType`, Bedrock Converse's `stopReason`. The two
+# `malformed_*` values concern tool use and output parsing, which a text generation does not ask for, so the
+# text is returned as before.
+BEDROCK_CONVERSE_STOP_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "end_turn": _NORMAL,
+ "stop_sequence": _NORMAL,
+ "tool_use": _NORMAL,
+ "malformed_model_output": _NORMAL,
+ "malformed_tool_use": _NORMAL,
+ "max_tokens": _TRUNCATED,
+ "model_context_window_exceeded": _CONTEXT_WINDOW_EXCEEDED,
+ "content_filtered": _REFUSED,
+ "guardrail_intervened": _REFUSED,
+ }
+)
+
+# `google.genai.types.FinishReason`. Every safety, recitation, blocklist, privacy and language stop is a filter,
+# the image ones included; the unspecified, other, tool-call and no-image values say nothing about the text, so
+# it is returned as before, as a gateway in strict mode also maps them to `stop`.
+GEMINI_FINISH_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "STOP": _NORMAL,
+ "FINISH_REASON_UNSPECIFIED": _NORMAL,
+ "OTHER": _NORMAL,
+ "MALFORMED_FUNCTION_CALL": _NORMAL,
+ "UNEXPECTED_TOOL_CALL": _NORMAL,
+ "NO_IMAGE": _NORMAL,
+ "IMAGE_OTHER": _NORMAL,
+ "MAX_TOKENS": _TRUNCATED,
+ "SAFETY": _REFUSED,
+ "RECITATION": _REFUSED,
+ "LANGUAGE": _REFUSED,
+ "BLOCKLIST": _REFUSED,
+ "PROHIBITED_CONTENT": _REFUSED,
+ "SPII": _REFUSED,
+ "IMAGE_SAFETY": _REFUSED,
+ "IMAGE_PROHIBITED_CONTENT": _REFUSED,
+ "IMAGE_RECITATION": _REFUSED,
+ }
+)
+
+# `mistralai.client.models.ChatCompletionChoiceFinishReason`. Its `length` is the request's output limit and its
+# `model_length` the model's context window. Its `error` is neither a cut nor a refusal, and a gateway in strict mode
+# maps it to `stop`, so the text is returned as before.
+MISTRAL_FINISH_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "stop": _NORMAL,
+ "tool_calls": _NORMAL,
+ "error": _NORMAL,
+ "length": _TRUNCATED,
+ "model_length": _CONTEXT_WINDOW_EXCEEDED,
+ }
+)
+
+# `openai.types.responses.response.IncompleteDetails.reason`, read when a Responses API answer's status is
+# `incomplete`.
+OPENAI_RESPONSES_INCOMPLETE_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "max_output_tokens": _TRUNCATED,
+ "content_filter": _REFUSED,
+ }
+)
+
+# `google.genai.types.BlockedReason`, a Gemini answer's `prompt_feedback.block_reason`, read when the answer holds no
+# candidate: the prompt was blocked before the model wrote anything, so every value is a refusal, the unspecified
+# one included. It is not read with the stop values, which share some of its words with another meaning.
+GEMINI_BLOCK_REASONS: Mapping[str, CompletionStopOutcome] = MappingProxyType(
+ {
+ "BLOCKED_REASON_UNSPECIFIED": _REFUSED,
+ "SAFETY": _REFUSED,
+ "OTHER": _REFUSED,
+ "BLOCKLIST": _REFUSED,
+ "PROHIBITED_CONTENT": _REFUSED,
+ "IMAGE_SAFETY": _REFUSED,
+ "MODEL_ARMOR": _REFUSED,
+ "JAILBREAK": _REFUSED,
+ }
+)
+
+# Every vocabulary a worker's stop value can come from, read as one, whatever worker receives it.
+STOP_REASON_VOCABULARIES: tuple[Mapping[str, CompletionStopOutcome], ...] = (
+ OPENAI_CHAT_FINISH_REASONS,
+ ANTHROPIC_STOP_REASONS,
+ BEDROCK_CONVERSE_STOP_REASONS,
+ GEMINI_FINISH_REASONS,
+ MISTRAL_FINISH_REASONS,
+)
+
+# The status a Responses API answer has when it stopped before the model finished.
+OPENAI_RESPONSES_INCOMPLETE_STATUS = "incomplete"
+
+
+def merge_stop_vocabularies(*, vocabularies: tuple[Mapping[str, CompletionStopOutcome], ...]) -> Mapping[str, CompletionStopOutcome]:
+ """Merge stop vocabularies into one table keyed by the case-folded value.
+
+ Gemini writes its values in capitals and the others in lower case, and no two of them read a shared value
+ differently, which a unit test holds, so the case-folded union classifies a value whichever provider sent it.
+ """
+ merged: dict[str, CompletionStopOutcome] = {}
+ for vocabulary in vocabularies:
+ for value, outcome in vocabulary.items():
+ merged[value.casefold()] = outcome
+ return MappingProxyType(merged)
+
+
+_ALL_STOP_REASONS = merge_stop_vocabularies(vocabularies=STOP_REASON_VOCABULARIES)
+
+
+def _log_unknown_stop(*, model_handle: str, stop_value: object) -> None:
+ log.warning(
+ "An LLM stop reason was not recognized, so it was taken as normal",
+ fields={"model_handle": model_handle, "stop_reason": str(stop_value)},
+ )
+
+
+def classify_stop_reason(*, stop_reason: str | None, model_handle: str) -> CompletionStopOutcome:
+ """Read a completion's stop value, from any vocabulary the classifier knows, as normal, cut short or refused.
+
+ Args:
+ stop_reason: The provider's stop value: a finish reason, a stop reason or Converse's stop reason. None when
+ the response carries none.
+ model_handle: The model's deck handle, named by the warning about an unknown value.
+
+ Returns:
+ The outcome the value stands for: normal for a missing or unknown value.
+ """
+ if stop_reason is None:
+ return _NORMAL
+ outcome = _ALL_STOP_REASONS.get(stop_reason.casefold())
+ if outcome is None:
+ _log_unknown_stop(model_handle=model_handle, stop_value=stop_reason)
+ return _NORMAL
+ return outcome
+
+
+def classify_responses_stop(*, status: str | None, incomplete_reason: str | None, model_handle: str) -> CompletionStopOutcome:
+ """Read a Responses API answer's status, and its incomplete reason, as normal, truncated or refused.
+
+ Only an `incomplete` status says the model did not finish; every other status is normal here, a failed or
+ cancelled answer being caught by the worker's own check for an answer with no text. An incomplete answer is never
+ normal, since the status alone says its text is unfinished: its reason only tells a filter from a cut, and a
+ reason that is missing, the SDK typing both the details and their reason as optional, or that the classifier does
+ not know is taken as a cut and logged at warning level.
+
+ Args:
+ status: The answer's `status`. None when it carries none.
+ incomplete_reason: The `incomplete_details.reason` of an incomplete answer. None when it carries none.
+ model_handle: The model's deck handle, named by the warning about a missing or unknown reason.
+
+ Returns:
+ The outcome the status and reason stand for: truncated for an incomplete answer whose reason is missing or unknown.
+ """
+ if status != OPENAI_RESPONSES_INCOMPLETE_STATUS:
+ return _NORMAL
+ outcome = OPENAI_RESPONSES_INCOMPLETE_REASONS.get(incomplete_reason) if incomplete_reason is not None else None
+ if outcome is None:
+ log.warning(
+ "An incomplete answer gave no known reason, so it was taken as truncated",
+ fields={"model_handle": model_handle, "stop_reason": status, "incomplete_reason": incomplete_reason},
+ )
+ return _TRUNCATED
+ return outcome
+
+
+def classify_prompt_block_reason(*, block_reason: str, model_handle: str) -> CompletionStopOutcome:
+ """Read the block reason of a Gemini answer that holds no candidate, which is always a refusal.
+
+ The prompt was blocked before the model wrote anything, so whatever reason is given the text cannot be had: a
+ reason the classifier does not know is a refusal too, logged at warning level so the table can learn it.
+
+ Args:
+ block_reason: The answer's `prompt_feedback.block_reason`, as its value.
+ model_handle: The model's deck handle, named by the warning about an unknown reason.
+
+ Returns:
+ The refusal outcome.
+ """
+ outcome = GEMINI_BLOCK_REASONS.get(block_reason.upper())
+ if outcome is None:
+ log.warning(
+ "A blocked prompt gave an unknown reason, so it was taken as a refusal",
+ fields={"model_handle": model_handle, "block_reason": block_reason},
+ )
+ return _REFUSED
+ return outcome
+
+
+def raise_for_completion_stop(
+ *,
+ outcome: CompletionStopOutcome,
+ stop_reason: str,
+ model_handle: str,
+ pipe_code: str | None,
+ max_tokens: int | None,
+ output_tokens: int | None,
+) -> None:
+ """Raise the error a stop that is not normal stands for, and return on a normal one.
+
+ Args:
+ outcome: The outcome the stop value was classified as.
+ stop_reason: The provider's stop value, as the error names it.
+ model_handle: The model's deck handle.
+ pipe_code: The code of the pipe whose text it was, when the job knows it.
+ max_tokens: The output limit the request sent, when it sent one.
+ output_tokens: The output tokens the provider counted, when it counted them.
+
+ Raises:
+ LLMCompletionTruncatedError: When the text was cut before the model finished it, at its output limit or at
+ its context window, which the error carries as its `truncation_limit`.
+ LLMCompletionRefusedError: When the model refused, or a content filter stopped the text.
+ """
+ truncation_limit: CompletionTruncationLimit
+ match outcome:
+ case CompletionStopOutcome.NORMAL:
+ return
+ case CompletionStopOutcome.REFUSED:
+ raise LLMCompletionRefusedError(model_handle=model_handle, stop_reason=stop_reason, pipe_code=pipe_code)
+ case CompletionStopOutcome.TRUNCATED:
+ truncation_limit = CompletionTruncationLimit.MAX_TOKENS
+ case CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED:
+ truncation_limit = CompletionTruncationLimit.CONTEXT_WINDOW
+ raise LLMCompletionTruncatedError(
+ model_handle=model_handle,
+ stop_reason=stop_reason,
+ truncation_limit=truncation_limit,
+ pipe_code=pipe_code,
+ max_tokens=max_tokens,
+ output_tokens=output_tokens,
+ )
diff --git a/pipelex/cogt/llm/llm_worker_abstract.py b/pipelex/cogt/llm/llm_worker_abstract.py
index 6e899d6b8..b24b96804 100644
--- a/pipelex/cogt/llm/llm_worker_abstract.py
+++ b/pipelex/cogt/llm/llm_worker_abstract.py
@@ -9,11 +9,12 @@ from opentelemetry.trace import NonRecordingSpan, Span, SpanContext, SpanKind, S
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import CogtError, LLMCapabilityError
+from pipelex.cogt.exceptions import CogtError, LLMCapabilityError, LLMCompletionRefusedError, LLMCompletionTruncatedError
from pipelex.cogt.inference.inference_call_summary import InferenceCallSummary, InferenceOperation
from pipelex.cogt.inference.inference_constants import InferenceOutputType
from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract
from pipelex.cogt.inference.prompt_file_checks import check_prompt_documents_are_read, check_prompt_images_are_images
+from pipelex.cogt.llm.completion_stop import CompletionStopOutcome, classify_stop_reason, raise_for_completion_stop
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.usage.token_category import TokenCategory
@@ -329,18 +330,21 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
return span
+ @classmethod
+ def _set_otel_span_usage(cls, *, span: Span | None, llm_job: LLMJob) -> None:
+ """Record the job's token usage on the OTel span, when the job recorded one. Safe to call if span is None."""
+ if span is None or not llm_job.job_report.llm_tokens_usage:
+ return
+ tokens = llm_job.job_report.llm_tokens_usage.nb_tokens_by_category
+ span.set_attribute(GenAISpanAttr.USAGE_INPUT_TOKENS, tokens.get(TokenCategory.INPUT, 0))
+ span.set_attribute(GenAISpanAttr.USAGE_OUTPUT_TOKENS, tokens.get(TokenCategory.OUTPUT, 0))
+
def _end_otel_span_with_completion_text(self, span: Span | None, *, llm_job: LLMJob, completion_text: str) -> None:
"""End the OTel span, recording usage and status. Safe to call if span is None."""
if span is None:
return
- # Record token usage if available
- if llm_job.job_report.llm_tokens_usage:
- tokens = llm_job.job_report.llm_tokens_usage.nb_tokens_by_category
- input_tokens = tokens.get(TokenCategory.INPUT, 0)
- output_tokens = tokens.get(TokenCategory.OUTPUT, 0)
- span.set_attribute(GenAISpanAttr.USAGE_INPUT_TOKENS, input_tokens)
- span.set_attribute(GenAISpanAttr.USAGE_OUTPUT_TOKENS, output_tokens)
+ self._set_otel_span_usage(span=span, llm_job=llm_job)
# Always capture full completion content - exporters handle redaction as needed
span.set_attribute(GenAISpanAttr.COMPLETION_CONTENT, completion_text)
@@ -355,13 +359,7 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
if span is None:
return
- # Record token usage if available
- if llm_job.job_report.llm_tokens_usage:
- tokens = llm_job.job_report.llm_tokens_usage.nb_tokens_by_category
- input_tokens = tokens.get(TokenCategory.INPUT, 0)
- output_tokens = tokens.get(TokenCategory.OUTPUT, 0)
- span.set_attribute(GenAISpanAttr.USAGE_INPUT_TOKENS, input_tokens)
- span.set_attribute(GenAISpanAttr.USAGE_OUTPUT_TOKENS, output_tokens)
+ self._set_otel_span_usage(span=span, llm_job=llm_job)
# Always capture full completion content - exporters handle redaction as needed
completion_object_json = OtelFactory.stringify_json(json_conent=completion_object.model_dump(serialize_as_any=True))
@@ -447,6 +445,23 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
# End OTel span with success status and usage data
self._end_otel_span_with_completion_text(span=span, llm_job=llm_job, completion_text=result_text)
+ def _report_stopped_text_job(self, *, span: Span | None, llm_job: LLMJob) -> None:
+ """Report the usage of a text its stop check refused, and record it on the span, which is left open.
+
+ The provider answered that text and billed it: every worker reads its response's token usage into the job
+ before its stop check raises, so a text cut at its limit or refused is reported as a returned text is, once,
+ and the run's cost report and its usage event count what the provider billed. The span is ended by the
+ caller's error path, with the error. A job that recorded no usage, its worker having raised before reading
+ any, has nothing to report.
+ """
+ tokens_usage = llm_job.job_report.llm_tokens_usage
+ if tokens_usage is None or not tokens_usage.nb_tokens_by_category:
+ return
+ llm_job.llm_job_after_complete()
+ if self.reporting_delegate:
+ self.reporting_delegate.report_inference_job(inference_job=llm_job)
+ self._set_otel_span_usage(span=span, llm_job=llm_job)
+
async def _after_object_job(
self,
*,
@@ -462,6 +477,39 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
# End OTel span with success status and usage data
self._end_otel_span_with_completion_object(span=span, llm_job=llm_job, completion_object=result_object)
+ def _check_completion_stop(self, *, llm_job: LLMJob, stop_reason: str | None, max_tokens: int | None) -> None:
+ """Raise when the text's stop value says it was cut or refused, before the worker hands the text back.
+
+ Every worker's text generation calls it with its response's stop value, after it has read the response's
+ token usage into the job, so a truncation names the output tokens used. A missing or unknown value is
+ normal (see `pipelex.cogt.llm.completion_stop`).
+
+ Args:
+ llm_job: The job whose text it is: its pipe and its token usage are named by the error.
+ stop_reason: The provider's stop value, None when the response carries none.
+ max_tokens: The output limit the request sent, None when it sent none.
+
+ Raises:
+ LLMCompletionTruncatedError: When the text was cut before the model finished it.
+ LLMCompletionRefusedError: When the model refused, or a content filter stopped the text.
+ """
+ outcome = classify_stop_reason(stop_reason=stop_reason, model_handle=self.inference_model.name)
+ self._raise_for_completion_stop(llm_job=llm_job, outcome=outcome, stop_reason=stop_reason or "", max_tokens=max_tokens)
+
+ def _raise_for_completion_stop(self, *, llm_job: LLMJob, outcome: CompletionStopOutcome, stop_reason: str, max_tokens: int | None) -> None:
+ """Raise the error a classified stop stands for, naming the job's pipe, the model and the output tokens used."""
+ output_tokens: int | None = None
+ if tokens_usage := llm_job.job_report.llm_tokens_usage:
+ output_tokens = tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT)
+ raise_for_completion_stop(
+ outcome=outcome,
+ stop_reason=stop_reason,
+ model_handle=self.inference_model.name,
+ pipe_code=llm_job.job_metadata.pipe_code,
+ max_tokens=max_tokens,
+ output_tokens=output_tokens,
+ )
+
def _check_can_perform_job(self, llm_job: LLMJob):
# This can be overridden by subclasses for specific checks
self._check_vision_support(llm_job=llm_job)
@@ -518,7 +566,12 @@ class LLMWorkerAbstract(InferenceWorkerAbstract, ABC):
with pipelex_span_active(span=span), call_summary.ends_here():
try:
self.check_request(inference_model=self.inference_model, job_params=self._sent_job_params(llm_job=llm_job), is_structured=False)
- text_result = await self._gen_text(llm_job=llm_job)
+ try:
+ text_result = await self._gen_text(llm_job=llm_job)
+ except (LLMCompletionTruncatedError, LLMCompletionRefusedError):
+ # The stop check refused a text the provider answered and billed: its usage is reported all the same
+ self._report_stopped_text_job(span=span, llm_job=llm_job)
+ raise
await self._after_text_job(span=span, llm_job=llm_job, result_text=text_result)
return text_result
except CogtError as exc:
diff --git a/pipelex/providers/anthropic/anthropic_llm_worker.py b/pipelex/providers/anthropic/anthropic_llm_worker.py
index 950ed6bcb..ec3c18789 100644
--- a/pipelex/providers/anthropic/anthropic_llm_worker.py
+++ b/pipelex/providers/anthropic/anthropic_llm_worker.py
@@ -3,6 +3,7 @@ from typing import TYPE_CHECKING, Any, ClassVar
from anthropic import (
APIConnectionError,
APIStatusError,
+ APITimeoutError,
AsyncAnthropic,
omit,
)
@@ -11,7 +12,8 @@ from anthropic.types import OutputConfigParam, ThinkingConfigParam
from pydantic.dataclasses import dataclass as pydantic_dataclass
from typing_extensions import override
-from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex import log
+from pipelex.cogt.exceptions import CogtError, InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
from pipelex.cogt.inference.error_classification import (
UserAction,
UserActionKind,
@@ -142,6 +144,81 @@ class AnthropicLLMWorker(LLMWorkerAbstract):
safe_max_tokens = AnthropicFactory.calculate_safe_max_tokens_for_timeout(timeout_seconds=timeout_seconds)
return min(requested_max_tokens, safe_max_tokens)
+ def _say_max_tokens_lowered(
+ self,
+ *,
+ is_caller_setting: bool,
+ requested_max_tokens: int,
+ effective_max_tokens: int,
+ timeout_seconds: int,
+ ) -> str | None:
+ """Say that a structured call sends a lower max_tokens than it was given, and return the sentence its errors end with.
+
+ A max_tokens the pipe's setting asked for and the call lowers is said at warning level, since the call then
+ sends less than the method wrote. The model's own default lowered is said at debug level: a model whose
+ default exceeds what the timeout allows has it lowered on every structured call, by design. Either way, an
+ error the call raises for a failure the lowered limit can explain, an output that failed validation or came
+ back incomplete, or a timeout, says the limit was lowered and to what (see `_with_note`).
+
+ Args:
+ is_caller_setting: Whether the requested max_tokens is the pipe's setting rather than the model's default.
+ requested_max_tokens: The max_tokens the call was given.
+ effective_max_tokens: The max_tokens the call sends.
+ timeout_seconds: The structured call's timeout, which the effective max_tokens fits.
+
+ Returns:
+ The sentence such an error of the call ends with, None when the call sends the max_tokens it was given.
+ """
+ if effective_max_tokens >= requested_max_tokens:
+ return None
+ fields = {
+ "model_handle": self.inference_model.name,
+ "requested_max_tokens": requested_max_tokens,
+ "effective_max_tokens": effective_max_tokens,
+ "timeout_seconds": timeout_seconds,
+ }
+ if is_caller_setting:
+ log.warning("A structured output's token limit was lowered to fit its timeout", fields=fields)
+ else:
+ log.debug("The model's default token limit was lowered to fit the structured output's timeout", fields=fields)
+ return (
+ f"The structured output's max_tokens was lowered from {requested_max_tokens} to {effective_max_tokens} "
+ f"to fit its {timeout_seconds}-second timeout."
+ )
+
+ @classmethod
+ def _lowered_max_tokens_can_explain(cls, *, failure: BaseException | None) -> bool:
+ """Whether a lowered max_tokens can explain the failure that ended a structured call.
+
+ A lower limit can cut the structured output short, which instructor reports, once its re-asks are spent, as
+ an output that failed validation or came back incomplete; and the limit is lowered to fit the call's
+ timeout, so a timeout concerns it too. A failure the API or the network answered for its own reason, a rate
+ limit, an authentication failure, an overload, a server error or a lost connection, has nothing to do with it.
+
+ Args:
+ failure: The exception that ended the call, None when instructor's wrapper carries none.
+
+ Returns:
+ True when the failure is a timeout or no API or connection error.
+ """
+ if isinstance(failure, APITimeoutError):
+ return True
+ return not isinstance(failure, (APIStatusError, APIConnectionError))
+
+ @classmethod
+ def _with_note(cls, *, error: CogtError, note: str | None, failure: BaseException | None) -> CogtError:
+ """The error a structured call raises, its message ending with the note when the lowered limit can explain the failure.
+
+ The note adds a fact to the error the call built and changes nothing else: its class, its category and
+ whether it is retried stay what they were. It is left off an error the lowered limit cannot explain, a rate
+ limit or an outage, so that error does not point at max_tokens.
+ """
+ if note is None or not cls._lowered_max_tokens_can_explain(failure=failure):
+ return error
+ error.message = f"{error.message} {note}"
+ error.args = (error.message,)
+ return error
+
@classmethod
def _build_thinking_params(cls, *, inference_model: InferenceModelSpec, job_params: LLMJobParams, max_tokens: int | None) -> _ThinkingParams:
"""Build thinking-related SDK parameters from job params and model spec.
@@ -301,6 +378,12 @@ class AnthropicLLMWorker(LLMWorkerAbstract):
model_handle=self.inference_model.name,
) from sdk_exc
+ if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and final_message.usage:
+ llm_tokens_usage.nb_tokens_by_category = AnthropicFactory.make_nb_tokens_by_category(usage=final_message.usage)
+
+ # Read before the text, so a text cut at max_tokens with nothing written, thinking having used the budget, is a truncation too
+ self._check_completion_stop(llm_job=llm_job, stop_reason=final_message.stop_reason, max_tokens=max_tokens)
+
# Collect all text blocks (adaptive thinking enables interleaved thinking,
# so the response may contain multiple text blocks interspersed with thinking blocks)
text_parts: list[str] = []
@@ -331,12 +414,7 @@ class AnthropicLLMWorker(LLMWorkerAbstract):
),
)
- full_reply_content = "\n\n".join(text_parts)
-
- if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and final_message.usage:
- llm_tokens_usage.nb_tokens_by_category = AnthropicFactory.make_nb_tokens_by_category(usage=final_message.usage)
-
- return full_reply_content
+ return "\n\n".join(text_parts)
def _request_extras_kwargs(self, *, llm_job: LLMJob, output_desc: str) -> dict[str, Any]:
"""The per-request headers and body additions this call sends, as SDK keyword arguments.
@@ -395,7 +473,14 @@ class AnthropicLLMWorker(LLMWorkerAbstract):
# The structured call sets an explicit timeout, and its max_tokens is held to what that timeout allows
timeout_seconds = get_config().inference.llm.anthropic.structured_output_timeout_seconds
- effective_max_tokens = self._sent_max_tokens(requested_max_tokens=job_params.max_tokens or self.default_max_tokens, is_structured=True)
+ requested_max_tokens = job_params.max_tokens or self.default_max_tokens
+ effective_max_tokens = self._sent_max_tokens(requested_max_tokens=requested_max_tokens, is_structured=True)
+ lowered_max_tokens_note = self._say_max_tokens_lowered(
+ is_caller_setting=llm_job.job_params.max_tokens is not None,
+ requested_max_tokens=requested_max_tokens,
+ effective_max_tokens=effective_max_tokens,
+ timeout_seconds=timeout_seconds,
+ )
# The thinking budget is fitted against the max_tokens this call actually sends
thinking_params = self._build_thinking_params(inference_model=self.inference_model, job_params=job_params, max_tokens=effective_max_tokens)
@@ -432,35 +517,38 @@ class AnthropicLLMWorker(LLMWorkerAbstract):
if underlying_exc is not None:
metadata = extract_anthropic_metadata(underlying_exc)
classification = classify_inference_error(metadata)
- raise render_inference_error(
+ rendered_error = render_inference_error(
metadata=metadata,
classification=classification,
family=InferenceErrorFamily.LLM,
model_desc=self.inference_model.desc,
model_handle=self.inference_model.name,
- ) from instructor_exc
+ )
+ raise self._with_note(error=rendered_error, note=lowered_max_tokens_note, failure=underlying_exc) from instructor_exc
msg = (
f"Anthropic structured generation via 'instructor' failed with model: {self.inference_model.desc} "
f"trying to generate schema: {schema} with error: {instructor_exc}"
)
- raise LLMCompletionError(
+ fallback_error = LLMCompletionError(
msg,
error_category=InferenceErrorCategory.UNKNOWN,
user_action=UserAction(
kind=UserActionKind.CONTACT_SUPPORT,
detail="Structured generation failed for an unrecognized reason — retry, and report this if it persists",
),
- ) from instructor_exc
+ )
+ raise self._with_note(error=fallback_error, note=lowered_max_tokens_note, failure=None) from instructor_exc
except (APIStatusError, APIConnectionError) as sdk_exc:
metadata = extract_anthropic_metadata(sdk_exc)
classification = classify_inference_error(metadata)
- raise render_inference_error(
+ rendered_error = render_inference_error(
metadata=metadata,
classification=classification,
family=InferenceErrorFamily.LLM,
model_desc=self.inference_model.desc,
model_handle=self.inference_model.name,
- ) from sdk_exc
+ )
+ raise self._with_note(error=rendered_error, note=lowered_max_tokens_note, failure=sdk_exc) from sdk_exc
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := completion.usage):
llm_tokens_usage.nb_tokens_by_category = AnthropicFactory.make_nb_tokens_by_category(usage=usage)
diff --git a/pipelex/providers/bedrock/bedrock_client_aiobotocore.py b/pipelex/providers/bedrock/bedrock_client_aiobotocore.py
index ff48be2dd..8c15978a1 100644
--- a/pipelex/providers/bedrock/bedrock_client_aiobotocore.py
+++ b/pipelex/providers/bedrock/bedrock_client_aiobotocore.py
@@ -4,8 +4,7 @@ from aiobotocore.session import get_session
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.providers.bedrock.bedrock_client_protocol import BedrockClientProtocol, make_converse_params
+from pipelex.providers.bedrock.bedrock_client_protocol import BedrockChatResult, BedrockClientProtocol, make_converse_params, read_converse_response
from pipelex.providers.bedrock.bedrock_message import BedrockMessageDictList
if TYPE_CHECKING:
@@ -27,16 +26,9 @@ class BedrockClientAiobotocore(BedrockClientProtocol):
model: str,
temperature: float | None,
max_tokens: int | None = None,
- ) -> tuple[str, NbTokensByCategoryDict]:
+ ) -> BedrockChatResult:
params = make_converse_params(messages=messages, system_text=system_text, model=model, temperature=temperature, max_tokens=max_tokens)
async with self.session.create_client("bedrock-runtime", region_name=self.aws_region) as bedrock_runtime_client: # pyright: ignore[reportUnknownMemberType]
conversation_response: ConverseResponseTypeDef = await bedrock_runtime_client.converse(**params)
- resp_dict: dict[str, Any] = cast("dict[str, Any]", conversation_response)
- usage_dict: dict[str, Any] = resp_dict["usage"]
- nb_tokens_by_category: NbTokensByCategoryDict = {
- TokenCategory.INPUT: usage_dict["inputTokens"],
- TokenCategory.OUTPUT: usage_dict["outputTokens"],
- }
- response_text: str = resp_dict["output"]["message"]["content"][0]["text"]
- return response_text, nb_tokens_by_category
+ return read_converse_response(response=cast("dict[str, Any]", conversation_response))
diff --git a/pipelex/providers/bedrock/bedrock_client_boto3.py b/pipelex/providers/bedrock/bedrock_client_boto3.py
index c96079a86..6088f7f72 100644
--- a/pipelex/providers/bedrock/bedrock_client_boto3.py
+++ b/pipelex/providers/bedrock/bedrock_client_boto3.py
@@ -5,8 +5,7 @@ import boto3
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
-from pipelex.providers.bedrock.bedrock_client_protocol import BedrockClientProtocol, make_converse_params
+from pipelex.providers.bedrock.bedrock_client_protocol import BedrockChatResult, BedrockClientProtocol, make_converse_params, read_converse_response
from pipelex.providers.bedrock.bedrock_message import BedrockMessageDictList
@@ -24,7 +23,7 @@ class BedrockClientBoto3(BedrockClientProtocol):
model: str,
temperature: float | None,
max_tokens: int | None = None,
- ) -> tuple[str, NbTokensByCategoryDict]:
+ ) -> BedrockChatResult:
params = make_converse_params(messages=messages, system_text=system_text, model=model, temperature=temperature, max_tokens=max_tokens)
# ``to_thread`` rather than ``run_in_executor``: it carries the context over, so the SDK's own log
@@ -32,10 +31,4 @@ class BedrockClientBoto3(BedrockClientProtocol):
# standard trace fields, and a span its instrumentation opens has that current span as its parent.
resp_dict: dict[str, Any] = await asyncio.to_thread(self.boto3_client.converse, **params) # pyright: ignore[reportUnknownMemberType, reportUnknownArgumentType]
- usage_dict: dict[str, Any] = resp_dict["usage"]
- nb_tokens_by_category: NbTokensByCategoryDict = {
- TokenCategory.INPUT: usage_dict["inputTokens"],
- TokenCategory.OUTPUT: usage_dict["outputTokens"],
- }
- response_text: str = resp_dict["output"]["message"]["content"][0]["text"]
- return response_text, nb_tokens_by_category
+ return read_converse_response(response=resp_dict)
diff --git a/pipelex/providers/bedrock/bedrock_client_protocol.py b/pipelex/providers/bedrock/bedrock_client_protocol.py
index 548eeb6c6..c664054b5 100644
--- a/pipelex/providers/bedrock/bedrock_client_protocol.py
+++ b/pipelex/providers/bedrock/bedrock_client_protocol.py
@@ -1,9 +1,21 @@
-from typing import Any, Protocol, runtime_checkable
+from typing import Any, NamedTuple, Protocol, cast, runtime_checkable
-from pipelex.cogt.usage.token_category import NbTokensByCategoryDict
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.providers.bedrock.bedrock_message import BedrockMessageDictList
+class BedrockChatResult(NamedTuple):
+ """What a Bedrock Converse call answered: its text, its token counts and why it stopped.
+
+ `stop_reason` is Converse's `stopReason`, which the worker reads to tell a text cut at its limit or stopped by
+ a filter from a finished one, None when the answer carries none.
+ """
+
+ text: str
+ nb_tokens_by_category: NbTokensByCategoryDict
+ stop_reason: str | None
+
+
@runtime_checkable
class BedrockClientProtocol(Protocol):
async def chat(
@@ -14,7 +26,32 @@ class BedrockClientProtocol(Protocol):
model: str,
temperature: float | None,
max_tokens: int | None = None,
- ) -> tuple[str, NbTokensByCategoryDict]: ...
+ ) -> BedrockChatResult: ...
+
+
+def read_converse_response(*, response: dict[str, Any]) -> BedrockChatResult:
+ """Read a Bedrock Converse answer into its text, its token counts and its stop reason.
+
+ The text is every text block of the answer's message joined, empty when none came back, as when a content
+ filter stopped the answer: the worker then raises on the stop reason rather than on a missing block.
+ """
+ usage_dict: dict[str, Any] = response["usage"]
+ nb_tokens_by_category: NbTokensByCategoryDict = {
+ TokenCategory.INPUT: usage_dict["inputTokens"],
+ TokenCategory.OUTPUT: usage_dict["outputTokens"],
+ }
+ content_blocks = cast("list[dict[str, Any]]", response.get("output", {}).get("message", {}).get("content", []))
+ text_parts: list[str] = []
+ for content_block in content_blocks:
+ block_text = content_block.get("text")
+ if isinstance(block_text, str):
+ text_parts.append(block_text)
+ stop_reason = response.get("stopReason")
+ return BedrockChatResult(
+ text="".join(text_parts),
+ nb_tokens_by_category=nb_tokens_by_category,
+ stop_reason=stop_reason if isinstance(stop_reason, str) else None,
+ )
def make_converse_params(
diff --git a/pipelex/providers/bedrock/bedrock_llm_worker.py b/pipelex/providers/bedrock/bedrock_llm_worker.py
index d41ef6487..fb0cc3a6e 100644
--- a/pipelex/providers/bedrock/bedrock_llm_worker.py
+++ b/pipelex/providers/bedrock/bedrock_llm_worker.py
@@ -3,8 +3,8 @@ from typing import Any
from botocore.exceptions import ClientError
from typing_extensions import override
-from pipelex.cogt.exceptions import LLMCapabilityError, SdkTypeError
-from pipelex.cogt.inference.error_classification import extract_bedrock_metadata
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.inference.error_classification import UserAction, UserActionKind, extract_bedrock_metadata
from pipelex.cogt.inference.error_classify import classify_inference_error
from pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_error
from pipelex.cogt.llm.llm_job import LLMJob
@@ -67,13 +67,14 @@ class BedrockLLMWorker(LLMWorkerAbstract):
job_params = llm_job.applied_job_params or llm_job.job_params
message = BedrockFactory.make_simple_message(llm_job=llm_job)
+ max_tokens = job_params.max_tokens or self.default_max_tokens
try:
- bedrock_response_text, nb_tokens_by_category = await self.bedrock_client_for_text.chat(
+ chat_result = await self.bedrock_client_for_text.chat(
messages=message.to_dict_list(),
system_text=llm_job.llm_prompt.system_text,
model=self.inference_model.model_id,
temperature=job_params.temperature if self.inference_model.accepts_temperature else None,
- max_tokens=job_params.max_tokens or self.default_max_tokens,
+ max_tokens=max_tokens,
)
except ClientError as exc:
metadata = extract_bedrock_metadata(exc)
@@ -86,9 +87,21 @@ class BedrockLLMWorker(LLMWorkerAbstract):
model_handle=self.inference_model.name,
) from exc
- if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and nb_tokens_by_category:
- llm_tokens_usage.nb_tokens_by_category = nb_tokens_by_category
- return bedrock_response_text
+ if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and chat_result.nb_tokens_by_category:
+ llm_tokens_usage.nb_tokens_by_category = chat_result.nb_tokens_by_category
+
+ self._check_completion_stop(llm_job=llm_job, stop_reason=chat_result.stop_reason, max_tokens=max_tokens)
+ if not chat_result.text:
+ msg = f"Bedrock returned no text with model: {self.inference_model.desc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action=UserAction(
+ kind=UserActionKind.CHANGE_INPUT,
+ detail="Bedrock returned an empty text response — try rephrasing the prompt or using a different model",
+ ),
+ )
+ return chat_result.text
@override
async def _gen_object(
diff --git a/pipelex/providers/google/google_llm_worker.py b/pipelex/providers/google/google_llm_worker.py
index 86f06b833..4cc42418e 100644
--- a/pipelex/providers/google/google_llm_worker.py
+++ b/pipelex/providers/google/google_llm_worker.py
@@ -17,6 +17,7 @@ from pipelex.cogt.inference.error_classification import (
)
from pipelex.cogt.inference.error_classify import classify_inference_error
from pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_error
+from pipelex.cogt.llm.completion_stop import classify_prompt_block_reason
from pipelex.cogt.llm.instructor_retry import make_instructor_schema_retrying
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams, ReasoningEffort
@@ -260,14 +261,25 @@ class GoogleLLMWorker(LLMWorkerAbstract):
model_handle=self.inference_model.name,
) from sdk_exc
- # Extract text from response (skips thinking parts)
- text_content = GoogleFactory.extract_text_from_response(response=response, model_desc=self.inference_model.desc)
-
# Track token usage if available
if llm_job.job_report.llm_tokens_usage and response.usage_metadata:
llm_job.job_report.llm_tokens_usage.nb_tokens_by_category = GoogleFactory.extract_token_usage(response.usage_metadata)
- return text_content
+ # Read before the text, so a candidate cut or filtered with no text left is reported for what stopped it
+ if response.candidates:
+ if finish_reason := response.candidates[0].finish_reason:
+ self._check_completion_stop(llm_job=llm_job, stop_reason=finish_reason.value, max_tokens=job_params.max_tokens)
+ elif response.prompt_feedback and (block_reason := response.prompt_feedback.block_reason):
+ # A blocked prompt gets no candidate at all, its block reason in the prompt feedback: a refusal, named for its reason
+ self._raise_for_completion_stop(
+ llm_job=llm_job,
+ outcome=classify_prompt_block_reason(block_reason=block_reason.value, model_handle=self.inference_model.name),
+ stop_reason=block_reason.value,
+ max_tokens=job_params.max_tokens,
+ )
+
+ # Extract text from response (skips thinking parts)
+ return GoogleFactory.extract_text_from_response(response=response, model_desc=self.inference_model.desc)
def _validates_structured_output_strictly(self) -> bool:
"""Whether ``instructor`` validates a structured response in pydantic's strict mode.
diff --git a/pipelex/providers/mistral/mistral_llm_worker.py b/pipelex/providers/mistral/mistral_llm_worker.py
index 1810c1797..29d99a89c 100644
--- a/pipelex/providers/mistral/mistral_llm_worker.py
+++ b/pipelex/providers/mistral/mistral_llm_worker.py
@@ -146,12 +146,13 @@ class MistralLLMWorker(LLMWorkerAbstract):
messages = await self.mistral_factory.make_simple_messages(llm_job=llm_job)
reasoning_effort = self._resolve_reasoning_effort(inference_model=self.inference_model, job_params=job_params)
self._log_reasoning_sent(api_name="Mistral", settings={"reasoning_effort": None if reasoning_effort is UNSET else reasoning_effort})
+ max_tokens = job_params.max_tokens or self.default_max_tokens
try:
response: ChatCompletionResponse | None = await self.mistral_client_for_text.chat.complete_async(
messages=messages,
model=self.inference_model.model_id,
temperature=job_params.temperature if self.inference_model.accepts_temperature else UNSET,
- max_tokens=job_params.max_tokens or self.default_max_tokens,
+ max_tokens=max_tokens,
reasoning_effort=reasoning_effort,
)
except (MistralError, httpx.TransportError) as sdk_exc:
@@ -199,6 +200,12 @@ class MistralLLMWorker(LLMWorkerAbstract):
detail="Mistral returned a choice with no message — wait a moment, then run it again",
),
)
+ if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
+ llm_tokens_usage.nb_tokens_by_category = self.mistral_factory.make_nb_tokens_by_category(usage=usage)
+
+ # Read before the content, so a text cut with nothing written is reported for what stopped it
+ self._check_completion_stop(llm_job=llm_job, stop_reason=response.choices[0].finish_reason, max_tokens=max_tokens)
+
mistral_response_content = message.content
result_text: str
if isinstance(mistral_response_content, str):
@@ -239,9 +246,6 @@ class MistralLLMWorker(LLMWorkerAbstract):
),
)
- if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
- llm_tokens_usage.nb_tokens_by_category = self.mistral_factory.make_nb_tokens_by_category(usage=usage)
-
return result_text
@override
diff --git a/pipelex/providers/openai/openai_completions_llm_worker.py b/pipelex/providers/openai/openai_completions_llm_worker.py
index 91821fde8..a206fec3a 100644
--- a/pipelex/providers/openai/openai_completions_llm_worker.py
+++ b/pipelex/providers/openai/openai_completions_llm_worker.py
@@ -176,17 +176,12 @@ class OpenAICompletionsLLMWorker(LLMWorkerAbstract):
),
)
- finish_reason = response.choices[0].finish_reason
- if finish_reason == "content_filter":
- msg = f"OpenAI response was filtered by content policy for model: {self.inference_model.desc}"
- raise LLMCompletionError(
- msg,
- error_category=InferenceErrorCategory.CONTENT,
- user_action=UserAction(
- kind=UserActionKind.CHANGE_INPUT,
- detail="Content was rejected by safety filters — revise the prompt",
- ),
- )
+ if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
+ llm_tokens_usage.nb_tokens_by_category = self.openai_completions_factory.make_nb_tokens_by_category(usage=usage)
+
+ # The finish reason is read in every vocabulary: a gateway in non-strict mode passes each provider's own
+ # value through this shape, Claude's `max_tokens` and `refusal` or Bedrock's `content_filtered` among them.
+ self._check_completion_stop(llm_job=llm_job, stop_reason=response.choices[0].finish_reason, max_tokens=job_params.max_tokens)
openai_message: ChatCompletionMessage = response.choices[0].message
response_text = openai_message.content
@@ -202,8 +197,6 @@ class OpenAICompletionsLLMWorker(LLMWorkerAbstract):
),
)
- if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := response.usage):
- llm_tokens_usage.nb_tokens_by_category = self.openai_completions_factory.make_nb_tokens_by_category(usage=usage)
return response_text
@override
diff --git a/pipelex/providers/openai/openai_responses_llm_worker.py b/pipelex/providers/openai/openai_responses_llm_worker.py
index ef0ea65ae..812d12991 100644
--- a/pipelex/providers/openai/openai_responses_llm_worker.py
+++ b/pipelex/providers/openai/openai_responses_llm_worker.py
@@ -24,6 +24,7 @@ from pipelex.cogt.inference.error_classification import (
)
from pipelex.cogt.inference.error_classify import classify_inference_error
from pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_error
+from pipelex.cogt.llm.completion_stop import classify_responses_stop
from pipelex.cogt.llm.instructor_retry import make_instructor_schema_retrying
from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen
from pipelex.cogt.llm.llm_worker_abstract import LLMWorkerAbstract
@@ -172,6 +173,19 @@ class OpenAIResponsesLLMWorker(LLMWorkerAbstract):
model_handle=self.inference_model.name,
) from sdk_exc
+ if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and response.usage:
+ llm_tokens_usage.nb_tokens_by_category = self.openai_responses_factory.make_nb_tokens_by_category(usage=response.usage)
+
+ # An answer cut at its output limit or stopped by a filter has the `incomplete` status, its reason in its details
+ incomplete_reason = response.incomplete_details.reason if response.incomplete_details else None
+ stop_outcome = classify_responses_stop(status=response.status, incomplete_reason=incomplete_reason, model_handle=self.inference_model.name)
+ self._raise_for_completion_stop(
+ llm_job=llm_job,
+ outcome=stop_outcome,
+ stop_reason=incomplete_reason or response.status or "",
+ max_tokens=job_params.max_tokens,
+ )
+
if not response.output_text:
msg = f"OpenAI Responses message content is empty: {response}\nmodel: {self.inference_model.desc}"
raise LLMCompletionError(
@@ -184,8 +198,6 @@ class OpenAIResponsesLLMWorker(LLMWorkerAbstract):
),
)
- if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and response.usage:
- llm_tokens_usage.nb_tokens_by_category = self.openai_responses_factory.make_nb_tokens_by_category(usage=response.usage)
return response.output_text
@override
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index 9e13da2cf..bc4392536 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -158,6 +158,8 @@ KitIndexLoadingError | Kit index loading | https://docs.pipelex.com/latest/error
LLMAssignmentError | LLM assignment | https://docs.pipelex.com/latest/errors/llm-assignment-error/
LLMCapabilityError | LLM capability | https://docs.pipelex.com/latest/errors/llm-capability-error/
LLMCompletionError | LLM completion | https://docs.pipelex.com/latest/errors/llm-completion-error/
+LLMCompletionRefusedError | LLM completion refused | https://docs.pipelex.com/latest/errors/llm-completion-refused-error/
+LLMCompletionTruncatedError | LLM completion truncated | https://docs.pipelex.com/latest/errors/llm-completion-truncated-error/
LLMConfigError | LLM config | https://docs.pipelex.com/latest/errors/llm-config-error/
LLMHandleNotFoundError | LLM handle not found | https://docs.pipelex.com/latest/errors/llm-handle-not-found-error/
LLMModelNotFoundError | LLM model not found | https://docs.pipelex.com/latest/errors/llm-model-not-found-error/
diff --git a/tests/helpers/completion_stop.py b/tests/helpers/completion_stop.py
new file mode 100644
index 000000000..2242b59f1
--- /dev/null
+++ b/tests/helpers/completion_stop.py
@@ -0,0 +1,40 @@
+"""A real text job for driving an LLM worker's text generation into its stop check.
+
+The stop check names the job's pipe and the output tokens the worker read into the job's usage, so the job is a
+real `LLMJob` whose usage is seeded empty, as `llm_job_before_start` seeds it, rather than a mock whose every
+attribute answers.
+"""
+
+from __future__ import annotations
+
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.system.job_metadata import JobMetadata, RunMetadata
+
+STOP_TEST_PIPE_CODE = "summarize_contract"
+STOP_TEST_PARTIAL_TEXT = "The contract binds the parties to deliver the goods by the end of"
+
+
+def make_text_llm_job(*, pipe_code: str | None = STOP_TEST_PIPE_CODE, max_tokens: int | None = None) -> LLMJob:
+ """A text job for the given pipe, its token usage seeded empty and its max_tokens as given."""
+ job_metadata = JobMetadata(
+ run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="test-user", pipeline_run_id="test-run"),
+ pipe_code=pipe_code,
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 1.0},
+ nb_tokens_by_category={},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(system_text="You are a careful summarizer.", user_text="Summarize the contract."),
+ job_params=LLMJobParams(temperature=0.5, max_tokens=max_tokens),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
diff --git a/tests/integration/pipelex/cli/test_data.py b/tests/integration/pipelex/cli/test_data.py
new file mode 100644
index 000000000..711f6f170
--- /dev/null
+++ b/tests/integration/pipelex/cli/test_data.py
@@ -0,0 +1,21 @@
+from typing import ClassVar
+
+
+class TruncatedCompletionRunTestData:
+ """A method whose text step the model cannot finish: its completion stops at the output limit."""
+
+ PIPE_CODE: ClassVar[str] = "write_summary"
+ MODEL_HANDLE: ClassVar[str] = "stand-in-gpt"
+ PARTIAL_TEXT: ClassVar[str] = "Cats are small carnivorous mammals that"
+
+ MTHDS: ClassVar[str] = """
+domain = "truncated_completion"
+description = "A text step whose completion the model cannot finish"
+
+[pipe.write_summary]
+type = "PipeLLM"
+description = "Summarize the topic"
+inputs = { topic = "Text" }
+output = "Text"
+prompt = "Write a long summary about $topic"
+"""
diff --git a/tests/integration/pipelex/cli/test_run_truncated_completion.py b/tests/integration/pipelex/cli/test_run_truncated_completion.py
new file mode 100644
index 000000000..6ed560315
--- /dev/null
+++ b/tests/integration/pipelex/cli/test_run_truncated_completion.py
@@ -0,0 +1,113 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import pytest
+import typer
+from openai import AsyncOpenAI
+from rich.console import Console
+
+from pipelex.cli.commands.run._run_core import _execute_run # pyright: ignore[reportPrivateUsage]
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.providers.openai.openai_completions_factory import OpenAICompletionsFactory
+from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
+from tests.integration.pipelex.cli.test_data import TruncatedCompletionRunTestData
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from pytest_mock import MockerFixture
+
+
+def _truncating_worker() -> OpenAICompletionsLLMWorker:
+ """A real chat-completions worker whose provider answers every call with a text cut at its output limit."""
+ body: dict[str, Any] = {
+ "id": "chatcmpl_truncated",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "stand-in-gpt-1",
+ "choices": [
+ {"index": 0, "finish_reason": "length", "message": {"role": "assistant", "content": TruncatedCompletionRunTestData.PARTIAL_TEXT}}
+ ],
+ "usage": {"prompt_tokens": 12, "completion_tokens": 64, "total_tokens": 76},
+ }
+
+ def answer(request: httpx.Request) -> httpx.Response:
+ return httpx.Response(200, json=body, request=request)
+
+ sdk_client = AsyncOpenAI(
+ api_key="test-key",
+ base_url="https://provider.test/v1",
+ max_retries=0,
+ http_client=httpx.AsyncClient(transport=httpx.MockTransport(answer)),
+ )
+ inference_model = InferenceModelSpec(
+ backend_name="stand_in",
+ name=TruncatedCompletionRunTestData.MODEL_HANDLE,
+ sdk="openai",
+ model_type=ModelType.LLM,
+ model_id="stand-in-gpt-1",
+ inputs=["text"],
+ outputs=["text", "structured"],
+ costs={},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+ return OpenAICompletionsLLMWorker(
+ openai_completions_factory=OpenAICompletionsFactory(is_http_url_enabled=False),
+ sdk_instance=sdk_client,
+ inference_model=inference_model,
+ reporting_delegate=None,
+ )
+
+
[email protected](loop_scope="class")
+class TestRunTruncatedCompletion:
+ async def test_a_truncated_text_fails_the_run_naming_the_pipe_and_the_error(
+ self,
+ mocker: MockerFixture,
+ tmp_path: Path,
+ capsys: pytest.CaptureFixture[str],
+ ) -> None:
+ """`pipelex run` on a step whose completion stops at its limit exits non-zero, naming the pipe and the truncation, and prints no output."""
+ mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker", return_value=_truncating_worker())
+ console = Console(width=400, record=True, color_system=None)
+ mocker.patch("pipelex.cli.error_handlers.get_console", return_value=console)
+ mocker.patch("pipelex.cli.commands.run._run_core.get_console", return_value=console)
+ bundle_path = tmp_path / "truncated_completion.mthds"
+ bundle_path.write_text(TruncatedCompletionRunTestData.MTHDS, encoding="utf-8")
+
+ with pytest.raises(typer.Exit) as exit_info:
+ await _execute_run(
+ pipe_code=TruncatedCompletionRunTestData.PIPE_CODE,
+ bundle_path=str(bundle_path),
+ inputs='{"topic": "cats"}',
+ save_working_memory=False,
+ working_memory_path=None,
+ save_main_stuff=False,
+ no_pretty_print=True,
+ graph=False,
+ graph_full_data=None,
+ output_dir=str(tmp_path / "outputs"),
+ dry_run=False,
+ mock_usage=False,
+ mock_inputs=False,
+ library_dir=None,
+ )
+
+ assert exit_info.value.exit_code == 1
+ stderr = capsys.readouterr().err
+ pipe_code = TruncatedCompletionRunTestData.PIPE_CODE
+ model_handle = TruncatedCompletionRunTestData.MODEL_HANDLE
+ assert f"Failed to execute pipeline '{pipe_code}': Pipe '{pipe_code}' failed: " in stderr
+ assert (
+ f"The model '{model_handle}' was cut off before it finished the text of pipe '{pipe_code}' "
+ "(stop reason 'length', 64 output tokens used, no max_tokens sent), so the text is incomplete. "
+ "Set the pipe's max_tokens up to the model's limit, lower its reasoning effort, or shorten its input."
+ ) in stderr
+ assert TruncatedCompletionRunTestData.PARTIAL_TEXT not in stderr
+ assert TruncatedCompletionRunTestData.PARTIAL_TEXT not in console.export_text()
diff --git a/tests/unit/pipelex/cogt/llm/test_completion_stop.py b/tests/unit/pipelex/cogt/llm/test_completion_stop.py
new file mode 100644
index 000000000..56e02ba6c
--- /dev/null
+++ b/tests/unit/pipelex/cogt/llm/test_completion_stop.py
@@ -0,0 +1,179 @@
+from __future__ import annotations
+
+import typing
+from typing import TYPE_CHECKING
+
+import pytest
+from anthropic.types import StopReason
+from google.genai.types import BlockedReason, FinishReason
+from mistralai.client.models.chatcompletionchoice import ChatCompletionChoiceFinishReason
+from openai.types.chat.chat_completion import Choice
+from openai.types.responses.response import IncompleteDetails
+from types_aiobotocore_bedrock_runtime.literals import StopReasonType
+
+from pipelex.cogt.exceptions import CompletionTruncationLimit, LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm import completion_stop
+from pipelex.cogt.llm.completion_stop import (
+ ANTHROPIC_STOP_REASONS,
+ BEDROCK_CONVERSE_STOP_REASONS,
+ GEMINI_BLOCK_REASONS,
+ GEMINI_FINISH_REASONS,
+ MISTRAL_FINISH_REASONS,
+ OPENAI_CHAT_FINISH_REASONS,
+ OPENAI_RESPONSES_INCOMPLETE_REASONS,
+ STOP_REASON_VOCABULARIES,
+ CompletionStopOutcome,
+ classify_prompt_block_reason,
+ classify_responses_stop,
+ classify_stop_reason,
+ raise_for_completion_stop,
+)
+from tests.unit.pipelex.cogt.llm.test_data import CompletionStopTestData
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+def _literal_values(annotation: object) -> set[str]:
+ """The string values of a `Literal`, looked for inside an `Optional` or a union with an open string type."""
+ values: set[str] = set()
+ for argument in typing.get_args(annotation):
+ if isinstance(argument, str):
+ values.add(argument)
+ else:
+ values.update(_literal_values(argument))
+ return values
+
+
+class TestCompletionStop:
+ @pytest.mark.parametrize(("_topic", "stop_reason", "expected_outcome"), CompletionStopTestData.STOP_REASON_CASES)
+ def test_each_vocabulary_is_classified(self, _topic: str, stop_reason: str, expected_outcome: CompletionStopOutcome) -> None:
+ assert classify_stop_reason(stop_reason=stop_reason, model_handle="test-model") == expected_outcome
+
+ def test_a_gemini_value_is_read_whatever_its_case(self) -> None:
+ assert classify_stop_reason(stop_reason="max_tokens", model_handle="gemini-test") == CompletionStopOutcome.TRUNCATED
+ assert classify_stop_reason(stop_reason="Safety", model_handle="gemini-test") == CompletionStopOutcome.REFUSED
+
+ def test_an_unknown_value_is_logged_and_taken_as_normal(self, mocker: MockerFixture) -> None:
+ """A value no vocabulary knows warns, naming the model and the value, and fails nothing."""
+ warning = mocker.patch.object(completion_stop.log, "warning")
+
+ outcome = classify_stop_reason(stop_reason="brand_new_stop", model_handle="claude-test")
+
+ assert outcome == CompletionStopOutcome.NORMAL
+ warning.assert_called_once_with(
+ "An LLM stop reason was not recognized, so it was taken as normal",
+ fields={"model_handle": "claude-test", "stop_reason": "brand_new_stop"},
+ )
+
+ def test_a_missing_value_is_normal_without_a_word(self, mocker: MockerFixture) -> None:
+ warning = mocker.patch.object(completion_stop.log, "warning")
+
+ assert classify_stop_reason(stop_reason=None, model_handle="test-model") == CompletionStopOutcome.NORMAL
+ warning.assert_not_called()
+
+ @pytest.mark.parametrize(("_topic", "status", "incomplete_reason", "expected_outcome"), CompletionStopTestData.RESPONSES_CASES)
+ def test_a_responses_answer_is_classified(
+ self, _topic: str, status: str | None, incomplete_reason: str | None, expected_outcome: CompletionStopOutcome
+ ) -> None:
+ outcome = classify_responses_stop(status=status, incomplete_reason=incomplete_reason, model_handle="gpt-test")
+ assert outcome == expected_outcome
+
+ @pytest.mark.parametrize("incomplete_reason", ["brand_new_reason", None])
+ def test_an_incomplete_answer_without_a_known_reason_is_logged_and_taken_as_truncated(
+ self, mocker: MockerFixture, incomplete_reason: str | None
+ ) -> None:
+ """The incomplete status alone says the text is unfinished, so a missing or unknown reason never makes it normal."""
+ warning = mocker.patch.object(completion_stop.log, "warning")
+
+ outcome = classify_responses_stop(status="incomplete", incomplete_reason=incomplete_reason, model_handle="gpt-test")
+
+ assert outcome == CompletionStopOutcome.TRUNCATED
+ warning.assert_called_once_with(
+ "An incomplete answer gave no known reason, so it was taken as truncated",
+ fields={"model_handle": "gpt-test", "stop_reason": "incomplete", "incomplete_reason": incomplete_reason},
+ )
+
+ @pytest.mark.parametrize("block_reason", [member.value for member in BlockedReason])
+ def test_every_gemini_block_reason_is_a_refusal(self, mocker: MockerFixture, block_reason: str) -> None:
+ warning = mocker.patch.object(completion_stop.log, "warning")
+
+ assert classify_prompt_block_reason(block_reason=block_reason, model_handle="gemini-test") == CompletionStopOutcome.REFUSED
+ warning.assert_not_called()
+
+ def test_an_unknown_gemini_block_reason_is_logged_and_taken_as_a_refusal(self, mocker: MockerFixture) -> None:
+ warning = mocker.patch.object(completion_stop.log, "warning")
+
+ outcome = classify_prompt_block_reason(block_reason="BRAND_NEW_BLOCK", model_handle="gemini-test")
+
+ assert outcome == CompletionStopOutcome.REFUSED
+ warning.assert_called_once_with(
+ "A blocked prompt gave an unknown reason, so it was taken as a refusal",
+ fields={"model_handle": "gemini-test", "block_reason": "BRAND_NEW_BLOCK"},
+ )
+
+ def test_the_vocabularies_agree_on_every_shared_value(self) -> None:
+ """Read as one table, no two vocabularies may read the same case-folded value differently."""
+ readings: dict[str, set[CompletionStopOutcome]] = {}
+ for vocabulary in STOP_REASON_VOCABULARIES:
+ for value, outcome in vocabulary.items():
+ readings.setdefault(value.casefold(), set()).add(outcome)
+ conflicting = {value: outcomes for value, outcomes in readings.items() if len(outcomes) > 1}
+ assert not conflicting
+
+ def test_each_vocabulary_covers_its_sdk_type(self) -> None:
+ """Each table holds every value its provider SDK's own type lists, plus only the additions it documents."""
+ assert set(OPENAI_CHAT_FINISH_REASONS) == _literal_values(Choice.model_fields["finish_reason"].annotation) | {"refusal"}
+ assert set(ANTHROPIC_STOP_REASONS) == _literal_values(StopReason) | {"model_context_window_exceeded"}
+ assert set(BEDROCK_CONVERSE_STOP_REASONS) == _literal_values(StopReasonType)
+ assert set(GEMINI_FINISH_REASONS) == {member.value for member in FinishReason}
+ assert set(MISTRAL_FINISH_REASONS) == _literal_values(ChatCompletionChoiceFinishReason)
+ assert set(OPENAI_RESPONSES_INCOMPLETE_REASONS) == _literal_values(IncompleteDetails.model_fields["reason"].annotation)
+ assert set(GEMINI_BLOCK_REASONS) == {member.value for member in BlockedReason}
+
+ def test_a_normal_stop_raises_nothing(self) -> None:
+ raise_for_completion_stop(
+ outcome=CompletionStopOutcome.NORMAL,
+ stop_reason="end_turn",
+ model_handle="claude-test",
+ pipe_code="summarize",
+ max_tokens=4096,
+ output_tokens=12,
+ )
+
+ @pytest.mark.parametrize(
+ ("outcome", "stop_reason", "expected_limit"),
+ [
+ (CompletionStopOutcome.TRUNCATED, "max_tokens", CompletionTruncationLimit.MAX_TOKENS),
+ (CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED, "model_context_window_exceeded", CompletionTruncationLimit.CONTEXT_WINDOW),
+ ],
+ )
+ def test_a_truncated_stop_raises_the_truncation_naming_its_limit(
+ self, outcome: CompletionStopOutcome, stop_reason: str, expected_limit: CompletionTruncationLimit
+ ) -> None:
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ raise_for_completion_stop(
+ outcome=outcome,
+ stop_reason=stop_reason,
+ model_handle="claude-test",
+ pipe_code="summarize",
+ max_tokens=4096,
+ output_tokens=4096,
+ )
+ assert exc_info.value.stop_reason == stop_reason
+ assert exc_info.value.truncation_limit == expected_limit
+ assert exc_info.value.max_tokens == 4096
+ assert exc_info.value.output_tokens == 4096
+
+ def test_a_refused_stop_raises_the_refusal(self) -> None:
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ raise_for_completion_stop(
+ outcome=CompletionStopOutcome.REFUSED,
+ stop_reason="content_filtered",
+ model_handle="claude-test",
+ pipe_code="summarize",
+ max_tokens=4096,
+ output_tokens=0,
+ )
+ assert exc_info.value.stop_reason == "content_filtered"
+ assert exc_info.value.pipe_code == "summarize"
diff --git a/tests/unit/pipelex/cogt/llm/test_data.py b/tests/unit/pipelex/cogt/llm/test_data.py
new file mode 100644
index 000000000..729b54d89
--- /dev/null
+++ b/tests/unit/pipelex/cogt/llm/test_data.py
@@ -0,0 +1,44 @@
+from typing import ClassVar
+
+from pipelex.cogt.llm.completion_stop import CompletionStopOutcome
+
+
+class CompletionStopTestData:
+ # (topic, stop value, outcome): a normal, a truncated and a refused value from each vocabulary
+ STOP_REASON_CASES: ClassVar[list[tuple[str, str, CompletionStopOutcome]]] = [
+ ("openai normal", "stop", CompletionStopOutcome.NORMAL),
+ ("openai tool call", "tool_calls", CompletionStopOutcome.NORMAL),
+ ("openai truncated", "length", CompletionStopOutcome.TRUNCATED),
+ ("openai filtered", "content_filter", CompletionStopOutcome.REFUSED),
+ ("gateway strict refusal", "refusal", CompletionStopOutcome.REFUSED),
+ ("anthropic normal", "end_turn", CompletionStopOutcome.NORMAL),
+ ("anthropic paused", "pause_turn", CompletionStopOutcome.NORMAL),
+ ("anthropic truncated", "max_tokens", CompletionStopOutcome.TRUNCATED),
+ ("anthropic context window", "model_context_window_exceeded", CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED),
+ ("anthropic refused", "refusal", CompletionStopOutcome.REFUSED),
+ ("converse normal", "stop_sequence", CompletionStopOutcome.NORMAL),
+ ("converse truncated", "max_tokens", CompletionStopOutcome.TRUNCATED),
+ ("converse context window", "model_context_window_exceeded", CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED),
+ ("converse filtered", "content_filtered", CompletionStopOutcome.REFUSED),
+ ("converse guardrail", "guardrail_intervened", CompletionStopOutcome.REFUSED),
+ ("gemini normal", "STOP", CompletionStopOutcome.NORMAL),
+ ("gemini truncated", "MAX_TOKENS", CompletionStopOutcome.TRUNCATED),
+ ("gemini safety", "SAFETY", CompletionStopOutcome.REFUSED),
+ ("gemini recitation", "RECITATION", CompletionStopOutcome.REFUSED),
+ ("gemini blocklist", "BLOCKLIST", CompletionStopOutcome.REFUSED),
+ ("gemini prohibited", "PROHIBITED_CONTENT", CompletionStopOutcome.REFUSED),
+ ("gemini privacy", "SPII", CompletionStopOutcome.REFUSED),
+ ("mistral normal", "stop", CompletionStopOutcome.NORMAL),
+ ("mistral truncated", "length", CompletionStopOutcome.TRUNCATED),
+ ("mistral context window", "model_length", CompletionStopOutcome.CONTEXT_WINDOW_EXCEEDED),
+ ]
+
+ # (topic, status, incomplete reason, outcome)
+ RESPONSES_CASES: ClassVar[list[tuple[str, str | None, str | None, CompletionStopOutcome]]] = [
+ ("completed", "completed", None, CompletionStopOutcome.NORMAL),
+ ("no status", None, None, CompletionStopOutcome.NORMAL),
+ ("truncated", "incomplete", "max_output_tokens", CompletionStopOutcome.TRUNCATED),
+ ("filtered", "incomplete", "content_filter", CompletionStopOutcome.REFUSED),
+ ("incomplete with no reason", "incomplete", None, CompletionStopOutcome.TRUNCATED),
+ ("failed", "failed", None, CompletionStopOutcome.NORMAL),
+ ]
diff --git a/tests/unit/pipelex/cogt/llm/test_llm_worker_stopped_text_usage.py b/tests/unit/pipelex/cogt/llm/test_llm_worker_stopped_text_usage.py
new file mode 100644
index 000000000..884e8476f
--- /dev/null
+++ b/tests/unit/pipelex/cogt/llm/test_llm_worker_stopped_text_usage.py
@@ -0,0 +1,176 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from openai import AsyncOpenAI
+from openai.types import CompletionUsage
+from openai.types.chat import ChatCompletion, ChatCompletionMessage
+from openai.types.chat.chat_completion import Choice
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import SimpleSpanProcessor
+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
+from opentelemetry.trace import StatusCode
+
+from pipelex.cogt.exceptions import LLMCompletionError, LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.providers.openai.openai_completions_factory import OpenAICompletionsFactory
+from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
+from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.system.job_metadata import JobMetadata, OtelContext, RunMetadata
+from pipelex.system.telemetry.otel_constants import GenAISpanAttr
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE
+
+if TYPE_CHECKING:
+ from typing import Literal
+
+ from opentelemetry.sdk.trace import ReadableSpan
+ from pytest_mock import MockerFixture, MockType
+
+INPUT_TOKENS = 50
+OUTPUT_TOKENS = 256
+
+
+def _completion(*, finish_reason: Literal["stop", "length", "content_filter"]) -> ChatCompletion:
+ return ChatCompletion(
+ id="chatcmpl_test",
+ object="chat.completion",
+ created=0,
+ model="gpt-test-id",
+ choices=[Choice(index=0, finish_reason=finish_reason, message=ChatCompletionMessage(role="assistant", content=STOP_TEST_PARTIAL_TEXT))],
+ usage=CompletionUsage(prompt_tokens=INPUT_TOKENS, completion_tokens=OUTPUT_TOKENS, total_tokens=INPUT_TOKENS + OUTPUT_TOKENS),
+ )
+
+
+def _make_worker(mocker: MockerFixture, *, response: ChatCompletion) -> tuple[OpenAICompletionsLLMWorker, MockType]:
+ """The worker built as its factory builds it, its SDK call answering with the response, its delegate recording."""
+ sdk_client = AsyncOpenAI(api_key="test-key")
+ mocker.patch.object(sdk_client.chat.completions, "create", new=mocker.AsyncMock(return_value=response))
+ reporting_delegate = mocker.MagicMock(spec=ReportingProtocol)
+ inference_model = InferenceModelSpec(
+ backend_name="openai",
+ name="gpt-test",
+ sdk="openai",
+ model_type=ModelType.LLM,
+ model_id="gpt-test-id",
+ inputs=["text"],
+ outputs=["text", "structured"],
+ costs={CostCategory.INPUT: 1, CostCategory.OUTPUT: 2},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+ worker = OpenAICompletionsLLMWorker(
+ openai_completions_factory=OpenAICompletionsFactory(is_http_url_enabled=False),
+ sdk_instance=sdk_client,
+ inference_model=inference_model,
+ reporting_delegate=reporting_delegate,
+ )
+ return worker, reporting_delegate
+
+
+def _make_llm_job() -> LLMJob:
+ """A text job run under a pipe whose span the metadata names as the parent, so the call starts a span."""
+ job_metadata = JobMetadata(
+ run_metadata=RunMetadata(user_id="pytest", pipeline_run_id="plr-stop", storage_scope="test/scope", read_scope=None),
+ pipe_code=STOP_TEST_PIPE_CODE,
+ otel_context=OtelContext(
+ trace_id=0x0123456789ABCDEF0123456789ABCDEF,
+ trace_name=STOP_TEST_PIPE_CODE,
+ trace_name_redacted=STOP_TEST_PIPE_CODE,
+ span_id=0x00000000000000AB,
+ ),
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(user_text="Summarize the contract."),
+ job_params=LLMJobParams(temperature=0.5, max_tokens=OUTPUT_TOKENS),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ )
+
+
+def _only_span(exporter: InMemorySpanExporter) -> ReadableSpan:
+ (span,) = exporter.get_finished_spans()
+ return span
+
+
+def _assert_reported_once_with_its_usage(*, reporting_delegate: MockType, llm_job: LLMJob) -> None:
+ reporting_delegate.report_inference_job.assert_called_once_with(inference_job=llm_job)
+ tokens_usage = llm_job.job_report.llm_tokens_usage
+ assert tokens_usage is not None
+ assert tokens_usage.nb_tokens_by_category[TokenCategory.INPUT] == INPUT_TOKENS
+ assert tokens_usage.nb_tokens_by_category[TokenCategory.OUTPUT] == OUTPUT_TOKENS
+ assert llm_job.job_metadata.completed_at is not None
+
+
+def span_exporter(mocker: MockerFixture) -> InMemorySpanExporter:
+ """The runtime's tracer, replaced by one exporting to memory, so the test reads back the span the worker started."""
+ exporter = InMemorySpanExporter()
+ provider = TracerProvider()
+ provider.add_span_processor(SimpleSpanProcessor(exporter))
+ mocker.patch.object(TelemetryManagerAbstract, "get_instance_tracer", return_value=provider.get_tracer(__name__))
+ return exporter
+
+
[email protected](loop_scope="class")
+class TestLLMWorkerStoppedTextUsage:
+ @pytest.mark.parametrize(
+ ("finish_reason", "expected_error"),
+ [
+ ("length", LLMCompletionTruncatedError),
+ ("content_filter", LLMCompletionRefusedError),
+ ],
+ )
+ async def test_a_stopped_text_raises_and_reports_its_billed_usage(
+ self,
+ mocker: MockerFixture,
+ span_exporter: InMemorySpanExporter,
+ finish_reason: Literal["length", "content_filter"],
+ expected_error: type[LLMCompletionError],
+ ) -> None:
+ """The provider answered and billed the text the stop check refused: its usage is reported once and put on the span."""
+ worker, reporting_delegate = _make_worker(mocker, response=_completion(finish_reason=finish_reason))
+ llm_job = _make_llm_job()
+
+ with pytest.raises(expected_error):
+ await worker.gen_text(llm_job=llm_job)
+
+ _assert_reported_once_with_its_usage(reporting_delegate=reporting_delegate, llm_job=llm_job)
+ llm_span = _only_span(span_exporter)
+ assert llm_span.status.status_code == StatusCode.ERROR
+ assert llm_span.attributes is not None
+ assert llm_span.attributes[GenAISpanAttr.USAGE_INPUT_TOKENS] == INPUT_TOKENS
+ assert llm_span.attributes[GenAISpanAttr.USAGE_OUTPUT_TOKENS] == OUTPUT_TOKENS
+
+ async def test_a_returned_text_is_reported_once(self, mocker: MockerFixture, span_exporter: InMemorySpanExporter) -> None:
+ worker, reporting_delegate = _make_worker(mocker, response=_completion(finish_reason="stop"))
+ llm_job = _make_llm_job()
+
+ assert await worker.gen_text(llm_job=llm_job) == STOP_TEST_PARTIAL_TEXT
+
+ _assert_reported_once_with_its_usage(reporting_delegate=reporting_delegate, llm_job=llm_job)
+ llm_span = _only_span(span_exporter)
+ assert llm_span.status.status_code == StatusCode.OK
+ assert llm_span.attributes is not None
+ assert llm_span.attributes[GenAISpanAttr.USAGE_OUTPUT_TOKENS] == OUTPUT_TOKENS
+
+ async def test_a_failure_with_no_usage_recorded_reports_nothing(self, mocker: MockerFixture, span_exporter: InMemorySpanExporter) -> None:
+ """A provider answer with no usage leaves nothing to report, even when its stop check refuses the text."""
+ response = _completion(finish_reason="length")
+ response.usage = None
+ worker, reporting_delegate = _make_worker(mocker, response=response)
+
+ with pytest.raises(LLMCompletionTruncatedError):
+ await worker.gen_text(llm_job=_make_llm_job())
+
+ reporting_delegate.report_inference_job.assert_not_called()
+ assert _only_span(span_exporter).status.status_code == StatusCode.ERROR
diff --git a/tests/unit/pipelex/cogt/test_llm_completion_stop_errors.py b/tests/unit/pipelex/cogt/test_llm_completion_stop_errors.py
new file mode 100644
index 000000000..f8c1c3843
--- /dev/null
+++ b/tests/unit/pipelex/cogt/test_llm_completion_stop_errors.py
@@ -0,0 +1,185 @@
+from __future__ import annotations
+
+import pytest
+
+from pipelex.base_exceptions import DisclosureMode, ErrorDomain
+from pipelex.cogt.exceptions import (
+ CompletionTruncationLimit,
+ InferenceErrorCategory,
+ LLMCompletionError,
+ LLMCompletionRefusedError,
+ LLMCompletionTruncatedError,
+)
+from pipelex.cogt.inference.error_classification import UserActionKind
+from pipelex.pipe_run.exceptions import PipeRouterError
+from pipelex.system.pipe_run_mode import PipeRunMode
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT
+
+TRUNCATED_MESSAGE = (
+ "The model 'claude-5-sonnet' was cut off before it finished the text of pipe 'ui_designer' "
+ "(stop reason 'max_tokens', 4096 output tokens used, max_tokens set to 4096), so the text is incomplete. "
+ "Raise the pipe's max_tokens, or shorten its input."
+)
+NO_MAX_TOKENS_NEXT_STEP = "Set the pipe's max_tokens up to the model's limit, lower its reasoning effort, or shorten its input."
+CONTEXT_WINDOW_NEXT_STEP = "Shorten the pipe's input, lower its reasoning effort, or choose a model with a larger context window."
+CONTEXT_WINDOW_MESSAGE = (
+ "The model 'claude-5-sonnet' filled its context window before it finished the text of pipe 'ui_designer' "
+ "(stop reason 'model_context_window_exceeded', 812 output tokens used, max_tokens set to 4096), so the text is incomplete. "
+ f"{CONTEXT_WINDOW_NEXT_STEP}"
+)
+REFUSED_MESSAGE = (
+ "The model 'claude-5-sonnet' declined to finish the text of pipe 'ui_designer', or a content filter stopped it "
+ "(stop reason 'content_filtered'), so the text cannot be used. Revise the pipe's prompt or its input."
+)
+
+
+def _truncated_error() -> LLMCompletionTruncatedError:
+ return LLMCompletionTruncatedError(
+ model_handle="claude-5-sonnet",
+ stop_reason="max_tokens",
+ truncation_limit=CompletionTruncationLimit.MAX_TOKENS,
+ pipe_code="ui_designer",
+ max_tokens=4096,
+ output_tokens=4096,
+ )
+
+
+def _context_window_error() -> LLMCompletionTruncatedError:
+ return LLMCompletionTruncatedError(
+ model_handle="claude-5-sonnet",
+ stop_reason="model_context_window_exceeded",
+ truncation_limit=CompletionTruncationLimit.CONTEXT_WINDOW,
+ pipe_code="ui_designer",
+ max_tokens=4096,
+ output_tokens=812,
+ )
+
+
+def _refused_error() -> LLMCompletionRefusedError:
+ return LLMCompletionRefusedError(model_handle="claude-5-sonnet", stop_reason="content_filtered", pipe_code="ui_designer")
+
+
+class TestLLMCompletionStopErrors:
+ def test_the_truncation_names_the_pipe_model_stop_counts_and_next_step(self) -> None:
+ error = _truncated_error()
+
+ assert error.message == TRUNCATED_MESSAGE
+ assert error.user_action is not None
+ assert error.user_action.kind == UserActionKind.CHANGE_INPUT
+ assert error.user_action.detail == "Raise the pipe's max_tokens, or shorten its input."
+
+ def test_a_cut_with_no_max_tokens_sent_advises_setting_one_rather_than_raising_it(self) -> None:
+ """The provider's default cut the text, so there is no max_tokens of the pipe's to raise: the advice is to set one."""
+ error = LLMCompletionTruncatedError(
+ model_handle="gpt-test",
+ stop_reason="length",
+ truncation_limit=CompletionTruncationLimit.MAX_TOKENS,
+ pipe_code="ui_designer",
+ output_tokens=16384,
+ )
+
+ assert error.message == (
+ "The model 'gpt-test' was cut off before it finished the text of pipe 'ui_designer' "
+ "(stop reason 'length', 16384 output tokens used, no max_tokens sent), so the text is incomplete. "
+ f"{NO_MAX_TOKENS_NEXT_STEP}"
+ )
+ assert error.user_action is not None
+ assert error.user_action.kind == UserActionKind.CHANGE_INPUT
+ assert error.user_action.detail == NO_MAX_TOKENS_NEXT_STEP
+ assert "Raise" not in error.message
+
+ def test_a_context_window_stop_advises_a_shorter_input_never_a_higher_max_tokens(self) -> None:
+ """The input and the output filled the window together, so a higher max_tokens cannot lift it."""
+ error = _context_window_error()
+
+ assert error.message == CONTEXT_WINDOW_MESSAGE
+ assert error.user_action is not None
+ assert error.user_action.kind == UserActionKind.CHANGE_INPUT
+ assert error.user_action.detail == CONTEXT_WINDOW_NEXT_STEP
+ assert "Raise" not in error.message
+
+ def test_the_refusal_names_the_pipe_model_stop_and_next_step(self) -> None:
+ error = _refused_error()
+
+ assert error.message == REFUSED_MESSAGE
+ assert error.user_action is not None
+ assert error.user_action.kind == UserActionKind.CHANGE_INPUT
+ assert error.user_action.detail == "Revise the pipe's prompt or its input."
+
+ def test_without_a_pipe_or_counts_the_messages_still_read_as_sentences(self) -> None:
+ truncated = LLMCompletionTruncatedError(
+ model_handle="gpt-test", stop_reason="length", truncation_limit=CompletionTruncationLimit.MAX_TOKENS, max_tokens=2048
+ )
+ truncated_by_default = LLMCompletionTruncatedError(
+ model_handle="gpt-test", stop_reason="length", truncation_limit=CompletionTruncationLimit.MAX_TOKENS
+ )
+ context_window = LLMCompletionTruncatedError(
+ model_handle="mistral-test", stop_reason="model_length", truncation_limit=CompletionTruncationLimit.CONTEXT_WINDOW
+ )
+ refused = LLMCompletionRefusedError(model_handle="gpt-test", stop_reason="refusal")
+
+ assert truncated.message == (
+ "The model 'gpt-test' was cut off before it finished its text (stop reason 'length', max_tokens set to 2048), "
+ "so the text is incomplete. Raise max_tokens, or shorten the input."
+ )
+ assert truncated_by_default.message == (
+ "The model 'gpt-test' was cut off before it finished its text (stop reason 'length', no max_tokens sent), "
+ "so the text is incomplete. Set max_tokens up to the model's limit, lower the reasoning effort, or shorten the input."
+ )
+ assert context_window.message == (
+ "The model 'mistral-test' filled its context window before it finished its text (stop reason 'model_length'), "
+ "so the text is incomplete. Shorten the input, lower the reasoning effort, or choose a model with a larger context window."
+ )
+ assert refused.message == (
+ "The model 'gpt-test' declined to finish its text, or a content filter stopped it (stop reason 'refusal'), "
+ "so the text cannot be used. Revise the prompt or the input."
+ )
+
+ @pytest.mark.parametrize(
+ ("error_type", "error", "user_action_detail"),
+ [
+ ("LLMCompletionTruncatedError", _truncated_error(), "Raise the pipe's max_tokens, or shorten its input."),
+ ("LLMCompletionTruncatedError", _context_window_error(), CONTEXT_WINDOW_NEXT_STEP),
+ ("LLMCompletionRefusedError", _refused_error(), "Revise the pipe's prompt or its input."),
+ ],
+ )
+ def test_the_report_carries_the_fields_an_agent_branches_on(self, error_type: str, error: LLMCompletionError, user_action_detail: str) -> None:
+ """A content error in the input domain, not retryable, its next step a change of input, its type its own."""
+ assert isinstance(error, LLMCompletionError)
+ report = error.to_error_report()
+
+ assert report.error_type == error_type
+ assert report.error_category == InferenceErrorCategory.CONTENT
+ assert report.error_domain == ErrorDomain.INPUT
+ assert report.retryable is False
+ assert report.model == "claude-5-sonnet"
+ assert report.user_action is not None
+ assert report.user_action.kind == UserActionKind.CHANGE_INPUT
+ assert report.user_action.detail == user_action_detail
+ assert report.http_status == 422
+
+ @pytest.mark.parametrize("error", [_truncated_error(), _context_window_error(), _refused_error()])
+ def test_the_message_survives_strict_disclosure_without_the_partial_text(self, error: LLMCompletionError) -> None:
+ payload = error.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)
+
+ assert payload["message"] == error.message
+ assert payload["user_action"]["kind"] == "change_input"
+ assert payload["error_domain"] == "input"
+ assert payload["retryable"] is False
+ assert STOP_TEST_PARTIAL_TEXT not in payload["message"]
+
+ def test_a_run_failure_located_at_the_pipe_reports_the_truncation(self) -> None:
+ """The pipe router's located report keeps the truncation's identity, classification and caller-facing message."""
+ truncation = _truncated_error()
+ located = PipeRouterError.make_located(
+ failure=truncation, run_mode=PipeRunMode.LIVE, pipe_code="ui_designer", output_name=None, pipe_stack=["flow", "ui_designer"]
+ )
+ located.__cause__ = truncation
+
+ report = located.to_error_report()
+
+ assert report.error_type == "LLMCompletionTruncatedError"
+ assert report.message == f"Pipe 'ui_designer' failed (flow → ui_designer): {TRUNCATED_MESSAGE}"
+ assert report.retryable is False
+ assert report.error_domain == ErrorDomain.INPUT
+ assert report.to_dict(disclosure_mode=DisclosureMode.STRICT)["message"] == report.message
diff --git a/tests/unit/pipelex/providers/anthropic/test_anthropic_structured_clamp.py b/tests/unit/pipelex/providers/anthropic/test_anthropic_structured_clamp.py
new file mode 100644
index 000000000..a5766fcb7
--- /dev/null
+++ b/tests/unit/pipelex/providers/anthropic/test_anthropic_structured_clamp.py
@@ -0,0 +1,139 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import anthropic
+import httpx
+import pytest
+from anthropic.types import ToolUseBlock
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.cogt.llm.structured_output import StructureMethod
+from pipelex.providers.anthropic import anthropic_llm_worker
+from tests.helpers.anthropic_structured_request import make_message, make_structured_worker, sent_request
+from tests.helpers.instructor_test_utils import DummySchema, make_llm_job, wrap_in_instructor_retry
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+ from pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorker
+
+# What a 1200-second structured timeout lets the SDK's heuristic produce: 1200 * 128000 / 3600
+TIMEOUT_SAFE_MAX_TOKENS = 42666
+LOWERED_NOTE = f"The structured output's max_tokens was lowered from 64000 to {TIMEOUT_SAFE_MAX_TOKENS} to fit its 1200-second timeout."
+
+
+MESSAGES_REQUEST = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+
+
+def _status_error(error_class: type[anthropic.APIStatusError], *, status_code: int, message: str) -> anthropic.APIStatusError:
+ return error_class(message, response=httpx.Response(status_code, request=MESSAGES_REQUEST), body=None)
+
+
+# Failures the API or the network answered for its own reason, which no lowered max_tokens explains
+UNRELATED_FAILURES: list[tuple[str, Exception]] = [
+ ("rate limit", _status_error(anthropic.RateLimitError, status_code=429, message="Rate limit exceeded")),
+ ("authentication", _status_error(anthropic.AuthenticationError, status_code=401, message="Invalid API key")),
+ ("overloaded", _status_error(anthropic.InternalServerError, status_code=529, message="Overloaded")),
+ ("server error", _status_error(anthropic.InternalServerError, status_code=500, message="Internal server error")),
+ ("lost connection", anthropic.APIConnectionError(request=MESSAGES_REQUEST)),
+]
+
+
+def _fail_the_structured_call(mocker: MockerFixture, *, worker: AnthropicLLMWorker, failure: Exception, is_wrapped: bool) -> None:
+ """Make the structured call fail with the failure given, wrapped as instructor wraps one after an attempt, or raised as is."""
+ instructor_client = mocker.MagicMock()
+ side_effect = wrap_in_instructor_retry(failure) if is_wrapped else failure
+ instructor_client.chat.completions.create_with_completion = mocker.AsyncMock(side_effect=side_effect)
+ worker.instructor_for_objects = instructor_client
+
+
[email protected](loop_scope="class")
+class TestAnthropicStructuredClamp:
+ async def test_a_lowered_caller_max_tokens_is_logged_as_a_warning(self, mocker: MockerFixture) -> None:
+ worker, create = make_structured_worker(mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS)
+ llm_job = make_llm_job(mocker)
+ llm_job.job_params.max_tokens = 64000
+ warning = mocker.patch.object(anthropic_llm_worker.log, "warning")
+
+ result = await worker._gen_object(llm_job=llm_job, schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert result.text == "answer"
+ assert sent_request(create)["max_tokens"] == TIMEOUT_SAFE_MAX_TOKENS
+ warning.assert_called_once_with(
+ "A structured output's token limit was lowered to fit its timeout",
+ fields={
+ "model_handle": "claude-test",
+ "requested_max_tokens": 64000,
+ "effective_max_tokens": TIMEOUT_SAFE_MAX_TOKENS,
+ "timeout_seconds": 1200,
+ },
+ )
+
+ async def test_a_lowered_model_default_is_logged_at_debug_level(self, mocker: MockerFixture) -> None:
+ """A model default above the timeout's limit is lowered on every structured call, so it does not warn."""
+ worker, create = make_structured_worker(mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS, default_max_tokens=64000)
+ warning = mocker.patch.object(anthropic_llm_worker.log, "warning")
+ debug = mocker.patch.object(anthropic_llm_worker.log, "debug")
+
+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert sent_request(create)["max_tokens"] == TIMEOUT_SAFE_MAX_TOKENS
+ warning.assert_not_called()
+ lowered_calls = [
+ call
+ for call in debug.call_args_list
+ if call.args == ("The model's default token limit was lowered to fit the structured output's timeout",)
+ ]
+ assert len(lowered_calls) == 1
+
+ @pytest.mark.parametrize("is_wrapped", [True, False])
+ @pytest.mark.parametrize(("_topic", "failure"), UNRELATED_FAILURES)
+ async def test_an_error_the_lowered_limit_cannot_explain_carries_no_note(
+ self, mocker: MockerFixture, _topic: str, failure: Exception, is_wrapped: bool
+ ) -> None:
+ """A rate limit, an authentication failure, an overload, a server error or a lost connection never points at max_tokens."""
+ worker, _ = make_structured_worker(mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS, default_max_tokens=64000)
+ _fail_the_structured_call(mocker, worker=worker, failure=failure, is_wrapped=is_wrapped)
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert "was lowered" not in exc_info.value.message
+
+ @pytest.mark.parametrize("is_wrapped", [True, False])
+ async def test_a_timeout_of_the_lowered_call_says_the_limit_was_lowered(self, mocker: MockerFixture, is_wrapped: bool) -> None:
+ """The note is added to the error the call raises, whose class and retryable category stay as they were."""
+ worker, _ = make_structured_worker(mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS, default_max_tokens=64000)
+ _fail_the_structured_call(mocker, worker=worker, failure=anthropic.APITimeoutError(request=MESSAGES_REQUEST), is_wrapped=is_wrapped)
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.message.endswith(LOWERED_NOTE)
+ assert str(error) == error.message
+ assert error.error_category == InferenceErrorCategory.TRANSIENT
+ assert error.to_error_report().retryable is True
+
+ async def test_an_invalid_output_of_the_lowered_call_says_the_limit_was_lowered(self, mocker: MockerFixture) -> None:
+ """An output that still fails validation once instructor's re-asks are spent may have been cut by the lowered limit."""
+ invalid_output = make_message(ToolUseBlock(type="tool_use", id="toolu_test", name="DummySchema", input={"unexpected": "field"}))
+ worker, create = make_structured_worker(
+ mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS, default_max_tokens=64000, responses=[invalid_output]
+ )
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert sent_request(create)["max_tokens"] == TIMEOUT_SAFE_MAX_TOKENS
+ assert exc_info.value.message.endswith(LOWERED_NOTE)
+
+ async def test_an_error_of_a_call_sent_as_given_carries_no_note(self, mocker: MockerFixture) -> None:
+ worker, _ = make_structured_worker(mocker, structure_method=StructureMethod.INSTRUCTOR_ANTHROPIC_TOOLS)
+ _fail_the_structured_call(mocker, worker=worker, failure=anthropic.APITimeoutError(request=MESSAGES_REQUEST), is_wrapped=True)
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert "was lowered" not in exc_info.value.message
diff --git a/tests/unit/pipelex/providers/anthropic/test_anthropic_text_stop.py b/tests/unit/pipelex/providers/anthropic/test_anthropic_text_stop.py
new file mode 100644
index 000000000..815ab0b86
--- /dev/null
+++ b/tests/unit/pipelex/providers/anthropic/test_anthropic_text_stop.py
@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from anthropic.types import ContentBlock, Message, TextBlock, ThinkingBlock, Usage
+
+from pipelex.cogt.exceptions import LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.providers.anthropic.anthropic_config import AnthropicConfig
+from pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorker
+from tests.helpers.anthropic_structured_request import ANTHROPIC_LEVEL_MAP, ANTHROPIC_MIN_THINKING_BUDGET, anthropic_budget
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+
+if TYPE_CHECKING:
+ from anthropic.types import StopReason
+ from pytest_mock import MockerFixture
+
+
+def _final_message(*, stop_reason: StopReason, content: list[ContentBlock]) -> Message:
+ return Message(
+ id="msg_test",
+ type="message",
+ role="assistant",
+ model="claude-test",
+ content=content,
+ stop_reason=stop_reason,
+ stop_sequence=None,
+ usage=Usage(input_tokens=120, output_tokens=4096),
+ )
+
+
+def _text_worker(mocker: MockerFixture, *, final_message: Message) -> AnthropicLLMWorker:
+ """A worker whose streaming text call hands back the final message given."""
+ stream = mocker.MagicMock()
+ stream.get_final_message = mocker.AsyncMock(return_value=final_message)
+ stream_context = mocker.MagicMock()
+ stream_context.__aenter__ = mocker.AsyncMock(return_value=stream)
+ stream_context.__aexit__ = mocker.AsyncMock(return_value=False)
+ sdk_client = mocker.MagicMock()
+ sdk_client.messages.stream = mocker.MagicMock(return_value=stream_context)
+
+ worker = object.__new__(AnthropicLLMWorker)
+ worker.extras_factory = None
+ model = mocker.MagicMock()
+ model.desc = "test-model-desc"
+ model.model_id = "claude-test"
+ model.name = "claude-test"
+ model.thinking_mode = ThinkingMode.NONE
+ model.accepts_temperature = True
+ model.min_thinking_budget = ANTHROPIC_MIN_THINKING_BUDGET
+ model.max_thinking_budget = None
+ worker.inference_model = model
+ worker.default_max_tokens = 4096
+ worker.anthropic_async_client = sdk_client
+
+ config = mocker.MagicMock()
+ config.inference.llm.anthropic = AnthropicConfig(structured_output_timeout_seconds=1200, effort_to_level_map=ANTHROPIC_LEVEL_MAP)
+ config.inference.llm.get_reasoning_budget = mocker.MagicMock(side_effect=anthropic_budget)
+ mocker.patch("pipelex.providers.anthropic.anthropic_llm_worker.get_config", return_value=config)
+ return worker
+
+
[email protected](loop_scope="class")
+class TestAnthropicTextStop:
+ @pytest.mark.parametrize(
+ "content",
+ [
+ [TextBlock(type="text", text=STOP_TEST_PARTIAL_TEXT)],
+ [],
+ [ThinkingBlock(type="thinking", thinking="Weighing the clauses.", signature="sig")],
+ ],
+ )
+ async def test_a_text_cut_at_max_tokens_raises_the_truncation(self, mocker: MockerFixture, content: list[ContentBlock]) -> None:
+ """Partial text, no text, and only thinking with no text all stop on max_tokens and raise the truncation."""
+ worker = _text_worker(mocker, final_message=_final_message(stop_reason="max_tokens", content=content))
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "max_tokens"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 4096
+ assert error.output_tokens == 4096
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_a_refusal_raises_the_refusal(self, mocker: MockerFixture) -> None:
+ final_message = _final_message(stop_reason="refusal", content=[TextBlock(type="text", text=STOP_TEST_PARTIAL_TEXT)])
+ worker = _text_worker(mocker, final_message=final_message)
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == "refusal"
+
+ async def test_a_normal_stop_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ final_message = _final_message(stop_reason="end_turn", content=[TextBlock(type="text", text=STOP_TEST_PARTIAL_TEXT)])
+ worker = _text_worker(mocker, final_message=final_message)
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT
diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
index e3f619564..ee0b5c5d0 100644
--- a/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_boto3_current_span.py
@@ -46,9 +46,9 @@ class TestBedrockBoto3CurrentSpan:
llm_span = tracer.start_span("llm call")
with tracer.start_as_current_span("host") as host_span, pipelex_span_active(span=llm_span):
- text, _ = await client.chat(messages=[], system_text=None, model="some-model", temperature=0.5)
+ chat_result = await client.chat(messages=[], system_text=None, model="some-model", temperature=0.5)
- assert text == "answer"
+ assert chat_result.text == "answer"
assert recording.seen_log_span_ids == [llm_span.get_span_context().span_id]
assert recording.seen_log_current_span_ids == [host_span.get_span_context().span_id]
assert recording.seen_current_spans == [host_span]
diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_converse_request.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_converse_request.py
index 686b4ec23..db152a76b 100644
--- a/tests/unit/pipelex/providers/bedrock/test_bedrock_converse_request.py
+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_converse_request.py
@@ -12,7 +12,7 @@ from typing import TYPE_CHECKING, Any
import pytest
from pipelex.providers.bedrock.bedrock_client_boto3 import BedrockClientBoto3
-from pipelex.providers.bedrock.bedrock_client_protocol import make_converse_params
+from pipelex.providers.bedrock.bedrock_client_protocol import BedrockChatResult, make_converse_params
from pipelex.providers.bedrock.bedrock_llm_worker import BedrockLLMWorker
if TYPE_CHECKING:
@@ -43,7 +43,7 @@ def _make_worker(mocker: MockerFixture, *, accepts_temperature: bool) -> tuple[B
model.accepts_temperature = accepts_temperature
worker.inference_model = model
worker.default_max_tokens = 4096
- chat = mocker.AsyncMock(return_value=("answer", {}))
+ chat = mocker.AsyncMock(return_value=BedrockChatResult(text="answer", nb_tokens_by_category={}, stop_reason="end_turn"))
client = mocker.MagicMock()
client.chat = chat
worker.bedrock_client_for_text = client
@@ -88,9 +88,9 @@ class TestBedrockConverseRequest:
recording = _RecordingConverse()
client.boto3_client = recording
- text, _ = await client.chat(messages=_MESSAGES, system_text=None, model="mistral.test", temperature=None, max_tokens=100)
+ chat_result = await client.chat(messages=_MESSAGES, system_text=None, model="mistral.test", temperature=None, max_tokens=100)
- assert text == "answer"
+ assert chat_result.text == "answer"
assert recording.params == [
make_converse_params(messages=_MESSAGES, system_text=None, model="mistral.test", temperature=None, max_tokens=100)
]
diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_stop.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_stop.py
new file mode 100644
index 000000000..1180e3632
--- /dev/null
+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_stop.py
@@ -0,0 +1,95 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.providers.bedrock.bedrock_client_boto3 import BedrockClientBoto3
+from pipelex.providers.bedrock.bedrock_llm_worker import BedrockLLMWorker
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+class _AnsweringConverse:
+ """A boto3 Bedrock runtime whose Converse call answers with the response given."""
+
+ def __init__(self, *, response: dict[str, Any]) -> None:
+ self.response = response
+
+ def converse(self, **params: Any) -> dict[str, Any]: # ruff: ignore[unused-method-argument]
+ return self.response
+
+
+def _converse_response(*, stop_reason: str, text: str) -> dict[str, Any]:
+ content: list[dict[str, Any]] = [{"text": text}] if text else []
+ return {
+ "usage": {"inputTokens": 30, "outputTokens": 2048},
+ "output": {"message": {"role": "assistant", "content": content}},
+ "stopReason": stop_reason,
+ }
+
+
+def _worker(mocker: MockerFixture, *, stop_reason: str, text: str) -> BedrockLLMWorker:
+ """A worker over the real boto3 client, so the stop reason travels the client's own reading of the Converse answer."""
+ client = BedrockClientBoto3(aws_region="us-east-1")
+ client.boto3_client = _AnsweringConverse(response=_converse_response(stop_reason=stop_reason, text=text))
+ worker = object.__new__(BedrockLLMWorker)
+ model = mocker.MagicMock()
+ model.desc = "test-model-desc"
+ model.model_id = "mistral.test"
+ model.name = "mistral-bedrock"
+ model.accepts_temperature = True
+ worker.inference_model = model
+ worker.default_max_tokens = 2048
+ worker.bedrock_client_for_text = client
+ return worker
+
+
[email protected](loop_scope="class")
+class TestBedrockWorkerStop:
+ @pytest.mark.parametrize("text", [STOP_TEST_PARTIAL_TEXT, ""])
+ async def test_a_text_cut_at_max_tokens_raises_the_truncation(self, mocker: MockerFixture, text: str) -> None:
+ worker = _worker(mocker, stop_reason="max_tokens", text=text)
+ llm_job = make_text_llm_job()
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=llm_job) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "max_tokens"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 2048
+ assert error.output_tokens == 2048
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+ assert llm_job.job_report.llm_tokens_usage is not None
+ assert llm_job.job_report.llm_tokens_usage.nb_tokens_by_category == {TokenCategory.INPUT: 30, TokenCategory.OUTPUT: 2048}
+
+ @pytest.mark.parametrize("stop_reason", ["content_filtered", "guardrail_intervened"])
+ async def test_a_filtered_text_raises_the_refusal(self, mocker: MockerFixture, stop_reason: str) -> None:
+ worker = _worker(mocker, stop_reason=stop_reason, text="")
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == stop_reason
+
+ async def test_a_normal_stop_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ worker = _worker(mocker, stop_reason="end_turn", text=STOP_TEST_PARTIAL_TEXT)
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT
+
+ async def test_a_normal_stop_with_no_text_is_an_empty_text_error(self, mocker: MockerFixture) -> None:
+ """An answer with no text block no longer crashes the client: the worker refuses the empty text as other workers do."""
+ worker = _worker(mocker, stop_reason="end_turn", text="")
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert type(exc_info.value) is LLMCompletionError
+ assert exc_info.value.error_category == InferenceErrorCategory.CONTENT
diff --git a/tests/unit/pipelex/providers/google/test_google_llm_worker_text_stop.py b/tests/unit/pipelex/providers/google/test_google_llm_worker_text_stop.py
new file mode 100644
index 000000000..e9ad452ff
--- /dev/null
+++ b/tests/unit/pipelex/providers/google/test_google_llm_worker_text_stop.py
@@ -0,0 +1,107 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from google import genai
+from google.genai import types as genai_types
+
+from pipelex.cogt.exceptions import LLMCompletionError, LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.providers.google.google_config import GoogleConfig
+from pipelex.providers.google.google_llm_worker import GoogleLLMWorker
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+from tests.helpers.google_structured_request import GOOGLE_LEVEL_MAP
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+def _response(*, finish_reason: genai_types.FinishReason, text: str) -> genai_types.GenerateContentResponse:
+ content = genai_types.Content(role="model", parts=[genai_types.Part(text=text)]) if text else None
+ return genai_types.GenerateContentResponse(
+ candidates=[genai_types.Candidate(content=content, finish_reason=finish_reason)],
+ usage_metadata=genai_types.GenerateContentResponseUsageMetadata(prompt_token_count=80, candidates_token_count=1024),
+ )
+
+
+def _blocked_response(*, block_reason: genai_types.BlockedReason | None) -> genai_types.GenerateContentResponse:
+ """An answer with no candidate, as Gemini gives a blocked prompt, its block reason in the prompt feedback when given."""
+ return genai_types.GenerateContentResponse(
+ candidates=None,
+ prompt_feedback=genai_types.GenerateContentResponsePromptFeedback(block_reason=block_reason) if block_reason else None,
+ usage_metadata=genai_types.GenerateContentResponseUsageMetadata(prompt_token_count=80),
+ )
+
+
+def _text_worker(mocker: MockerFixture, *, response: genai_types.GenerateContentResponse) -> GoogleLLMWorker:
+ """A worker whose text call answers with the response given."""
+ client = genai.Client(api_key="test-key")
+ mocker.patch.object(client.aio.models, "generate_content", new_callable=mocker.AsyncMock, return_value=response)
+ worker = object.__new__(GoogleLLMWorker)
+ model = mocker.MagicMock()
+ model.desc = "test-model-desc"
+ model.model_id = "gemini-pro"
+ model.name = "gemini-pro"
+ model.thinking_mode = None
+ model.accepts_temperature = True
+ worker.inference_model = model
+ worker.genai_async_client = client.aio
+
+ config = mocker.MagicMock()
+ config.inference.llm.google = GoogleConfig(effort_to_level_map=GOOGLE_LEVEL_MAP)
+ mocker.patch("pipelex.providers.google.google_llm_worker.get_config", return_value=config)
+ return worker
+
+
[email protected](loop_scope="class")
+class TestGoogleLLMWorkerTextStop:
+ @pytest.mark.parametrize("text", [STOP_TEST_PARTIAL_TEXT, ""])
+ async def test_a_candidate_cut_at_max_tokens_raises_the_truncation(self, mocker: MockerFixture, text: str) -> None:
+ worker = _text_worker(mocker, response=_response(finish_reason=genai_types.FinishReason.MAX_TOKENS, text=text))
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job(max_tokens=1024)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "MAX_TOKENS"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 1024
+ assert error.output_tokens == 1024
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_a_safety_stop_after_partial_text_raises_the_refusal(self, mocker: MockerFixture) -> None:
+ worker = _text_worker(mocker, response=_response(finish_reason=genai_types.FinishReason.SAFETY, text=STOP_TEST_PARTIAL_TEXT))
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == "SAFETY"
+
+ @pytest.mark.parametrize("block_reason", [genai_types.BlockedReason.SAFETY, genai_types.BlockedReason.PROHIBITED_CONTENT])
+ async def test_a_blocked_prompt_raises_the_refusal_naming_its_reason(
+ self, mocker: MockerFixture, block_reason: genai_types.BlockedReason
+ ) -> None:
+ worker = _text_worker(mocker, response=_blocked_response(block_reason=block_reason))
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == block_reason.value
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert f"stop reason '{block_reason.value}'" in error.message
+
+ async def test_no_candidate_and_no_block_reason_keeps_the_generic_error(self, mocker: MockerFixture) -> None:
+ worker = _text_worker(mocker, response=_blocked_response(block_reason=None))
+
+ with pytest.raises(LLMCompletionError, match="No candidates returned") as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert not isinstance(exc_info.value, LLMCompletionRefusedError)
+
+ async def test_a_normal_stop_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ worker = _text_worker(mocker, response=_response(finish_reason=genai_types.FinishReason.STOP, text=STOP_TEST_PARTIAL_TEXT))
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT
diff --git a/tests/unit/pipelex/providers/mistral/test_mistral_text_stop.py b/tests/unit/pipelex/providers/mistral/test_mistral_text_stop.py
new file mode 100644
index 000000000..dc2958884
--- /dev/null
+++ b/tests/unit/pipelex/providers/mistral/test_mistral_text_stop.py
@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from mistralai.client.models import AssistantMessage, ChatCompletionChoice, ChatCompletionResponse, UsageInfo
+from mistralai.client.types import UnrecognizedStr
+
+from pipelex.cogt.exceptions import CompletionTruncationLimit, LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+from tests.helpers.mistral_request import make_worker
+
+if TYPE_CHECKING:
+ from mistralai.client.models.chatcompletionchoice import ChatCompletionChoiceFinishReason
+ from pytest_mock import MockerFixture
+
+
+def _response(*, finish_reason: ChatCompletionChoiceFinishReason, text: str) -> ChatCompletionResponse:
+ return ChatCompletionResponse(
+ id="cmpl_test",
+ object="chat.completion",
+ model="mistral-test",
+ created=0,
+ usage=UsageInfo(prompt_tokens=20, completion_tokens=4096, total_tokens=4116),
+ choices=[ChatCompletionChoice(index=0, finish_reason=finish_reason, message=AssistantMessage(content=text))],
+ )
+
+
[email protected](loop_scope="class")
+class TestMistralTextStop:
+ @pytest.mark.parametrize(
+ ("finish_reason", "text", "expected_limit"),
+ [
+ ("length", STOP_TEST_PARTIAL_TEXT, CompletionTruncationLimit.MAX_TOKENS),
+ ("length", "", CompletionTruncationLimit.MAX_TOKENS),
+ ("model_length", STOP_TEST_PARTIAL_TEXT, CompletionTruncationLimit.CONTEXT_WINDOW),
+ ],
+ )
+ async def test_a_text_cut_at_its_length_raises_the_truncation(
+ self, mocker: MockerFixture, finish_reason: ChatCompletionChoiceFinishReason, text: str, expected_limit: CompletionTruncationLimit
+ ) -> None:
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_response(finish_reason=finish_reason, text=text))
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == finish_reason
+ assert error.truncation_limit == expected_limit
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 4096
+ assert error.output_tokens == 4096
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_a_filter_value_raises_the_refusal(self, mocker: MockerFixture) -> None:
+ """Mistral's own vocabulary has no refusal value, but its SDK takes any string, and the classifier reads them all."""
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_response(finish_reason=UnrecognizedStr("content_filter"), text=""))
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == "content_filter"
+
+ async def test_a_normal_stop_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_response(finish_reason="stop", text=STOP_TEST_PARTIAL_TEXT))
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT
diff --git a/tests/unit/pipelex/providers/openai/test_data.py b/tests/unit/pipelex/providers/openai/test_data.py
index 2579f9768..a5bf915f2 100644
--- a/tests/unit/pipelex/providers/openai/test_data.py
+++ b/tests/unit/pipelex/providers/openai/test_data.py
@@ -60,3 +60,23 @@ class OpenAIErrorHandlingTestData:
"safety filters",
),
]
+
+
+class GatewayStopTestData:
+ """Stop values a gateway in non-strict mode passes through the chat-completions shape in each provider's vocabulary.
+
+ Each tuple: (topic, finish_reason, content); the content is partial or empty, as the gateway sets it.
+ """
+
+ TRUNCATION_CASES: ClassVar[list[tuple[str, str, str]]] = [
+ ("claude cut at max_tokens", "max_tokens", "The contract binds the parties to"),
+ ("claude cut with nothing written", "max_tokens", ""),
+ ("gemini cut at max_tokens", "MAX_TOKENS", "The contract binds the parties to"),
+ ("strict mode cut", "length", "The contract binds the parties to"),
+ ]
+
+ REFUSAL_CASES: ClassVar[list[tuple[str, str, str]]] = [
+ ("bedrock content filter", "content_filtered", ""),
+ ("claude refusal", "refusal", ""),
+ ("gemini safety stop after partial text", "SAFETY", "The contract binds the parties to"),
+ ]
diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_gateway_stop.py b/tests/unit/pipelex/providers/openai/test_openai_completions_gateway_stop.py
new file mode 100644
index 000000000..32ec2a5f2
--- /dev/null
+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_gateway_stop.py
@@ -0,0 +1,102 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import pytest
+from openai import AsyncOpenAI
+
+from pipelex.cogt.exceptions import LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.providers.openai.openai_completions_factory import OpenAICompletionsFactory
+from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
+from tests.helpers.completion_stop import STOP_TEST_PIPE_CODE, make_text_llm_job
+from tests.unit.pipelex.providers.openai.test_data import GatewayStopTestData
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+GATEWAY_MODEL_HANDLE = "claude-5-sonnet"
+
+
+def _gateway_body(*, finish_reason: str, content: str) -> dict[str, Any]:
+ return {
+ "id": "chatcmpl_gateway",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "claude-5-sonnet",
+ "choices": [{"index": 0, "finish_reason": finish_reason, "message": {"role": "assistant", "content": content}}],
+ "usage": {"prompt_tokens": 5758, "completion_tokens": 4096, "total_tokens": 9854},
+ }
+
+
+def _gateway_worker(mocker: MockerFixture, *, finish_reason: str, content: str) -> OpenAICompletionsLLMWorker:
+ """A completions worker over a real OpenAI SDK client, whose transport answers as a gateway in non-strict mode does.
+
+ The SDK parses the body itself, so the finish reason reaches the worker exactly as the SDK hands it over,
+ a value outside OpenAI's own vocabulary included.
+ """
+ body = _gateway_body(finish_reason=finish_reason, content=content)
+
+ def answer(request: httpx.Request) -> httpx.Response:
+ return httpx.Response(200, json=body, request=request)
+
+ sdk_client = AsyncOpenAI(
+ api_key="test-key",
+ base_url="https://gateway.test/v1",
+ max_retries=0,
+ http_client=httpx.AsyncClient(transport=httpx.MockTransport(answer)),
+ )
+ worker = object.__new__(OpenAICompletionsLLMWorker)
+ model = mocker.MagicMock()
+ model.desc = "gateway-model-desc"
+ model.model_id = "claude-5-sonnet"
+ model.name = GATEWAY_MODEL_HANDLE
+ model.thinking_mode = ThinkingMode.NONE
+ model.accepts_temperature = True
+ model.extra_headers = None
+ worker.inference_model = model
+ worker.openai_completions_factory = OpenAICompletionsFactory(is_http_url_enabled=False)
+ worker.openai_client_for_text = sdk_client
+ return worker
+
+
[email protected](loop_scope="class")
+class TestOpenAICompletionsGatewayStop:
+ @pytest.mark.parametrize(("_topic", "finish_reason", "content"), GatewayStopTestData.TRUNCATION_CASES)
+ async def test_a_passed_through_truncation_raises_the_truncation(
+ self, mocker: MockerFixture, _topic: str, finish_reason: str, content: str
+ ) -> None:
+ worker = _gateway_worker(mocker, finish_reason=finish_reason, content=content)
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job(max_tokens=4096)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == finish_reason
+ assert error.model_handle == GATEWAY_MODEL_HANDLE
+ assert error.output_tokens == 4096
+ assert error.message == (
+ f"The model '{GATEWAY_MODEL_HANDLE}' was cut off before it finished the text of pipe '{STOP_TEST_PIPE_CODE}' "
+ f"(stop reason '{finish_reason}', 4096 output tokens used, max_tokens set to 4096), so the text is incomplete. "
+ "Raise the pipe's max_tokens, or shorten its input."
+ )
+
+ @pytest.mark.parametrize(("_topic", "finish_reason", "content"), GatewayStopTestData.REFUSAL_CASES)
+ async def test_a_passed_through_refusal_or_filter_raises_the_refusal(
+ self, mocker: MockerFixture, _topic: str, finish_reason: str, content: str
+ ) -> None:
+ worker = _gateway_worker(mocker, finish_reason=finish_reason, content=content)
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == finish_reason
+ assert exc_info.value.pipe_code == STOP_TEST_PIPE_CODE
+
+ async def test_a_passed_through_normal_stop_returns_the_text(self, mocker: MockerFixture) -> None:
+ worker = _gateway_worker(mocker, finish_reason="end_turn", content="The whole summary.")
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == "The whole summary."
diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_retries_kept.py b/tests/unit/pipelex/providers/openai/test_openai_completions_retries_kept.py
new file mode 100644
index 000000000..e3f9c71c2
--- /dev/null
+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_retries_kept.py
@@ -0,0 +1,79 @@
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING
+
+import httpx
+import openai
+import pytest
+from openai.types.chat import ChatCompletion, ChatCompletionMessage, ChatCompletionMessageToolCall
+from openai.types.chat.chat_completion import Choice
+from openai.types.chat.chat_completion_message_tool_call import Function
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from tests.helpers.completion_stop import make_text_llm_job
+from tests.helpers.instructor_test_utils import DummySchema
+from tests.helpers.openai_completions_request import make_worker, text_completion, tool_call_completion
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+def _invalid_tool_call_completion() -> ChatCompletion:
+ """A tool call whose arguments miss the schema's required field, so validation fails and instructor re-asks."""
+ tool_call = ChatCompletionMessageToolCall(
+ id="call_invalid",
+ type="function",
+ function=Function(name="DummySchema", arguments=json.dumps({"wrong_field": "answer"})),
+ )
+ message = ChatCompletionMessage(role="assistant", content=None, tool_calls=[tool_call])
+ return ChatCompletion(
+ id="chatcmpl_invalid",
+ object="chat.completion",
+ created=0,
+ model="test-model",
+ choices=[Choice(index=0, finish_reason="tool_calls", message=message)],
+ )
+
+
+def _rate_limit_error() -> openai.RateLimitError:
+ request = httpx.Request("POST", "https://api.openai.com/v1/chat/completions")
+ return openai.RateLimitError("Rate limit exceeded", response=httpx.Response(429, request=request), body=None)
+
+
[email protected](loop_scope="class")
+class TestOpenAICompletionsRetriesKept:
+ async def test_a_structured_output_failing_validation_is_retried_as_configured(self, mocker: MockerFixture) -> None:
+ """The stop check reads text generation only: a structured attempt that fails validation is re-asked, up to the configured count."""
+ worker, create = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=tool_call_completion())
+ create.side_effect = [_invalid_tool_call_completion(), tool_call_completion()]
+ llm_job = make_text_llm_job()
+ llm_job.job_config.schema_reask_max_attempts = 2
+
+ result = await worker._gen_object(llm_job=llm_job, schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert result.text == "answer"
+ assert create.await_count == 2
+
+ async def test_the_configured_count_bounds_the_retries(self, mocker: MockerFixture) -> None:
+ worker, create = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=tool_call_completion())
+ create.side_effect = [_invalid_tool_call_completion() for _ in range(5)]
+ llm_job = make_text_llm_job()
+ llm_job.job_config.schema_reask_max_attempts = 3
+
+ with pytest.raises(LLMCompletionError):
+ await worker._gen_object(llm_job=llm_job, schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert create.await_count == 3
+
+ async def test_a_rate_limited_text_call_stays_retryable(self, mocker: MockerFixture) -> None:
+ """A provider failure on the text path is classified as before, its retryable category untouched by the stop check."""
+ worker, create = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=text_completion())
+ create.side_effect = _rate_limit_error()
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category == InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.to_error_report().retryable is True
diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_stop.py b/tests/unit/pipelex/providers/openai/test_openai_completions_stop.py
new file mode 100644
index 000000000..e300cd2dc
--- /dev/null
+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_stop.py
@@ -0,0 +1,62 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from openai.types import CompletionUsage
+from openai.types.chat import ChatCompletion, ChatCompletionMessage
+from openai.types.chat.chat_completion import Choice
+
+from pipelex.cogt.exceptions import LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+from tests.helpers.openai_completions_request import make_worker
+
+if TYPE_CHECKING:
+ from typing import Literal
+
+ from pytest_mock import MockerFixture
+
+
+def _completion(*, finish_reason: Literal["stop", "length", "tool_calls", "content_filter", "function_call"], content: str) -> ChatCompletion:
+ return ChatCompletion(
+ id="chatcmpl_test",
+ object="chat.completion",
+ created=0,
+ model="test-model",
+ choices=[Choice(index=0, finish_reason=finish_reason, message=ChatCompletionMessage(role="assistant", content=content))],
+ usage=CompletionUsage(prompt_tokens=50, completion_tokens=256, total_tokens=306),
+ )
+
+
[email protected](loop_scope="class")
+class TestOpenAICompletionsStop:
+ @pytest.mark.parametrize("content", [STOP_TEST_PARTIAL_TEXT, ""])
+ async def test_a_text_cut_at_its_length_raises_the_truncation(self, mocker: MockerFixture, content: str) -> None:
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_completion(finish_reason="length", content=content))
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job(max_tokens=256)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "length"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 256
+ assert error.output_tokens == 256
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_a_filtered_text_raises_the_refusal(self, mocker: MockerFixture) -> None:
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_completion(finish_reason="content_filter", content=""))
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == "content_filter"
+ assert exc_info.value.pipe_code == STOP_TEST_PIPE_CODE
+
+ async def test_a_normal_stop_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=_completion(finish_reason="stop", content=STOP_TEST_PARTIAL_TEXT))
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT
diff --git a/tests/unit/pipelex/providers/openai/test_openai_responses_stop.py b/tests/unit/pipelex/providers/openai/test_openai_responses_stop.py
new file mode 100644
index 000000000..1724309f7
--- /dev/null
+++ b/tests/unit/pipelex/providers/openai/test_openai_responses_stop.py
@@ -0,0 +1,106 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+from openai.types.responses import Response, ResponseOutputMessage, ResponseOutputText, ResponseUsage
+from openai.types.responses.response import IncompleteDetails
+from openai.types.responses.response_usage import InputTokensDetails, OutputTokensDetails
+
+from pipelex.cogt.exceptions import LLMCompletionRefusedError, LLMCompletionTruncatedError
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from tests.helpers.completion_stop import STOP_TEST_PARTIAL_TEXT, STOP_TEST_PIPE_CODE, make_text_llm_job
+from tests.helpers.openai_responses_request import make_worker
+
+if TYPE_CHECKING:
+ from typing import Literal
+
+ from openai.types.responses import ResponseOutputItem
+ from pytest_mock import MockerFixture
+
+
+def _response(
+ *,
+ status: Literal["completed", "incomplete"],
+ incomplete_reason: Literal["max_output_tokens", "content_filter"] | None,
+ text: str,
+) -> Response:
+ output: list[ResponseOutputItem] = []
+ if text:
+ output.append(
+ ResponseOutputMessage(
+ id="msg_test",
+ type="message",
+ role="assistant",
+ status=status,
+ content=[ResponseOutputText(type="output_text", text=text, annotations=[])],
+ )
+ )
+ return Response(
+ id="resp_test",
+ created_at=0,
+ model="gpt-test",
+ object="response",
+ output=output,
+ parallel_tool_calls=False,
+ tool_choice="auto",
+ tools=[],
+ status=status,
+ incomplete_details=IncompleteDetails(reason=incomplete_reason) if incomplete_reason else None,
+ usage=ResponseUsage(
+ input_tokens=40,
+ input_tokens_details=InputTokensDetails(cached_tokens=0),
+ output_tokens=512,
+ output_tokens_details=OutputTokensDetails(reasoning_tokens=0),
+ total_tokens=552,
+ ),
+ )
+
+
[email protected](loop_scope="class")
+class TestOpenAIResponsesStop:
+ @pytest.mark.parametrize("text", [STOP_TEST_PARTIAL_TEXT, ""])
+ async def test_an_answer_cut_at_its_output_limit_raises_the_truncation(self, mocker: MockerFixture, text: str) -> None:
+ response = _response(status="incomplete", incomplete_reason="max_output_tokens", text=text)
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=response)
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job(max_tokens=512)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "max_output_tokens"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.max_tokens == 512
+ assert error.output_tokens == 512
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_an_incomplete_answer_with_no_details_raises_the_truncation(self, mocker: MockerFixture) -> None:
+ """The incomplete status alone says the partial text is unfinished, whether or not its details give a reason."""
+ response = _response(status="incomplete", incomplete_reason=None, text=STOP_TEST_PARTIAL_TEXT)
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=response)
+
+ with pytest.raises(LLMCompletionTruncatedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job(max_tokens=512)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ error = exc_info.value
+ assert error.stop_reason == "incomplete"
+ assert error.pipe_code == STOP_TEST_PIPE_CODE
+ assert error.output_tokens == 512
+ assert STOP_TEST_PARTIAL_TEXT not in error.message
+
+ async def test_a_filtered_answer_raises_the_refusal(self, mocker: MockerFixture) -> None:
+ response = _response(status="incomplete", incomplete_reason="content_filter", text=STOP_TEST_PARTIAL_TEXT)
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=response)
+
+ with pytest.raises(LLMCompletionRefusedError) as exc_info:
+ await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.stop_reason == "content_filter"
+
+ async def test_a_completed_answer_returns_the_text_unchanged(self, mocker: MockerFixture) -> None:
+ response = _response(status="completed", incomplete_reason=None, text=STOP_TEST_PARTIAL_TEXT)
+ worker, _ = make_worker(mocker, thinking_mode=ThinkingMode.NONE, response=response)
+
+ text = await worker._gen_text(llm_job=make_text_llm_job()) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ assert text == STOP_TEST_PARTIAL_TEXT