Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Provider-quota-misread · L-261015-3d5225 (#1496)
Louis Choquelabf6e617d2ca
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6bf840a56..2cbc94cbf 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,19 @@# Changelog+## [Unreleased]++### Fixed++- **A direct provider's spent quota is read from its structure, not its wording**: a call with your own keys to OpenAI or a backend on the OpenAI SDK, Anthropic, Google Gemini, Mistral or Amazon Bedrock is a spent quota only on the status, error code or error detail that provider documents for one, and is otherwise retried as `provider.rate_limited`. A 429 whose message mentioned a quota, as Gemini's per-minute throttles do, used to fail at once as `provider.quota_exhausted`. Anthropic's credit-balance and self-set usage-limit refusals, which carry no code, are still read, by their opening words.+- **Throttles sent on another status than 429 are retried**: OpenRouter's 402 for its in-flight budget, Groq's 498 for its flex tier at capacity, and Bedrock's `ServiceQuotaExceededException`, from Bedrock's SDK or from Claude on Bedrock through the Anthropic SDK, are now `provider.rate_limited` where they failed at once.+- **Gemini's retry delay is carried**: the delay Gemini gives in its error's `RetryInfo` detail, not in a `Retry-After` header, is now the failure's `retry_after` and is named in its advice.+- **A delay in milliseconds is read**: a `retry-after-ms` header above zero, which Azure OpenAI sends beside `retry-after`, is now the delay a rate limit carries, read first as the OpenAI and Anthropic SDKs read it, on a call through either SDK, Azure's REST API or a gateway. The transport retry of a call to Azure's REST API now waits that delay too, where it read `Retry-After` alone.+- **More of Bedrock's wordings of an input over the window are recognized**: a `ValidationException` saying "Input Tokens Exceeded" (Amazon Nova), "Too many input tokens" (Amazon Titan) or "prompt tokens (N) exceed model maximum (M)" is now `model.context_exceeded` where it was `runtime.unexpected`.++### Removed++- **`ProviderErrorMetadata.is_quota_exhaustion` (Breaking)**: a plugin reads `ProviderErrorMetadata.limit_verdict` instead, a `ProviderLimitVerdict` of `quota_exhausted`, `rate_limited` or `neither`, the one entry point the classifier reads.+## [v0.92.3] - 2026-10-27### Fixeddiff --git a/docs/reliability/failure-classification.md b/docs/reliability/failure-classification.mdindex 57e53a5c8..7aba336ec 100644--- a/docs/reliability/failure-classification.md+++ b/docs/reliability/failure-classification.md@@ -22,8 +22,19 @@ Only `transient` failures are worth retrying — re-running a call that failed bA spent quota and a rate limit both refuse the call, often with the same HTTP status, and they differ only in what cures them: time cures a rate limit, while only a person, adding credit or raising a quota, cures a spent quota. For a failure a gateway relays, Pipelex tells them apart from what the provider says structurally, its status and its error code, and never from the words of its message: a gateway relays every provider's prose, and a throttle's message may well mention a quota, as Azure OpenAI's does by linking to its quota-increase page.-Through a gateway, a `402` is a spent quota, whatever its code says. A `429` is a spent quota only when it carries an error code its provider documents as a spent quota: OpenAI's broad `insufficient_quota`, or one of the billing codes OpenAI's error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, all of which Azure OpenAI and the OpenAI-compatible providers relay. Every other `429`, including one with no code at all, is a rate limit: it is `transient`, so it is retried, and if it still fails it reaches you as `provider.rate_limited`, with the delay the provider asked for when it gave one.+Through a gateway, a `402` is a spent quota, whatever its code says. A `429` is a spent quota only when it carries an error code its provider documents as a spent quota: OpenAI's broad `insufficient_quota`, or one of the billing codes OpenAI's error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, all of which Azure OpenAI and the OpenAI-compatible providers relay. Every other `429`, including one with no code at all, is a rate limit: it is `transient`, so it is retried, and if it still fails it reaches you as `provider.rate_limited`, with the delay the provider asked for when it gave one. These codes, read from the error's `code` and its `type` alike, are all a gateway shares with a provider called directly: the provider-specific rules below, the throttles some providers send on another status and the Anthropic and Google billing rules, apply only when Pipelex calls the provider itself.-When the structure says nothing either way, Pipelex reads a `429` as a rate limit on purpose, because the two mistakes do not cost the same. A spent quota read as a rate limit costs a few refused calls, which are not billed, and some waiting before the run fails with the provider's code in its diagnostics. A rate limit read as a spent quota fails the run at once and sends you to add credit that cannot help. Each provider Pipelex calls directly, with your own keys, has a rule of its own, and some of those still read the provider's wording.+When the structure says nothing either way, Pipelex reads a `429` as a rate limit on purpose, because the two mistakes do not cost the same. A spent quota read as a rate limit costs a few refused calls, which are not billed, and some waiting before the run fails with the provider's code in its diagnostics. A rate limit read as a spent quota fails the run at once and sends you to add credit that cannot help.++A provider Pipelex calls directly, with your own keys, is read the same way, from the status, the codes and the error details that provider documents, and every `429` none of them marks as a spent quota is a rate limit:++- **OpenAI, and every backend reached through the OpenAI SDK**, OpenRouter included: a `402` is a spent quota, and a `429` is one only when its `code` or its `type` is one of the OpenAI codes above. OpenRouter documents one `402` as a wait rather than a spent quota, the one whose `error.metadata.limit_source` is `openrouter_in_flight_budget`, sent when the requests already running would overspend a key's budget: Pipelex reads that one as a rate limit, and carries the delay of the `Retry-After` header OpenRouter sends with it when the header is there and readable. Groq's `498`, its own status for a flex tier with no capacity left, which it says to retry with backoff, is a rate limit too.+- **Anthropic**: a `402`, or an error whose type is `billing_error`, is a spent quota, and so is a `429` whose details carry the code `enforced_spend_limit_reached`, the spend cap of an organization or a workspace. Anthropic answers a credit balance too low, or a usage limit you set yourself, with a `400` `invalid_request_error` and no code of its own, so for those alone Pipelex reads the message, and only how it begins: "Your credit balance is too low", "You have reached your specified API usage limits" or "You have reached your specified workspace API usage limits". Claude on Amazon Bedrock is read the Bedrock way, below, from the error code Bedrock sends in its `x-amzn-errortype` header.+- **Google Gemini**: a `402` is a spent quota, and so is an error whose `google.rpc.ErrorInfo` detail has the reason `BILLING_DISABLED`, Google's reason for a project whose billing account is disabled, which arrives as a `403`. A `429` is a spent quota only when its details say the quota it hit has a limit of zero, an `ErrorInfo` whose `quota_limit_value` is `"0"` or a `google.rpc.QuotaFailure` violation whose `quotaValue` is `"0"`, as for a free-tier quota on a model the free tier does not include, since no wait can cure it. Every other `429` is a rate limit, whatever quota its message names, a per-day cap included, which resets by itself. Gemini gives the delay it wants in the `RetryInfo` detail of its error rather than in a `Retry-After` header, and Pipelex carries that delay just as it would the header.+- **Mistral**: a `402` is a spent quota and every `429` is a rate limit.+- **Amazon Bedrock**: no failure is a spent quota, because Bedrock bills your AWS account and keeps no balance to run out. A `ThrottlingException` is a rate limit, and so is a `ServiceQuotaExceededException`, which arrives as a `400` but which AWS documents as a request to send again later. A known limitation: unlike Google, Bedrock exposes no structured signal for an on-demand quota of zero, so the `429` `ThrottlingException` an account with no quota for a model receives is read as a rate limit too, and waited out until the retry budget is spent.+- **Every other provider**, Azure OpenAI included, reports no spent quota: its `429` is a rate limit.++When a provider gives its delay twice, as Azure OpenAI does in `retry-after-ms` and in `retry-after`, Pipelex carries the milliseconds, the more precise of the two, which the OpenAI and Anthropic SDKs also read first for their own retries. It does so only when the milliseconds are above zero: for a `retry-after-ms` of `0`, Pipelex deliberately falls back to `retry-after`, since a zero gives no delay to wait.For the full mechanics of how errors are classified, carried, and reported, see [Error Model](../under-the-hood/error-model.md).diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex 3105e339e..99312ce85 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -338,7 +338,7 @@ The three steps live in three modules. Only the per-provider Extract functions s| Module | Step | What it owns ||--------|------|--------------|-| `pipelex/cogt/inference/error_classification.py` | Extract | `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, the `extract_*_metadata` functions and the parsers they share (`parse_retry_after_seconds`, `error_code_from_body_code_first`), plus pure discriminators (`is_quota_exhaustion`, `is_content_policy_violation`, `is_context_window_exceeded`, `is_network_error`, `is_model_not_allowed`) exposed as `@property` on the metadata |+| `pipelex/cogt/inference/error_classification.py` | Extract | `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, the `extract_*_metadata` functions and the parsers they share (`parse_retry_after_seconds`, `retry_after_seconds_from_headers`, `error_code_from_body_code_first`), plus pure discriminators (`limit_verdict`, a `ProviderLimitVerdict` and the one entry point to the spent-quota and rate-limit verdicts, `is_throttle_on_another_status`, `is_content_policy_violation`, `is_context_window_exceeded`, `is_network_error`, `is_model_not_allowed`) exposed as `@property` on the metadata || `pipelex/cogt/inference/error_classify.py` | Classify | `classify_inference_error()` — provider-blind mapping from `ProviderErrorMetadata` → `ClassificationResult(category, user_action_kind, is_model_not_found, service_error_code, is_model_not_allowed)`, consulting the booted plugins' service error vocabulary ahead of the status ladder || `pipelex/cogt/inference/error_render.py` | Render | `render_inference_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |@@ -554,7 +554,7 @@ An adapter is therefore a trivial renderer: it must not redefine a status, a dom```mermaidflowchart TBSDK["Layer 0 — SDK exception<br/>(openai.RateLimitError)"]- W["Layer 1 — Worker classifies<br/>is_quota_exhaustion_*() → CogtError<br/>+ InferenceErrorCategory + ProviderErrorMetadata"]+ W["Layer 1 — Worker classifies<br/>limit_verdict → CogtError<br/>+ InferenceErrorCategory + ProviderErrorMetadata"]WRAP["Layers 2-3 — Wrappers<br/>PipeRouterError → PipelineExecutionError<br/>(locate the root fault)"]PROBLEM["PipelexProblem<br/>via to_problem(view=...) over the cause chain"]@@ -688,15 +688,15 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| Scenario | Behavior ||----------|----------|-| Rate limit hit | `TRANSIENT` → repeated within the run; `provider.rate_limited`, whose verdict invites a retry and which answers 429, with a `Retry-After` header when the provider gave a delay; transport retry honors `Retry-After` |-| Quota / billing exhausted | `CAPACITY` → not repeated; `UserAction(CHECK_BILLING)`; `provider.quota_exhausted`, the operator's → HTTP 502. Through a gateway, it is read from the status and the provider's error code alone, never the message: a 402 is a spent quota whatever its code, a 429 is one only when its provider code is in `_GATEWAY_QUOTA_ERROR_CODES` (OpenAI's broad `insufficient_quota` and the billing codes its error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, which Azure OpenAI and the OpenAI-compatible providers relay; the Extract step reads a body's `code` before its `type`, so a billing code beside a broad `type` is read as the billing code), and a gateway's 429 with any other code, or none, is a rate limit, whatever its message says |+| Rate limit hit | `TRANSIENT` → repeated within the run; `provider.rate_limited`, whose verdict invites a retry and which answers 429, with a `Retry-After` header when the provider gave a delay; transport retry honors `Retry-After`. `ProviderErrorMetadata.limit_verdict` reads it, once per classification, as `RATE_LIMITED`: the throttles a provider documents on another status first, OpenRouter's 402 whose `error.metadata.limit_source`, in any section of the body, is `openrouter_in_flight_budget`, its `Retry-After` the wait when it came and is readable but never the verdict, Groq's own 498 for its flex tier at capacity, and Bedrock's 400 `ServiceQuotaExceededException`, which Claude on Bedrock carries too, through the Anthropic SDK, in the `x-amzn-errortype` header the Extract step reads as its provider code when the body names none, the same code the Bedrock worker reads, so the header feeds the service error vocabulary and the content-policy check nothing Bedrock's own SDK does not; then every 429 that is not a spent quota. The OpenAI, Anthropic, Azure and gateway Extract steps read the delay from `retry-after-ms` before `Retry-After`, as the OpenAI and Anthropic SDKs do, since Azure OpenAI sends both and the milliseconds are the more precise, and only when it is above zero: for a `retry-after-ms` of 0, Pipelex deliberately falls back to `Retry-After`, where the SDKs' header parsers return the zero and only their retry-timeout step discards it. Gemini's delay arrives in the error's `google.rpc.RetryInfo` detail rather than in a header, and the Extract step reads its `retryDelay` as the retry-after when no header gave one |+| Quota / billing exhausted | `CAPACITY` → not repeated; `UserAction(CHECK_BILLING)`; `provider.quota_exhausted`, the operator's → HTTP 502. It is decided by `ProviderErrorMetadata.limit_verdict`, from the status and the codes and details each provider documents, never the message. Through a gateway and on OpenAI's dialect, which every backend on the OpenAI SDK speaks, one rule, `_is_quota_exhaustion_by_code`, reads the same codes: the `provider_error_code` the Extract step kept, the type on the OpenAI dialect and the code through a gateway, and every `code` and `type` `_error_codes_in_body` collects from the body, its nested `error` section and its top level alike. A 402 is a spent quota whatever its code; a 429 is one only when one of those codes is in `_QUOTA_ERROR_CODES` (OpenAI's broad `insufficient_quota` and the billing codes its error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, which Azure OpenAI and the OpenAI-compatible providers relay), and every other 429, one with no code included, is a rate limit, whatever its message says. That code read is all a gateway shares with the direct path: the provider-specific rules, the throttles on another status and the Anthropic and Google rules below, apply only to a provider called directly. On the OpenAI dialect, OpenRouter's in-flight-budget 402 above is a rate limit; Anthropic takes a 402 or a `billing_error`, a 429 whose `error.details.error_code` is `enforced_spend_limit_reached`, and a 400 `invalid_request_error` whose body message begins with one of `_ANTHROPIC_QUOTA_MESSAGE_PREFIXES`, the one place a message is read, because Anthropic sends a low credit balance and a self-set usage limit with no code of their own; Google takes a 402, an error whose `google.rpc.ErrorInfo` reason is `BILLING_DISABLED` on any status, a 403 in practice, and a 429 whose details say the quota it hit has a limit of zero, an `ErrorInfo` `quota_limit_value` or a `google.rpc.QuotaFailure` violation's `quotaValue` of `"0"`, which no wait cures, so its every other 429, a per-day cap included, is a rate limit; Mistral takes a 402 alone, so its every 429 is a rate limit; and Bedrock, Azure and the other direct providers report no spent quota at all || Bad API key | `CONFIGURATION` → not repeated; `provider.credential_refused`, the operator's → HTTP 502 || Model or deployment not found (provider HTTP 404) | Raises a dedicated `*ModelNotFoundError` sibling (`LLMModelNotFoundError`, `ImgGenModelNotFoundError`, `ExtractModelNotFoundError`, `SearchModelNotFoundError`) carrying `model.unavailable`, since a method cannot name a provider's raw model id; operator re-raises `PipeOperatorModelAvailabilityError` || Model reference unknown to the deck (a method naming `gpt-5.1`, a mistyped `@alias` or `$preset`) | `ModelChoiceNotFoundError` → `CONFIGURATION` category, but the code `model.unknown`, the method's → **HTTP 422**, whose sentence names the reference and whose `suggestions` parameter carries the close matches of every catalog the deck knows, an empty list when none is near, which are all a caller needs to fix the method. A handle the deck serves only as another type of model is `model.capability_unsupported` || The deck's own default or override names a model it does not define | `ModelDeckChoiceUnresolvedError`, naming the deck setting in its message for the operator, never the caller's `model.unknown`: `model.unavailable` → HTTP 501, with the reason `not_served`, for a bare handle no backend serves as the type the step needs, whose caller step offers another model, and `pipelex.config.invalid` → HTTP 500 for a preset, an alias or a waterfall the deck does not define; a bad override is refused when the deck is built || Content-policy violation | `CONTENT` → not repeated; `UserAction(CHANGE_INPUT)`; `model.content_refused` → **HTTP 422** || Malformed prompt image / bad prompt parameter | `CONTENT` class-level (`PromptImageFormatError`, `LLMPromptParameterError`, …), each with its own code (`input.file_type_unsupported`, `pipe.prompt_empty`, …) |-| A provider **HTTP 400**, or another 4xx such as the 422 or 413 a gateway may answer, saying the inputs are over what the model can read (OpenAI's `context_length_exceeded` code, which Azure OpenAI and OpenAI-compatible gateways relay, or a provider's own wording, read in the rendered message and in the body, where a raw `httpx` error such as Azure REST's keeps it: OpenAI's "maximum context length" and "exceeds the context window", Anthropic's "prompt is too long" and "exceed context limit", Bedrock's "Input is too long for requested model", Gemini's "exceeds the maximum number of tokens allowed") | `ProviderErrorMetadata.is_context_window_exceeded` → `CONTENT` / `CHANGE_INPUT`, not repeated; `model.context_exceeded`, the caller's → **HTTP 422**, whose parameters name the model and, as its `limit`, the context window, but neither the prompt's size nor the window's, which the provider words and never gives as numbers. The 400 arm and the other-4xx arm read it, ahead of the content-policy signal, for every family, and a 413 that carries no such signal keeps its payload-size reading, `input.too_large`. A language model's window holds the prompt and the output the request reserves with `max_tokens` together, and Anthropic's "exceed context limit" and OpenAI's "maximum context length" also refuse a request whose `max_tokens`, which may be the deck's default, pushes it over, so the language-model next step names both: shorten the input or lower `max_tokens`, or choose a model with a larger context window. The other families' next steps fit what their model reads: the prompt and its input images for an image model, the document for an extraction model, the inputs for a search or judgment model. Each phrase names the window or the inputs' length, so a 400 about a malformed request matches none of them |+| A provider **HTTP 400**, or another 4xx such as the 422 or 413 a gateway may answer, saying the inputs are over what the model can read (OpenAI's `context_length_exceeded` code, which Azure OpenAI and OpenAI-compatible gateways relay, or a provider's own wording, read in the rendered message and in the body, where a raw `httpx` error such as Azure REST's keeps it: OpenAI's "maximum context length" and "exceeds the context window", Anthropic's "prompt is too long" and "exceed context limit", Bedrock's "Input is too long for requested model", and, for the models Bedrock lets word it themselves and read only on an AWS `ValidationException`, since the words alone may be another provider's throttle, Amazon Nova's "Input Tokens Exceeded", Amazon Titan's "Too many input tokens" and the "prompt tokens (N) exceed model maximum (M)" of the models Bedrock serves through its OpenAI-compatible engine, read whole with its subject so that "max_tokens (N) exceed model maximum (M)", a parameter over its bound, is not read as the input's length, since Bedrock sends no code for it, Gemini's "exceeds the maximum number of tokens allowed") | `ProviderErrorMetadata.is_context_window_exceeded` → `CONTENT` / `CHANGE_INPUT`, not repeated; `model.context_exceeded`, the caller's → **HTTP 422**, whose parameters name the model and, as its `limit`, the context window, but neither the prompt's size nor the window's, which the provider words and never gives as numbers. The 400 arm and the other-4xx arm read it, ahead of the content-policy signal, for every family, and a 413 that carries no such signal keeps its payload-size reading, `input.too_large`. A language model's window holds the prompt and the output the request reserves with `max_tokens` together, and Anthropic's "exceed context limit" and OpenAI's "maximum context length" also refuse a request whose `max_tokens`, which may be the deck's default, pushes it over, so the language-model next step names both: shorten the input or lower `max_tokens`, or choose a model with a larger context window. The other families' next steps fit what their model reads: the prompt and its input images for an image model, the document for an extraction model, the inputs for a search or judgment model. Each phrase names the window or the inputs' length, so a 400 about a malformed request matches none of them || Any other provider **HTTP 400** | `CONTENT` / `CHANGE_INPUT`, but `runtime.unexpected` → HTTP 500: a 400 nothing recognises is most likely a request of ours the provider could not take, and only a context-window or a content-policy signal makes it the caller's, `model.context_exceeded` or `model.content_refused` || Local file extractor raises a builtin (docling, pypdfium2) | `FileNotFoundError` → `input.reference_not_found`, `ValueError` / `RuntimeError` → `input.file_unreadable`, both `CONTENT` → **HTTP 422**; `OSError` → `UNKNOWN`, `runtime.unexpected` (see `_LOCAL_EXTRACT_BY_TYPE_NAME`) || A text completion stops at its output limit or context window (`length`, `max_tokens`, `MAX_TOKENS`, an `incomplete` Responses answer at `max_output_tokens`, `model_context_window_exceeded`, `model_length`), with partial or empty text | Every LLM worker reads the stop before returning text and raises `LLMCompletionTruncatedError` → `CONTENT` / `CHANGE_INPUT`, not repeated: `model.output_truncated`, the method's, at the output limit, which the method's `max_tokens` sets, and `model.context_exceeded`, the input's, at the context window, both → **HTTP 422**. The message, for the operator, names the model, the pipe, the stop value, the output tokens used and the `max_tokens` sent, never the partial text; the caller reads the code's sentence, with the model and the token counts as its parameters. The error's `truncation_limit` tells the output limit from the context window (`model_context_window_exceeded`, `model_length`), and so does its code's next step: a higher `max_tokens` for the first, a shorter input for the second. A gateway in non-strict mode passing each provider's own value through the completions shape is read the same way, since the classifier reads the union of the provider vocabularies |diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.mdindex ef00aa0d9..1e67bb395 100644--- a/docs/under-the-hood/inference-backend-plugins.md+++ b/docs/under-the-hood/inference-backend-plugins.md@@ -332,7 +332,7 @@ A worker implements the provider half of a call, `_gen_text` or `_extract_pages`| Symbol | Module | Role ||--------|--------|------|| `ProviderErrorMetadata`, `UserAction`, `UserActionKind` | `pipelex.cogt.inference.error_classification` | the structured failure a worker distils, and the advice it carries |-| `parse_retry_after_seconds`, `error_code_from_body_code_first` | `pipelex.cogt.inference.error_classification` | the parsers a distiller of a plain-HTTP failure shares |+| `retry_after_seconds_from_headers`, `parse_retry_after_seconds`, `error_code_from_body_code_first` | `pipelex.cogt.inference.error_classification` | the parsers a distiller of a plain-HTTP failure shares: the delay from a response's headers (`retry-after-ms` first when it is above zero, then `Retry-After`), the delay from a lone `Retry-After` value, and the error code of an OpenAI-shaped body, read `code` first || `extract_gateway_metadata` | `pipelex.cogt.inference.error_classification` | the distiller of a Portkey-substrate failure || `ProviderName` | `pipelex.cogt.inference.provider_name` | which provider a distilled failure reports || `classify_inference_error`, `InferenceErrorFamily`, `ClassificationResult` | `pipelex.cogt.inference.error_classify` | the shared Classify step, called with the family of the failed call (`classify_inference_error(metadata, family=InferenceErrorFamily.LLM)`), which picks the category and the registry code together |diff --git a/docs/under-the-hood/service-error-codes.md b/docs/under-the-hood/service-error-codes.mdindex 1d05c0da3..ebb64e515 100644--- a/docs/under-the-hood/service-error-codes.md+++ b/docs/under-the-hood/service-error-codes.md@@ -67,7 +67,7 @@ At boot, every plugin's contributions are frozen into one `ServiceErrorVocabular**Codes are matched on the code alone, with no check on the provider.** A request reaches a service through whichever SDK its protocol calls for, so the provider a failure reports does not identify the service; the code does. A plugin that contributes a code vouches that no vendor emits it, which in practice means the codes live in a namespace of the service's own.-**For a code to reach the classifier, the worker's Extract step must recover it.** The vendor-facing distillers read an OpenAI-shaped error body's `type` before its `code`. A service that renders `{"error": {"type": "invalid_request_error", "code": "<specific_code>"}}` needs its distiller to read `code` first: `error_code_from_body_code_first` in `pipelex.cogt.inference.error_classification` does exactly that, and `parse_retry_after_seconds` beside it reads a `Retry-After` header, for a plugin that distils a plain-HTTP failure itself.+**For a code to reach the classifier, the worker's Extract step must recover it.** The vendor-facing distillers read an OpenAI-shaped error body's `type` before its `code`. A service that renders `{"error": {"type": "invalid_request_error", "code": "<specific_code>"}}` needs its distiller to read `code` first: `error_code_from_body_code_first` in `pipelex.cogt.inference.error_classification` does exactly that. For a plugin that distils a plain-HTTP failure itself, `retry_after_seconds_from_headers` beside it reads the delay from the response's headers, `retry-after-ms` first when it is above zero and then `Retry-After`, and `parse_retry_after_seconds` reads a lone `Retry-After` value.## Fail-loud guaranteesdiff --git a/pipelex/cogt/inference/error_classification.py b/pipelex/cogt/inference/error_classification.pyindex 4ccdb9043..a7e63225a 100644--- a/pipelex/cogt/inference/error_classification.py+++ b/pipelex/cogt/inference/error_classification.py@@ -1,13 +1,14 @@"""Helpers for classifying SDK errors into InferenceErrorCategory values.-Pure functions that inspect error statuses, codes and messages to discriminate-between quota exhaustion vs rate limiting, detect content policy violations, and-recover the underlying SDK exception that ``InstructorRetryException``+Pure functions that inspect error statuses and codes to discriminate a spent+quota from a rate limit, inspect messages to detect content policy violations,+and recover the underlying SDK exception that ``InstructorRetryException``wraps when ``instructor`` exhausts its retry loop."""import jsonimport math+import refrom datetime import UTC, datetimefrom email.utils import parsedate_to_datetimefrom enum import StrEnum@@ -47,6 +48,18 @@ _STATUSLESS_TRANSPORT_TYPE_NAMES: frozenset[str] = frozenset()+class ProviderLimitVerdict(StrEnum):+ """What a provider's refusal says about its limits: a spent quota, a rate limit, or neither.++ A spent quota is cured by a person adding credit or raising a quota; a rate limit by time. ``NEITHER`` leaves the+ refusal to the rest of the classifier.+ """++ QUOTA_EXHAUSTED = "quota_exhausted"+ RATE_LIMITED = "rate_limited"+ NEITHER = "neither"++# The one error code outside any service's own namespace that the runtime classifies itself:# Portkey's refusal of a model an integration serves but does not allow for this caller, because the# model is off the integration's allow-list or archived. Portkey's cloud emits it for a caller's own@@ -91,9 +104,11 @@ class ProviderErrorMetadata(BaseModel):provider: ProviderNamesdk_exception_type: str- # Human-readable error text from the SDK exception (``str(exc)``). Both the- # Classify step (quota / content-policy discrimination) and the Render step- # (message composition) read it, so the Extract step must capture it. A+ # Human-readable error text from the SDK exception (``str(exc)``). The+ # Classify step reads it for the content-policy and context-window checks, and+ # never for a spent quota or a rate limit (Anthropic's credit sentences are read+ # from the body's message), and the Render step composes its message from it,+ # so the Extract step must capture it. A# pydantic ``ValidationError``, which ends instructor's re-asks, is written by# its locations and reasons alone: its own text quotes the model's answer. Any# other failure to parse that answer, such as instructor's assertion carrying a@@ -109,29 +124,92 @@ class ProviderErrorMetadata(BaseModel):body: Any | None = Field(default=None, exclude=True)@property- def is_quota_exhaustion(self) -> bool:- """Whether this error is a quota/credits exhaustion rather than rate limiting.-- Dispatches on ``provider`` because each provider signals quota- exhaustion differently; Mistral and Gateway also use HTTP 402. A- gateway relays every provider's prose, so its rule reads the status and- the provider's error code alone, never the message.+ def limit_verdict(self) -> ProviderLimitVerdict:+ """Whether the provider refused the call for a spent quota, for a rate limit, or for neither.++ A spent quota and a rate limit both refuse the call, often on the same status, and differ only in what cures+ them: time cures a rate limit, and only a person cures a spent quota. Each provider signals a spent quota in its+ own structure, so the rule dispatches on ``provider`` and reads the provider's status and its error code, type+ or details, never the words of its message, with one exception: Anthropic, whose credit-balance and self-set+ spend limit refusals carry nothing but their sentence on a 400 (see ``_is_quota_exhaustion_anthropic``).++ A throttle a provider documents on another status than 429 is a rate limit first (see+ ``is_throttle_on_another_status``). Then a spent quota, by the provider's rule. Then every other 429, whatever+ its message says: when the structure says nothing either way, a quota read as a rate limit costs some refused+ calls and some waiting, while a rate limit read as a quota fails the run and sends a person to add credit that+ cannot help. The providers whose rule never reports a spent quota reach one through the classifier's status+ ladder alone, which reads every 402 as one, or, for Linkup, through its typed exception. The property computes+ the throttle, quota and rate-limit verdicts together, each once per read, and the classifier reads it once: it is+ the one entry point to them."""+ if self.is_throttle_on_another_status:+ return ProviderLimitVerdict.RATE_LIMITED+ if self._reads_as_spent_quota():+ return ProviderLimitVerdict.QUOTA_EXHAUSTED+ if self.status_code == 429:+ return ProviderLimitVerdict.RATE_LIMITED+ return ProviderLimitVerdict.NEITHER++ def _reads_as_spent_quota(self) -> bool:+ """Apply the provider's own spent-quota rule, once ``is_throttle_on_another_status`` has set its throttles aside."""+ status_code = self.status_code or 0match self.provider:- case ProviderName.OPENAI:- return _is_quota_exhaustion_openai(self.message)+ case ProviderName.OPENAI | ProviderName.GATEWAY:+ # OpenAI names a billing cause in `code` beside a broad `type`, and the two Extract steps keep a different+ # one of them as `provider_error_code`, the OpenAI one the type and the gateway one the code. The rule+ # reads both from the body as well, through `_error_codes_in_body`, so a spent-quota code gets the same+ # verdict directly and through a gateway, whose relayed message decides nothing. That code read is all+ # the gateway hop shares: the provider-specific rules, the throttles `is_throttle_on_another_status`+ # reads and the Anthropic and Google billing rules, apply on the direct path only.+ return _is_quota_exhaustion_by_code(status_code=status_code, error_codes=(self.provider_error_code, *_error_codes_in_body(self.body)))case ProviderName.ANTHROPIC:- return _is_quota_exhaustion_anthropic(self.message)+ return _is_quota_exhaustion_anthropic(status_code=status_code, error_type=self.provider_error_code, body=self.body)case ProviderName.GOOGLE:- return _is_quota_exhaustion_google(self.message)+ return _is_quota_exhaustion_google(status_code=status_code, body=self.body)case ProviderName.MISTRAL:- return _is_quota_exhaustion_mistral(self.message, status_code=self.status_code or 0)- case ProviderName.BEDROCK:- return _is_quota_exhaustion_aws(self.message, provider_error_code=self.provider_error_code)- case ProviderName.GATEWAY:- return _is_quota_exhaustion_gateway(status_code=self.status_code or 0, provider_error_code=self.provider_error_code)+ # Mistral answers a spent balance with a 402 and every rate limit with a 429, a monthly cap that a+ # person may have to raise included, so the status alone decides.+ return status_code == 402+ case (+ ProviderName.BEDROCK+ | ProviderName.AZURE+ | ProviderName.FAL+ | ProviderName.HUGGINGFACE+ | ProviderName.LINKUP+ | ProviderName.TYPESAFE+ | ProviderName.DOCLING+ | ProviderName.PYPDFIUM2+ ):+ # Bedrock is billed to the AWS account, with no balance to spend, and every error AWS calls a quota is a+ # rate quota: `ThrottlingException` on a 429, and `ServiceQuotaExceededException`, which AWS says can+ # be resubmitted later and which `is_throttle_on_another_status` reads.+ return False++ @property+ def is_throttle_on_another_status(self) -> bool:+ """Whether the provider documents this refusal, on a status other than 429, as a throttle that a wait cures.++ OpenRouter, an OpenAI-dialect backend, answers a request its in-flight spending budget cannot hold yet with a+ 402 whose ``error.metadata.limit_source`` is ``openrouter_in_flight_budget``, and documents it as a wait for+ its ``Retry-After``: the structure decides, and the header, when it came and was readable, is the wait. Groq,+ another OpenAI-dialect backend, answers a request its flex tier has no capacity for with its own status, 498,+ and documents retrying it with backoff. Bedrock answers ``ServiceQuotaExceededException`` on a 400, and AWS+ documents it as a request over the account's service quota that "you can resubmit later"; Claude on Bedrock,+ called through the Anthropic SDK, carries the same code, read from Bedrock's ``x-amzn-errortype`` header by+ ``extract_anthropic_metadata``.+ """+ match self.provider:+ case ProviderName.OPENAI:+ return self.status_code == _GROQ_FLEX_CAPACITY_STATUS_CODE or _is_in_flight_budget_throttle_openrouter(+ status_code=self.status_code, body=self.body+ )+ case ProviderName.BEDROCK | ProviderName.ANTHROPIC:+ return self.provider_error_code == _AWS_SERVICE_QUOTA_EXCEEDED_ERROR_CODEcase (- ProviderName.AZURE+ ProviderName.GOOGLE+ | ProviderName.MISTRAL+ | ProviderName.GATEWAY+ | ProviderName.AZURE| ProviderName.FAL| ProviderName.HUGGINGFACE| ProviderName.LINKUP@@ -187,9 +265,12 @@ class ProviderErrorMetadata(BaseModel):return Trueif error_code_from_body_code_first(self.body) in _CONTEXT_WINDOW_ERROR_CODES:return True- if _names_the_context_window(text=self.message):+ is_aws_validation_error = self.provider_error_code == _AWS_VALIDATION_ERROR_CODE+ if _names_the_context_window(text=self.message, is_aws_validation_error=is_aws_validation_error):return True- return self.body is not None and _names_the_context_window(text=_stringify_for_scan(self.body))+ return self.body is not None and _names_the_context_window(+ text=_stringify_for_scan(self.body), is_aws_validation_error=is_aws_validation_error+ )@propertydef is_network_error(self) -> bool:@@ -234,21 +315,6 @@ class UserAction(BaseModel):detail: str-_OPENAI_QUOTA_PATTERNS: tuple[str, ...] = (- "insufficient_quota",- "exceeded your current quota",-)--_ANTHROPIC_QUOTA_PATTERNS: tuple[str, ...] = (- "quota exceeded",- "quota has been",- "credit balance",- "out of credits",- "insufficient credit",- "billing limit",- "billing issue",-)-_CONTENT_POLICY_PATTERNS: tuple[str, ...] = ("content_policy","content_filter",@@ -286,40 +352,42 @@ _CONTEXT_WINDOW_PATTERNS: tuple[str, ...] = ("exceeds the maximum number of tokens allowed",)-_GOOGLE_QUOTA_PATTERNS: tuple[str, ...] = (- "quota exceeded",- "resource has been exhausted",- "billing limit",- "billing quota",- "billing exceeded",- "billing account",+# AWS's error code for a request Bedrock refuses as invalid, the only one an input over the model's window arrives with+_AWS_VALIDATION_ERROR_CODE = "ValidationException"++# How the Bedrock models that word it themselves refuse an input over their window, read only on an AWS+# `ValidationException`, since the words alone could be another provider's throttle ("Daily input tokens exceeded").+# Bedrock has no structured signal for it, only the model's sentence: Amazon Nova's "Input Tokens Exceeded: Number of+# input tokens exceeds maximum length." and Amazon Titan's "Too many input tokens", each matched the same way by+# langchain-aws's `_CONTEXT_OVERFLOW_MARKERS` (libs/aws/langchain_aws/chat_models/bedrock_converse.py). The bare "too+# many tokens" is left out on purpose: it is also Bedrock's throttle, "Too many tokens per day". The models Bedrock+# serves through its OpenAI-compatible engine word it with numbers inside, so `_PROMPT_TOKENS_OVER_MODEL_MAXIMUM` reads+# theirs.+_BEDROCK_CONTEXT_WINDOW_PATTERNS: tuple[str, ...] = (+ "input tokens exceeded",+ "too many input tokens",)-_MISTRAL_QUOTA_PATTERNS: tuple[str, ...] = (- "quota",- "billing limit",- "billing quota",- "out of credits",- "insufficient credits",-)--_AWS_QUOTA_PATTERNS: tuple[str, ...] = (- "quota",- "limit exceeded",- "service quota",-)--# The provider error codes that make a gateway's 429 a spent quota rather than a rate limit. A gateway relays every-# provider's prose, so the message decides nothing: Azure's throttle links to its quota-increase page, and Google's-# per-minute limit says "check quota". A code joins this set only when the provider that sends it documents it as a-# spent quota on a 429, with a comment naming that provider and a test case. A 402 needs no entry, since its status-# decides it, and a gateway's own codes are read from the service error vocabulary before this rule runs.-_GATEWAY_QUOTA_ERROR_CODES: frozenset[str] = frozenset(+# The models Bedrock serves through its OpenAI-compatible engine, such as Kimi, refuse an input over their window with+# "prompt tokens (N) exceed model maximum (M)", the sentence LiteLLM reads for them+# (litellm/litellm_core_utils/exception_mapping_utils.py, `_BEDROCK_MANTLE_CONTEXT_WINDOW_PATTERN`). It is read whole,+# with its subject: "exceed model maximum" alone is also how a parameter over its bound is refused, as in+# "max_tokens (200000) exceed model maximum (8192)", which is the request's fault, not the input's length.+_PROMPT_TOKENS_OVER_MODEL_MAXIMUM = re.compile(r"prompt tokens \(\d+\) exceed model maximum")++# The error codes that make a 429 a spent quota rather than a rate limit, on the OpenAI dialect and through a gateway+# alike, each one OpenAI's error-code guide documents for a billing refusal+# (https://developers.openai.com/api/docs/guides/error-codes), which Azure OpenAI and the OpenAI-compatible providers+# relay in the same fields. OpenAI names each billing cause in `error.code` and says `error.type` can still be+# `insufficient_quota`, so the rule reads both. Every other 429, the rate limits and OpenAI's `slow_down` included, is a+# rate limit, whatever its message says: "You exceeded your current quota" is also Gemini's wording for its per-minute+# throttle, which an OpenAI-dialect backend pointed at Google relays, and Azure's throttle links to its quota-increase+# page. A code joins this set only when the provider that sends it documents it as a spent quota on a 429, with a+# comment naming that provider and a test case. A 402 needs no entry, since its status decides it, and a gateway's own+# codes are read from the service error vocabulary before this rule runs.+_QUOTA_ERROR_CODES: frozenset[str] = frozenset({- # OpenAI's codes for a spent quota on a 429, which Azure OpenAI and the OpenAI-compatible providers relay in the- # same fields (https://developers.openai.com/api/docs/guides/error-codes). OpenAI names each billing cause in- # `error.code` and says `error.type` can still be `insufficient_quota`; the gateway's Extract step reads the- # body's code before its type, so the specific codes are listed beside the broad one.+ # the broad code and type, and the long-standing code of a spent quota"insufficient_quota",# no prepaid credits remaining"credit_balance_exhausted",@@ -332,65 +400,173 @@ _GATEWAY_QUOTA_ERROR_CODES: frozenset[str] = frozenset(})+# Groq's own status for a request its flex tier has no capacity for, sent with the error `capacity_exceeded`, which it+# says to retry with jittered backoff (https://console.groq.com/docs/flex-processing). No other OpenAI-dialect backend+# is known to answer it, and its error's field is not documented, so the status alone decides.+_GROQ_FLEX_CAPACITY_STATUS_CODE = 498++# OpenRouter's `error.metadata.limit_source` for a 402 that its in-flight spending budget, not the account's balance,+# caused: the budget frees itself as the running requests settle+# (https://openrouter.ai/docs/api_reference/limits.md#handling-402-errors).+_OPENROUTER_IN_FLIGHT_BUDGET_LIMIT_SOURCE = "openrouter_in_flight_budget"++# Anthropic's error type for a billing refusal, documented on a 402 (https://platform.claude.com/docs/en/api/errors)+_ANTHROPIC_BILLING_ERROR_TYPE = "billing_error"++# Anthropic's `error.details.error_code` on the 429 its tier's monthly spend cap answers, whose type is+# `rate_limit_error` like a rate limit's and which carries no `retry-after`+# (https://platform.claude.com/docs/en/api/rate-limits#reaching-your-spend-cap)+_ANTHROPIC_SPEND_CAP_ERROR_CODE = "enforced_spend_limit_reached"++# Anthropic's error type for a malformed request, which its credit-balance and self-set spend-limit refusals share+_ANTHROPIC_INVALID_REQUEST_ERROR_TYPE = "invalid_request_error"++# How Anthropic's 400 `invalid_request_error` refusals for a spent credit or spend limit begin, lowercased. Anthropic+# sends no structured signal for them: they share their status and type with every malformed request, so their sentence+# is the only signal, and it is read on that status and type alone, anchored at the message's start.+_ANTHROPIC_QUOTA_MESSAGE_PREFIXES: tuple[str, ...] = (+ # The credit balance, documented nowhere; an SDK maintainer confirmed its status and type+ # (https://github.com/anthropics/anthropic-sdk-typescript/issues/618)+ "your credit balance is too low",+ # The organization's and the workspace's own spend limits+ # (https://platform.claude.com/docs/en/api/rate-limits#setting-your-own-spend-limit)+ "you have reached your specified api usage limits",+ "you have reached your specified workspace api usage limits",+)-def _is_quota_exhaustion_openai(error_message: str) -> bool:- """Check if an OpenAI error message indicates quota/credits exhaustion rather than rate limiting."""- lower_message = error_message.lower()- return any(pattern in lower_message for pattern in _OPENAI_QUOTA_PATTERNS)+# AWS's error code for a request over the account's Bedrock service quota, answered on a 400, which "you can resubmit+# later" (https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html): a throttle, not a+# spent quota, since Bedrock is billed to the AWS account with no balance to spend.+_AWS_SERVICE_QUOTA_EXCEEDED_ERROR_CODE = "ServiceQuotaExceededException"+# Google's `google.rpc.ErrorInfo` reason for a request whose project's billing account is disabled, which the Gemini+# API and Vertex AI send as Google Cloud APIs (https://github.com/googleapis/googleapis/blob/master/google/api/error_reason.proto).+# Only a person can enable billing again, so it is a spent quota whatever status it arrives on.+_GOOGLE_BILLING_DISABLED_REASON = "BILLING_DISABLED"-def _is_quota_exhaustion_anthropic(error_message: str) -> bool:- """Check if an Anthropic error message indicates quota/credits exhaustion rather than rate limiting."""- lower_message = error_message.lower()- return any(pattern in lower_message for pattern in _ANTHROPIC_QUOTA_PATTERNS)+# The `@type` of the Google error details the rules read (https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto)+_GOOGLE_ERROR_INFO_TYPE = "type.googleapis.com/google.rpc.ErrorInfo"+_GOOGLE_QUOTA_FAILURE_TYPE = "type.googleapis.com/google.rpc.QuotaFailure"-def _is_quota_exhaustion_google(error_message: str) -> bool:- """Check if a Google error message indicates quota/credits exhaustion rather than rate limiting."""- lower_message = error_message.lower()- return any(pattern in lower_message for pattern in _GOOGLE_QUOTA_PATTERNS)+def _error_sections(body: Any) -> list[dict[str, Any]]:+ """The places an error body may hold its fields: its nested ``error`` section first, then its top level.++ Most providers nest their error under ``error``, and the OpenAI client pre-unwraps that section onto the body it+ keeps, which leaves the fields at the top level.+ """+ if not isinstance(body, dict):+ return []+ body_dict = cast("dict[str, Any]", body)+ sections: list[dict[str, Any]] = []+ error_section = body_dict.get("error")+ if isinstance(error_section, dict):+ sections.append(cast("dict[str, Any]", error_section))+ sections.append(body_dict)+ return sections-def _is_quota_exhaustion_mistral(error_message: str, *, status_code: int) -> bool:- """Check if a Mistral error indicates quota/credits exhaustion.+def _is_quota_exhaustion_by_code(*, status_code: int, error_codes: tuple[str | None, ...]) -> bool:+ """Check if an OpenAI-dialect or gateway error is a spent quota, from its status and its error codes alone.- HTTP 402 (Payment Required) is a definitive quota signal.- HTTP 429 requires message inspection to distinguish quota from rate limiting.+ A 402 is a spent quota whatever its code, as OpenRouter answers insufficient credits, except its in-flight budget's+ throttle, which ``ProviderErrorMetadata.is_throttle_on_another_status`` sets aside first. A 429 is a spent quota only+ when one of ``error_codes`` is in ``_QUOTA_ERROR_CODES``; any other 429, one with no code included, is a rate limit.+ The message is never read."""if status_code == 402:return True- lower_message = error_message.lower()- return status_code == 429 and any(pattern in lower_message for pattern in _MISTRAL_QUOTA_PATTERNS)+ return status_code == 429 and any(error_code in _QUOTA_ERROR_CODES for error_code in error_codes)-def _is_quota_exhaustion_aws(error_message: str, *, provider_error_code: str | None) -> bool:- """Check if an AWS error indicates quota/credits exhaustion rather than rate limiting.+def _is_in_flight_budget_throttle_openrouter(*, status_code: int | None, body: Any) -> bool:+ """Check if an OpenAI-dialect 402 is OpenRouter's in-flight budget throttle, from its ``error.metadata.limit_source``.- AWS botocore puts the canonical signal in the error ``Code`` (e.g.- ``ServiceQuotaExceededException``), which ``extract_bedrock_metadata`` surfaces- as ``provider_error_code``. Some payloads also echo the situation in the- ``Message`` text — we check both so a quota exception with a vague message is- still detected.+ The source is read in every section of the body, the nested ``error`` and the top level the OpenAI client unwraps it+ to. The ``Retry-After`` OpenRouter documents beside it is the wait, not the verdict: a throttle whose header is+ missing or unreadable is still one, waited for by the retry's own backoff."""- if provider_error_code == "ServiceQuotaExceededException":- return True- lower_message = error_message.lower()- return any(pattern in lower_message for pattern in _AWS_QUOTA_PATTERNS)+ if status_code != 402:+ return False+ for section in _error_sections(body):+ error_metadata = section.get("metadata")+ if not isinstance(error_metadata, dict):+ continue+ if cast("dict[str, Any]", error_metadata).get("limit_source") == _OPENROUTER_IN_FLIGHT_BUDGET_LIMIT_SOURCE:+ return True+ return False-def _is_quota_exhaustion_gateway(*, status_code: int, provider_error_code: str | None) -> bool:- """Check if a Portkey/Gateway error indicates quota/credits exhaustion, from its status and its provider code alone.+def _is_quota_exhaustion_anthropic(*, status_code: int, error_type: str | None, body: Any) -> bool:+ """Check if an Anthropic error is a spent quota or credit, from its status, its error type and its details.- HTTP 402 (Payment Required) is a spent quota whatever its code. HTTP 429 is a spent quota only when the provider's- error code is one of ``_GATEWAY_QUOTA_ERROR_CODES``; any other 429, one with no code included, is a rate limit,- which the classifier's 429 arm reads as ``provider.rate_limited``. The message is never read: a gateway relays- every provider's prose, and a word list keeps meeting a new provider's wording. When the structure says nothing- either way, a quota read as a rate limit costs some refused calls and some waiting, while a rate limit read as a- quota fails the run and sends a person to add credit that cannot help.+ A 402, or an error of type ``billing_error`` on any status, is a billing refusal. A 429 is a spent quota only when+ its ``error.details.error_code`` is ``enforced_spend_limit_reached``, the tier's monthly spend cap; every other 429+ is a rate limit. A 400 of type ``invalid_request_error`` is a spent quota when its message begins with one of+ ``_ANTHROPIC_QUOTA_MESSAGE_PREFIXES``: Anthropic answers a spent credit balance and a spend limit the organization+ set itself with nothing but that sentence, so it is the one refusal any rule reads from the provider's words.+ The details and the message are read from the error body, since the message the SDK renders prefixes them.+ """+ if status_code == 402 or error_type == _ANTHROPIC_BILLING_ERROR_TYPE:+ return True+ error_sections = _error_sections(body)+ if not error_sections:+ return False+ error_section = error_sections[0]+ if status_code == 429:+ details = error_section.get("details")+ return isinstance(details, dict) and cast("dict[str, Any]", details).get("error_code") == _ANTHROPIC_SPEND_CAP_ERROR_CODE+ if status_code == 400 and error_type == _ANTHROPIC_INVALID_REQUEST_ERROR_TYPE:+ error_message = error_section.get("message")+ return isinstance(error_message, str) and error_message.strip().lower().startswith(_ANTHROPIC_QUOTA_MESSAGE_PREFIXES)+ return False+++def _google_detail_entries(body: Any, *, detail_type: str) -> list[dict[str, Any]]:+ """The entries of a Google error's ``details`` whose ``@type`` is ``detail_type``, the nested ``error`` first."""+ entries: list[dict[str, Any]] = []+ for section in _error_sections(body):+ details = section.get("details")+ if not isinstance(details, list):+ continue+ for entry in cast("list[Any]", details):+ if isinstance(entry, dict) and cast("dict[str, Any]", entry).get("@type") == detail_type:+ entries.append(cast("dict[str, Any]", entry))+ return entries+++def _google_error_info_metadata(entry: dict[str, Any]) -> dict[str, Any]:+ metadata = entry.get("metadata")+ return cast("dict[str, Any]", metadata) if isinstance(metadata, dict) else {}+++def _is_quota_exhaustion_google(*, status_code: int, body: Any) -> bool:+ """Check if a Google error is a spent quota, from its status and its ``google.rpc`` details alone.++ A 402 is a spent balance. An ``ErrorInfo`` whose reason is ``BILLING_DISABLED`` is a project whose billing account is+ disabled, on whatever status it arrives, a 403 in practice. A 429 is a spent quota only when its details say the+ quota it hit has a limit of zero: the model is not included in the project's tier at all, as for a free-tier quota+ on a model the free tier leaves out, so no wait can cure it. Every other 429 is a rate limit, a per-day cap+ included: it resets by itself, and a run that waits for it within its budget may still succeed."""if status_code == 402:return True- return status_code == 429 and provider_error_code in _GATEWAY_QUOTA_ERROR_CODES+ error_infos = _google_detail_entries(body, detail_type=_GOOGLE_ERROR_INFO_TYPE)+ if any(entry.get("reason") == _GOOGLE_BILLING_DISABLED_REASON for entry in error_infos):+ return True+ if status_code != 429:+ return False+ if any(str(_google_error_info_metadata(entry).get("quota_limit_value")) == "0" for entry in error_infos):+ return True+ for quota_failure in _google_detail_entries(body, detail_type=_GOOGLE_QUOTA_FAILURE_TYPE):+ violations = quota_failure.get("violations")+ if not isinstance(violations, list):+ continue+ for violation in cast("list[Any]", violations):+ # `quotaValue` is an int64, which protobuf's JSON form writes as a string+ if isinstance(violation, dict) and str(cast("dict[str, Any]", violation).get("quotaValue")) == "0":+ return True+ return Falsedef _is_content_policy_violation(error_message: str) -> bool:@@ -399,10 +575,17 @@ def _is_content_policy_violation(error_message: str) -> bool:return any(pattern in lower_message for pattern in _CONTENT_POLICY_PATTERNS)-def _names_the_context_window(*, text: str) -> bool:- """Check if an error's text words a refusal of inputs over the model's context window, as some provider words it."""+def _names_the_context_window(*, text: str, is_aws_validation_error: bool) -> bool:+ """Check if an error's text words a refusal of inputs over the model's context window, as some provider words it.++ The Bedrock models' own wordings, ``_BEDROCK_CONTEXT_WINDOW_PATTERNS``, are read only on an AWS ``ValidationException``.+ """lower_text = text.lower()- return any(pattern in lower_text for pattern in _CONTEXT_WINDOW_PATTERNS)+ if any(pattern in lower_text for pattern in _CONTEXT_WINDOW_PATTERNS):+ return True+ if is_aws_validation_error and any(pattern in lower_text for pattern in _BEDROCK_CONTEXT_WINDOW_PATTERNS):+ return True+ return _PROMPT_TOKENS_OVER_MODEL_MAXIMUM.search(lower_text) is not Nonedef _stringify_for_scan(body: Any) -> str:@@ -530,6 +713,51 @@ def parse_retry_after_seconds(value: Any) -> float | None:return max(delta_seconds, 0.0)+def retry_after_seconds_from_headers(*, headers: Any) -> float | None:+ """Read the delay a provider asked for from a response's headers, in seconds.++ ``retry-after-ms``, a delay in milliseconds, comes first: it is not standard, but Azure OpenAI sends it beside+ ``retry-after`` on a 429 (https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/provisioned-get-started),+ it is the more precise of the two, and the OpenAI and Anthropic SDKs read it first for their own retries. It+ decides only when it is a finite number above zero. Below or at zero Pipelex deliberately falls back to+ ``retry-after``, through ``parse_retry_after_seconds``, where the SDKs' header parsers return the zero and leave+ their retry-timeout step to discard it: a zero is no delay, and the seconds the provider sent beside it are one.+ """+ if headers is None:+ return None+ retry_after_ms = headers.get("retry-after-ms")+ if retry_after_ms is not None:+ try:+ milliseconds = float(retry_after_ms)+ except (TypeError, ValueError):+ pass+ else:+ if math.isfinite(milliseconds) and milliseconds > 0:+ return milliseconds / 1000+ return parse_retry_after_seconds(headers.get("retry-after"))+++# Bedrock's error type header, which its REST-JSON protocol sends on every error answer: Claude on Bedrock, called+# through the Anthropic SDK, keeps it, while its body carries only a message.+_AWS_ERROR_TYPE_HEADER = "x-amzn-errortype"+++def _aws_error_code_from_headers(*, headers: Any) -> str | None:+ """Read the AWS error code from Bedrock's ``x-amzn-errortype`` header, normalized as botocore normalizes it.++ The header may carry colon-separated data after the code and may name it as a namespaced shape id, as in+ ``ThrottlingException:http://internal.amazon.com/coral/com.amazon.bedrock/``; botocore's ``RestJSONParser`` keeps+ what precedes the first colon and follows the last ``#``.+ """+ if headers is None:+ return None+ error_type = headers.get(_AWS_ERROR_TYPE_HEADER)+ if not isinstance(error_type, str):+ return None+ code = error_type.split(":", 1)[0].rsplit("#", 1)[-1].strip()+ return code or None++def _provider_error_code_from_body(body: Any) -> str | None:if not isinstance(body, dict):return None@@ -558,7 +786,7 @@ def extract_openai_metadata(- ``APIStatusError`` subclasses (``BadRequestError``, ``RateLimitError``,``AuthenticationError`` …) expose ``status_code``, ``request_id``,- ``response.headers`` (for ``Retry-After``), and ``body``. The SDK+ ``response.headers`` (for ``retry-after-ms`` and ``Retry-After``), and ``body``. The SDKpre-unwraps ``body["error"]`` so ``body["type"]`` / ``body["code"]``sit at the top level — and are also mirrored to ``exc.type`` /``exc.code`` as instance attributes.@@ -572,10 +800,7 @@ def extract_openai_metadata(if not isinstance(request_id, str):request_id = Noneresponse = getattr(exc, "response", None)- headers = getattr(response, "headers", None)- retry_after_seconds: float | None = None- if headers is not None:- retry_after_seconds = parse_retry_after_seconds(headers.get("retry-after"))+ retry_after_seconds = retry_after_seconds_from_headers(headers=getattr(response, "headers", None))body = getattr(exc, "body", None)# OpenAI's _make_status_error pre-unwraps body["error"] onto exc.type / exc.code,# so we read those attributes directly rather than re-parsing the body.@@ -612,7 +837,8 @@ def extract_anthropic_metadata(Tolerates the two exception shapes in the Anthropic SDK:- ``APIStatusError`` subclasses expose ``status_code``, ``request_id``,- ``response.headers`` (for ``Retry-After``) and ``body``.+ ``response.headers`` (for ``retry-after-ms`` and ``Retry-After``, and, for Claude on Bedrock,+ ``x-amzn-errortype``, Bedrock's error code) and ``body``.- ``APIConnectionError`` / ``APITimeoutError`` expose neither``status_code`` nor ``response``; every status-related field comes backas ``None``.@@ -625,10 +851,10 @@ def extract_anthropic_metadata(request_id = Noneresponse = getattr(exc, "response", None)headers = getattr(response, "headers", None)- retry_after_seconds: float | None = None- if headers is not None:- retry_after_seconds = parse_retry_after_seconds(headers.get("retry-after"))+ retry_after_seconds = retry_after_seconds_from_headers(headers=headers)body = getattr(exc, "body", None)+ # Claude on Bedrock answers with Bedrock's error, whose body holds a message alone and whose code is in a header+ provider_error_code = _provider_error_code_from_body(body) or _aws_error_code_from_headers(headers=headers)return ProviderErrorMetadata(provider=ProviderName.ANTHROPIC,sdk_exception_type=type(exc).__name__,@@ -636,7 +862,7 @@ def extract_anthropic_metadata(status_code=status_code,request_id=request_id,retry_after_seconds=retry_after_seconds,- provider_error_code=_provider_error_code_from_body(body),+ provider_error_code=provider_error_code,body=body,)@@ -657,6 +883,13 @@ def _provider_error_code_from_flat_body(body: Any) -> str | None:return None+def _error_codes_in_body(body: Any) -> tuple[str, ...]:+ """Every ``code`` and ``type`` an error body carries, in its nested ``error`` section and at its top level."""+ return tuple(+ error_code for section in _error_sections(body) for error_code in (section.get("code"), section.get("type")) if isinstance(error_code, str)+ )++def error_code_from_body_code_first(body: Any) -> str | None:"""Read the error code off an error body, preferring ``code`` over ``type``.@@ -675,15 +908,7 @@ def error_code_from_body_code_first(body: Any) -> str | None:never reaches the classifier. A body carrying only ``code`` reads identicallythrough either precedence."""- if not isinstance(body, dict):- return None- body_dict = cast("dict[str, Any]", body)- error_section = body_dict.get("error")- sections: list[dict[str, Any]] = []- if isinstance(error_section, dict):- sections.append(cast("dict[str, Any]", error_section))- sections.append(body_dict)- for section in sections:+ for section in _error_sections(body):code = section.get("code") or section.get("type")if isinstance(code, str):return code@@ -748,6 +973,34 @@ def _google_provider_error_code_from_details(details: Any) -> str | None:return None+# The type URL of the detail in which a Google error states the wait it asks for+_GOOGLE_RETRY_INFO_TYPE = "type.googleapis.com/google.rpc.RetryInfo"+++def _google_retry_delay_seconds(details: Any) -> float | None:+ """Read the wait a Google error asks for from its ``google.rpc.RetryInfo`` detail, in seconds.++ The Gemini API states the wait for a 429 ``RESOURCE_EXHAUSTED`` in its body rather than in a ``Retry-After``+ header: among the error's ``details``, an entry whose ``@type`` is ``google.rpc.RetryInfo`` carries a+ ``retryDelay``, a protobuf ``Duration`` in its JSON form, seconds with an ``s`` suffix and up to nine fractional+ digits (``"33s"``, ``"33.470052706s"``). The nested ``error`` section is read first, then the top level, as for the+ status. A delay that is not a finite, non-negative number of seconds reads as no delay.+ """+ retry_infos = _google_detail_entries(details, detail_type=_GOOGLE_RETRY_INFO_TYPE)+ if not retry_infos:+ return None+ retry_delay = retry_infos[0].get("retryDelay")+ if not isinstance(retry_delay, str) or not retry_delay.endswith("s"):+ return None+ try:+ seconds = float(retry_delay.removesuffix("s"))+ except ValueError:+ return None+ if math.isfinite(seconds) and seconds >= 0:+ return seconds+ return None++def extract_google_metadata(exc: BaseException, *, validated_type: ValidatedType | None = None, sdk_error_types: SdkErrorTypes | None = None) -> ProviderErrorMetadata:@@ -768,6 +1021,9 @@ def extract_google_metadata(- ``response`` may be ``None`` or any of ``httpx.Response`` /``requests.Response`` / ``ReplayResponse``. We read ``x-goog-request-id``and ``retry-after`` from ``response.headers`` when present.+ - The Gemini API states the wait for a 429 in the body's ``google.rpc.RetryInfo``+ detail rather than in a header, so ``retry_after_seconds`` falls back to its+ ``retryDelay`` when no ``retry-after`` header gave one."""code = getattr(exc, "code", None)status_code = code if isinstance(code, int) else None@@ -781,6 +1037,8 @@ def extract_google_metadata(request_id = request_id_valueretry_after_seconds = parse_retry_after_seconds(headers.get("retry-after"))details = getattr(exc, "details", None)+ if retry_after_seconds is None:+ retry_after_seconds = _google_retry_delay_seconds(details)return ProviderErrorMetadata(provider=ProviderName.GOOGLE,sdk_exception_type=_resolve_sdk_exception_type(exc, status_code=status_code),@@ -832,7 +1090,7 @@ def _build_azure_metadata(response: Any, *, sdk_exception_type: str, message: strequest_id_value = headers.get("x-ms-request-id") or headers.get("apim-request-id") or headers.get("x-request-id")if isinstance(request_id_value, str):request_id = request_id_value- retry_after_seconds = parse_retry_after_seconds(headers.get("retry-after"))+ retry_after_seconds = retry_after_seconds_from_headers(headers=headers)body, provider_error_code = _parse_response_text_body(response)return ProviderErrorMetadata(provider=ProviderName.AZURE,@@ -947,7 +1205,7 @@ def extract_gateway_metadata(exc: BaseException) -> ProviderErrorMetadata:request_id_value = headers.get("x-request-id") or headers.get("x-portkey-trace-id")if isinstance(request_id_value, str):request_id = request_id_value- retry_after_seconds = parse_retry_after_seconds(headers.get("retry-after"))+ retry_after_seconds = retry_after_seconds_from_headers(headers=headers)body: Any = getattr(exc, "body", None)provider_error_code = error_code_from_body_code_first(body)if provider_error_code is None:diff --git a/pipelex/cogt/inference/error_classify.py b/pipelex/cogt/inference/error_classify.pyindex e4877d5e1..21b5ee7d0 100644--- a/pipelex/cogt/inference/error_classify.py+++ b/pipelex/cogt/inference/error_classify.py@@ -8,7 +8,8 @@ provider-specific branching and no SDK imports.Every provider-specific nuance has already been normalized into the metadataby the Extract step (e.g. Google's ``code`` becomes ``status_code``;-quota-vs-rate-limit is decided by the ``is_quota_exhaustion`` property). The+quota-vs-rate-limit is decided by the ``limit_verdict``+property). TheHTTP status code drives classification; status-less errors dispatch on theSDK exception type name.@@ -28,7 +29,7 @@ from mthds.protocol.problem.registry import CoreCodefrom pydantic import BaseModel, model_validatorfrom pipelex.cogt.exceptions import INFERENCE_FAILURE_CODES, InferenceErrorCategory-from pipelex.cogt.inference.error_classification import SDKErrorEnvelope, UserActionKind+from pipelex.cogt.inference.error_classification import ProviderLimitVerdict, SDKErrorEnvelope, UserActionKindfrom pipelex.cogt.inference.service_error_vocabulary import ServiceErrorCodefrom pipelex.errors.error_codes import ErrorCode, error_code_specfrom pipelex.errors.registry.pipelex_codes import PipelexErrorCode@@ -310,19 +311,22 @@ def classify_inference_error(metadata: SDKErrorEnvelope, *, family: InferenceErris_model_not_allowed=True,)- # Quota exhaustion is decided by the provider-aware ``is_quota_exhaustion``- # property and takes precedence over the HTTP status: providers signal it on- # different statuses (OpenAI/Anthropic 429, Mistral/Gateway 402, AWS 400). A- # gateway's 429 is a quota only on the provider's quota code; any other 429- # falls through to the arm below and is a rate limit.- if metadata.is_quota_exhaustion:+ # A spent quota or a rate limit is decided once, by the provider-aware+ # ``limit_verdict``, and takes precedence over the HTTP status: providers signal+ # a spent quota on different statuses (a 429 with a quota code, a 402, Google's+ # 403 for disabled billing, Anthropic's 400 for a spent credit), and a throttle+ # on statuses other than 429 (OpenRouter's 402 for its in-flight budget, Groq's+ # 498, Bedrock's 400 `ServiceQuotaExceededException`). Every rule reads the+ # provider's structure; a 429 that carries no quota signal is a rate limit.+ limit_verdict = metadata.limit_verdict+ if limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED:return ClassificationResult(category=InferenceErrorCategory.CAPACITY,user_action_kind=UserActionKind.CHECK_BILLING,error_code=CoreCode.PROVIDER_QUOTA_EXHAUSTED,)- if status_code == 429:+ if limit_verdict is ProviderLimitVerdict.RATE_LIMITED:return ClassificationResult(category=InferenceErrorCategory.TRANSIENT,user_action_kind=UserActionKind.WAIT_AND_RETRY,diff --git a/pipelex/cogt/inference/transport_retry.py b/pipelex/cogt/inference/transport_retry.pyindex 9774b2c01..bbe9b2cf1 100644--- a/pipelex/cogt/inference/transport_retry.py+++ b/pipelex/cogt/inference/transport_retry.py@@ -11,13 +11,13 @@ uses) rather than hand-rolling a retry loop, and it must only ever wrap a path tretry of its own — layering it on top of a retrying SDK would double-retry."""-import email.utilsfrom collections.abc import Awaitable, Callable-from datetime import UTC, datetimeimport httpxfrom tenacity import AsyncRetrying, RetryCallState, retry_if_exception, stop_after_attempt, wait_random_exponential+from pipelex.cogt.inference.error_classification import retry_after_seconds_from_headers+# Transient HTTP statuses worth retrying in the default (ambiguous-failure-tolerant) mode — the# same floor the provider SDKs apply. Connection-level failures (no HTTP response at all) are# handled separately by the exception-type check.@@ -46,33 +46,18 @@ _MAX_RETRY_AFTER_SECONDS: float = 60.0_exponential_wait = wait_random_exponential(multiplier=1.0, max=_MAX_RETRY_AFTER_SECONDS)-def _parse_retry_after(raw_value: str | None) -> float | None:- """Parse a ``Retry-After`` header value (delta-seconds or HTTP-date) into seconds, or None."""- if raw_value is None:- return None- cleaned = raw_value.strip()- if not cleaned:- return None- if cleaned.isdigit():- return float(cleaned)- try:- parsed_date = email.utils.parsedate_to_datetime(cleaned)- except (TypeError, ValueError):- return None- if parsed_date.tzinfo is None:- # A zoneless or "-0000" HTTP-date parses to a naive datetime; subtracting it from an- # aware "now" would raise TypeError. Treat such a malformed header as unusable.- return None- return max((parsed_date - datetime.now(UTC)).total_seconds(), 0.0)--def _transport_retry_wait(retry_state: RetryCallState) -> float:- """Honor a ``Retry-After`` header when the failure carried one, else full-jitter exponential backoff."""+ """Honor the delay the failure's headers asked for, else full-jitter exponential backoff.++ The delay is read by ``retry_after_seconds_from_headers``, the reader every Extract step shares: ``retry-after-ms``+ above zero first, which Azure OpenAI sends, then ``Retry-After``, in seconds or as an HTTP-date. A header that gives+ no usable delay leaves the wait to the backoff.+ """outcome = retry_state.outcomeif outcome is not None and outcome.failed:exc = outcome.exception()if isinstance(exc, httpx.HTTPStatusError):- retry_after = _parse_retry_after(exc.response.headers.get("Retry-After"))+ retry_after = retry_after_seconds_from_headers(headers=exc.response.headers)if retry_after is not None:return min(retry_after, _MAX_RETRY_AFTER_SECONDS)return _exponential_wait(retry_state)diff --git a/subject_grants.toml b/subject_grants.tomlindex 8e5c897dc..2ceb3f645 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -1608,31 +1608,31 @@ rationale = "Factory from source: builds the result from the inference_model; siparam = "response"rationale = "Extracts/derives from the response; single operand."-["pipelex/cogt/inference/error_classification.py::_google_provider_error_code_from_details"]-param = "details"-rationale = "Extracts/derives from the details; single operand."+["pipelex/cogt/inference/error_classification.py::_error_codes_in_body"]+param = "body"+rationale = "Lists the codes and types of one error body, the operand the OpenAI and gateway spent-quota rule reads; single operand."-["pipelex/cogt/inference/error_classification.py::_is_content_policy_violation"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."+["pipelex/cogt/inference/error_classification.py::_error_sections"]+param = "body"+rationale = "Lists the sections of one error body, the operand its readers error_code_from_body_code_first and _google_detail_entries pass straight through; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_anthropic"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."+["pipelex/cogt/inference/error_classification.py::_google_detail_entries"]+param = "body"+rationale = "Lists the entries of one Google error body's details of one type, the operand its readers _google_retry_delay_seconds and _is_quota_exhaustion_google pass straight through; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_aws"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."+["pipelex/cogt/inference/error_classification.py::_google_error_info_metadata"]+param = "entry"+rationale = "Reads the metadata map of one google.rpc.ErrorInfo entry; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_google"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."+["pipelex/cogt/inference/error_classification.py::_google_provider_error_code_from_details"]+param = "details"+rationale = "Extracts/derives from the details; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_mistral"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."+["pipelex/cogt/inference/error_classification.py::_google_retry_delay_seconds"]+param = "details"+rationale = "Extracts the retry delay from a Google error's details, as its sibling _google_provider_error_code_from_details reads the status; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_openai"]+["pipelex/cogt/inference/error_classification.py::_is_content_policy_violation"]param = "error_message"rationale = "Predicate on the error_message; single operand."@@ -1772,10 +1772,6 @@ rationale = "Keyed accessor by the llm_handle; single operand."param = "exc"rationale = "Positional-Callable tenacity retry predicate: exc is the single operand tenacity passes."-["pipelex/cogt/inference/transport_retry.py::_parse_retry_after"]-param = "raw_value"-rationale = "Parses the raw_value; single operand."-["pipelex/cogt/inference/transport_retry.py::_transport_retry_wait"]param = "retry_state"rationale = "Positional-Callable tenacity wait strategy: retry_state is the single operand tenacity passes."diff --git a/tests/unit/pipelex/cogt/inference/test_classify_inference_error.py b/tests/unit/pipelex/cogt/inference/test_classify_inference_error.pyindex 18c97ecb1..db7ea2f48 100644--- a/tests/unit/pipelex/cogt/inference/test_classify_inference_error.py+++ b/tests/unit/pipelex/cogt/inference/test_classify_inference_error.py@@ -57,16 +57,18 @@ class _TestCases:CoreCode.PROVIDER_RATE_LIMITED,False,),+ # A 429 whose words name a quota but which carries no quota code is a rate limit; the quota code's verdict is+ # pinned in `test_classify_extra_signals`, which carries the provider's code(- "quota_429",+ "quota_wording_429_without_a_quota_code",ProviderName.OPENAI,"RateLimitError","You exceeded your current quota",429,_LLM,- InferenceErrorCategory.CAPACITY,- UserActionKind.CHECK_BILLING,- CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ InferenceErrorCategory.TRANSIENT,+ UserActionKind.WAIT_AND_RETRY,+ CoreCode.PROVIDER_RATE_LIMITED,False,),(@@ -105,16 +107,18 @@ class _TestCases:CoreCode.PROVIDER_CREDENTIAL_REFUSED,False,),+ # Anthropic's credit sentence is a spent credit on a 400 `invalid_request_error` alone; a 403 is a credential+ # refusal whatever it says(- "permission_denied_403_quota_anthropic",+ "permission_denied_403_with_a_credit_sentence_anthropic",ProviderName.ANTHROPIC,"PermissionDeniedError","Your credit balance is too low",403,_LLM,- InferenceErrorCategory.CAPACITY,- UserActionKind.CHECK_BILLING,- CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ InferenceErrorCategory.CONFIGURATION,+ UserActionKind.CHECK_CREDENTIALS,+ CoreCode.PROVIDER_CREDENTIAL_REFUSED,False,),(@@ -629,14 +633,25 @@ class TestClassifyInferenceError:CoreCode.RUNTIME_UNEXPECTED,),(- "bedrock_service_quota_via_error_code",+ "openai_quota_code_on_429",+ ProviderName.OPENAI,+ "RateLimitError",+ "You exceeded your current quota",+ 429,+ "insufficient_quota",+ InferenceErrorCategory.CAPACITY,+ CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ ),+ # AWS documents it as resubmittable later: a throttle on its 400, read as a rate limit+ (+ "bedrock_service_quota_exceeded_is_a_throttle",ProviderName.BEDROCK,"ClientError","An error occurred",400,"ServiceQuotaExceededException",- InferenceErrorCategory.CAPACITY,- CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ InferenceErrorCategory.TRANSIENT,+ CoreCode.PROVIDER_RATE_LIMITED,),],)diff --git a/tests/unit/pipelex/cogt/inference/test_direct_provider_quota_classification.py b/tests/unit/pipelex/cogt/inference/test_direct_provider_quota_classification.pynew file mode 100644index 000000000..a8e9a521d--- /dev/null+++ b/tests/unit/pipelex/cogt/inference/test_direct_provider_quota_classification.py@@ -0,0 +1,404 @@+from __future__ import annotations++import json+from typing import TYPE_CHECKING, Any, cast++import anthropic+import botocore.session+import httpx+import openai+import pytest+from anthropic.lib.bedrock import AnthropicBedrock+from botocore.exceptions import ClientError+from google.genai import errors as genai_errors+from mistralai.client.errors import SDKError+from mthds.protocol.problem.registry import CoreCode++from pipelex.cogt.exceptions import InferenceErrorCategory+from pipelex.cogt.inference.error_classification import (+ RUNTIME_CLASSIFIED_ERROR_CODES,+ ProviderErrorMetadata,+ ProviderLimitVerdict,+ UserActionKind,+ extract_anthropic_metadata,+ extract_azure_metadata,+ extract_bedrock_metadata,+ extract_google_metadata,+ extract_mistral_metadata,+ extract_openai_metadata,+)+from pipelex.cogt.inference.error_classify import ClassificationResult, InferenceErrorFamily, classify_inference_error+from pipelex.cogt.inference.error_render import render_inference_error+from pipelex.errors.problem import ProblemView+from pipelex.errors.registry.pipelex_codes import PipelexErrorCode+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestData++if TYPE_CHECKING:+ from pipelex.errors.error_codes import ErrorCode++_TestData = DirectProviderQuotaTestData++# Every error Bedrock's runtime API declares, with its status, from botocore's own model of the service, so a code AWS+# adds is covered without editing this list+_BEDROCK_RUNTIME_ERRORS: list[tuple[str, int]] = sorted(+ (shape.name, shape.metadata["error"]["httpStatusCode"])+ for shape in botocore.session.get_session().get_service_model("bedrock-runtime").error_shapes+)+++def _response(*, url: str, status_code: int, content: str, headers: dict[str, str] | None = None) -> httpx.Response:+ request = httpx.Request("POST", url)+ return httpx.Response(+ status_code=status_code,+ request=request,+ content=content.encode(),+ headers={"content-type": "application/json", **(headers or {})},+ )+++def _as_openai_raises_it(*, status_code: int, body: dict[str, Any], headers: dict[str, str] | None = None) -> BaseException:+ """Build the exception through the OpenAI client's own factory, which keeps the body's ``error`` section."""+ response = _response(url="https://api.openai.com/v1/chat/completions", status_code=status_code, content=json.dumps(body), headers=headers)+ client = openai.OpenAI(api_key="unused-in-this-test")+ return client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _as_anthropic_raises_it(*, status_code: int, body: dict[str, Any], headers: dict[str, str] | None = None) -> BaseException:+ """Build the exception through the Anthropic client's own factory, which keeps the whole body."""+ response = _response(url="https://api.anthropic.com/v1/messages", status_code=status_code, content=json.dumps(body), headers=headers)+ client = anthropic.Anthropic(api_key="unused-in-this-test")+ return client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _as_google_raises_it(*, status_code: int, body: dict[str, Any]) -> BaseException:+ """Build the exception as google-genai raises a client error, the response's JSON as its details."""+ response = _response(+ url="https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent",+ status_code=status_code,+ content=json.dumps(body),+ )+ return genai_errors.ClientError(status_code, body, response)+++def _as_mistral_raises_it(*, status_code: int, body: str) -> BaseException:+ """Build the exception as the Mistral SDK raises an error answer, with the response's text as its body."""+ response = _response(url="https://api.mistral.ai/v1/chat/completions", status_code=status_code, content=body)+ return SDKError("API error occurred", response, body)+++def _as_botocore_raises_it(*, status_code: int, error_code: str, message: str) -> BaseException:+ """Build the ``ClientError`` botocore raises from Bedrock's parsed answer to ``Converse``."""+ error_response: dict[str, Any] = {+ "Error": {"Code": error_code, "Message": message},+ "ResponseMetadata": {"RequestId": "req-bedrock-1", "HTTPStatusCode": status_code, "HTTPHeaders": {}},+ }+ return ClientError(error_response=cast("Any", error_response), operation_name="Converse")+++def _assert_classified(*, metadata: ProviderErrorMetadata, family: InferenceErrorFamily, expected_error_code: ErrorCode) -> ClassificationResult:+ """Classify the metadata and assert the verdict the code implies: a spent quota or a rate limit, or neither."""+ classification = classify_inference_error(metadata, family=family)+ assert classification.error_code == expected_error_code+ match expected_error_code:+ case CoreCode.PROVIDER_QUOTA_EXHAUSTED:+ assert metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED+ assert classification.category == InferenceErrorCategory.CAPACITY+ assert classification.user_action_kind == UserActionKind.CHECK_BILLING+ case CoreCode.PROVIDER_RATE_LIMITED:+ assert metadata.limit_verdict is ProviderLimitVerdict.RATE_LIMITED+ assert classification.category == InferenceErrorCategory.TRANSIENT+ assert classification.category.is_retryable is True+ assert classification.user_action_kind == UserActionKind.WAIT_AND_RETRY+ case CoreCode() | PipelexErrorCode():+ assert metadata.limit_verdict is ProviderLimitVerdict.NEITHER+ return classification+++class TestDirectProviderQuotaClassification:+ @pytest.mark.parametrize(+ ("_topic", "status_code", "body", "headers", "expected_error_code", "expected_retry_after"),+ [+ ("long_standing_insufficient_quota", 429, _TestData.OPENAI_INSUFFICIENT_QUOTA_BODY, None, CoreCode.PROVIDER_QUOTA_EXHAUSTED, None),+ # The billing code is in `code` beside the broad `type`, which the Extract step keeps: the rule reads both+ (+ "billing_code_beside_the_broad_type",+ 429,+ _TestData.OPENAI_CREDIT_BALANCE_EXHAUSTED_BODY,+ None,+ CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ None,+ ),+ ("billing_code_with_no_type", 429, _TestData.OPENAI_PROJECT_SPEND_LIMIT_NO_TYPE_BODY, None, CoreCode.PROVIDER_QUOTA_EXHAUSTED, None),+ ("per_minute_rate_limit", 429, _TestData.OPENAI_RATE_LIMIT_BODY, {"retry-after": "2"}, CoreCode.PROVIDER_RATE_LIMITED, 2),+ # The words say "exceeded your current quota", which the former word list read as a spent quota+ ("gemini_throttle_wording_with_no_code", 429, _TestData.OPENAI_DIALECT_GEMINI_THROTTLE_BODY, None, CoreCode.PROVIDER_RATE_LIMITED, None),+ (+ "openrouter_in_flight_budget_with_its_wait",+ 402,+ _TestData.OPENROUTER_IN_FLIGHT_BUDGET_BODY,+ {"retry-after": _TestData.OPENROUTER_RETRY_AFTER},+ CoreCode.PROVIDER_RATE_LIMITED,+ 2,+ ),+ # The limit source decides; the header, when it is missing or unreadable, only leaves the wait to the backoff+ (+ "openrouter_in_flight_budget_with_no_wait",+ 402,+ _TestData.OPENROUTER_IN_FLIGHT_BUDGET_BODY,+ None,+ CoreCode.PROVIDER_RATE_LIMITED,+ None,+ ),+ (+ "openrouter_in_flight_budget_with_an_unreadable_wait",+ 402,+ _TestData.OPENROUTER_IN_FLIGHT_BUDGET_BODY,+ {"retry-after": "soon"},+ CoreCode.PROVIDER_RATE_LIMITED,+ None,+ ),+ (+ "openrouter_insufficient_credits",+ 402,+ _TestData.OPENROUTER_INSUFFICIENT_CREDITS_BODY,+ {"retry-after": _TestData.OPENROUTER_RETRY_AFTER},+ CoreCode.PROVIDER_QUOTA_EXHAUSTED,+ 2,+ ),+ # Groq's own status for its flex tier at capacity, which it says to retry with backoff+ ("groq_flex_capacity_498", 498, _TestData.GROQ_FLEX_CAPACITY_BODY, None, CoreCode.PROVIDER_RATE_LIMITED, None),+ # Azure OpenAI's throttle, its delay read from the milliseconds before the rounded seconds+ (+ "azure_throttle_with_its_delay_in_milliseconds",+ 429,+ _TestData.OPENAI_RATE_LIMIT_BODY,+ _TestData.AZURE_RETRY_AFTER_HEADERS,+ CoreCode.PROVIDER_RATE_LIMITED,+ _TestData.AZURE_RETRY_AFTER_MS_SECONDS,+ ),+ ],+ )+ def test_openai_dialect(+ self,+ _topic: str,+ status_code: int,+ body: dict[str, Any],+ headers: dict[str, str] | None,+ expected_error_code: ErrorCode,+ expected_retry_after: float | None,+ ) -> None:+ """An OpenAI-dialect 429 is a spent quota on one of OpenAI's billing codes or types alone; a 402 is one unless OpenRouter says it waits."""+ metadata = extract_openai_metadata(_as_openai_raises_it(status_code=status_code, body=body, headers=headers))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)+ assert metadata.retry_after_seconds == expected_retry_after++ @pytest.mark.parametrize(+ ("_topic", "status_code", "body", "expected_error_code"),+ [+ ("monthly_spend_cap_429", 429, _TestData.ANTHROPIC_SPEND_CAP_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("billing_error_402", 402, _TestData.ANTHROPIC_BILLING_ERROR_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("credit_balance_400", 400, _TestData.ANTHROPIC_CREDIT_BALANCE_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("self_set_spend_limit_400", 400, _TestData.ANTHROPIC_SELF_SET_LIMIT_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("workspace_spend_limit_400", 400, _TestData.ANTHROPIC_WORKSPACE_LIMIT_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("per_minute_rate_limit_429", 429, _TestData.ANTHROPIC_RATE_LIMIT_BODY, CoreCode.PROVIDER_RATE_LIMITED),+ ("malformed_request_400", 400, _TestData.ANTHROPIC_INVALID_REQUEST_BODY, CoreCode.RUNTIME_UNEXPECTED),+ # The credit sentence decides a 400 `invalid_request_error` alone: on a 403 it is a credential refusal+ ("credit_sentence_on_a_403", 403, _TestData.ANTHROPIC_CREDIT_BALANCE_BODY, CoreCode.PROVIDER_CREDENTIAL_REFUSED),+ ],+ )+ def test_anthropic(self, _topic: str, status_code: int, body: dict[str, Any], expected_error_code: ErrorCode) -> None:+ """Anthropic's quota is its 402 or `billing_error`, its spend cap's code on a 429, or one of its spend sentences on a 400."""+ metadata = extract_anthropic_metadata(_as_anthropic_raises_it(status_code=status_code, body=body))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)++ def test_anthropic_rate_limit_carries_its_wait(self) -> None:+ """A plain Anthropic 429 is a rate limit, and its `retry-after` reaches the metadata."""+ exc = _as_anthropic_raises_it(status_code=429, body=_TestData.ANTHROPIC_RATE_LIMIT_BODY, headers={"retry-after": "12"})++ metadata = extract_anthropic_metadata(exc)++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=CoreCode.PROVIDER_RATE_LIMITED)+ assert metadata.retry_after_seconds == 12++ def test_google_per_minute_throttle_is_a_rate_limit_waited_for_the_delay_it_asks(self) -> None:+ """Gemini's throttle, verbatim, is a rate limit whatever its words, and its `RetryInfo` wait reaches the problem."""+ exc = _as_google_raises_it(status_code=429, body=_TestData.GOOGLE_PER_MINUTE_THROTTLE_BODY)++ metadata = extract_google_metadata(exc)+ classification = _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=CoreCode.PROVIDER_RATE_LIMITED)++ assert metadata.provider_error_code == "RESOURCE_EXHAUSTED"+ assert "quota" in metadata.message+ assert metadata.retry_after_seconds == _TestData.GOOGLE_RETRY_DELAY_SECONDS++ rendered = render_inference_error(+ metadata=metadata,+ classification=classification,+ family=InferenceErrorFamily.LLM,+ model_desc="gemini-2.5-flash → Model[gemini-2.5-flash]",+ model_handle="gemini-2.5-flash",+ )+ # As the worker's error enrichment does, which names the model and the backend the problem's sentence reads+ rendered.fill_model_and_provider("gemini-2.5-flash", backend_name="google")+ problem = rendered.to_problem(view=ProblemView.OPERATOR)++ assert problem.code == CoreCode.PROVIDER_RATE_LIMITED+ assert problem.is_retryable is True+ assert problem.retry_after == _TestData.GOOGLE_RETRY_DELAY_SECONDS++ def test_google_spent_prepay_balance_is_a_spent_quota(self) -> None:+ """Gemini answers a Prepay balance at zero with a 402, which its status alone makes a spent quota."""+ metadata = extract_google_metadata(_as_google_raises_it(status_code=402, body=_TestData.GOOGLE_PREPAY_DEPLETED_BODY))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=CoreCode.PROVIDER_QUOTA_EXHAUSTED)+ assert metadata.retry_after_seconds is None++ @pytest.mark.parametrize(+ ("_topic", "status_code", "body", "expected_error_code"),+ [+ # Billing disabled on the project, read from the ErrorInfo reason whatever the message says+ ("billing_disabled_403", 403, _TestData.GOOGLE_BILLING_DISABLED_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ # Another reason on the same status is a refused credential, as any 403+ ("service_disabled_403", 403, _TestData.GOOGLE_SERVICE_DISABLED_BODY, CoreCode.PROVIDER_CREDENTIAL_REFUSED),+ # A quota whose limit is zero, which no wait cures+ ("zero_limit_quota_429", 429, _TestData.GOOGLE_ZERO_LIMIT_QUOTA_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ("zero_quota_value_429", 429, _TestData.GOOGLE_ZERO_QUOTA_VALUE_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ # The same quota at a limit above zero, which a wait cures+ ("nonzero_limit_quota_429", 429, _TestData.GOOGLE_NONZERO_LIMIT_QUOTA_BODY, CoreCode.PROVIDER_RATE_LIMITED),+ ],+ )+ def test_google_details(self, _topic: str, status_code: int, body: dict[str, Any], expected_error_code: ErrorCode) -> None:+ """Google's billing and zero-limit refusals are read from the `google.rpc` details of its error, never its words."""+ metadata = extract_google_metadata(_as_google_raises_it(status_code=status_code, body=body))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)++ @pytest.mark.parametrize(+ ("_topic", "status_code", "body", "expected_error_code"),+ [+ ("capacity_429", 429, _TestData.MISTRAL_CAPACITY_BODY, CoreCode.PROVIDER_RATE_LIMITED),+ # The words name a quota, which the former word list read as a spent one+ ("quota_wording_429", 429, _TestData.MISTRAL_QUOTA_WORDING_BODY, CoreCode.PROVIDER_RATE_LIMITED),+ ("balance_depleted_402", 402, _TestData.MISTRAL_BALANCE_DEPLETED_BODY, CoreCode.PROVIDER_QUOTA_EXHAUSTED),+ ],+ )+ def test_mistral(self, _topic: str, status_code: int, body: str, expected_error_code: ErrorCode) -> None:+ """Mistral's spent balance is its 402, and every 429 is a rate limit."""+ metadata = extract_mistral_metadata(_as_mistral_raises_it(status_code=status_code, body=body))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)++ @pytest.mark.parametrize(+ ("_topic", "status_code", "error_code", "message", "expected_error_code"),+ [+ ("daily_tokens_throttle", 429, "ThrottlingException", _TestData.BEDROCK_DAILY_TOKENS_THROTTLE_MESSAGE, CoreCode.PROVIDER_RATE_LIMITED),+ # AWS documents it as resubmittable later, so it is a throttle on its 400+ ("service_quota_exceeded", 400, "ServiceQuotaExceededException", _TestData.BEDROCK_SERVICE_QUOTA_MESSAGE, CoreCode.PROVIDER_RATE_LIMITED),+ # An input over the model's window, which Bedrock words per model and signals no other way+ ("nova_input_overflow", 400, "ValidationException", _TestData.BEDROCK_NOVA_INPUT_OVERFLOW_MESSAGE, CoreCode.MODEL_CONTEXT_EXCEEDED),+ ("titan_input_overflow", 400, "ValidationException", _TestData.BEDROCK_TITAN_INPUT_OVERFLOW_MESSAGE, CoreCode.MODEL_CONTEXT_EXCEEDED),+ (+ "openai_engine_input_overflow",+ 400,+ "ValidationException",+ _TestData.BEDROCK_OPENAI_ENGINE_INPUT_OVERFLOW_MESSAGE,+ CoreCode.MODEL_CONTEXT_EXCEEDED,+ ),+ ],+ )+ def test_bedrock(self, _topic: str, status_code: int, error_code: str, message: str, expected_error_code: ErrorCode) -> None:+ """Bedrock has no balance to spend, so none of its answers is a spent quota."""+ metadata = extract_bedrock_metadata(_as_botocore_raises_it(status_code=status_code, error_code=error_code, message=message))++ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)++ @pytest.mark.parametrize(+ ("_topic", "status_code", "error_type_header", "message", "expected_error_code", "expected_provider_error_code"),+ [+ # AWS documents it as resubmittable later: a throttle on its 400, read as the Bedrock worker reads it+ (+ "service_quota_exceeded",+ 400,+ _TestData.BEDROCK_SERVICE_QUOTA_ERROR_TYPE_HEADER,+ _TestData.BEDROCK_SERVICE_QUOTA_MESSAGE,+ CoreCode.PROVIDER_RATE_LIMITED,+ "ServiceQuotaExceededException",+ ),+ (+ "throttling",+ 429,+ _TestData.BEDROCK_THROTTLING_ERROR_TYPE_HEADER,+ _TestData.BEDROCK_DAILY_TOKENS_THROTTLE_MESSAGE,+ CoreCode.PROVIDER_RATE_LIMITED,+ "ThrottlingException",+ ),+ ],+ )+ def test_claude_on_bedrock(+ self,+ _topic: str,+ status_code: int,+ error_type_header: str,+ message: str,+ expected_error_code: ErrorCode,+ expected_provider_error_code: str,+ ) -> None:+ """Claude on Bedrock, through the Anthropic SDK, carries Bedrock's error code in a header, read as on Bedrock."""+ response = _response(+ url="https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-sonnet-4-5/invoke",+ status_code=status_code,+ content=json.dumps({"message": message}),+ headers={"x-amzn-errortype": error_type_header},+ )+ client = AnthropicBedrock(aws_region="us-east-1", api_key="unused-in-this-test")+ exc = client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ metadata = extract_anthropic_metadata(exc)++ assert metadata.provider_error_code == expected_provider_error_code+ _assert_classified(metadata=metadata, family=InferenceErrorFamily.LLM, expected_error_code=expected_error_code)++ def test_azure_rest_reads_the_delay_in_milliseconds(self) -> None:+ """Azure OpenAI's REST answer, which no SDK wraps, carries its delay in milliseconds before its rounded seconds."""+ response = _response(+ url="https://test.openai.azure.com/openai/deployments/gpt-image-1/images/generations",+ status_code=429,+ content=json.dumps({"error": {"code": "429", "message": "Requests to the deployment have exceeded the rate limit."}}),+ headers=_TestData.AZURE_RETRY_AFTER_HEADERS,+ )+ with pytest.raises(httpx.HTTPStatusError) as exc_info:+ response.raise_for_status()++ metadata = extract_azure_metadata(exc_info.value)++ assert metadata.retry_after_seconds == _TestData.AZURE_RETRY_AFTER_MS_SECONDS+ _assert_classified(metadata=metadata, family=InferenceErrorFamily.IMG_GEN, expected_error_code=CoreCode.PROVIDER_RATE_LIMITED)++ @pytest.mark.parametrize(("aws_error_code", "status_code"), _BEDROCK_RUNTIME_ERRORS)+ def test_claude_on_bedrock_reads_each_aws_code_as_the_bedrock_worker_does(self, aws_error_code: str, status_code: int) -> None:+ """An AWS code read from the header reaches only the lookups the Bedrock worker already feeds it to, with the same verdict.++ The header puts no code in front of the service error vocabulary, the content-policy check or the context-window+ check that Bedrock's own SDK does not already put there, and the runtime classifies none of them itself.+ """+ message = "The request was refused."+ response = _response(+ url="https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-sonnet-4-5/invoke",+ status_code=status_code,+ content=json.dumps({"message": message}),+ headers={"x-amzn-errortype": f"{aws_error_code}:http://internal.amazon.com/coral/com.amazon.bedrock/"},+ )+ client = AnthropicBedrock(aws_region="us-east-1", api_key="unused-in-this-test")+ claude_metadata = extract_anthropic_metadata(client._make_status_error_from_response(response)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+ bedrock_metadata = extract_bedrock_metadata(_as_botocore_raises_it(status_code=status_code, error_code=aws_error_code, message=message))++ assert claude_metadata.provider_error_code == bedrock_metadata.provider_error_code == aws_error_code+ assert aws_error_code not in RUNTIME_CLASSIFIED_ERROR_CODES+ assert claude_metadata.is_content_policy_violation is False+ assert claude_metadata.is_context_window_exceeded is False+ assert claude_metadata.is_model_not_allowed is False+ claude_verdict = classify_inference_error(claude_metadata, family=InferenceErrorFamily.LLM)+ bedrock_verdict = classify_inference_error(bedrock_metadata, family=InferenceErrorFamily.LLM)+ assert (claude_verdict.category, claude_verdict.error_code) == (bedrock_verdict.category, bedrock_verdict.error_code)diff --git a/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.py b/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.pyindex 554d596d4..78f9943ea 100644--- a/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.py+++ b/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.py@@ -5,13 +5,14 @@ import jsonfrom typing import TYPE_CHECKING, Anyimport httpx+import openaiimport pytestfrom mthds.protocol.problem.registry import CoreCodefrom portkey_ai import Portkeyfrom portkey_ai._vendor import openai as portkey_vendored_openaifrom pipelex.cogt.exceptions import InferenceErrorCategory-from pipelex.cogt.inference.error_classification import UserActionKind, extract_gateway_metadata+from pipelex.cogt.inference.error_classification import ProviderLimitVerdict, UserActionKind, extract_gateway_metadata, extract_openai_metadatafrom pipelex.cogt.inference.error_classify import InferenceErrorFamily, classify_inference_errorfrom pipelex.cogt.inference.error_render import render_inference_errorfrom pipelex.cogt.inference.provider_name import ProviderName@@ -24,31 +25,31 @@ if TYPE_CHECKING:_ORIGIN = "https://gateway.example.com"-def _gateway_response(*, body: dict[str, Any], retry_after: str) -> tuple[httpx.Request, httpx.Response]:+def _gateway_response(*, body: dict[str, Any], retry_after: str, extra_headers: dict[str, str] | None = None) -> tuple[httpx.Request, httpx.Response]:request = httpx.Request("POST", f"{_ORIGIN}/v1/images/generations")response = httpx.Response(status_code=429,request=request,content=json.dumps(body).encode(),- headers={"content-type": "application/json", "retry-after": retry_after},+ headers={"content-type": "application/json", "retry-after": retry_after, **(extra_headers or {})},)return request, response-def _as_portkey_raises_it(*, body: dict[str, Any], retry_after: str) -> BaseException:+def _as_portkey_raises_it(*, body: dict[str, Any], retry_after: str, extra_headers: dict[str, str] | None = None) -> BaseException:"""Build the exception through Portkey's own factory, which keeps only the message string on ``body``."""- request, response = _gateway_response(body=body, retry_after=retry_after)+ request, response = _gateway_response(body=body, retry_after=retry_after, extra_headers=extra_headers)client = Portkey(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1")return client._make_status_error_from_response(request=request, response=response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]-def _as_portkey_s_vendored_openai_raises_it(*, body: dict[str, Any], retry_after: str) -> BaseException:+def _as_portkey_s_vendored_openai_raises_it(*, body: dict[str, Any], retry_after: str, extra_headers: dict[str, str] | None = None) -> BaseException:"""Build the exception through the OpenAI client Portkey vendors, the hop a gateway's image call takes.Its factory renders the whole body into the message and keeps the ``error`` section on ``body``, which is theshape of the refusal's diagnostics on the hosted API."""- _, response = _gateway_response(body=body, retry_after=retry_after)+ _, response = _gateway_response(body=body, retry_after=retry_after, extra_headers=extra_headers)client = portkey_vendored_openai.OpenAI(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1")return client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]@@ -78,7 +79,7 @@ class TestGatewayThrottleClassification:assert metadata.provider_error_code == GatewayThrottleTestData.AZURE_RATE_LIMIT_CODEassert metadata.retry_after_seconds == 57assert "quota" in metadata.message- assert metadata.is_quota_exhaustion is False+ assert metadata.limit_verdict is ProviderLimitVerdict.RATE_LIMITEDassert classification.error_code == CoreCode.PROVIDER_RATE_LIMITEDassert classification.category == InferenceErrorCategory.TRANSIENTassert classification.category.is_retryable is True@@ -99,6 +100,42 @@ class TestGatewayThrottleClassification:assert problem.is_retryable is Trueassert problem.retry_after == 57+ @pytest.mark.parametrize("raise_it", _HOPS)+ def test_azure_s_delay_in_milliseconds_is_read_before_its_seconds(self, raise_it: Callable[..., BaseException]) -> None:+ """Azure sends its delay twice, in milliseconds and in rounded seconds, and the milliseconds are the wait read."""+ exc = raise_it(+ body=GatewayThrottleTestData.AZURE_RATE_LIMIT_BODY,+ retry_after=GatewayThrottleTestData.AZURE_RETRY_AFTER,+ extra_headers={"retry-after-ms": GatewayThrottleTestData.AZURE_RETRY_AFTER_MS},+ )++ metadata = extract_gateway_metadata(exc)++ assert metadata.retry_after_seconds == GatewayThrottleTestData.AZURE_RETRY_AFTER_MS_SECONDS+ assert classify_inference_error(metadata, family=InferenceErrorFamily.IMG_GEN).error_code == CoreCode.PROVIDER_RATE_LIMITED++ @pytest.mark.parametrize("raise_it", _HOPS)+ def test_the_broad_quota_type_beside_another_code_reads_as_it_does_directly(self, raise_it: Callable[..., BaseException]) -> None:+ """OpenAI's broad quota `type` beside a code the set does not hold is a spent quota through a gateway, as it is directly.++ The gateway's Extract step keeps the code and the OpenAI one the type, so the rule reads both off the body. The+ code is built: one OpenAI's guide does not name.+ """+ body = copy.deepcopy(GatewayThrottleTestData.AZURE_RATE_LIMIT_BODY)+ body["error"]["type"] = GatewayThrottleTestData.OPENAI_QUOTA_CODE+ body["error"]["code"] = "quota_code_not_in_the_set"+ _, response = _gateway_response(body=body, retry_after=GatewayThrottleTestData.AZURE_RETRY_AFTER)+ direct_client = openai.OpenAI(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1")+ direct_metadata = extract_openai_metadata(direct_client._make_status_error_from_response(response)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ gateway_metadata = extract_gateway_metadata(raise_it(body=body, retry_after=GatewayThrottleTestData.AZURE_RETRY_AFTER))++ assert gateway_metadata.provider_error_code == "quota_code_not_in_the_set"+ assert direct_metadata.provider_error_code == GatewayThrottleTestData.OPENAI_QUOTA_CODE+ for metadata in (gateway_metadata, direct_metadata):+ assert metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED+ assert classify_inference_error(metadata, family=InferenceErrorFamily.IMG_GEN).error_code == CoreCode.PROVIDER_QUOTA_EXHAUSTED+@pytest.mark.parametrize(("error_type", "error_code"),[@@ -125,7 +162,7 @@ class TestGatewayThrottleClassification:classification = classify_inference_error(metadata, family=InferenceErrorFamily.IMG_GEN)assert metadata.provider_error_code == error_code- assert metadata.is_quota_exhaustion is True+ assert metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTEDassert classification.error_code == CoreCode.PROVIDER_QUOTA_EXHAUSTEDassert classification.category == InferenceErrorCategory.CAPACITYassert classification.category.is_retryable is Falsediff --git a/tests/unit/pipelex/cogt/inference/test_provider_error_metadata.py b/tests/unit/pipelex/cogt/inference/test_provider_error_metadata.pyindex 72f90c939..b01c65740 100644--- a/tests/unit/pipelex/cogt/inference/test_provider_error_metadata.py+++ b/tests/unit/pipelex/cogt/inference/test_provider_error_metadata.py@@ -14,7 +14,7 @@ import pytestfrom mthds.protocol.problem.registry import CoreCodefrom pydantic import ValidationError-from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, extract_azure_metadata+from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, ProviderLimitVerdict, extract_azure_metadatafrom pipelex.cogt.inference.error_classify import InferenceErrorFamily, classify_inference_errorfrom pipelex.cogt.inference.provider_name import ProviderName@@ -94,29 +94,36 @@ class TestProviderErrorMetadata:@pytest.mark.parametrize(("_topic", "provider", "message", "status_code", "provider_error_code", "expected"),[- ("openai_quota", ProviderName.OPENAI, "You exceeded your current quota", None, None, True),- ("openai_rate_limit", ProviderName.OPENAI, "Rate limit reached", None, None, False),- ("anthropic_credit_balance", ProviderName.ANTHROPIC, "Your credit balance is too low", None, None, True),+ ("openai_quota_code_429", ProviderName.OPENAI, "You exceeded your current quota", 429, "insufficient_quota", True),+ # The words decide nothing: a 429 with no quota code is a rate limit+ ("openai_quota_wording_429_no_code", ProviderName.OPENAI, "You exceeded your current quota", 429, None, False),+ ("openai_rate_limit", ProviderName.OPENAI, "Rate limit reached", 429, None, False),+ ("anthropic_billing_error_402", ProviderName.ANTHROPIC, "Error code: 402", 402, "billing_error", True),+ # Anthropic's spend sentences are read from the body's message alone, which this metadata does not carry+ ("anthropic_credit_wording_no_body", ProviderName.ANTHROPIC, "Your credit balance is too low", 400, "invalid_request_error", False),+ ("google_402_is_quota", ProviderName.GOOGLE, "payment required", 402, None, True),+ ("google_429_quota_wording", ProviderName.GOOGLE, "Resource has been exhausted (e.g. check quota)", 429, "RESOURCE_EXHAUSTED", False),("mistral_402_is_quota", ProviderName.MISTRAL, "payment required", 402, None, True),- ("mistral_429_quota_pattern", ProviderName.MISTRAL, "quota exceeded", 429, None, True),+ ("mistral_429_quota_wording", ProviderName.MISTRAL, "quota exceeded", 429, None, False),("mistral_429_plain_rate_limit", ProviderName.MISTRAL, "too many requests", 429, None, False),("gateway_402_is_quota", ProviderName.GATEWAY, "payment required", 402, None, True),("azure_never_quota", ProviderName.AZURE, "quota exceeded", 429, None, False),+ # AWS documents it as resubmittable later: a throttle on its 400, not a spent quota(- "bedrock_quota_by_aws_code_no_message_hint",+ "bedrock_service_quota_exceeded_is_a_throttle",ProviderName.BEDROCK,"An error occurred",400,"ServiceQuotaExceededException",- True,+ False,),(- "bedrock_quota_message_keyword",+ "bedrock_quota_wording_no_code",ProviderName.BEDROCK,"Service quota exceeded for this model",400,None,- True,+ False,),("bedrock_plain_validation_not_quota",@@ -128,7 +135,7 @@ class TestProviderErrorMetadata:),],)- def test_is_quota_exhaustion_dispatches_on_provider(+ def test_spent_quota_verdict_dispatches_on_provider(self,_topic: str,provider: ProviderName,@@ -145,7 +152,7 @@ class TestProviderErrorMetadata:provider_error_code=provider_error_code,)- assert metadata.is_quota_exhaustion is expected+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) is expected@pytest.mark.parametrize(("_topic", "message", "provider_error_code", "expected"),@@ -177,6 +184,17 @@ class TestProviderErrorMetadata:("anthropic", "prompt is too long: 208310 tokens > 200000 maximum", None, None, True),("anthropic_with_max_tokens", "input length and `max_tokens` exceed context limit: 197202 + 8192 > 200000", None, None, True),("bedrock", "Input is too long for requested model.", None, None, True),+ ("bedrock_nova", "Input Tokens Exceeded: Number of input tokens exceeds maximum length.", "ValidationException", None, True),+ ("bedrock_titan", "Too many input tokens. Max input tokens: 8192, request input token count: 9000", "ValidationException", None, True),+ ("bedrock_openai_engine", "prompt tokens (300000) exceed model maximum (262144)", "ValidationException", None, True),+ # Nova's and Titan's words are read on an AWS ValidationException alone: from another provider they may be a throttle+ ("input_tokens_exceeded_from_another_provider", "Daily input tokens exceeded", None, None, False),+ ("too_many_input_tokens_from_another_provider", "Too many input tokens", "rate_limit_exceeded", None, False),+ ("bedrock_nova_wording_on_a_throttle", "Input Tokens Exceeded for this minute", "ThrottlingException", None, False),+ # The same words about a parameter over its bound, which is the request's fault, not the input's length+ ("a_parameter_over_the_model_maximum", "max_tokens (200000) exceed model maximum (8192)", "ValidationException", None, False),+ # Bedrock's throttle, whose "too many tokens" names no window+ ("bedrock_throttle_wording", "Too many tokens per day, please wait before trying again.", "ThrottlingException", None, False),("gemini", "The input token count (1234567) exceeds the maximum number of tokens allowed (1048576).", None, None, True),("a_parameter_refused",diff --git a/tests/unit/pipelex/cogt/inference/test_retry_after_seconds_from_headers.py b/tests/unit/pipelex/cogt/inference/test_retry_after_seconds_from_headers.pynew file mode 100644index 000000000..1e32f69ad--- /dev/null+++ b/tests/unit/pipelex/cogt/inference/test_retry_after_seconds_from_headers.py@@ -0,0 +1,30 @@+from __future__ import annotations++import httpx+import pytest++from pipelex.cogt.inference.error_classification import retry_after_seconds_from_headers+++class TestRetryAfterSecondsFromHeaders:+ @pytest.mark.parametrize(+ ("headers", "expected_seconds"),+ [+ pytest.param(None, None, id="no_headers"),+ pytest.param({}, None, id="neither_header"),+ pytest.param({"retry-after": "2"}, 2.0, id="seconds_alone"),+ pytest.param({"retry-after-ms": "1500"}, 1.5, id="milliseconds_alone"),+ # Azure OpenAI's shape: both, the milliseconds the more precise+ pytest.param({"retry-after-ms": "4500", "retry-after": "5"}, 4.5, id="milliseconds_before_seconds"),+ pytest.param({"Retry-After-Ms": "250"}, 0.25, id="any_case"),+ pytest.param({"retry-after-ms": "soon", "retry-after": "3"}, 3.0, id="unreadable_milliseconds_leave_the_seconds"),+ pytest.param({"retry-after-ms": "-5", "retry-after": "3"}, 3.0, id="negative_milliseconds_leave_the_seconds"),+ # The milliseconds decide only above zero; for a zero, Pipelex deliberately falls back to the seconds+ pytest.param({"retry-after-ms": "0", "retry-after": "30"}, 30.0, id="zero_milliseconds_leave_the_seconds"),+ pytest.param({"retry-after-ms": "0"}, None, id="zero_milliseconds_alone_read_as_none"),+ pytest.param({"retry-after-ms": "inf"}, None, id="infinite_milliseconds_read_as_none"),+ ],+ )+ def test_reads_the_milliseconds_first(self, headers: dict[str, str] | None, expected_seconds: float | None) -> None:+ """``retry-after-ms``, which the OpenAI and Anthropic SDKs also read first, decides when it is above zero; ``retry-after`` otherwise."""+ assert retry_after_seconds_from_headers(headers=None if headers is None else httpx.Headers(headers)) == expected_secondsdiff --git a/tests/unit/pipelex/cogt/inference/test_transport_retry.py b/tests/unit/pipelex/cogt/inference/test_transport_retry.pyindex 764cb2f48..f29349fbb 100644--- a/tests/unit/pipelex/cogt/inference/test_transport_retry.py+++ b/tests/unit/pipelex/cogt/inference/test_transport_retry.py@@ -114,10 +114,35 @@ class TestRequestWithTransportRetry:assert send.call_count == 2assert response.status_code == 200+ async def test_honors_azure_s_delay_in_milliseconds(self, mocker: MockerFixture) -> None:+ """Azure OpenAI's REST answer gives its delay twice; the milliseconds, above zero, are the wait."""+ sleep_mock = mocker.patch("asyncio.sleep", new_callable=mocker.AsyncMock)+ send = mocker.AsyncMock(side_effect=[_make_status_error(429, headers={"retry-after-ms": "4500", "retry-after": "5"}), _make_ok_response()])++ await request_with_transport_retry(send_request=send, max_retries=3)+ sleep_mock.assert_awaited_once_with(4.5)++ @pytest.mark.parametrize(+ "headers",+ [+ pytest.param({"Retry-After": "soon"}, id="unparseable_seconds"),+ pytest.param({"retry-after-ms": "never"}, id="unparseable_milliseconds"),+ pytest.param({}, id="no_header"),+ ],+ )+ async def test_a_header_with_no_usable_delay_leaves_the_wait_to_the_backoff(self, mocker: MockerFixture, headers: dict[str, str]) -> None:+ sleep_mock = mocker.patch("asyncio.sleep", new_callable=mocker.AsyncMock)+ backoff_mock = mocker.patch("pipelex.cogt.inference.transport_retry._exponential_wait", return_value=0.25)+ send = mocker.AsyncMock(side_effect=[_make_status_error(429, headers=headers), _make_ok_response()])++ await request_with_transport_retry(send_request=send, max_retries=3)+ backoff_mock.assert_called_once()+ sleep_mock.assert_awaited_once_with(0.25)+async def test_naive_retry_after_header_does_not_crash(self, mocker: MockerFixture) -> None:mocker.patch("asyncio.sleep", new_callable=mocker.AsyncMock)- # A "-0000" HTTP-date is a valid RFC date that parses to a *naive* datetime; the wait- # calculation must treat it as unusable and fall back to backoff, not raise TypeError.+ # A "-0000" HTTP-date is a valid RFC date that parses to a *naive* datetime; the shared+ # reader takes it as UTC, so a date in the past is no wait at all, and nothing raises.send = mocker.AsyncMock(side_effect=[_make_status_error(429, headers={"Retry-After": "Wed, 21 Oct 2015 07:28:00 -0000"}),diff --git a/tests/unit/pipelex/cogt/test_data.py b/tests/unit/pipelex/cogt/test_data.pyindex f1aad2775..4218a4e10 100644--- a/tests/unit/pipelex/cogt/test_data.py+++ b/tests/unit/pipelex/cogt/test_data.py@@ -44,6 +44,10 @@ class GatewayThrottleTestData:AZURE_RATE_LIMIT_CODE: ClassVar[str] = "RateLimitReached"# The delay the gateway forwards from Azure in the `retry-after` headerAZURE_RETRY_AFTER: ClassVar[str] = "57"+ # The same delay in milliseconds, the `retry-after-ms` Azure OpenAI sends beside `retry-after` on a 429+ # (https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/provisioned-get-started); the value is built+ AZURE_RETRY_AFTER_MS: ClassVar[str] = "56400"+ AZURE_RETRY_AFTER_MS_SECONDS: ClassVar[float] = 56.4# The whole body the gateway answered with, at HTTP 429AZURE_RATE_LIMIT_BODY: ClassVar[dict[str, Any]] = {"error": {@@ -61,3 +65,349 @@ class GatewayThrottleTestData:# One of the billing codes OpenAI's error-code guide documents for a 429 in `error.code`, whose `error.type` can# still be `insufficient_quota`OPENAI_CREDIT_BALANCE_EXHAUSTED_CODE: ClassVar[str] = "credit_balance_exhausted"+++class DirectProviderQuotaTestData:+ """Error bodies of the providers Pipelex calls directly, each as the provider sends it over the wire.++ Each one is a real answer a user posted or a body the provider documents, with its source named; where neither+ exists, the comment says what the body was built from. They pin how each provider tells a spent quota from a rate+ limit: the status and the structured code or type, never the wording.+ """++ # OpenAI's long-standing spent-quota answer, on a 429, as users post it+ # (https://community.openai.com/t/429-insufficient-quota-when-enough-quota-and-credits-are-left/567740)+ OPENAI_INSUFFICIENT_QUOTA_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "message": (+ "You exceeded your current quota, please check your plan and billing details. For more information on "+ "this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors."+ ),+ "type": "insufficient_quota",+ "param": None,+ "code": "insufficient_quota",+ }+ }+ # OpenAI's billing 429s carry the cause in `error.code`, and the broad `insufficient_quota` can still be the type+ # (https://developers.openai.com/api/docs/guides/error-codes). The guide gives the codes and their descriptions, no+ # verbatim body, so these are built from them.+ OPENAI_CREDIT_BALANCE_EXHAUSTED_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "message": "Your organization has no prepaid credits remaining.",+ "type": "insufficient_quota",+ "param": None,+ "code": "credit_balance_exhausted",+ }+ }+ OPENAI_PROJECT_SPEND_LIMIT_NO_TYPE_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "message": "Your project reached its enforced spend limit.",+ "param": None,+ "code": "project_spend_limit_exceeded",+ }+ }+ # OpenAI's per-minute token limit, in the shape users post it, the organization's id replaced; the guide lists the+ # rate limit with no code, and this type and code are not in it+ OPENAI_RATE_LIMIT_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "message": (+ "Rate limit reached for gpt-4o in organization org-abc123 on tokens per min (TPM): Limit 30000, Used "+ "29000, Requested 2000. Please try again in 2s. Visit https://platform.openai.com/account/rate-limits "+ "to learn more."+ ),+ "type": "tokens",+ "param": None,+ "code": "rate_limit_exceeded",+ }+ }+ # Built from Gemini's per-minute throttle, whose words say "quota" and "billing", in Google's error shape, for an+ # OpenAI-dialect backend pointed at a Google endpoint: its `code` is the status, a number, and it carries no type+ OPENAI_DIALECT_GEMINI_THROTTLE_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 429,+ "message": (+ "You exceeded your current quota, please check your plan and billing details. For more information on "+ "this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits."+ ),+ "status": "RESOURCE_EXHAUSTED",+ }+ }+ # OpenRouter's 402 for a request its in-flight spending budget cannot hold until the running requests settle, its+ # documented example with the remedy hint left out (https://openrouter.ai/docs/api_reference/limits.md)+ OPENROUTER_IN_FLIGHT_BUDGET_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 402,+ "message": (+ "This request would exceed your available credits given your current in-flight requests. Retry after "+ "in-flight requests settle, or add credits."+ ),+ "metadata": {"reason": "in_flight_budget_exhausted", "limit_source": "openrouter_in_flight_budget"},+ }+ }+ # The wait OpenRouter sends with it in the `retry-after` header, as its documentation describes+ OPENROUTER_RETRY_AFTER: ClassVar[str] = "2"+ # OpenRouter's 402 for a balance that cannot cover the request, with its documented message and limit source+ # (https://openrouter.ai/docs/api-reference/errors)+ OPENROUTER_INSUFFICIENT_CREDITS_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 402,+ "message": "Your account or API key has insufficient credits. Add more credits and retry the request.",+ "metadata": {"limit_source": "openrouter_credits"},+ }+ }++ # Anthropic's 429 for its tier's monthly spend cap, verbatim from its rate-limits page+ # (https://platform.claude.com/docs/en/api/rate-limits#reaching-your-spend-cap)+ ANTHROPIC_SPEND_CAP_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {+ "type": "rate_limit_error",+ "message": (+ "You have reached your API usage limits: your organization has crossed its monthly API usage threshold, "+ "set based on your organization's API tier. You will regain access on 2026-09-01 at 00:00 UTC."+ ),+ "details": {"error_code": "enforced_spend_limit_reached"},+ },+ "request_id": "req_018EeWyXxfu5pfWkrYcMdjWG",+ }+ # Anthropic's 400 for a spent credit balance, verbatim from the SDK issue that confirms its status and type+ # (https://github.com/anthropics/anthropic-sdk-typescript/issues/618)+ ANTHROPIC_CREDIT_BALANCE_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "message": "Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.",+ },+ "request_id": "req_011Cf75wfCXCbniKKrP6wSut",+ }+ # Anthropic's 400 for a spend limit the organization or a workspace set itself. Its rate-limits page documents the+ # sentence's start and no verbatim body (https://platform.claude.com/docs/en/api/rate-limits#setting-your-own-spend-limit),+ # so these are built on it.+ ANTHROPIC_SELF_SET_LIMIT_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "message": "You have reached your specified API usage limits. You will regain access on 2026-11-01 at 00:00 UTC.",+ },+ }+ ANTHROPIC_WORKSPACE_LIMIT_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {+ "type": "invalid_request_error",+ "message": "You have reached your specified workspace API usage limits. You will regain access on 2026-11-01 at 00:00 UTC.",+ },+ }+ # Anthropic's 402, with the description its errors reference gives `billing_error`+ # (https://platform.claude.com/docs/en/api/errors)+ ANTHROPIC_BILLING_ERROR_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {"type": "billing_error", "message": "There's an issue with your billing or payment information."},+ }+ # Anthropic's per-minute token limit, in the shape users post it, the organization's id left out+ ANTHROPIC_RATE_LIMIT_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {+ "type": "rate_limit_error",+ "message": (+ "This request would exceed the rate limit for your organization of 400,000 input tokens per minute. "+ "For details, refer to: https://docs.claude.com/en/api/rate-limits. You can see the response headers "+ "for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later."+ ),+ },+ }+ # A malformed request, on the status and type the two spend refusals share+ ANTHROPIC_INVALID_REQUEST_BODY: ClassVar[dict[str, Any]] = {+ "type": "error",+ "error": {"type": "invalid_request_error", "message": "messages: text content blocks must be non-empty"},+ }++ # Gemini's per-minute throttle on a paid tier, verbatim as a user posted it+ # (https://discuss.ai.google.dev/t/you-exceeded-your-current-quota-please-check-your-plan-and-billing-details/109580):+ # its words say "quota" and "billing", and the wait it asks for is in the body's `google.rpc.RetryInfo`+ GOOGLE_PER_MINUTE_THROTTLE_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 429,+ "message": (+ "You exceeded your current quota, please check your plan and billing details. For more information on "+ "this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. To monitor your current usage, "+ "head to: https://ai.dev/usage?tab=rate-limit. \n* Quota exceeded for metric: "+ "generativelanguage.googleapis.com/generate_content_paid_tier_input_token_count, limit: 1000000, model: "+ "gemini-2.5-flash\nPlease retry in 33.470052706s."+ ),+ "status": "RESOURCE_EXHAUSTED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.Help",+ "links": [{"description": "Learn more about Gemini API quotas", "url": "https://ai.google.dev/gemini-api/docs/rate-limits"}],+ },+ {+ "@type": "type.googleapis.com/google.rpc.QuotaFailure",+ "violations": [+ {+ "quotaMetric": "generativelanguage.googleapis.com/generate_content_paid_tier_input_token_count",+ "quotaId": "GenerateContentPaidTierInputTokensPerModelPerMinute",+ "quotaDimensions": {"location": "global", "model": "gemini-2.5-flash"},+ "quotaValue": "1000000",+ }+ ],+ },+ {"@type": "type.googleapis.com/google.rpc.RetryInfo", "retryDelay": "33s"},+ ],+ }+ }+ # The wait that throttle's `RetryInfo` asks for, in seconds+ GOOGLE_RETRY_DELAY_SECONDS: ClassVar[float] = 33.0+ # Gemini's 402 for a Prepay balance at zero, which its billing page documents with no body, so only the status is+ # Google's (https://ai.google.dev/gemini-api/docs/billing)+ GOOGLE_PREPAY_DEPLETED_BODY: ClassVar[dict[str, Any]] = {"error": {"code": 402, "message": "Payment required."}}+ # Google's refusal of a project whose billing account is disabled: the `google.rpc.ErrorInfo` is the BILLING_DISABLED+ # example of Google's ErrorReason definitions, with the Gemini API's service named+ # (https://github.com/googleapis/googleapis/blob/master/google/api/error_reason.proto), inside the error envelope every+ # Google Cloud API answers with (https://google.aip.dev/193); the message is built+ GOOGLE_BILLING_DISABLED_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 403,+ "message": "This API method requires billing to be enabled. Please enable billing on project #123 then retry.",+ "status": "PERMISSION_DENIED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.ErrorInfo",+ "reason": "BILLING_DISABLED",+ "domain": "googleapis.com",+ "metadata": {"consumer": "projects/123", "service": "generativelanguage.googleapis.com"},+ }+ ],+ }+ }+ # The same envelope with another documented reason, SERVICE_DISABLED: the API is off for the project, which the+ # project's owner turns on, not a billing matter (same source)+ GOOGLE_SERVICE_DISABLED_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 403,+ "message": "Generative Language API has not been used in project 123 before or it is disabled.",+ "status": "PERMISSION_DENIED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.ErrorInfo",+ "reason": "SERVICE_DISABLED",+ "domain": "googleapis.com",+ "metadata": {"consumer": "projects/123", "service": "generativelanguage.googleapis.com"},+ }+ ],+ }+ }+ # Gemini's 429 for a quota whose limit is zero, verbatim from the Gemini developer forum, the project number as posted+ # (https://discuss.ai.google.dev/t/quota-limits-on-generativelanguage-googleapis-com/108535)+ GOOGLE_ZERO_LIMIT_QUOTA_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 429,+ "message": (+ "Quota exceeded for quota metric 'Generate Content API requests per minute' and limit 'GenerateContent "+ "request limit per minute for a region' of service 'generativelanguage.googleapis.com' for consumer "+ "'project_number:MYPROJECTNO'."+ ),+ "status": "RESOURCE_EXHAUSTED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.ErrorInfo",+ "reason": "RATE_LIMIT_EXCEEDED",+ "domain": "googleapis.com",+ "metadata": {+ "quota_location": "europe-west1",+ "quota_limit": "GenerateContentRequestsPerMinutePerProjectPerRegion",+ "quota_unit": "1/min/{project}/{region}",+ "consumer": "projects/MYPROJECTNO",+ "service": "generativelanguage.googleapis.com",+ "quota_limit_value": "0",+ "quota_metric": "generativelanguage.googleapis.com/generate_content_requests",+ },+ }+ ],+ }+ }+ # Built: the same quota at a limit of 60, which a wait cures+ GOOGLE_NONZERO_LIMIT_QUOTA_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 429,+ "message": "Quota exceeded for quota metric 'Generate Content API requests per minute'.",+ "status": "RESOURCE_EXHAUSTED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.ErrorInfo",+ "reason": "RATE_LIMIT_EXCEEDED",+ "domain": "googleapis.com",+ "metadata": {"quota_limit": "GenerateContentRequestsPerMinutePerProjectPerRegion", "quota_limit_value": "60"},+ }+ ],+ }+ }+ # Built: a free-tier quota on a model the free tier leaves out, stated by a `google.rpc.QuotaFailure` violation's+ # `quotaValue`, the enforced quota value, an int64 that protobuf's JSON form writes as a string+ # (https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto)+ GOOGLE_ZERO_QUOTA_VALUE_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "code": 429,+ "message": "You exceeded your current quota, please check your plan and billing details.",+ "status": "RESOURCE_EXHAUSTED",+ "details": [+ {+ "@type": "type.googleapis.com/google.rpc.QuotaFailure",+ "violations": [+ {+ "quotaMetric": "generativelanguage.googleapis.com/generate_content_free_tier_requests",+ "quotaId": "GenerateRequestsPerDayPerProjectPerModel-FreeTier",+ "quotaDimensions": {"location": "global", "model": "gemini-2.5-pro"},+ "quotaValue": "0",+ }+ ],+ }+ ],+ }+ }++ # Mistral's capacity refusal on a 429, verbatim from a production log+ # (https://github.com/EOSC-Data-Commons/data-commons-search/issues/7)+ MISTRAL_CAPACITY_BODY: ClassVar[str] = (+ '{"object":"error","message":"Service tier capacity exceeded for this model.",'+ '"type":"service_tier_capacity_exceeded","param":null,"code":"3505"}'+ )+ # Built: a 429 whose words name a quota and which carries no type, as Mistral's monthly token cap may read, since no+ # verbatim body of it was found; a rate limit all the same+ MISTRAL_QUOTA_WORDING_BODY: ClassVar[str] = '{"object":"error","message":"Rate limit reached: your monthly token quota has been exceeded."}'+ # Mistral's 402 for a balance at zero, which its billing page documents with no body, so only the status is Mistral's+ # (https://docs.mistral.ai/admin/billing-usage/billing)+ MISTRAL_BALANCE_DEPLETED_BODY: ClassVar[str] = '{"object":"error","message":"Payment required."}'++ # Bedrock's throttle for an account whose daily token quota is spent, verbatim from AWS re:Post+ # (https://repost.aws/questions/QUt6wvrkLHQwq6yq52nVADvA/all-bedrock-model-quotas-stuck-at-0-tokens-day-cannot-make-any-api-calls)+ BEDROCK_DAILY_TOKENS_THROTTLE_MESSAGE: ClassVar[str] = "Too many tokens per day, please wait before trying again."+ # Amazon Nova's refusal of an input over its window, as LibreChat's measured table of overflow signatures records it+ # (https://github.com/LibreChat-AI/agents/blob/main/docs/context-overflow-signatures.md)+ BEDROCK_NOVA_INPUT_OVERFLOW_MESSAGE: ClassVar[str] = "Input Tokens Exceeded: Number of input tokens exceeds maximum length."+ # Built around the fragment langchain-aws, LiteLLM and inspect_ai match for Amazon Titan, no full sentence of which+ # was found+ BEDROCK_TITAN_INPUT_OVERFLOW_MESSAGE: ClassVar[str] = "Too many input tokens. Max input tokens: 8192, request input token count: 9000"+ # Built from the pattern LiteLLM reads for the models Bedrock serves through its OpenAI-compatible engine, such as Kimi+ BEDROCK_OPENAI_ENGINE_INPUT_OVERFLOW_MESSAGE: ClassVar[str] = "prompt tokens (300000) exceed model maximum (262144)"+ # The header Bedrock's REST-JSON protocol names its error in, with the namespace suffix AWS services append, which+ # Claude on Bedrock answers through the Anthropic SDK beside a body holding a message alone+ BEDROCK_SERVICE_QUOTA_ERROR_TYPE_HEADER: ClassVar[str] = "ServiceQuotaExceededException:http://internal.amazon.com/coral/com.amazon.bedrock/"+ BEDROCK_THROTTLING_ERROR_TYPE_HEADER: ClassVar[str] = "ThrottlingException:http://internal.amazon.com/coral/com.amazon.bedrock/"++ # Groq's refusal when its flex tier has no capacity, sent on its own status, 498, with the error it names+ # `capacity_exceeded` (https://console.groq.com/docs/flex-processing). Built: Groq does not say which field holds that+ # name, so it sits in the `type` its error guide shows, and the message is ours; the status alone decides.+ GROQ_FLEX_CAPACITY_BODY: ClassVar[dict[str, Any]] = {+ "error": {"message": "Flex tier capacity exceeded, please try again later.", "type": "capacity_exceeded"}+ }++ # Azure OpenAI's two delay headers on a 429, the milliseconds first, which the OpenAI SDK prefers+ # (https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/provisioned-get-started)+ AZURE_RETRY_AFTER_HEADERS: ClassVar[dict[str, str]] = {"retry-after-ms": "4500", "retry-after": "5"}+ AZURE_RETRY_AFTER_MS_SECONDS: ClassVar[float] = 4.5+ # Bedrock's service-quota refusal, with AWS's documented description+ # (https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html)+ BEDROCK_SERVICE_QUOTA_MESSAGE: ClassVar[str] = (+ "Your request exceeds the service quota for your account. You can view your quotas at Viewing service quotas. "+ "You can resubmit your request later."+ )diff --git a/tests/unit/pipelex/cogt/test_error_classification.py b/tests/unit/pipelex/cogt/test_error_classification.pyindex afa5c5944..007b4b852 100644--- a/tests/unit/pipelex/cogt/test_error_classification.py+++ b/tests/unit/pipelex/cogt/test_error_classification.py@@ -1,23 +1,32 @@-"""Tests for quota and content-policy detection on ``ProviderErrorMetadata``.+"""Tests for quota, rate-limit and content-policy detection on ``ProviderErrorMetadata``.-The quota and content-policy probes are exposed via the-``ProviderErrorMetadata.is_quota_exhaustion`` / ``is_content_policy_violation``-properties; the helpers backing them are module-private. These tests pin the-per-provider phrase matching so changes to the patterns surface as test diffs,-and the gateway's rule, which reads the status and the provider's code alone.+The probes are exposed via the ``ProviderErrorMetadata.limit_verdict`` /+``is_throttle_on_another_status`` / ``is_content_policy_violation`` properties; the+helpers backing them are module-private. These tests pin each+provider's quota rule, which reads the status and the provider's structured code or+type, never its message, except for Anthropic's spend sentences on a 400, and the+content-policy phrase matching."""-from typing import ClassVar+from typing import Any, ClassVarimport pytest-from pipelex.cogt.inference.error_classification import ProviderErrorMetadata+from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, ProviderLimitVerdictfrom pipelex.cogt.inference.provider_name import ProviderName-from tests.unit.pipelex.cogt.test_data import GatewayThrottleTestData+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestData, GatewayThrottleTestData++_Bodies = DirectProviderQuotaTestDatadef _metadata(- provider: ProviderName, message: str, status_code: int | None = None, *, provider_error_code: str | None = None+ provider: ProviderName,+ message: str,+ status_code: int | None = None,+ *,+ provider_error_code: str | None = None,+ body: Any | None = None,+ retry_after_seconds: float | None = None,) -> ProviderErrorMetadata:return ProviderErrorMetadata(provider=provider,@@ -25,70 +34,223 @@ def _metadata(message=message,status_code=status_code,provider_error_code=provider_error_code,+ body=body,+ retry_after_seconds=retry_after_seconds,)class _TestCases:- # (topic, message, expected_is_quota)- OPENAI_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [- ("insufficient_quota", "Error: insufficient_quota - you have exceeded your billing limit", True),- ("exceeded_quota", "You exceeded your current quota, please check your plan", True),- ("generic_rate_limit", "Rate limit exceeded. Please retry after 20s", False),- ("empty_message", "", False),- ("unrelated_error", "Invalid API key provided", False),- ("case_insensitive", "Insufficient_Quota error occurred", True),+ # (topic, message, status_code, provider_error_code, body, expected_is_quota)+ # As the OpenAI client keeps it: `provider_error_code` is the type when there is one, the code otherwise, and the+ # body is the error section, which carries the code. A 429 is a quota on one of OpenAI's billing codes or types+ # alone, whatever its words; a 402 is a quota.+ OPENAI_QUOTA_CASES: ClassVar[list[tuple[str, str, int | None, str | None, Any, bool]]] = [+ (+ "insufficient_quota_type_and_code_429",+ GatewayThrottleTestData.OPENAI_QUOTA_MESSAGE,+ 429,+ "insufficient_quota",+ _Bodies.OPENAI_INSUFFICIENT_QUOTA_BODY["error"],+ True,+ ),+ (+ "billing_code_beside_the_broad_type_429",+ "Your organization has no prepaid credits remaining.",+ 429,+ "insufficient_quota",+ _Bodies.OPENAI_CREDIT_BALANCE_EXHAUSTED_BODY["error"],+ True,+ ),+ # The Extract step keeps the type, so the billing code is read from the body+ (+ "billing_code_beside_another_type_429",+ "Your organization reached its enforced spend limit.",+ 429,+ "invalid_request_error",+ {+ "message": "Your organization reached its enforced spend limit.",+ "type": "invalid_request_error",+ "code": "organization_spend_limit_exceeded",+ },+ True,+ ),+ (+ "billing_code_with_no_type_429",+ "Your project reached its enforced spend limit.",+ 429,+ "project_spend_limit_exceeded",+ _Bodies.OPENAI_PROJECT_SPEND_LIMIT_NO_TYPE_BODY["error"],+ True,+ ),+ (+ "usage_limit_code_429",+ "Your organization reached its OpenAI-assigned usage limit.",+ 429,+ "organization_usage_limit_exceeded",+ {"code": "organization_usage_limit_exceeded"},+ True,+ ),+ ("payment_required_402", "Insufficient credits", 402, None, None, True),+ ("rate_limit_code_429", "Rate limit reached for gpt-4o", 429, "tokens", _Bodies.OPENAI_RATE_LIMIT_BODY["error"], False),+ ("slow_down_429", "Slow down", 429, "rate_limit_error", {"type": "rate_limit_error", "code": "slow_down"}, False),+ # Each message the former word list read as a quota, now a rate limit since no code says otherwise+ ("quota_wording_429_no_code", "You exceeded your current quota, please check your plan", 429, None, None, False),+ ("insufficient_quota_wording_429_no_code", "Error: insufficient_quota - you have exceeded your billing limit", 429, None, None, False),+ ("gemini_throttle_relayed_429", "You exceeded your current quota", 429, None, _Bodies.OPENAI_DIALECT_GEMINI_THROTTLE_BODY["error"], False),+ # The quota code decides a 429 only; on any other status the status ladder reads it+ ("quota_code_500_not_quota", "Internal server error", 500, "insufficient_quota", {"type": "insufficient_quota"}, False),+ ("no_status", "", None, None, None, False),]- ANTHROPIC_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [- ("quota_in_message", "Your account quota has been exceeded", True),- ("billing_limit", "You have reached your billing limit", True),- ("billing_issue", "There is a billing issue with your account", True),- ("credit_balance", "Your credit balance is too low to make this request", True),- ("out_of_credits", "Account is out of credits", True),- ("insufficient_credit", "Insufficient credit on this API key", True),- ("generic_rate_limit", "rate_limit_error: Number of request tokens has exceeded your per-minute limit", False),- ("empty_message", "", False),- ("unrelated_error", "Invalid API key", False),- ("billing_word_alone_not_matched", "Please check your billing dashboard", False),- ("credit_card_declined_not_matched", "Your credit card was declined, please update payment method", False),- ("credit_account_not_matched", "We will credit your account within 24 hours", False),- ("no_credit_check_not_matched", "No credit check required", False),- ("case_insensitive", "Account Billing Issue detected", True),+ # (topic, message, status_code, provider_error_code, body, expected_is_quota)+ # As the Anthropic client keeps it: `provider_error_code` is the error's type, and the body is the whole document.+ ANTHROPIC_QUOTA_CASES: ClassVar[list[tuple[str, str, int | None, str | None, Any, bool]]] = [+ ("billing_error_402", "Error code: 402", 402, "billing_error", _Bodies.ANTHROPIC_BILLING_ERROR_BODY, True),+ ("billing_error_type_on_another_status", "Error code: 400", 400, "billing_error", _Bodies.ANTHROPIC_BILLING_ERROR_BODY, True),+ ("payment_required_402_no_type", "Payment required", 402, None, None, True),+ ("spend_cap_code_429", "Error code: 429", 429, "rate_limit_error", _Bodies.ANTHROPIC_SPEND_CAP_BODY, True),+ ("credit_balance_400", "Error code: 400", 400, "invalid_request_error", _Bodies.ANTHROPIC_CREDIT_BALANCE_BODY, True),+ ("self_set_limit_400", "Error code: 400", 400, "invalid_request_error", _Bodies.ANTHROPIC_SELF_SET_LIMIT_BODY, True),+ ("workspace_limit_400", "Error code: 400", 400, "invalid_request_error", _Bodies.ANTHROPIC_WORKSPACE_LIMIT_BODY, True),+ ("rate_limit_429", "Error code: 429", 429, "rate_limit_error", _Bodies.ANTHROPIC_RATE_LIMIT_BODY, False),+ (+ "quota_wording_429_no_code",+ "Your account quota has been exceeded",+ 429,+ "rate_limit_error",+ {"type": "error", "error": {"type": "rate_limit_error", "message": "Your account quota has been exceeded"}},+ False,+ ),+ ("malformed_request_400", "Error code: 400", 400, "invalid_request_error", _Bodies.ANTHROPIC_INVALID_REQUEST_BODY, False),+ # The spend sentences decide a 400 `invalid_request_error` alone, at the start of the body's message+ ("credit_sentence_on_a_403", "Error code: 403", 403, "invalid_request_error", _Bodies.ANTHROPIC_CREDIT_BALANCE_BODY, False),+ (+ "credit_sentence_on_another_type_400",+ "Error code: 400",+ 400,+ "api_error",+ {"type": "error", "error": {"type": "api_error", "message": "Your credit balance is too low to access the Anthropic API."}},+ False,+ ),+ (+ "credit_sentence_not_at_the_start_400",+ "Error code: 400",+ 400,+ "invalid_request_error",+ {"type": "error", "error": {"type": "invalid_request_error", "message": "messages.0: says your credit balance is too low"}},+ False,+ ),+ # The rendered message is not read: the sentence is the body's+ ("credit_sentence_in_the_message_alone_400", "Your credit balance is too low", 400, "invalid_request_error", None, False),+ ("no_status", "", None, None, None, False),]- GOOGLE_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [- ("quota_exceeded", "Quota exceeded for aiplatform.googleapis.com", True),- ("resource_exhausted", "Resource has been exhausted (e.g. check quota)", True),- ("billing_quota_exceeded", "Billing quota exceeded for project foo", True),- ("billing_limit_reached", "Billing limit reached on this account", True),- ("billing_exceeded", "Billing exceeded for the current period", True),- ("billing_account_not_active", "Billing account is not active", True),- ("billing_account_disabled", "billing account is disabled for project", True),- ("generic_rate_limit", "Rate limit exceeded per minute", False),- ("empty_message", "", False),- ("unrelated_error", "Model not found", False),- ("billing_setup_not_quota", "billing project not configured", False),- ("billing_enable_not_quota", "Please enable billing for this project to use this API", False),- ("billing_api_not_enabled_not_quota", "billing API not enabled", False),+ # (topic, message, status_code, provider_error_code, expected_is_quota)+ # A 402 is a quota; every 429 RESOURCE_EXHAUSTED is a rate limit, whatever its message says.+ GOOGLE_QUOTA_CASES: ClassVar[list[tuple[str, str, int | None, str | None, bool]]] = [+ ("prepay_depleted_402", "402 None. Payment required.", 402, None, True),+ (+ "per_minute_throttle_429",+ _Bodies.GOOGLE_PER_MINUTE_THROTTLE_BODY["error"]["message"],+ 429,+ "RESOURCE_EXHAUSTED",+ False,+ ),+ ("resource_exhausted_wording_429", "Resource has been exhausted (e.g. check quota)", 429, "RESOURCE_EXHAUSTED", False),+ ("billing_wording_429", "Billing quota exceeded for project foo", 429, "RESOURCE_EXHAUSTED", False),+ # The words decide nothing: a disabled billing account is read from its ErrorInfo reason, which a body carries+ ("billing_account_wording_403", "Billing account is not active", 403, "PERMISSION_DENIED", False),+ ("quota_wording_400", "Quota exceeded for aiplatform.googleapis.com", 400, "INVALID_ARGUMENT", False),+ ("no_status", "", None, None, False),]# (topic, message, status_code, expected_is_quota)+ # A 402 is a quota; every 429 is a rate limit, the monthly token cap included, whatever its message says.MISTRAL_QUOTA_CASES: ClassVar[list[tuple[str, str, int, bool]]] = [("payment_required_402", "Payment required", 402, True),- ("quota_429", "Quota exceeded for your account", 429, True),- ("billing_429", "Billing limit reached", 429, True),+ ("payment_required_402_any_message", "anything at all", 402, True),+ ("quota_wording_429", "Quota exceeded for your account", 429, False),+ ("billing_wording_429", "Billing limit reached", 429, False),("generic_rate_limit_429", "Rate limit exceeded. Please retry after 20s", 429, False),("server_error_500", "Internal server error", 500, False),("not_found_404", "Model not found", 404, False),- ("payment_required_402_any_message", "anything at all", 402, True),]- AWS_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [- ("quota_exceeded", "Service quota exceeded for this account", True),- ("limit_exceeded", "Token limit exceeded for model", True),- ("generic_throttle", "Rate exceeded", False),+ # (topic, message, status_code, provider_error_code, expected_is_quota)+ # Bedrock is billed to the AWS account, with no balance to spend: none of its answers is a spent quota.+ BEDROCK_QUOTA_CASES: ClassVar[list[tuple[str, str, int, str | None, bool]]] = [+ ("daily_tokens_throttle_429", _Bodies.BEDROCK_DAILY_TOKENS_THROTTLE_MESSAGE, 429, "ThrottlingException", False),+ ("service_quota_exceeded_400", _Bodies.BEDROCK_SERVICE_QUOTA_MESSAGE, 400, "ServiceQuotaExceededException", False),+ ("input_overflow_validation_400", _Bodies.BEDROCK_NOVA_INPUT_OVERFLOW_MESSAGE, 400, "ValidationException", False),+ ("quota_wording_no_code_400", "Service quota exceeded for this account", 400, None, False),+ ]++ # (topic, provider, status_code, provider_error_code, body, retry_after_seconds, expected_is_throttle)+ THROTTLE_ON_ANOTHER_STATUS_CASES: ClassVar[list[tuple[str, ProviderName, int | None, str | None, Any, float | None, bool]]] = [+ # OpenRouter's in-flight budget, its error section as the OpenAI client keeps it, sent with its wait+ (+ "openrouter_in_flight_budget_with_its_wait",+ ProviderName.OPENAI,+ 402,+ None,+ _Bodies.OPENROUTER_IN_FLIGHT_BUDGET_BODY["error"],+ 2.0,+ True,+ ),+ (+ "openrouter_in_flight_budget_nested_with_its_wait",+ ProviderName.OPENAI,+ 402,+ None,+ _Bodies.OPENROUTER_IN_FLIGHT_BUDGET_BODY,+ 2.0,+ True,+ ),+ # The limit source decides; the header is the wait, not the verdict+ ("openrouter_in_flight_budget_with_no_wait", ProviderName.OPENAI, 402, None, _Bodies.OPENROUTER_IN_FLIGHT_BUDGET_BODY["error"], None, True),+ # The source is read in every section, past one whose metadata says something else+ (+ "openrouter_in_flight_budget_in_the_second_section",+ ProviderName.OPENAI,+ 402,+ None,+ {"error": {"metadata": {"reason": "in_flight_budget_exhausted"}}, "metadata": {"limit_source": "openrouter_in_flight_budget"}},+ None,+ True,+ ),+ (+ "openrouter_insufficient_credits_with_a_wait",+ ProviderName.OPENAI,+ 402,+ None,+ _Bodies.OPENROUTER_INSUFFICIENT_CREDITS_BODY["error"],+ 2.0,+ False,+ ),+ ("openai_402_with_no_body", ProviderName.OPENAI, 402, None, None, 2.0, False),+ ("in_flight_budget_on_a_429", ProviderName.OPENAI, 429, None, _Bodies.OPENROUTER_IN_FLIGHT_BUDGET_BODY["error"], 2.0, False),+ # OpenRouter's structure from another provider says nothing+ ("in_flight_budget_from_anthropic", ProviderName.ANTHROPIC, 402, None, _Bodies.OPENROUTER_IN_FLIGHT_BUDGET_BODY, 2.0, False),+ ("bedrock_service_quota_exceeded", ProviderName.BEDROCK, 400, "ServiceQuotaExceededException", None, None, True),+ ("bedrock_throttling_is_a_plain_429", ProviderName.BEDROCK, 429, "ThrottlingException", None, None, False),+ # Claude on Bedrock, through the Anthropic SDK, carries Bedrock's code from its error-type header+ ("claude_on_bedrock_service_quota_exceeded", ProviderName.ANTHROPIC, 400, "ServiceQuotaExceededException", None, None, True),+ ("anthropic_invalid_request_is_no_throttle", ProviderName.ANTHROPIC, 400, "invalid_request_error", None, None, False),+ # Groq's flex tier at capacity, on its own status+ ("groq_flex_capacity_498", ProviderName.OPENAI, 498, None, _Bodies.GROQ_FLEX_CAPACITY_BODY["error"], None, True),+ ("a_498_from_another_dialect", ProviderName.MISTRAL, 498, None, None, None, False),+ ]++ CONTENT_POLICY_CASES: ClassVar[list[tuple[str, str, bool]]] = [+ ("content_policy", "Your request was rejected due to content_policy_violation", True),+ ("blocked_by_safety", "The response was blocked by safety systems", True),+ ("content_filter", "content_filter triggered for this request", True),+ ("safety_system", "Rejected by safety system", True),+ ("safety_filter", "Blocked by safety filter", True),+ ("generic_bad_request", "Invalid parameter: temperature must be between 0 and 2", False),("empty_message", "", False),- ("unrelated_error", "Access denied", False),+ ("safety_word_alone_not_matched", "Please ensure thread safety when using this", False),+ ("case_insensitive", "Content_Policy violation detected", True),]# (topic, message, status_code, provider_error_code, expected_is_quota)@@ -133,37 +295,31 @@ class _TestCases:("server_error_500", "Internal server error", 500, None, False),]- CONTENT_POLICY_CASES: ClassVar[list[tuple[str, str, bool]]] = [- ("content_policy", "Your request was rejected due to content_policy_violation", True),- ("blocked_by_safety", "The response was blocked by safety systems", True),- ("content_filter", "content_filter triggered for this request", True),- ("safety_system", "Rejected by safety system", True),- ("safety_filter", "Blocked by safety filter", True),- ("generic_bad_request", "Invalid parameter: temperature must be between 0 and 2", False),- ("empty_message", "", False),- ("safety_word_alone_not_matched", "Please ensure thread safety when using this", False),- ("case_insensitive", "Content_Policy violation detected", True),- ]-class TestErrorClassification:- """Tests for quota detection and content policy classification via the public ``ProviderErrorMetadata`` properties."""+ """Tests for quota, rate-limit and content-policy detection via the public ``ProviderErrorMetadata`` properties."""@pytest.mark.parametrize(- ("_topic", "error_message", "expected"),+ ("_topic", "error_message", "status_code", "provider_error_code", "body", "expected"),_TestCases.OPENAI_QUOTA_CASES,)- def test_is_quota_exhaustion_openai(self, _topic: str, error_message: str, expected: bool) -> None:- """Discriminate OpenAI quota exhaustion from generic rate limiting."""- assert _metadata(ProviderName.OPENAI, error_message).is_quota_exhaustion == expected+ def test_is_quota_exhaustion_openai(+ self, _topic: str, error_message: str, status_code: int | None, provider_error_code: str | None, body: Any, expected: bool+ ) -> None:+ """Discriminate an OpenAI-dialect spent quota from a rate limit by the status and the error code or type alone."""+ metadata = _metadata(ProviderName.OPENAI, error_message, status_code, provider_error_code=provider_error_code, body=body)+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected@pytest.mark.parametrize(- ("_topic", "error_message", "expected"),+ ("_topic", "error_message", "status_code", "provider_error_code", "body", "expected"),_TestCases.ANTHROPIC_QUOTA_CASES,)- def test_is_quota_exhaustion_anthropic(self, _topic: str, error_message: str, expected: bool) -> None:- """Discriminate Anthropic quota exhaustion from generic rate limiting."""- assert _metadata(ProviderName.ANTHROPIC, error_message).is_quota_exhaustion == expected+ def test_is_quota_exhaustion_anthropic(+ self, _topic: str, error_message: str, status_code: int | None, provider_error_code: str | None, body: Any, expected: bool+ ) -> None:+ """Discriminate an Anthropic spent quota or credit by its status, its type, its spend cap's code and its spend sentences on a 400."""+ metadata = _metadata(ProviderName.ANTHROPIC, error_message, status_code, provider_error_code=provider_error_code, body=body)+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected@pytest.mark.parametrize(("_topic", "error_message", "expected"),@@ -174,28 +330,61 @@ class TestErrorClassification:assert _metadata(ProviderName.OPENAI, error_message).is_content_policy_violation == expected@pytest.mark.parametrize(- ("_topic", "error_message", "expected"),+ ("_topic", "error_message", "status_code", "provider_error_code", "expected"),_TestCases.GOOGLE_QUOTA_CASES,)- def test_is_quota_exhaustion_google(self, _topic: str, error_message: str, expected: bool) -> None:- """Discriminate Google quota exhaustion from generic rate limiting."""- assert _metadata(ProviderName.GOOGLE, error_message).is_quota_exhaustion == expected+ def test_is_quota_exhaustion_google(+ self, _topic: str, error_message: str, status_code: int | None, provider_error_code: str | None, expected: bool+ ) -> None:+ """A Google spent quota is its 402 or a detail of its body; with no body, every 429 is a rate limit, whatever it says."""+ metadata = _metadata(ProviderName.GOOGLE, error_message, status_code, provider_error_code=provider_error_code)+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected@pytest.mark.parametrize(("_topic", "error_message", "status_code", "expected"),_TestCases.MISTRAL_QUOTA_CASES,)def test_is_quota_exhaustion_mistral(self, _topic: str, error_message: str, status_code: int, expected: bool) -> None:- """Discriminate Mistral quota exhaustion from generic rate limiting."""- assert _metadata(ProviderName.MISTRAL, error_message, status_code).is_quota_exhaustion == expected+ """A Mistral spent balance is its 402; every 429 is a rate limit, whatever it says."""+ assert (_metadata(ProviderName.MISTRAL, error_message, status_code).limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected@pytest.mark.parametrize(- ("_topic", "error_message", "expected"),- _TestCases.AWS_QUOTA_CASES,+ ("_topic", "error_message", "status_code", "provider_error_code", "expected"),+ _TestCases.BEDROCK_QUOTA_CASES,+ )+ def test_is_quota_exhaustion_bedrock(+ self, _topic: str, error_message: str, status_code: int, provider_error_code: str | None, expected: bool+ ) -> None:+ """No Bedrock answer is a spent quota, whatever its code or its words."""+ metadata = _metadata(ProviderName.BEDROCK, error_message, status_code, provider_error_code=provider_error_code)+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected++ @pytest.mark.parametrize(+ ("_topic", "provider", "status_code", "provider_error_code", "body", "retry_after_seconds", "expected"),+ _TestCases.THROTTLE_ON_ANOTHER_STATUS_CASES,)- def test_is_quota_exhaustion_aws(self, _topic: str, error_message: str, expected: bool) -> None:- """Discriminate AWS quota exhaustion from generic throttling."""- assert _metadata(ProviderName.BEDROCK, error_message).is_quota_exhaustion == expected+ def test_is_throttle_on_another_status(+ self,+ _topic: str,+ provider: ProviderName,+ status_code: int | None,+ provider_error_code: str | None,+ body: Any,+ retry_after_seconds: float | None,+ expected: bool,+ ) -> None:+ """A throttle a provider documents on another status than 429 is a rate limit, never a spent quota."""+ metadata = _metadata(+ provider,+ "Error",+ status_code,+ provider_error_code=provider_error_code,+ body=body,+ retry_after_seconds=retry_after_seconds,+ )+ assert metadata.is_throttle_on_another_status == expected+ if expected:+ assert metadata.limit_verdict is ProviderLimitVerdict.RATE_LIMITED@pytest.mark.parametrize(("_topic", "error_message", "status_code", "provider_error_code", "expected"),@@ -206,4 +395,28 @@ class TestErrorClassification:) -> None:"""Discriminate Gateway quota exhaustion from rate limiting by the status and the provider's code alone."""metadata = _metadata(ProviderName.GATEWAY, error_message, status_code, provider_error_code=provider_error_code)- assert metadata.is_quota_exhaustion == expected+ assert (metadata.limit_verdict is ProviderLimitVerdict.QUOTA_EXHAUSTED) == expected++ @pytest.mark.parametrize(+ ("_topic", "provider", "status_code", "provider_error_code", "expected"),+ [+ ("openai_plain_429", ProviderName.OPENAI, 429, None, ProviderLimitVerdict.RATE_LIMITED),+ ("openai_quota_code_429", ProviderName.OPENAI, 429, "insufficient_quota", ProviderLimitVerdict.QUOTA_EXHAUSTED),+ ("google_resource_exhausted_429", ProviderName.GOOGLE, 429, "RESOURCE_EXHAUSTED", ProviderLimitVerdict.RATE_LIMITED),+ ("mistral_402", ProviderName.MISTRAL, 402, None, ProviderLimitVerdict.QUOTA_EXHAUSTED),+ ("bedrock_throttling_429", ProviderName.BEDROCK, 429, "ThrottlingException", ProviderLimitVerdict.RATE_LIMITED),+ ("anthropic_400", ProviderName.ANTHROPIC, 400, "invalid_request_error", ProviderLimitVerdict.NEITHER),+ ("no_status", ProviderName.OPENAI, None, None, ProviderLimitVerdict.NEITHER),+ ],+ )+ def test_limit_verdict(+ self,+ _topic: str,+ provider: ProviderName,+ status_code: int | None,+ provider_error_code: str | None,+ expected: ProviderLimitVerdict,+ ) -> None:+ """Every 429 with no spent-quota signal is a rate limit, and nothing else is unless its provider documents a throttle."""+ metadata = _metadata(provider, "Too many requests", status_code, provider_error_code=provider_error_code)+ assert metadata.limit_verdict is expecteddiff --git a/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_error_handling.py b/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_error_handling.pyindex 7b26c47a6..cb3a0209c 100644--- a/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_error_handling.py+++ b/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_error_handling.py@@ -15,6 +15,7 @@ from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError,from pipelex.cogt.inference.error_classification import UserActionKindfrom pipelex.errors.problem import ProblemViewfrom pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorker+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDatafrom tests.unit.pipelex.providers.anthropic.test_data import AnthropicErrorHandlingTestData@@ -24,12 +25,12 @@ def _mock_httpx_response(status_code: int = 429) -> httpx.Response:return httpx.Response(status_code=status_code, request=request)-def _make_anthropic_rate_limit_error(message: str) -> anthropic.RateLimitError:- return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=None)+def _make_anthropic_rate_limit_error(message: str, *, body: dict[str, Any] | None = None) -> anthropic.RateLimitError:+ return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=body)-def _make_anthropic_bad_request_error(message: str) -> anthropic.BadRequestError:- return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=None)+def _make_anthropic_bad_request_error(message: str, *, body: dict[str, Any] | None = None) -> anthropic.BadRequestError:+ return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=body)def _make_anthropic_timeout_error() -> anthropic.APITimeoutError:@@ -106,7 +107,7 @@ class TestAnthropicWorkerErrorHandling:# ---- RateLimitError tests ----@pytest.mark.parametrize(- ("_topic", "error_message", "expected_category", "expected_action_substring"),+ ("_topic", "error_message", "body", "expected_category", "expected_action_substring"),AnthropicErrorHandlingTestData.RATE_LIMIT_CASES,)async def test_rate_limit(@@ -114,12 +115,13 @@ class TestAnthropicWorkerErrorHandling:mocker: MockerFixture,_topic: str,error_message: str,+ body: dict[str, Any] | None,expected_category: InferenceErrorCategory,expected_action_substring: str | None,) -> None:- """RateLimitError is caught and categorized correctly (TRANSIENT or CAPACITY)."""+ """RateLimitError is TRANSIENT unless its body carries the spend cap's code, which makes it CAPACITY."""worker = _make_worker(mocker)- sdk_exc = _make_anthropic_rate_limit_error(error_message)+ sdk_exc = _make_anthropic_rate_limit_error(error_message, body=body)worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:@@ -203,10 +205,10 @@ class TestAnthropicWorkerErrorHandling:# ---- PermissionDeniedError tests ----- async def test_permission_denied_quota_is_capacity(self, mocker: MockerFixture) -> None:- """PermissionDeniedError with quota message is CAPACITY."""+ async def test_credit_balance_bad_request_is_capacity(self, mocker: MockerFixture) -> None:+ """Anthropic's 400 for a spent credit balance, whose sentence is its only signal, is CAPACITY."""worker = _make_worker(mocker)- sdk_exc = _make_anthropic_permission_denied_error("Your account quota has been exceeded")+ sdk_exc = _make_anthropic_bad_request_error("Error code: 400", body=DirectProviderQuotaTestData.ANTHROPIC_CREDIT_BALANCE_BODY)worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:@@ -215,9 +217,9 @@ class TestAnthropicWorkerErrorHandling:assert exc_info.value.error_category is InferenceErrorCategory.CAPACITYasync def test_permission_denied_generic_is_configuration(self, mocker: MockerFixture) -> None:- """PermissionDeniedError without quota message is CONFIGURATION."""+ """PermissionDeniedError is CONFIGURATION, even when its words name a quota."""worker = _make_worker(mocker)- sdk_exc = _make_anthropic_permission_denied_error("You do not have access to this resource")+ sdk_exc = _make_anthropic_permission_denied_error("Your account quota has been exceeded")worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:diff --git a/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_object_error_handling.py b/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_object_error_handling.pyindex 87a902913..f9ef167e6 100644--- a/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_object_error_handling.py+++ b/tests/unit/pipelex/providers/anthropic/test_anthropic_worker_object_error_handling.py@@ -7,7 +7,7 @@ are categorized correctly instead of being flattened to ``CONTENT``.from __future__ import annotations-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, Anyimport anthropicimport httpx@@ -21,6 +21,7 @@ from pipelex.cogt.inference.error_classification import UserActionKindfrom pipelex.errors.problem import ProblemViewfrom pipelex.providers.anthropic.anthropic_llm_worker import AnthropicLLMWorkerfrom tests.helpers.instructor_test_utils import DummySchema, make_llm_job, wrap_in_instructor_retry+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDatadef _mock_httpx_response(status_code: int) -> httpx.Response:@@ -28,12 +29,12 @@ def _mock_httpx_response(status_code: int) -> httpx.Response:return httpx.Response(status_code=status_code, request=request)-def _make_anthropic_rate_limit_error(message: str) -> anthropic.RateLimitError:- return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=None)+def _make_anthropic_rate_limit_error(message: str, *, body: dict[str, Any] | None = None) -> anthropic.RateLimitError:+ return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=body)-def _make_anthropic_bad_request_error(message: str) -> anthropic.BadRequestError:- return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=None)+def _make_anthropic_bad_request_error(message: str, *, body: dict[str, Any] | None = None) -> anthropic.BadRequestError:+ return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=body)def _make_anthropic_timeout_error() -> anthropic.APITimeoutError:@@ -127,10 +128,13 @@ class TestAnthropicWorkerObjectErrorHandling:assert metadata.sdk_exception_type == "RateLimitError"assert metadata.status_code == 429- async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_spend_cap_is_capacity(self, mocker: MockerFixture) -> None:+ """The spend cap's 429, recovered from instructor's wrapper with its body, is a spent quota by its code."""_patch_gen_object_dependencies(mocker)worker = _make_worker(mocker)- sdk_exc = _make_anthropic_rate_limit_error("Your account quota has been exceeded")+ sdk_exc = _make_anthropic_rate_limit_error(+ "You have reached your API usage limits", body=DirectProviderQuotaTestData.ANTHROPIC_SPEND_CAP_BODY+ )wrapped = wrap_in_instructor_retry(sdk_exc)worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]@@ -200,10 +204,13 @@ class TestAnthropicWorkerObjectErrorHandling:assert exc_info.value.user_action is not Noneassert exc_info.value.user_action.kind is UserActionKind.CHECK_CREDENTIALS- async def test_wrapped_permission_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_credit_balance_is_capacity(self, mocker: MockerFixture) -> None:+ """The credit balance's 400, recovered from instructor's wrapper with its body, is a spent credit by its sentence."""_patch_gen_object_dependencies(mocker)worker = _make_worker(mocker)- wrapped = wrap_in_instructor_retry(_make_anthropic_permission_denied_error("Your account quota has been exceeded"))+ wrapped = wrap_in_instructor_retry(+ _make_anthropic_bad_request_error("Error code: 400", body=DirectProviderQuotaTestData.ANTHROPIC_CREDIT_BALANCE_BODY)+ )worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:@@ -213,6 +220,20 @@ class TestAnthropicWorkerObjectErrorHandling:assert exc_info.value.user_action is not Noneassert exc_info.value.user_action.kind is UserActionKind.CHECK_BILLING+ async def test_wrapped_permission_denied_with_quota_wording_is_configuration(self, mocker: MockerFixture) -> None:+ """A 403 whose words name a quota is still a refused credential: its wording decides nothing."""+ _patch_gen_object_dependencies(mocker)+ worker = _make_worker(mocker)+ wrapped = wrap_in_instructor_retry(_make_anthropic_permission_denied_error("Your account quota has been exceeded"))+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]++ with pytest.raises(LLMCompletionError) as exc_info:+ await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION+ assert exc_info.value.user_action is not None+ assert exc_info.value.user_action.kind is UserActionKind.CHECK_CREDENTIALS+async def test_wrapped_server_error_is_transient(self, mocker: MockerFixture) -> None:"""A wrapped 5xx APIStatusError is categorized TRANSIENT via the generic fallback branch."""_patch_gen_object_dependencies(mocker)@@ -290,7 +311,7 @@ class TestAnthropicWorkerObjectErrorHandling:"""The operator's problem carries what the provider answered, and the error advises its local reader what to do."""_patch_gen_object_dependencies(mocker)worker = _make_worker(mocker)- sdk_exc = _make_anthropic_rate_limit_error("Your account quota has been exceeded")+ sdk_exc = _make_anthropic_rate_limit_error("Your account quota has been exceeded", body=DirectProviderQuotaTestData.ANTHROPIC_SPEND_CAP_BODY)wrapped = wrap_in_instructor_retry(sdk_exc)worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]@@ -303,6 +324,7 @@ class TestAnthropicWorkerObjectErrorHandling:# The provider's `name` is the backend the error names, which the worker leaves to its caller to fill.assert problem_wire["diagnostics"]["provider"] == {"status": 429,+ "code": "rate_limit_error","message": "Your account quota has been exceeded",}# What the provider answered is the operator's alone.diff --git a/tests/unit/pipelex/providers/anthropic/test_data.py b/tests/unit/pipelex/providers/anthropic/test_data.pyindex da18ada53..113a0a20c 100644--- a/tests/unit/pipelex/providers/anthropic/test_data.py+++ b/tests/unit/pipelex/providers/anthropic/test_data.py@@ -1,32 +1,39 @@"""Test data for Anthropic worker error handling tests."""-from typing import ClassVar+from typing import Any, ClassVarfrom pipelex.cogt.exceptions import InferenceErrorCategory+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDataclass AnthropicErrorHandlingTestData:"""Test cases for Anthropic worker SDK exception handling.- Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none)+ Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none), the rate-limit+ cases carrying the error body after the message."""- RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [+ # (topic, error_message, body, expected_category, expected_user_action_substring_or_none): a 429 is a spent quota+ # on the spend cap's code in its body alone, whatever its words say+ RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, dict[str, Any] | None, InferenceErrorCategory, str | None]]] = [("generic_rate_limit","rate_limit_error: Number of request tokens has exceeded your per-minute limit",+ DirectProviderQuotaTestData.ANTHROPIC_RATE_LIMIT_BODY,InferenceErrorCategory.TRANSIENT,"run it again",),(- "quota_exhaustion",+ "quota_wording_without_the_spend_cap_code","Your account quota has been exceeded",- InferenceErrorCategory.CAPACITY,- "billing",+ None,+ InferenceErrorCategory.TRANSIENT,+ "run it again",),(- "billing_limit",- "You have reached your billing limit",+ "monthly_spend_cap",+ "You have reached your API usage limits",+ DirectProviderQuotaTestData.ANTHROPIC_SPEND_CAP_BODY,InferenceErrorCategory.CAPACITY,"billing",),diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_error_handling.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_error_handling.pyindex 4b3355cd3..6474dd6e1 100644--- a/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_error_handling.py+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_error_handling.py@@ -6,6 +6,7 @@ from typing import TYPE_CHECKING, Anyimport pytestfrom botocore.exceptions import ClientError+from mthds.protocol.problem.registry import CoreCodeif TYPE_CHECKING:from pytest_mock import MockerFixture@@ -133,8 +134,8 @@ class TestBedrockWorkerErrorHandling:@pytest.mark.builds_problems_without_required_params(reason="calls the worker below its public method, whose enrichment names the model and the provider")- async def test_problem_capacity_not_retryable(self, mocker: MockerFixture) -> None:- """The problem of a CAPACITY Bedrock error invites no retry."""+ async def test_problem_service_quota_exceeded_is_retryable(self, mocker: MockerFixture) -> None:+ """Bedrock's service-quota refusal, which AWS says can be resubmitted later, is a rate limit whose problem invites a retry."""worker = _make_bedrock_llm_worker(mocker)sdk_exc = _make_bedrock_client_error("ServiceQuotaExceededException", "Service quota exceeded")worker.bedrock_client_for_text.chat.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]@@ -151,8 +152,9 @@ class TestBedrockWorkerErrorHandling:error = exc_info.valueproblem = error.to_problem(view=ProblemView.OPERATOR)- assert find_inference_error_category_in_chain(error) == "capacity"- assert problem.is_retryable is False+ assert find_inference_error_category_in_chain(error) == "transient"+ assert problem.code == CoreCode.PROVIDER_RATE_LIMITED+ assert problem.is_retryable is True@pytest.mark.builds_problems_without_required_params(reason="calls the worker below its public method, whose enrichment names the model and the provider"diff --git a/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_semantic.py b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_semantic.pyindex 394192ac3..5631558c4 100644--- a/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_semantic.py+++ b/tests/unit/pipelex/providers/bedrock/test_bedrock_worker_semantic.py@@ -63,8 +63,9 @@ class TestBedrockWorkerSemantic:@pytest.mark.parametrize(("error_code", "error_message", "status_code", "expected_category", "expected_kind"),[- ("ServiceQuotaExceededException", "Quota exceeded", 400, InferenceErrorCategory.CAPACITY, UserActionKind.CHECK_BILLING),- ("ThrottlingException", "Quota limit exceeded for this account", 429, InferenceErrorCategory.CAPACITY, UserActionKind.CHECK_BILLING),+ # Bedrock has no balance to spend: none of its answers is a spent quota, whatever its words+ ("ServiceQuotaExceededException", "Quota exceeded", 400, InferenceErrorCategory.TRANSIENT, UserActionKind.WAIT_AND_RETRY),+ ("ThrottlingException", "Quota limit exceeded for this account", 429, InferenceErrorCategory.TRANSIENT, UserActionKind.WAIT_AND_RETRY),("ThrottlingException", "Rate exceeded", 429, InferenceErrorCategory.TRANSIENT, UserActionKind.WAIT_AND_RETRY),("AccessDeniedException", "Not authorized", 403, InferenceErrorCategory.CONFIGURATION, UserActionKind.CHECK_CREDENTIALS),("ValidationException", "Invalid request", 400, InferenceErrorCategory.CONTENT, UserActionKind.CHANGE_INPUT),diff --git a/tests/unit/pipelex/providers/bedrock/test_data.py b/tests/unit/pipelex/providers/bedrock/test_data.pyindex e6e1b1991..a5643dd20 100644--- a/tests/unit/pipelex/providers/bedrock/test_data.py+++ b/tests/unit/pipelex/providers/bedrock/test_data.py@@ -19,19 +19,21 @@ class BedrockLLMErrorHandlingTestData:InferenceErrorCategory.TRANSIENT,"run it again",),+ # Bedrock has no balance to spend: a throttle is a rate limit whatever its words, and AWS documents+ # `ServiceQuotaExceededException` as resubmittable later(- "throttling_quota",+ "throttling_with_quota_wording","ThrottlingException","Account quota limit exceeded for this model",- InferenceErrorCategory.CAPACITY,- "billing",+ InferenceErrorCategory.TRANSIENT,+ "run it again",),("service_quota_exceeded","ServiceQuotaExceededException","You have exceeded the service quota for this resource",- InferenceErrorCategory.CAPACITY,- "billing",+ InferenceErrorCategory.TRANSIENT,+ "run it again",),("access_denied",diff --git a/tests/unit/pipelex/providers/google/test_data.py b/tests/unit/pipelex/providers/google/test_data.pyindex 4dcddeb0b..08cffbb87 100644--- a/tests/unit/pipelex/providers/google/test_data.py+++ b/tests/unit/pipelex/providers/google/test_data.py@@ -29,17 +29,25 @@ class GoogleErrorHandlingTestData:InferenceErrorCategory.CONFIGURATION,None,),+ # Every 429 is a rate limit, whatever its words say; a spent Prepay balance answers 402(- "rate_limit_429_quota",+ "rate_limit_429_quota_wording",429,"Resource has been exhausted (e.g. check quota)",- InferenceErrorCategory.CAPACITY,- "billing",+ InferenceErrorCategory.TRANSIENT,+ "run it again",),(- "rate_limit_429_billing",+ "rate_limit_429_billing_wording",429,"Billing account is not active",+ InferenceErrorCategory.TRANSIENT,+ "run it again",+ ),+ (+ "prepay_depleted_402",+ 402,+ "Payment required.",InferenceErrorCategory.CAPACITY,"billing",),diff --git a/tests/unit/pipelex/providers/google/test_google_img_gen_worker_error_handling.py b/tests/unit/pipelex/providers/google/test_google_img_gen_worker_error_handling.pyindex 1cd023445..09a663afe 100644--- a/tests/unit/pipelex/providers/google/test_google_img_gen_worker_error_handling.py+++ b/tests/unit/pipelex/providers/google/test_google_img_gen_worker_error_handling.py@@ -22,6 +22,7 @@ from pipelex.cogt.img_gen.img_gen_model_rules import AspectRatioTaxonomy, ImgGenfrom pipelex.cogt.inference.error_classification import UserActionKindfrom pipelex.errors.problem import PipelexProblem, ProblemDiagnostics, ProblemView, ProviderDiagnosticsfrom pipelex.providers.google.google_img_gen_worker import GoogleImgGenWorker+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDatadef _make_google_client_error(code: int, message: str, status: str = "ERROR") -> genai_errors.ClientError:@@ -122,9 +123,25 @@ class TestGoogleImgGenWorkerErrorHandling:assert exc_info.value.user_action is not Noneassert exc_info.value.user_action.kind is UserActionKind.CHECK_CREDENTIALS- async def test_quota_429_is_check_billing(self, mocker: MockerFixture) -> None:+ async def test_per_minute_throttle_whose_words_name_a_quota_waits_its_delay(self, mocker: MockerFixture) -> None:+ """Gemini's throttle says "quota" and "billing" and is a rate limit, carrying the wait its `RetryInfo` asks for."""worker = _make_worker(mocker)- sdk_exc = _make_google_client_error(429, "Resource has been exhausted (e.g. check quota)", status="RESOURCE_EXHAUSTED")+ sdk_exc = genai_errors.ClientError(429, DirectProviderQuotaTestData.GOOGLE_PER_MINUTE_THROTTLE_BODY, None)+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]++ with pytest.raises(ImgGenGenerationError) as exc_info:+ await worker._gen_image(img_gen_job=_make_img_gen_job(mocker)) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT+ assert exc_info.value.user_action is not None+ assert exc_info.value.user_action.kind is UserActionKind.WAIT_AND_RETRY+ assert exc_info.value.provider_metadata is not None+ assert exc_info.value.provider_metadata.retry_after_seconds == DirectProviderQuotaTestData.GOOGLE_RETRY_DELAY_SECONDS++ async def test_prepay_depleted_402_is_check_billing(self, mocker: MockerFixture) -> None:+ """Gemini answers a Prepay balance at zero with a 402, a spent quota by its status."""+ worker = _make_worker(mocker)+ sdk_exc = genai_errors.ClientError(402, DirectProviderQuotaTestData.GOOGLE_PREPAY_DEPLETED_BODY, None)worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(ImgGenGenerationError) as exc_info:@@ -248,8 +265,8 @@ class TestGoogleImgGenWorkerErrorHandling:error = exc_info.valueproblem = error.to_problem(view=ProblemView.OPERATOR)- assert find_inference_error_category_in_chain(error) == "capacity"- assert problem.is_retryable is False+ assert find_inference_error_category_in_chain(error) == "transient"+ assert problem.is_retryable is True# The provider's `name` is the backend the error names, which the worker leaves to its caller to fill.assert _provider_of(problem).name is Noneassert _provider_of(problem).status == 429diff --git a/tests/unit/pipelex/providers/google/test_google_llm_worker_object_error_handling.py b/tests/unit/pipelex/providers/google/test_google_llm_worker_object_error_handling.pyindex c4755e40b..1318a2e07 100644--- a/tests/unit/pipelex/providers/google/test_google_llm_worker_object_error_handling.py+++ b/tests/unit/pipelex/providers/google/test_google_llm_worker_object_error_handling.py@@ -121,7 +121,8 @@ class TestGoogleLLMWorkerObjectErrorHandling:assert metadata.request_id == "req_rate"assert metadata.retry_after_seconds == 5.0- async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_resource_exhausted_whose_words_name_a_quota_is_transient(self, mocker: MockerFixture) -> None:+ """Every Gemini 429 `RESOURCE_EXHAUSTED` is a rate limit, whatever its words say."""worker = _make_worker(mocker)sdk_exc = _make_client_error(429,@@ -134,10 +135,10 @@ class TestGoogleLLMWorkerObjectErrorHandling:with pytest.raises(LLMCompletionError) as exc_info:await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]- assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENTassert exc_info.value.user_action is not None- assert exc_info.value.user_action.kind is UserActionKind.CHECK_BILLING- assert "billing" in exc_info.value.user_action.detail.lower()+ assert exc_info.value.user_action.kind is UserActionKind.WAIT_AND_RETRY+ assert "run it again" in exc_info.value.user_action.detailmetadata = exc_info.value.provider_metadataassert metadata is not Noneassert metadata.status_code == 429diff --git a/tests/unit/pipelex/providers/google/test_google_worker_error_handling.py b/tests/unit/pipelex/providers/google/test_google_worker_error_handling.pyindex 28bc45708..e97169bf8 100644--- a/tests/unit/pipelex/providers/google/test_google_worker_error_handling.py+++ b/tests/unit/pipelex/providers/google/test_google_worker_error_handling.py@@ -269,9 +269,9 @@ class TestGoogleWorkerErrorHandling:reason="calls the worker below its public method, whose enrichment names the model and the provider")async def test_problem_capacity_not_retryable(self, mocker: MockerFixture) -> None:- """The problem of a CAPACITY error invites no retry."""+ """The problem of a CAPACITY error, Gemini's 402 for a Prepay balance at zero, invites no retry."""worker = _make_google_llm_worker(mocker)- sdk_exc = _make_google_client_error(code=429, message="Resource has been exhausted (e.g. check quota)")+ sdk_exc = _make_google_client_error(code=402, message="Payment required.")worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:diff --git a/tests/unit/pipelex/providers/mistral/test_data.py b/tests/unit/pipelex/providers/mistral/test_data.pyindex 7ff96258d..62d261b80 100644--- a/tests/unit/pipelex/providers/mistral/test_data.py+++ b/tests/unit/pipelex/providers/mistral/test_data.py@@ -19,12 +19,13 @@ class MistralLLMErrorHandlingTestData:InferenceErrorCategory.CAPACITY,"billing",),+ # Every 429 is a rate limit, the monthly token cap included, whatever its words say; a spent balance answers 402(- "rate_limit_quota_429",+ "rate_limit_429_quota_wording",429,"Rate limit reached: your account quota has been exceeded",- InferenceErrorCategory.CAPACITY,- "billing",+ InferenceErrorCategory.TRANSIENT,+ "run it again",),("rate_limit_generic_429",@@ -87,12 +88,13 @@ class MistralExtractErrorHandlingTestData:InferenceErrorCategory.CAPACITY,"billing",),+ # Every 429 is a rate limit, the monthly token cap included, whatever its words say; a spent balance answers 402(- "rate_limit_quota_429",+ "rate_limit_429_quota_wording",429,"Rate limit reached: your account quota has been exceeded",- InferenceErrorCategory.CAPACITY,- "billing",+ InferenceErrorCategory.TRANSIENT,+ "run it again",),("rate_limit_generic_429",diff --git a/tests/unit/pipelex/providers/mistral/test_mistral_extract_worker_semantic.py b/tests/unit/pipelex/providers/mistral/test_mistral_extract_worker_semantic.pyindex 03228f4ca..d5e3df063 100644--- a/tests/unit/pipelex/providers/mistral/test_mistral_extract_worker_semantic.py+++ b/tests/unit/pipelex/providers/mistral/test_mistral_extract_worker_semantic.py@@ -52,7 +52,8 @@ class TestMistralExtractWorkerSemantic:("status_code", "error_message", "expected_category", "expected_kind"),[(402, "Payment required: insufficient credits", InferenceErrorCategory.CAPACITY, UserActionKind.CHECK_BILLING),- (429, "Too many requests, quota exceeded", InferenceErrorCategory.CAPACITY, UserActionKind.CHECK_BILLING),+ # Every 429 is a rate limit, whatever its words say+ (429, "Too many requests, quota exceeded", InferenceErrorCategory.TRANSIENT, UserActionKind.WAIT_AND_RETRY),(429, "Rate limit exceeded", InferenceErrorCategory.TRANSIENT, UserActionKind.WAIT_AND_RETRY),(401, "Invalid API key", InferenceErrorCategory.CONFIGURATION, UserActionKind.CHECK_CREDENTIALS),(403, "Forbidden", InferenceErrorCategory.CONFIGURATION, UserActionKind.CHECK_CREDENTIALS),diff --git a/tests/unit/pipelex/providers/mistral/test_mistral_llm_worker_object_error_handling.py b/tests/unit/pipelex/providers/mistral/test_mistral_llm_worker_object_error_handling.pyindex 23ce93963..c59bbf0c5 100644--- a/tests/unit/pipelex/providers/mistral/test_mistral_llm_worker_object_error_handling.py+++ b/tests/unit/pipelex/providers/mistral/test_mistral_llm_worker_object_error_handling.py@@ -93,7 +93,8 @@ class TestMistralLLMWorkerObjectErrorHandling:assert metadata.retry_after_seconds == 3.0assert metadata.provider_error_code == "rate_limit_error"- async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_rate_limit_whose_words_name_a_quota_is_transient(self, mocker: MockerFixture) -> None:+ """Every Mistral 429 is a rate limit, whatever its words say; a spent balance answers 402."""worker = _make_worker(mocker)sdk_exc = _make_mistral_error(429,@@ -105,10 +106,10 @@ class TestMistralLLMWorkerObjectErrorHandling:with pytest.raises(LLMCompletionError) as exc_info:await worker._gen_object(llm_job=make_llm_job(mocker), schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]- assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENTassert exc_info.value.user_action is not None- assert exc_info.value.user_action.kind is UserActionKind.CHECK_BILLING- assert "billing" in exc_info.value.user_action.detail.lower()+ assert exc_info.value.user_action.kind is UserActionKind.WAIT_AND_RETRY+ assert "run it again" in exc_info.value.user_action.detailmetadata = exc_info.value.provider_metadataassert metadata is not Noneassert metadata.status_code == 429diff --git a/tests/unit/pipelex/providers/openai/test_data.py b/tests/unit/pipelex/providers/openai/test_data.pyindex a5bf915f2..92dd99892 100644--- a/tests/unit/pipelex/providers/openai/test_data.py+++ b/tests/unit/pipelex/providers/openai/test_data.py@@ -1,35 +1,49 @@"""Test data for OpenAI worker error handling tests."""-from typing import ClassVar+from typing import Any, ClassVarfrom pipelex.cogt.exceptions import InferenceErrorCategory+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDataclass OpenAIErrorHandlingTestData:"""Test cases for OpenAI worker SDK exception handling.- Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none)+ Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none), the rate-limit+ cases carrying the error body after the message."""- RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [+ # (topic, error_message, body, expected_category, expected_user_action_substring_or_none), the body being the error+ # section the OpenAI client keeps: a 429 is a spent quota on OpenAI's quota code or type alone, whatever its words+ RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, dict[str, Any] | None, InferenceErrorCategory, str | None]]] = [("generic_rate_limit","Rate limit exceeded. Please retry after 20s",+ DirectProviderQuotaTestData.OPENAI_RATE_LIMIT_BODY["error"],InferenceErrorCategory.TRANSIENT,"run it again",),("quota_insufficient","Error: insufficient_quota - you have exceeded your billing limit",+ DirectProviderQuotaTestData.OPENAI_INSUFFICIENT_QUOTA_BODY["error"],InferenceErrorCategory.CAPACITY,"billing",),(- "quota_exceeded",- "You exceeded your current quota, please check your plan",+ "billing_code_beside_the_broad_type",+ "Your organization has no prepaid credits remaining.",+ DirectProviderQuotaTestData.OPENAI_CREDIT_BALANCE_EXHAUSTED_BODY["error"],InferenceErrorCategory.CAPACITY,"billing",),+ (+ "quota_wording_without_a_quota_code",+ "You exceeded your current quota, please check your plan",+ None,+ InferenceErrorCategory.TRANSIENT,+ "run it again",+ ),]TIMEOUT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory]]] = [diff --git a/tests/unit/pipelex/providers/openai/test_openai_completions_worker_object_error_handling.py b/tests/unit/pipelex/providers/openai/test_openai_completions_worker_object_error_handling.pyindex 8e1bcd47c..93a128a07 100644--- a/tests/unit/pipelex/providers/openai/test_openai_completions_worker_object_error_handling.py+++ b/tests/unit/pipelex/providers/openai/test_openai_completions_worker_object_error_handling.py@@ -28,11 +28,13 @@ def _mock_httpx_response(status_code: int, headers: dict[str, str] | None = Nonereturn httpx.Response(status_code=status_code, request=request, headers=headers or {})-def _make_openai_rate_limit_error(message: str) -> openai.RateLimitError:+def _make_openai_rate_limit_error(+ message: str, *, error_type: str = "rate_limit_error", error_code: str = "rate_limit_exceeded"+) -> openai.RateLimitError:return openai.RateLimitError(message,response=_mock_httpx_response(429, headers={"x-request-id": "req_rate", "retry-after": "3"}),- body={"type": "rate_limit_error", "code": "rate_limit_exceeded", "message": message},+ body={"type": error_type, "code": error_code, "message": message},)@@ -119,9 +121,12 @@ class TestOpenAICompletionsWorkerObjectErrorHandling:assert metadata.retry_after_seconds == 3.0assert metadata.provider_error_code == "rate_limit_error"- async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_insufficient_quota_is_capacity(self, mocker: MockerFixture) -> None:+ """OpenAI's quota type and code make the 429 a spent quota."""worker = _make_worker(mocker)- sdk_exc = _make_openai_rate_limit_error("Error: insufficient_quota — you exceeded your billing limit")+ sdk_exc = _make_openai_rate_limit_error(+ "Error: insufficient_quota — you exceeded your billing limit", error_type="insufficient_quota", error_code="insufficient_quota"+ )wrapped = wrap_in_instructor_retry(sdk_exc)worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]diff --git a/tests/unit/pipelex/providers/openai/test_openai_responses_worker_object_error_handling.py b/tests/unit/pipelex/providers/openai/test_openai_responses_worker_object_error_handling.pyindex 40559a2ed..bf05e0980 100644--- a/tests/unit/pipelex/providers/openai/test_openai_responses_worker_object_error_handling.py+++ b/tests/unit/pipelex/providers/openai/test_openai_responses_worker_object_error_handling.py@@ -30,11 +30,13 @@ def _mock_httpx_response(status_code: int, headers: dict[str, str] | None = Nonereturn httpx.Response(status_code=status_code, request=request, headers=headers or {})-def _make_openai_rate_limit_error(message: str) -> openai.RateLimitError:+def _make_openai_rate_limit_error(+ message: str, *, error_type: str = "rate_limit_error", error_code: str = "rate_limit_exceeded"+) -> openai.RateLimitError:return openai.RateLimitError(message,response=_mock_httpx_response(429, headers={"x-request-id": "req_rate", "retry-after": "3"}),- body={"type": "rate_limit_error", "code": "rate_limit_exceeded", "message": message},+ body={"type": error_type, "code": error_code, "message": message},)@@ -120,9 +122,12 @@ class TestOpenAIResponsesWorkerObjectErrorHandling:assert metadata.retry_after_seconds == 3.0assert metadata.provider_error_code == "rate_limit_error"- async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:+ async def test_wrapped_billing_code_is_capacity(self, mocker: MockerFixture) -> None:+ """OpenAI's billing code beside its broad quota type makes the 429 a spent quota."""worker = _make_worker(mocker)- sdk_exc = _make_openai_rate_limit_error("Error: insufficient_quota — you exceeded your billing limit")+ sdk_exc = _make_openai_rate_limit_error(+ "Error: insufficient_quota — you exceeded your billing limit", error_type="insufficient_quota", error_code="credit_balance_exhausted"+ )wrapped = wrap_in_instructor_retry(sdk_exc)worker.instructor_for_objects.responses.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]diff --git a/tests/unit/pipelex/providers/openai/test_openai_worker_error_handling.py b/tests/unit/pipelex/providers/openai/test_openai_worker_error_handling.pyindex 50b2444f4..e0b8201a8 100644--- a/tests/unit/pipelex/providers/openai/test_openai_worker_error_handling.py+++ b/tests/unit/pipelex/providers/openai/test_openai_worker_error_handling.py@@ -23,6 +23,7 @@ from pipelex.errors.problem import PipelexProblem, ProblemDiagnostics, ProblemVifrom pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorkerfrom pipelex.providers.openai.openai_img_gen_worker import OpenAIImgGenWorkerfrom pipelex.providers.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker+from tests.unit.pipelex.cogt.test_data import DirectProviderQuotaTestDatafrom tests.unit.pipelex.providers.openai.test_data import OpenAIErrorHandlingTestData@@ -32,8 +33,9 @@ def _mock_httpx_response(status_code: int = 429) -> httpx.Response:return httpx.Response(status_code=status_code, request=request)-def _make_openai_rate_limit_error(message: str) -> openai.RateLimitError:- return openai.RateLimitError(message, response=_mock_httpx_response(429), body=None)+def _make_openai_rate_limit_error(message: str, *, body: dict[str, Any] | None = None) -> openai.RateLimitError:+ """Build the 429, its ``body`` being the error section the OpenAI client keeps."""+ return openai.RateLimitError(message, response=_mock_httpx_response(429), body=body)def _make_openai_bad_request_error(message: str) -> openai.BadRequestError:@@ -174,7 +176,7 @@ class TestOpenAIWorkerErrorHandling:# ---- RateLimitError tests (completions LLM worker) ----@pytest.mark.parametrize(- ("_topic", "error_message", "expected_category", "expected_action_substring"),+ ("_topic", "error_message", "body", "expected_category", "expected_action_substring"),OpenAIErrorHandlingTestData.RATE_LIMIT_CASES,)async def test_completions_llm_rate_limit(@@ -182,12 +184,13 @@ class TestOpenAIWorkerErrorHandling:mocker: MockerFixture,_topic: str,error_message: str,+ body: dict[str, Any] | None,expected_category: InferenceErrorCategory,expected_action_substring: str | None,) -> None:- """RateLimitError is caught and categorized correctly (TRANSIENT or CAPACITY)."""+ """RateLimitError is TRANSIENT unless its code or type is one of OpenAI's quota codes, which makes it CAPACITY."""worker = _make_completions_llm_worker(mocker)- sdk_exc = _make_openai_rate_limit_error(error_message)+ sdk_exc = _make_openai_rate_limit_error(error_message, body=body)worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:@@ -349,9 +352,9 @@ class TestOpenAIWorkerErrorHandling:assert exc_info.value.__cause__ is sdk_excasync def test_responses_llm_rate_limit_quota(self, mocker: MockerFixture) -> None:- """Responses worker: quota exhaustion rate limit is CAPACITY."""+ """Responses worker: a 429 carrying OpenAI's quota code is CAPACITY."""worker = _make_responses_llm_worker(mocker)- sdk_exc = _make_openai_rate_limit_error("insufficient_quota")+ sdk_exc = _make_openai_rate_limit_error("insufficient_quota", body=DirectProviderQuotaTestData.OPENAI_INSUFFICIENT_QUOTA_BODY["error"])worker.openai_client_for_responses.responses.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info:@@ -411,7 +414,7 @@ class TestOpenAIWorkerErrorHandling:async def test_problem_capacity_not_retryable(self, mocker: MockerFixture) -> None:"""The problem of a CAPACITY error invites no retry."""worker = _make_completions_llm_worker(mocker)- sdk_exc = _make_openai_rate_limit_error("insufficient_quota")+ sdk_exc = _make_openai_rate_limit_error("insufficient_quota", body=DirectProviderQuotaTestData.OPENAI_INSUFFICIENT_QUOTA_BODY["error"])worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]with pytest.raises(LLMCompletionError) as exc_info: