Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Gateway-429-read-as-quota · L-261027-4b60ef (#1494)
Louis Choquel651b28a3e9d0
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 9f037b55c..f1fdc40d8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,11 @@# Changelog+## [Unreleased]++### Fixed++- **A gateway's rate limit is no longer read as a spent quota**: a 429 relayed by a Portkey-dialect gateway is now `provider.rate_limited`, retried with the provider's `Retry-After`, unless its error code is one the provider documents for a spent quota: OpenAI's `insufficient_quota`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` or `organization_usage_limit_exceeded`. A 429 whose message mentioned a quota, as Azure OpenAI's throttles do, used to fail at once as `provider.quota_exhausted` with advice to add credit. A 402 is still a spent quota.+## [v0.92.2] - 2026-10-26### Addeddiff --git a/docs/reliability/failure-classification.md b/docs/reliability/failure-classification.mdindex 47ec27695..57e53a5c8 100644--- a/docs/reliability/failure-classification.md+++ b/docs/reliability/failure-classification.md@@ -18,4 +18,12 @@ Whichever path runs, every inference failure is classified the moment it happensOnly `transient` failures are worth retrying — re-running a call that failed because your API key is wrong just wastes time and money. On the [durable path](durable-execution.md), the activity retry policy uses exactly this signal: a `transient` failure retries, every other category fails fast.+## A spent quota or a rate limit++A spent quota and a rate limit both refuse the call, often with the same HTTP status, and they differ only in what cures them: time cures a rate limit, while only a person, adding credit or raising a quota, cures a spent quota. For a failure a gateway relays, Pipelex tells them apart from what the provider says structurally, its status and its error code, and never from the words of its message: a gateway relays every provider's prose, and a throttle's message may well mention a quota, as Azure OpenAI's does by linking to its quota-increase page.++Through a gateway, a `402` is a spent quota, whatever its code says. A `429` is a spent quota only when it carries an error code its provider documents as a spent quota: OpenAI's broad `insufficient_quota`, or one of the billing codes OpenAI's error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, all of which Azure OpenAI and the OpenAI-compatible providers relay. Every other `429`, including one with no code at all, is a rate limit: it is `transient`, so it is retried, and if it still fails it reaches you as `provider.rate_limited`, with the delay the provider asked for when it gave one.++When the structure says nothing either way, Pipelex reads a `429` as a rate limit on purpose, because the two mistakes do not cost the same. A spent quota read as a rate limit costs a few refused calls, which are not billed, and some waiting before the run fails with the provider's code in its diagnostics. A rate limit read as a spent quota fails the run at once and sends you to add credit that cannot help. Each provider Pipelex calls directly, with your own keys, has a rule of its own, and some of those still read the provider's wording.+For the full mechanics of how errors are classified, carried, and reported, see [Error Model](../under-the-hood/error-model.md).diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex 42d369f9f..3105e339e 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -689,7 +689,7 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| Scenario | Behavior ||----------|----------|| Rate limit hit | `TRANSIENT` → repeated within the run; `provider.rate_limited`, whose verdict invites a retry and which answers 429, with a `Retry-After` header when the provider gave a delay; transport retry honors `Retry-After` |-| Quota / billing exhausted | `CAPACITY` → not repeated; `UserAction(CHECK_BILLING)`; `provider.quota_exhausted`, the operator's → HTTP 502 |+| Quota / billing exhausted | `CAPACITY` → not repeated; `UserAction(CHECK_BILLING)`; `provider.quota_exhausted`, the operator's → HTTP 502. Through a gateway, it is read from the status and the provider's error code alone, never the message: a 402 is a spent quota whatever its code, a 429 is one only when its provider code is in `_GATEWAY_QUOTA_ERROR_CODES` (OpenAI's broad `insufficient_quota` and the billing codes its error-code guide names in `error.code`, `credit_balance_exhausted`, `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and `organization_usage_limit_exceeded`, which Azure OpenAI and the OpenAI-compatible providers relay; the Extract step reads a body's `code` before its `type`, so a billing code beside a broad `type` is read as the billing code), and a gateway's 429 with any other code, or none, is a rate limit, whatever its message says || Bad API key | `CONFIGURATION` → not repeated; `provider.credential_refused`, the operator's → HTTP 502 || Model or deployment not found (provider HTTP 404) | Raises a dedicated `*ModelNotFoundError` sibling (`LLMModelNotFoundError`, `ImgGenModelNotFoundError`, `ExtractModelNotFoundError`, `SearchModelNotFoundError`) carrying `model.unavailable`, since a method cannot name a provider's raw model id; operator re-raises `PipeOperatorModelAvailabilityError` || Model reference unknown to the deck (a method naming `gpt-5.1`, a mistyped `@alias` or `$preset`) | `ModelChoiceNotFoundError` → `CONFIGURATION` category, but the code `model.unknown`, the method's → **HTTP 422**, whose sentence names the reference and whose `suggestions` parameter carries the close matches of every catalog the deck knows, an empty list when none is near, which are all a caller needs to fix the method. A handle the deck serves only as another type of model is `model.capability_unsupported` |diff --git a/pipelex/cogt/inference/error_classification.py b/pipelex/cogt/inference/error_classification.pyindex e842d041c..4ccdb9043 100644--- a/pipelex/cogt/inference/error_classification.py+++ b/pipelex/cogt/inference/error_classification.py@@ -1,7 +1,7 @@"""Helpers for classifying SDK errors into InferenceErrorCategory values.-Pure functions that inspect error messages to discriminate between-quota exhaustion vs rate limiting, detect content policy violations, and+Pure functions that inspect error statuses, codes and messages to discriminate+between quota exhaustion vs rate limiting, detect content policy violations, andrecover the underlying SDK exception that ``InstructorRetryException``wraps when ``instructor`` exhausts its retry loop."""@@ -112,8 +112,10 @@ class ProviderErrorMetadata(BaseModel):def is_quota_exhaustion(self) -> bool:"""Whether this error is a quota/credits exhaustion rather than rate limiting.- Dispatches on ``provider`` because each provider phrases quota- exhaustion differently; Mistral and Gateway also use HTTP 402.+ Dispatches on ``provider`` because each provider signals quota+ exhaustion differently; Mistral and Gateway also use HTTP 402. A+ gateway relays every provider's prose, so its rule reads the status and+ the provider's error code alone, never the message."""match self.provider:case ProviderName.OPENAI:@@ -127,7 +129,7 @@ class ProviderErrorMetadata(BaseModel):case ProviderName.BEDROCK:return _is_quota_exhaustion_aws(self.message, provider_error_code=self.provider_error_code)case ProviderName.GATEWAY:- return _is_quota_exhaustion_gateway(self.message, status_code=self.status_code or 0)+ return _is_quota_exhaustion_gateway(status_code=self.status_code or 0, provider_error_code=self.provider_error_code)case (ProviderName.AZURE| ProviderName.FAL@@ -307,15 +309,27 @@ _AWS_QUOTA_PATTERNS: tuple[str, ...] = ("service quota",)-_GATEWAY_QUOTA_PATTERNS: tuple[str, ...] = (- "quota",- "billing limit",- "billing quota",- "insufficient_quota",- "insufficient credit",- "insufficient funds",- "insufficient balance",- "credits exhausted",+# The provider error codes that make a gateway's 429 a spent quota rather than a rate limit. A gateway relays every+# provider's prose, so the message decides nothing: Azure's throttle links to its quota-increase page, and Google's+# per-minute limit says "check quota". A code joins this set only when the provider that sends it documents it as a+# spent quota on a 429, with a comment naming that provider and a test case. A 402 needs no entry, since its status+# decides it, and a gateway's own codes are read from the service error vocabulary before this rule runs.+_GATEWAY_QUOTA_ERROR_CODES: frozenset[str] = frozenset(+ {+ # OpenAI's codes for a spent quota on a 429, which Azure OpenAI and the OpenAI-compatible providers relay in the+ # same fields (https://developers.openai.com/api/docs/guides/error-codes). OpenAI names each billing cause in+ # `error.code` and says `error.type` can still be `insufficient_quota`; the gateway's Extract step reads the+ # body's code before its type, so the specific codes are listed beside the broad one.+ "insufficient_quota",+ # no prepaid credits remaining+ "credit_balance_exhausted",+ # the organization's enforced spend limit+ "organization_spend_limit_exceeded",+ # the project's enforced spend limit+ "project_spend_limit_exceeded",+ # the organization's OpenAI-assigned usage limit+ "organization_usage_limit_exceeded",+ })@@ -364,16 +378,19 @@ def _is_quota_exhaustion_aws(error_message: str, *, provider_error_code: str | Nreturn any(pattern in lower_message for pattern in _AWS_QUOTA_PATTERNS)-def _is_quota_exhaustion_gateway(error_message: str, *, status_code: int) -> bool:- """Check if a Portkey/Gateway error indicates quota/credits exhaustion.+def _is_quota_exhaustion_gateway(*, status_code: int, provider_error_code: str | None) -> bool:+ """Check if a Portkey/Gateway error indicates quota/credits exhaustion, from its status and its provider code alone.- HTTP 402 (Payment Required) is a definitive quota signal.- HTTP 429 requires message inspection to distinguish quota from rate limiting.+ HTTP 402 (Payment Required) is a spent quota whatever its code. HTTP 429 is a spent quota only when the provider's+ error code is one of ``_GATEWAY_QUOTA_ERROR_CODES``; any other 429, one with no code included, is a rate limit,+ which the classifier's 429 arm reads as ``provider.rate_limited``. The message is never read: a gateway relays+ every provider's prose, and a word list keeps meeting a new provider's wording. When the structure says nothing+ either way, a quota read as a rate limit costs some refused calls and some waiting, while a rate limit read as a+ quota fails the run and sends a person to add credit that cannot help."""if status_code == 402:return True- lower_message = error_message.lower()- return status_code == 429 and any(pattern in lower_message for pattern in _GATEWAY_QUOTA_PATTERNS)+ return status_code == 429 and provider_error_code in _GATEWAY_QUOTA_ERROR_CODESdef _is_content_policy_violation(error_message: str) -> bool:diff --git a/pipelex/cogt/inference/error_classify.py b/pipelex/cogt/inference/error_classify.pyindex e92144bc2..e4877d5e1 100644--- a/pipelex/cogt/inference/error_classify.py+++ b/pipelex/cogt/inference/error_classify.py@@ -312,7 +312,9 @@ def classify_inference_error(metadata: SDKErrorEnvelope, *, family: InferenceErr# Quota exhaustion is decided by the provider-aware ``is_quota_exhaustion``# property and takes precedence over the HTTP status: providers signal it on- # different statuses (OpenAI/Anthropic 429, Mistral/Gateway 402, AWS 400).+ # different statuses (OpenAI/Anthropic 429, Mistral/Gateway 402, AWS 400). A+ # gateway's 429 is a quota only on the provider's quota code; any other 429+ # falls through to the arm below and is a rate limit.if metadata.is_quota_exhaustion:return ClassificationResult(category=InferenceErrorCategory.CAPACITY,diff --git a/subject_grants.toml b/subject_grants.tomlindex d1ae791b5..8e5c897dc 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -1624,10 +1624,6 @@ rationale = "Predicate on the error_message; single operand."param = "error_message"rationale = "Predicate on the error_message; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_gateway"]-param = "error_message"-rationale = "Predicate on the error_message; single operand."-["pipelex/cogt/inference/error_classification.py::_is_quota_exhaustion_google"]param = "error_message"rationale = "Predicate on the error_message; single operand."diff --git a/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.py b/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.pynew file mode 100644index 000000000..554d596d4--- /dev/null+++ b/tests/unit/pipelex/cogt/inference/test_gateway_throttle_classification.py@@ -0,0 +1,132 @@+from __future__ import annotations++import copy+import json+from typing import TYPE_CHECKING, Any++import httpx+import pytest+from mthds.protocol.problem.registry import CoreCode+from portkey_ai import Portkey+from portkey_ai._vendor import openai as portkey_vendored_openai++from pipelex.cogt.exceptions import InferenceErrorCategory+from pipelex.cogt.inference.error_classification import UserActionKind, extract_gateway_metadata+from pipelex.cogt.inference.error_classify import InferenceErrorFamily, classify_inference_error+from pipelex.cogt.inference.error_render import render_inference_error+from pipelex.cogt.inference.provider_name import ProviderName+from pipelex.errors.problem import ProblemView+from tests.unit.pipelex.cogt.test_data import GatewayThrottleTestData++if TYPE_CHECKING:+ from collections.abc import Callable++_ORIGIN = "https://gateway.example.com"+++def _gateway_response(*, body: dict[str, Any], retry_after: str) -> tuple[httpx.Request, httpx.Response]:+ request = httpx.Request("POST", f"{_ORIGIN}/v1/images/generations")+ response = httpx.Response(+ status_code=429,+ request=request,+ content=json.dumps(body).encode(),+ headers={"content-type": "application/json", "retry-after": retry_after},+ )+ return request, response+++def _as_portkey_raises_it(*, body: dict[str, Any], retry_after: str) -> BaseException:+ """Build the exception through Portkey's own factory, which keeps only the message string on ``body``."""+ request, response = _gateway_response(body=body, retry_after=retry_after)+ client = Portkey(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1")+ return client._make_status_error_from_response(request=request, response=response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _as_portkey_s_vendored_openai_raises_it(*, body: dict[str, Any], retry_after: str) -> BaseException:+ """Build the exception through the OpenAI client Portkey vendors, the hop a gateway's image call takes.++ Its factory renders the whole body into the message and keeps the ``error`` section on ``body``, which is the+ shape of the refusal's diagnostics on the hosted API.+ """+ _, response = _gateway_response(body=body, retry_after=retry_after)+ client = portkey_vendored_openai.OpenAI(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1")+ return client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++_HOPS = [+ pytest.param(_as_portkey_raises_it, id="portkey-factory"),+ pytest.param(_as_portkey_s_vendored_openai_raises_it, id="portkey-vendored-openai"),+]+++class TestGatewayThrottleClassification:+ @pytest.mark.parametrize("raise_it", _HOPS)+ def test_azure_s_throttle_is_a_rate_limit_waited_for_the_delay_it_asks(self, raise_it: Callable[..., BaseException]) -> None:+ """Azure's refusal, verbatim, through each hop that raises a gateway's failure, is a retryable rate limit.++ The gateway's quota rule reads the status and the provider's code alone, so the message's link to Azure's+ quota-increase page decides nothing: the refusal is ``provider.rate_limited``, repeated within the run, and+ carries Azure's 57 seconds, where it used to fail at once as ``provider.quota_exhausted``.+ """+ exc = raise_it(body=GatewayThrottleTestData.AZURE_RATE_LIMIT_BODY, retry_after=GatewayThrottleTestData.AZURE_RETRY_AFTER)++ metadata = extract_gateway_metadata(exc)+ classification = classify_inference_error(metadata, family=InferenceErrorFamily.IMG_GEN)++ assert metadata.provider == ProviderName.GATEWAY+ assert metadata.status_code == 429+ assert metadata.provider_error_code == GatewayThrottleTestData.AZURE_RATE_LIMIT_CODE+ assert metadata.retry_after_seconds == 57+ assert "quota" in metadata.message+ assert metadata.is_quota_exhaustion is False+ assert classification.error_code == CoreCode.PROVIDER_RATE_LIMITED+ assert classification.category == InferenceErrorCategory.TRANSIENT+ assert classification.category.is_retryable is True+ assert classification.user_action_kind == UserActionKind.WAIT_AND_RETRY++ rendered = render_inference_error(+ metadata=metadata,+ classification=classification,+ family=InferenceErrorFamily.IMG_GEN,+ model_desc="gpt-image-2 → Model[gpt-image-2]",+ model_handle="gpt-image-2",+ )+ # As the worker's error enrichment does, which names the model and the backend the problem's sentence reads+ rendered.fill_model_and_provider("gpt-image-2", backend_name="pipelex_manifold")+ problem = rendered.to_problem(view=ProblemView.OPERATOR)++ assert problem.code == CoreCode.PROVIDER_RATE_LIMITED+ assert problem.is_retryable is True+ assert problem.retry_after == 57++ @pytest.mark.parametrize(+ ("error_type", "error_code"),+ [+ pytest.param(None, GatewayThrottleTestData.OPENAI_QUOTA_CODE, id="broad-code"),+ # OpenAI's billing shape: the cause in `code`, which the Extract step reads before the broad `type`+ pytest.param(+ GatewayThrottleTestData.OPENAI_QUOTA_CODE,+ GatewayThrottleTestData.OPENAI_CREDIT_BALANCE_EXHAUSTED_CODE,+ id="billing-code-beside-the-broad-type",+ ),+ ],+ )+ @pytest.mark.parametrize("raise_it", _HOPS)+ def test_the_same_refusal_on_openai_s_quota_code_is_a_spent_quota(+ self, raise_it: Callable[..., BaseException], error_type: str | None, error_code: str+ ) -> None:+ """The contrast: only the provider's code changes, and the code alone makes the 429 a spent quota."""+ body = copy.deepcopy(GatewayThrottleTestData.AZURE_RATE_LIMIT_BODY)+ body["error"]["type"] = error_type+ body["error"]["code"] = error_code+ exc = raise_it(body=body, retry_after=GatewayThrottleTestData.AZURE_RETRY_AFTER)++ metadata = extract_gateway_metadata(exc)+ classification = classify_inference_error(metadata, family=InferenceErrorFamily.IMG_GEN)++ assert metadata.provider_error_code == error_code+ assert metadata.is_quota_exhaustion is True+ assert classification.error_code == CoreCode.PROVIDER_QUOTA_EXHAUSTED+ assert classification.category == InferenceErrorCategory.CAPACITY+ assert classification.category.is_retryable is False+ assert classification.user_action_kind == UserActionKind.CHECK_BILLINGdiff --git a/tests/unit/pipelex/cogt/test_data.py b/tests/unit/pipelex/cogt/test_data.pyindex 121fe01e8..f1aad2775 100644--- a/tests/unit/pipelex/cogt/test_data.py+++ b/tests/unit/pipelex/cogt/test_data.py@@ -1,4 +1,4 @@-from typing import ClassVar+from typing import Any, ClassVarfrom mthds.protocol.problem.registry import CoreCode, ProblemDomain@@ -28,3 +28,36 @@ class ExceptionTestData:("ambiguous_is_an_unknown_outcome", InferenceErrorCategory.AMBIGUOUS, CoreCode.PROVIDER_OUTCOME_UNKNOWN, ProblemDomain.RUNTIME, True),("unknown_keeps_the_fallback", InferenceErrorCategory.UNKNOWN, CoreCode.RUNTIME_UNEXPECTED, ProblemDomain.RUNTIME, False),]+++class GatewayThrottleTestData:+ # Azure OpenAI's refusal of a `gpt-image-2` call over its deployment's call rate, verbatim as a run on the hosted API+ # received it through the gateway, which prefixes the provider's message with the provider's name. Its code is+ # Azure's rate-limit code, and its last sentence links to Azure's quota-increase page, whose address holds the word+ # "quota": a rule reading the message for that word took this throttle for a spent quota.+ AZURE_RATE_LIMIT_MESSAGE: ClassVar[str] = (+ "azure-openai error: Your requests to gpt-image-2 for gpt-image-2 in East US 2 have exceeded the call rate limit "+ "for your current AIServices S0 pricing tier. This request was for ImageGenerations_Create under Azure OpenAI API "+ "version 2025-04-01-preview. Please retry after 57 seconds. To increase your default rate limit, visit: "+ "https://aka.ms/oai/quotaincrease."+ )+ AZURE_RATE_LIMIT_CODE: ClassVar[str] = "RateLimitReached"+ # The delay the gateway forwards from Azure in the `retry-after` header+ AZURE_RETRY_AFTER: ClassVar[str] = "57"+ # The whole body the gateway answered with, at HTTP 429+ AZURE_RATE_LIMIT_BODY: ClassVar[dict[str, Any]] = {+ "error": {+ "message": AZURE_RATE_LIMIT_MESSAGE,+ "type": None,+ "param": None,+ "code": AZURE_RATE_LIMIT_CODE,+ },+ "provider": "azure-openai",+ }++ # OpenAI's broad code for a spent quota on a 429, which Azure OpenAI and the OpenAI-compatible providers relay+ OPENAI_QUOTA_CODE: ClassVar[str] = "insufficient_quota"+ OPENAI_QUOTA_MESSAGE: ClassVar[str] = "You exceeded your current quota, please check your plan and billing details."+ # One of the billing codes OpenAI's error-code guide documents for a 429 in `error.code`, whose `error.type` can+ # still be `insufficient_quota`+ OPENAI_CREDIT_BALANCE_EXHAUSTED_CODE: ClassVar[str] = "credit_balance_exhausted"diff --git a/tests/unit/pipelex/cogt/test_error_classification.py b/tests/unit/pipelex/cogt/test_error_classification.pyindex b4ae421c9..afa5c5944 100644--- a/tests/unit/pipelex/cogt/test_error_classification.py+++ b/tests/unit/pipelex/cogt/test_error_classification.py@@ -3,7 +3,8 @@The quota and content-policy probes are exposed via the``ProviderErrorMetadata.is_quota_exhaustion`` / ``is_content_policy_violation``properties; the helpers backing them are module-private. These tests pin the-per-provider phrase matching so changes to the patterns surface as test diffs.+per-provider phrase matching so changes to the patterns surface as test diffs,+and the gateway's rule, which reads the status and the provider's code alone."""from typing import ClassVar@@ -12,14 +13,18 @@ import pytestfrom pipelex.cogt.inference.error_classification import ProviderErrorMetadatafrom pipelex.cogt.inference.provider_name import ProviderName+from tests.unit.pipelex.cogt.test_data import GatewayThrottleTestData-def _metadata(provider: ProviderName, message: str, status_code: int | None = None) -> ProviderErrorMetadata:+def _metadata(+ provider: ProviderName, message: str, status_code: int | None = None, *, provider_error_code: str | None = None+) -> ProviderErrorMetadata:return ProviderErrorMetadata(provider=provider,sdk_exception_type="RateLimitError",message=message,status_code=status_code,+ provider_error_code=provider_error_code,)@@ -86,22 +91,46 @@ class _TestCases:("unrelated_error", "Access denied", False),]- # (topic, message, status_code, expected_is_quota)- GATEWAY_QUOTA_CASES: ClassVar[list[tuple[str, str, int, bool]]] = [- ("payment_required_402", "Payment required", 402, True),- ("quota_429", "Your quota has been exhausted", 429, True),- ("billing_429", "Billing limit exceeded", 429, True),- ("insufficient_credits_429", "Insufficient credits", 429, True),- ("insufficient_quota_429", "insufficient_quota for this account", 429, True),- ("insufficient_funds_429", "Insufficient funds in your account", 429, True),- ("insufficient_balance_429", "Insufficient balance to complete request", 429, True),- ("credits_exhausted_429", "Credits exhausted on this key", 429, True),- ("insufficient_permissions_429_not_quota", "Insufficient permissions to use this model", 429, False),- ("insufficient_role_429_not_quota", "Insufficient role for the requested action", 429, False),- ("insufficient_access_429_not_quota", "Insufficient access privileges", 429, False),- ("generic_rate_limit_429", "Rate limit exceeded. Please retry", 429, False),- ("server_error_500", "Internal server error", 500, False),- ("payment_required_402_any_message", "random text", 402, True),+ # (topic, message, status_code, provider_error_code, expected_is_quota)+ # A gateway's rule reads the status and the provider's code, never the message: a 402 is a quota whatever it says,+ # a 429 is a quota only on a documented quota code, and every other 429 is a rate limit, whatever its words.+ GATEWAY_QUOTA_CASES: ClassVar[list[tuple[str, str, int, str | None, bool]]] = [+ ("payment_required_402", "Payment required", 402, None, True),+ ("payment_required_402_any_message", "random text", 402, None, True),+ ("payment_required_402_any_code", "Payment required", 402, GatewayThrottleTestData.AZURE_RATE_LIMIT_CODE, True),+ ("openai_insufficient_quota_code_429", GatewayThrottleTestData.OPENAI_QUOTA_MESSAGE, 429, GatewayThrottleTestData.OPENAI_QUOTA_CODE, True),+ ("insufficient_quota_code_429_any_message", "random text", 429, GatewayThrottleTestData.OPENAI_QUOTA_CODE, True),+ # OpenAI's billing codes, each naming the cause of a spent quota in `error.code`+ (+ "openai_credit_balance_exhausted_code_429",+ "Credit balance exhausted",+ 429,+ GatewayThrottleTestData.OPENAI_CREDIT_BALANCE_EXHAUSTED_CODE,+ True,+ ),+ ("openai_organization_spend_limit_code_429", "Organization spend limit reached", 429, "organization_spend_limit_exceeded", True),+ ("openai_project_spend_limit_code_429", "Project spend limit reached", 429, "project_spend_limit_exceeded", True),+ ("openai_organization_usage_limit_code_429", "Organization usage limit reached", 429, "organization_usage_limit_exceeded", True),+ (+ "azure_rate_limit_reached_429_linking_its_quota_increase_page",+ GatewayThrottleTestData.AZURE_RATE_LIMIT_MESSAGE,+ 429,+ GatewayThrottleTestData.AZURE_RATE_LIMIT_CODE,+ False,+ ),+ ("azure_message_429_no_code", GatewayThrottleTestData.AZURE_RATE_LIMIT_MESSAGE, 429, None, False),+ ("generic_rate_limit_429_no_code", "Rate limit exceeded. Please retry", 429, None, False),+ # Each message the former word list read as a quota, now a rate limit since no code says otherwise+ ("quota_message_429_no_code", "Your quota has been exhausted", 429, None, False),+ ("billing_message_429_no_code", "Billing limit exceeded", 429, None, False),+ ("insufficient_credits_message_429_no_code", "Insufficient credits", 429, None, False),+ ("insufficient_quota_message_429_no_code", "insufficient_quota for this account", 429, None, False),+ ("insufficient_funds_message_429_no_code", "Insufficient funds in your account", 429, None, False),+ ("insufficient_balance_message_429_no_code", "Insufficient balance to complete request", 429, None, False),+ ("credits_exhausted_message_429_no_code", "Credits exhausted on this key", 429, None, False),+ # The quota code decides a 429 only; on any other status the status ladder reads it+ ("insufficient_quota_code_500_not_quota", "Internal server error", 500, GatewayThrottleTestData.OPENAI_QUOTA_CODE, False),+ ("server_error_500", "Internal server error", 500, None, False),]CONTENT_POLICY_CASES: ClassVar[list[tuple[str, str, bool]]] = [@@ -169,9 +198,12 @@ class TestErrorClassification:assert _metadata(ProviderName.BEDROCK, error_message).is_quota_exhaustion == expected@pytest.mark.parametrize(- ("_topic", "error_message", "status_code", "expected"),+ ("_topic", "error_message", "status_code", "provider_error_code", "expected"),_TestCases.GATEWAY_QUOTA_CASES,)- def test_is_quota_exhaustion_gateway(self, _topic: str, error_message: str, status_code: int, expected: bool) -> None:- """Discriminate Gateway quota exhaustion from generic rate limiting."""- assert _metadata(ProviderName.GATEWAY, error_message, status_code).is_quota_exhaustion == expected+ def test_is_quota_exhaustion_gateway(+ self, _topic: str, error_message: str, status_code: int, provider_error_code: str | None, expected: bool+ ) -> None:+ """Discriminate Gateway quota exhaustion from rate limiting by the status and the provider's code alone."""+ metadata = _metadata(ProviderName.GATEWAY, error_message, status_code, provider_error_code=provider_error_code)+ assert metadata.is_quota_exhaustion == expected