Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix/Structured-error-classification · L-260927-086ea4 — model_not_allowed says change input (#1293)
Louis Choquel8ab46edd182b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8ffeca761..f645fe76b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -27,7 +27,8 @@- **`bedrock_anthropic` models run with `AWS_BEARER_TOKEN_BEDROCK` set**: under `bedrock_access_variant = "aws_access"`, the default, Anthropic models on Bedrock sign their requests with the configured AWS access keys and ignore a Bedrock bearer token in the environment, where every one of them used to fail at client construction with a raw `ValueError`. When the access keys are missing and the environment carries such a token, the `AwsCredentialsError` now says so and names `bedrock_access_variant = "bedrock_token"` as the way to authenticate with a bearer token.- **Structured output on the `google` backend accepts enum fields**: under `instructor/genai_tools`, `instructor` validated Gemini's function-call arguments strictly and refused a string for any enum field, so every schema with an enum failed on every attempt; those arguments are now validated in lax mode, so a `google.toml` an earlier `pipelex init` copied keeps working unchanged. The backend file we ship now sets `structure_method = "instructor/genai_structured_outputs"`, Gemini's native JSON output.- **Structured output on the `google` backend sends the system prompt, temperature and token limit**: `instructor` ignored the Google config pipelex handed it, so every structured call ran with no system prompt, the default temperature and the model's default output token limit, whichever structure method the backend used; they now reach Gemini.-- **A transport error inside a structured-output call is classified again under `instructor` 1.16 and later**: those releases wrap a provider's rate limit, timeout or refused connection in their own retry exception, which pipelex classified as an unknown error, or after a re-ask as the earlier parse failure; the provider's error is now read first, so it keeps its category, its retryability and its provider metadata.+- **A transport error inside a structured-output call is classified again under `instructor` 1.15.3 and later**: those releases wrap a provider's rate limit, timeout or refused connection in their own retry exception, which pipelex classified as an unknown error, or after a re-ask as the earlier parse failure; the provider's error is now read first, so it keeps its category, its retryability and its provider metadata.+- **A gateway that does not allow the model asks for another model**: the inference gateway's `model_not_allowed_error` refusal (HTTP 412) now reads as a configuration error whose `change_model` action names the model handle and says who settles a model deck that lists a model the gateway refuses, where it said to review the prompt, parameters and inputs. The HTTP status stays 500.- **A structured-output call is re-asked when the model returns no tool call or no JSON**: under `instructor` 1.16 and later such a response raises a parse error that pipelex's re-ask loop did not retry, so it failed at once; the loop now retries exactly the parse errors `instructor` itself re-asks.- **A model whose `structure_method` names a provider-specific `instructor` mode sets up again**: `instructor` 1.17 refuses another provider's mode on an OpenAI-compatible client, which failed the setup of such a model, the gateway's `mistral-large` among them; each structure method now builds with the core mode `instructor` itself resolves it to.- **Mistral OCR on an image returns the figures it finds**: an extract from an image, rather than a document, failed with `MistralExtractResponseError` whenever Mistral OCR detected a figure inside it, because the request never asked for the figures' pixels; it now sends the same image options as a document extract, so `max_nb_images` and `image_min_size` apply to images too, and a request for captions, which Mistral OCR cannot write, is refused on an image as it already was on a document instead of returning none.diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex 8f4b12fcc..fbe8f83e1 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -352,7 +352,7 @@ Everything said above about the request limits holds here too — the code is th#### When the model cannot be routed-The third family, and the one no request escapes: whatever else it asks, it has to be *routed* first — the gateway reads the model out of it, looks that model up in its own routing table, and hands the call to the integration that serves it. When that resolution fails there is no provider to send the request to, so the gateway refuses it itself. `GatewayRoutingRefusal` is the runtime's name for each outcome.+The third family, and the one no request escapes: whatever else it asks, it has to be *routed* first — the gateway reads the model out of it, looks that model up in its own routing table, and hands the call to the integration that serves it. When that resolution fails there is no provider to send the request to, so the gateway refuses it itself; and when it succeeds, the integration it lands on may still refuse a model it does not allow. `GatewayRoutingRefusal` is the runtime's name for each outcome.| Gateway code | HTTP | `GatewayRoutingRefusal` | Category / action | What the caller is told ||---|---|---|---|---|@@ -360,15 +360,17 @@ The third family, and the one no request escapes: whatever else it asks, it has| `pig-02` | 400 | `DISABLED_INTEGRATION` | `CONFIGURATION` / `CONTACT_SUPPORT` | the model's integration is not enabled on this deployment — its credentials are unset || `pig-05` | 400 | `WRONG_PROTOCOL` | `CONFIGURATION` / `CHANGE_MODEL` | the model is served, but not over the protocol the request used for it || `pig-06` | 400 | `UNSERVED_CAPABILITY` | `CONFIGURATION` / `CHANGE_MODEL` | the model's integration does not serve that capability (extract, search) |+| `model_not_allowed_error` | 412 | `MODEL_NOT_ALLOWED` | `CONFIGURATION` / `CHANGE_MODEL` | the gateway does not allow the model, named by its handle, for this account — pick another; if the pipe named it, leaving the pipe's model unset uses the default |-Everything said about the two families above holds here too — the code is the discriminator rather than the provider, the check runs ahead of the status ladder, the advice defers every specific to the gateway's own message, and nothing is ever retried. Four things are particular to this family:+Everything said about the two families above holds here too — the code is the discriminator rather than the provider, the check runs ahead of the status ladder, the advice defers every specific to the gateway's own message, with the one exception below, and nothing is ever retried. What is particular to this family:-- **The category is `CONFIGURATION`, never `CONTENT`, and that is the whole bug this family fixes.** Nothing in the prompt, the parameters or the inputs causes any of these, and no edit to them avoids one. Before the family existed all four fell through to the status ladder's 400 arm, so a caller who named a model the deployment does not serve was told *"The provider rejected the request — review the prompt, parameters, and inputs"* and received an `LLMCompletionError`.-- **Every member is its own wire code, unlike the two families above.** Grouping is by remedy in all three, and here each code names a different thing that has to change — so the grouping happens to be one-to-one rather than by design.-- **Only `pig-01` sets `is_model_not_found`.** The flag is not a category: it selects the family's `*ModelNotFoundError` class, which `pipe_operator.py` re-raises as a `PipeOperatorModelAvailabilityError` carrying the model handle — the pipe-level error a caller already gets when the model deck itself cannot find a model. `pig-01` is exactly that case seen from the gateway. For `pig-05` and `pig-06` the model *does* exist and *is* served, so claiming it was not found would be false; they stay on the generic failure class and carry the distinction in the advice.+- **The category is `CONFIGURATION`, never `CONTENT`, and that is the whole bug this family fixes.** Nothing in the prompt, the parameters or the inputs causes any of these, and no edit to them avoids one. Before the family existed the `pig-` members fell through to the status ladder's 400 arm, so a caller who named a model the deployment does not serve was told *"The provider rejected the request — review the prompt, parameters, and inputs"* and received an `LLMCompletionError`. `model_not_allowed_error` fell through to the ladder's generic 4xx arm instead, which gave it the right category and the same wrong advice.+- **Every member is its own wire code, unlike the two families above.** Grouping is by remedy in all three, and here each code names a different thing that has to change — so the grouping happens to be one-to-one rather than by design. One code is not in the `pig-` namespace: `model_not_allowed_error` is the code of the Portkey substrate the gateway is built on, answered at 412 when an integration does not allow every model and does not list this one, or lists it as archived. No model vendor uses it. Portkey's cloud emits it both for the integrations `pipelex_gateway` reaches by default and for a user's own workspace behind the `portkey` backend, and the manifold's vendored middleware emits it too. The refusal means the same thing and calls for the same move from any of them, so it is matched on the code alone like the rest. It carries its code in `error.type` with `error.code` null, and every Extract hop recovers it from there.+- **Only `pig-01` sets `is_model_not_found`.** The flag is not a category: it selects the family's `*ModelNotFoundError` class, which `pipe_operator.py` re-raises as a `PipeOperatorModelAvailabilityError` carrying the model handle — the pipe-level error a caller already gets when the model deck itself cannot find a model. `pig-01` is exactly that case seen from the gateway. For `pig-05`, `pig-06` and `model_not_allowed_error` the model *does* exist and *is* served, so claiming it was not found would be false; they stay on the generic failure class and carry the distinction in the advice.- **`pig-02` is the family's `CONTACT_SUPPORT` arm**, the same call `STORAGE_NOT_SERVED` and `BODY_LENGTH_REQUIRED` get. A switched-off integration is the gateway operator's fact: `CHECK_CREDENTIALS` would send a hosted caller to rotate their own perfectly valid key, and `CHANGE_MODEL` would send them shopping for a model over an unset variable. The gateway's message names the integration and the variables to set, and the advice points at it.+- **`model_not_allowed_error` is the one member whose advice names the model.** The others defer every specific to the gateway's message, but this one's message names only the backend's wire id (`us.anthropic.claude-sonnet-4-5-20250929-v1:0`), which the method's author never wrote. So its advice names the model handle the deck resolved the pipe's model to, which `render_inference_error` already receives. Both of its hints are conditional, because the Render step cannot tell the cases apart: leaving the pipe's model unset helps only when the pipe named the refused model rather than falling back to it as the default, and a model deck that lists a model the allow-list refuses is Pipelex's to settle on the Pipelex gateway and the user's own on a Portkey workspace of theirs, which report the same provider.-**Moving the family to `CONFIGURATION` moves its HTTP answer too, and that is worth stating out loud.** `InferenceErrorCategory.CONFIGURATION` implies `ErrorDomain.CONFIG`, which `error_domain_to_http_status` renders as **500**, where `CONTENT` implies `INPUT` and renders as 422. So on any surface built on `ErrorReport.http_status`, all four of these now answer 500 where they answered 422 — the same answer the status ladder's 404 arm has always given a missing model, and the right one in kind, since none of them is a property of what the caller submitted. Two consequences follow: these land in server-error alerting rather than client-error, and 500 is a status outer HTTP clients and orchestrators retry. "Never retried" above is a statement about `InferenceErrorCategory.is_retryable` inside this process; it says nothing about what a caller's own client does with the status, and an outer retry of any of these earns an identical refusal.+**Moving the family to `CONFIGURATION` moves its HTTP answer too, and that is worth stating out loud.** `InferenceErrorCategory.CONFIGURATION` implies `ErrorDomain.CONFIG`, which `error_domain_to_http_status` renders as **500**, where `CONTENT` implies `INPUT` and renders as 422. So on any surface built on `ErrorReport.http_status`, the `pig-` members now answer 500 where they answered 422 — the same answer the status ladder's 404 arm has always given a missing model, and the right one in kind, since none of them is a property of what the caller submitted. Two consequences follow: these land in server-error alerting rather than client-error, and 500 is a status outer HTTP clients and orchestrators retry. "Never retried" above is a statement about `InferenceErrorCategory.is_retryable` inside this process; it says nothing about what a caller's own client does with the status, and an outer retry of any of these earns an identical refusal. `model_not_allowed_error` is the exception: the ladder's generic 4xx arm already read its 412 as `CONFIGURATION`, so it answered 500 before the family took it in, and only its `user_action` moved.`pig-05` and `pig-06` are as often a deck-versus-gateway disagreement as a caller's mistake — the runtime picks the protocol and the route from its own model deck — so both details say "your model deck" out loud rather than only "pick another model", which would leave an operator hunting for a model problem that is a configuration problem.@@ -387,7 +389,7 @@ The gateway's remaining codes belong to no family and classify on their status l### The `instructor` Unwrap-On structured-generation paths, `instructor` raises an `InstructorRetryException` from the exception that ended its retry loop. The loop re-asks only a response that fails the schema, so that exception is either the raw SDK exception of a call that failed in transport, or, once the re-ask budget is spent, the last parse failure. `extract_underlying_sdk_exception()` recovers it from the wrapper's cause, not from `failed_attempts`, which lists only parse failures and would name an earlier one when a re-ask then fails in transport. The recovered exception routes through the same per-provider categorization as the plain-text path. A genuinely unrecognized inner exception (e.g. a `pydantic.ValidationError` from a schema mismatch) lands in `UNKNOWN` rather than being mis-labelled as a `CONTENT`-policy violation.+On structured-generation paths, `instructor` raises an `InstructorRetryException` from the exception that ended its retry loop. The loop re-asks only a response that fails the schema, so that exception is either the raw SDK exception of a call that failed in transport, or, once the re-ask budget is spent, the last parse failure. `extract_underlying_sdk_exception()` recovers it from the wrapper's cause, not from `failed_attempts`, which lists only parse failures and would name an earlier one when a re-ask then fails in transport. The recovered exception routes through the same per-provider categorization as the plain-text path. Once the re-ask budget is spent, the recovered exception is the last `pydantic.ValidationError`, which `_STATUSLESS_BY_TYPE_NAME` classifies as `CONTENT` / `CHANGE_INPUT`, so it answers 422. A recovered exception the classifier does not recognize lands in `UNKNOWN`.### Model and Provider Attribution@@ -637,7 +639,8 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| Malformed prompt image / bad prompt parameter | `CONTENT` class-level (`PromptImageFormatError`, `LLMPromptParameterError`, …) → `error_domain = INPUT` → **HTTP 422** || Any other provider **HTTP 400** | `CONTENT` → `error_domain = INPUT` → **HTTP 422**. This is the widest reach of the derivation: a 400 covers a context-length overflow and a parameter the model rejects alike, and an engine-side request-construction fault lands here too — reported as the caller's to fix, and absent from the 5xx rate || Local file extractor raises a builtin (docling, pypdfium2) | `ValueError` / `RuntimeError` / `FileNotFoundError` → `CONTENT` → `error_domain = INPUT` → **HTTP 422**; `OSError` → `TRANSIENT` (see `_LOCAL_EXTRACT_BY_TYPE_NAME`) |-| LLM returns schema-mismatched JSON | `instructor` re-asks; if exhausted → `UNKNOWN` → no `error_domain` asserted → HTTP 500 |+| LLM returns schema-mismatched JSON | `instructor` re-asks; if exhausted → the last `pydantic.ValidationError` → `CONTENT` / `CHANGE_INPUT` → `error_domain = INPUT` → **HTTP 422** |+| The gateway does not allow the model for this account (`model_not_allowed_error`, HTTP 412) | `CONFIGURATION` / `CHANGE_MODEL`, never retried; the advice names the model handle; `error_domain = CONFIG` → HTTP 500 || Connection dropped mid-request | `AMBIGUOUS` → non-retryable (outcome unknown); `error_domain = RUNTIME` || A step names, in an inline model setting, a model the deck neither defines nor names | `ModelNotFoundError`, classified by the lookup as the caller's: `error_domain = INPUT` → **HTTP 422**, caller-facing under STRICT, with a `CHANGE_MODEL` next step naming the model as the method wrote it and the type of model the step needs. A model the deck itself names but does not serve (a preset or alias on a backend that is not enabled) keeps `CONFIG` → HTTP 500, redacted || A `PipeParallel` cannot combine its branch results, or a pipe starts without a required input | The caller's own method or request: `error_domain = INPUT` → **HTTP 422**, caller-facing under STRICT, with a `CHANGE_INPUT` next step |diff --git a/pipelex/cogt/inference/error_classification.py b/pipelex/cogt/inference/error_classification.pyindex 38f146372..b65719455 100644--- a/pipelex/cogt/inference/error_classification.py+++ b/pipelex/cogt/inference/error_classification.py@@ -237,7 +237,12 @@ class GatewayRoutingRefusal(StrEnum):Each member corresponds to one of the gateway's own error codes, which is thecontract between the two repositories — the wording of a refusal is free to- change, the code is not.+ change, the code is not. All but one are the gateway's ``pig-`` codes, at HTTP+ 400. ``MODEL_NOT_ALLOWED`` is keyed on the code of the Portkey substrate the+ gateway is built on, which answers it at HTTP 412 from the middleware the+ gateway vendors; Portkey's cloud answers it the same way, for the integrations+ ``pipelex_gateway`` reaches by default and for a caller's own workspace behind+ the ``portkey`` backend alike."""#: ``pig-01`` at HTTP 400 — the request body names one that no integration@@ -275,6 +280,16 @@ class GatewayRoutingRefusal(StrEnum):#: model named on a pipe it cannot serve: the message names the integration,#: the provider and the capability.UNSERVED_CAPABILITY = "unserved_capability"+ #: ``model_not_allowed_error`` at HTTP 412 — an integration serves the model+ #: but does not allow it for this caller. The substrate raises it in two+ #: cases: the integration does not allow every model and does not list this+ #: one, or it lists it as archived. The caller can pick another model; for+ #: whoever operates the gateway, it means the model deck and the integration's+ #: allow-list disagree. The gateway's message names only the backend's wire+ #: id, which the method's author never wrote, so this is the one member whose+ #: advice names the model handle the deck resolved to (see+ #: ``_render_gateway_routing_refusal_detail``).+ MODEL_NOT_ALLOWED = "model_not_allowed"# The gateway's routing-refusal codes, mapped to what the runtime does about them.@@ -287,6 +302,18 @@ class GatewayRoutingRefusal(StrEnum):# Claude travels on the shared Anthropic driver — while ``pig-`` is the gateway's# own code namespace and no vendor emits into it.#+# **``model_not_allowed_error`` is the one code outside that namespace**, and it is+# matched on the code alone for a reason of its own: it is the code of Portkey, the+# substrate the gateway is built on, and no model vendor uses it. Portkey's cloud+# emits it for the integrations ``pipelex_gateway`` reaches and for a caller's own+# workspace behind the ``portkey`` backend alike, and so does the middleware the+# manifold vendors from it. The refusal means the same thing and calls for the same+# move whichever of them raised it, so only the advice has to hold for both. Both+# raise sites answer 412, so checking the status too would add nothing. It arrives with+# the code in ``type`` and ``code`` null, and every Extract hop recovers it from+# there: the vendor-facing hops read ``type`` first, and the two Pipelex-service+# hops fall back to it.+## **Two of the gateway's routing codes are deliberately absent**, and the omission# is the scope decision rather than an oversight:#@@ -320,6 +347,7 @@ _GATEWAY_ROUTING_REFUSAL_BY_CODE: dict[str, GatewayRoutingRefusal] = {"pig-02": GatewayRoutingRefusal.DISABLED_INTEGRATION,"pig-05": GatewayRoutingRefusal.WRONG_PROTOCOL,"pig-06": GatewayRoutingRefusal.UNSERVED_CAPABILITY,+ "model_not_allowed_error": GatewayRoutingRefusal.MODEL_NOT_ALLOWED,}@@ -436,8 +464,8 @@ class ProviderErrorMetadata(BaseModel):above do, and for the same reason: the gateway can refuse to route arequest before any provider sees it, and the code is the only thing thatsays so. Without this the whole family falls through to the status ladder's- 400 arm and a caller who named a model the deployment does not serve is- told to review their prompt.+ 4xx arms and a caller who named a model the deployment does not serve, or+ may not use, is told to review their prompt.Disjoint from both other families by construction, so the Classify step mayread the three in any order. Returns ``None`` for every other refusal,diff --git a/pipelex/cogt/inference/error_classify.py b/pipelex/cogt/inference/error_classify.pyindex 5852595c1..f1b5ea0ea 100644--- a/pipelex/cogt/inference/error_classify.py+++ b/pipelex/cogt/inference/error_classify.py@@ -195,8 +195,10 @@ def _classify_gateway_routing_refusal(*, refusal: GatewayRoutingRefusal) -> ClasEvery member is ``CONFIGURATION``: nothing in the prompt, the parameters or theinputs causes any of these, and no edit to them avoids one. That is the whole- point of the family — without it all four take the status ladder's 400 arm and- read as a provider rejecting the caller's content.+ point of the family — without it the ``pig-`` members take the status ladder's+ 400 arm and read as a provider rejecting the caller's content, and+ ``MODEL_NOT_ALLOWED`` takes its generic 4xx arm, whose domain is right but whose+ action sends the caller to edit their inputs.None is retryable, for the reason none of the other two gateway families is:the gateway refused before a provider saw the request, and an identical retry@@ -217,9 +219,10 @@ def _classify_gateway_routing_refusal(*, refusal: GatewayRoutingRefusal) -> Clasis_model_not_found=True,gateway_routing_refusal=refusal,)- case GatewayRoutingRefusal.WRONG_PROTOCOL | GatewayRoutingRefusal.UNSERVED_CAPABILITY:- # The model exists and is served — it just cannot do what was asked, or- # was asked over a protocol its integration does not speak. So the flag+ case GatewayRoutingRefusal.WRONG_PROTOCOL | GatewayRoutingRefusal.UNSERVED_CAPABILITY | GatewayRoutingRefusal.MODEL_NOT_ALLOWED:+ # The model exists and is served — it just cannot do what was asked, was+ # asked over a protocol its integration does not speak, or is served by+ # an integration that does not allow it for this caller. So the flag# stays unset: telling a caller the model was not found would be false,# and the Render step carries the real distinction. ``CHANGE_MODEL`` is# still what an end caller can do about it.@@ -281,9 +284,11 @@ def classify_inference_error(metadata: SDKErrorEnvelope) -> ClassificationResult# And its refusals to route the request at all, on the same footing again. All# three gateway code sets are disjoint, so the order among these branches# carries no meaning; what matters is that all three run ahead of the status- # ladder. Every routing refusal arrives on 400, which the ladder would read as- # a provider rejecting the prompt — for a model the gateway does not serve, is- # not configured to reach, or cannot ask what was asked.+ # ladder. The ``pig-`` refusals arrive on 400, which the ladder would read as a+ # provider rejecting the prompt — for a model the gateway does not serve, is+ # not configured to reach, or cannot ask what was asked. The substrate's+ # ``model_not_allowed_error`` arrives on 412, which the ladder would send to+ # the prompt, the parameters and the inputs too.gateway_routing_refusal = metadata.gateway_routing_refusalif gateway_routing_refusal is not None:return _classify_gateway_routing_refusal(refusal=gateway_routing_refusal)diff --git a/pipelex/cogt/inference/error_render.py b/pipelex/cogt/inference/error_render.pyindex 4c9c81fb9..6145f84e3 100644--- a/pipelex/cogt/inference/error_render.py+++ b/pipelex/cogt/inference/error_render.py@@ -139,15 +139,20 @@ def _render_gateway_unresolved_reference_detail(*, reference: GatewayUnresolvedR)-def _render_gateway_routing_refusal_detail(*, refusal: GatewayRoutingRefusal) -> str:+def _render_gateway_routing_refusal_detail(*, refusal: GatewayRoutingRefusal, model_handle: str) -> str:"""Produce the advice for a request the inference gateway refused to route.- Names no model, integration, protocol or capability, for the same reason+ Names no integration, protocol or capability, for the same reason``_render_gateway_limit_detail`` names no number: the gateway's own refusalmessage sits beside this detail and already states every specific. What thistext adds is what the caller — or whoever operates the deployment — should*do*, which the message does not say.+ It names no model either, with one exception: ``MODEL_NOT_ALLOWED``, whose+ message names only the backend's wire id (``us.anthropic.claude-…``), which the+ method's author never wrote. That member's advice names ``model_handle``, the+ handle the model deck resolved the pipe's model to; every other member ignores it.+Every member whose remedy could be a deck edit says "the model deck" out loud.The runtime picks the model, the protocol and the route from its own deck, sothose refusals usually mean the deck and the gateway disagree about a model@@ -187,9 +192,23 @@ def _render_gateway_routing_refusal_detail(*, refusal: GatewayRoutingRefusal) ->"The model's integration does not serve that capability on the inference gateway — the error message names ""the provider and what was asked of it. Pick a model whose provider serves it, or correct the model deck.")+ case GatewayRoutingRefusal.MODEL_NOT_ALLOWED:+ # Both halves are conditional because the Render step cannot tell the+ # cases apart. The handle may be one the pipe named or the deck's+ # default, and leaving the model unset only helps in the first case. And+ # the same code comes from the Pipelex gateway and from a caller's own+ # Portkey workspace behind the ``portkey`` backend, which report the+ # same provider: a deck that lists a model the allow-list refuses is+ # Pipelex's to settle on the first and the caller's on the second.+ return (+ f"The inference gateway does not allow the model '{model_handle}' for this account — pick another model for the "+ "pipe; if the pipe named this one, leaving its model unset uses the default instead. If your model deck lists "+ "it as available, the deck and the gateway's allow-list disagree: on the Pipelex gateway, contact support; on "+ "a Portkey workspace of your own, allow the model in the integration that serves it."+ )-def _render_detail(metadata: SDKErrorEnvelope, *, classification: ClassificationResult) -> str:+def _render_detail(metadata: SDKErrorEnvelope, *, classification: ClassificationResult, model_handle: str) -> str:"""Produce the free-form user-facing advice text for the classified error."""if classification.gateway_request_limit is not None:# Branches ahead of the action kind rather than inside it: the limit names@@ -206,11 +225,11 @@ def _render_detail(metadata: SDKErrorEnvelope, *, classification: Classification# Same reason a third time, and here the action kind is too coarse for# every member rather than for some: ``CHANGE_MODEL`` renders "the# requested model was not found", which is right only for ``pig-01`` and- # false for ``pig-05`` and ``pig-06``, whose model exists and is served —- # and for ``pig-02``, whose model resolves but to an integration that is- # switched off. ``CONTACT_SUPPORT`` renders "the error could not be- # classified" for a refusal that named itself precisely.- return _render_gateway_routing_refusal_detail(refusal=classification.gateway_routing_refusal)+ # false for ``pig-05``, ``pig-06`` and ``model_not_allowed_error``, whose+ # model exists and is served — and for ``pig-02``, whose model resolves but+ # to an integration that is switched off. ``CONTACT_SUPPORT`` renders "the+ # error could not be classified" for a refusal that named itself precisely.+ return _render_gateway_routing_refusal_detail(refusal=classification.gateway_routing_refusal, model_handle=model_handle)match classification.user_action_kind:case UserActionKind.WAIT_AND_RETRY:if metadata.retry_after_seconds is not None:@@ -245,7 +264,8 @@ def render_inference_error(classification: The result of the Classify step.family: The worker family, selecting the concrete exception class.model_desc: Human-readable model description for the error message.- model_handle: The pipelex model handle, carried on ``*ModelNotFoundError``.+ model_handle: The pipelex model handle, carried on ``*ModelNotFoundError`` and named by+ the advice of a routing refusal whose own message names only the backend's id.Returns:A ``CogtError`` subclass instance carrying the category, structured@@ -254,7 +274,7 @@ def render_inference_error(message = _format_message(metadata, model_desc=model_desc)user_action = UserAction(kind=classification.user_action_kind,- detail=_render_detail(metadata, classification=classification),+ detail=_render_detail(metadata, classification=classification, model_handle=model_handle),)if classification.is_model_not_found:not_found_class = _NOT_FOUND_CLASSES[family]diff --git a/tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.py b/tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.pyindex 49139c532..c069711bd 100644--- a/tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.py+++ b/tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.py@@ -7,14 +7,19 @@ resolution fails it refuses the request itself, with codes of its own — ``pig-when it serves no such model, ``pig-02`` when the model's integration is switchedoff for want of a credential, ``pig-05`` when a native-protocol path names a modelanother provider serves, and ``pig-06`` when a model reaches a route its provider-does not serve.+does not serve. One more comes from the Portkey substrate the gateway is built on+rather than from the ``pig-`` namespace: ``model_not_allowed_error``, which its+middleware answers when an integration serves the model but does not allow it for+this caller, because the model is off the integration's allow-list or archived.-Every one of them answers HTTP 400, so without a class for them the Classify step+Every ``pig-`` code answers HTTP 400, so without a class for them the Classify stepfalls through to the status ladder's 400 arm and a caller who named a model thedeployment does not serve is told *"The provider rejected the request — review theprompt, parameters, and inputs."* — and receives an ``LLMCompletionError`` ratherthan the ``LLMModelNotFoundError`` that case has always produced when the model-deck itself cannot find a model. These tests pin the whole chain: the code is+deck itself cannot find a model. ``model_not_allowed_error`` answers 412, which the+ladder's generic 4xx arm already read as a configuration problem, but with the+same advice to review the prompt. These tests pin the whole chain: the code isrecognized, it survives every Extract hop that can carry it, it classifies as aconfiguration problem that is never retried, the Render step says which routingdecision failed and who has to change what, and both consequences of the category@@ -23,7 +28,7 @@ and the advice surviving the pipe layer's re-raise.**Every member is its own wire code here**, unlike the two families beside it.That is not an accident of the code space: grouping is by remedy in all three-families, and each of these four names a different thing that has to change.+families, and each of these names a different thing that has to change.**Only ``pig-01`` sets ``is_model_not_found``.** The flag is not a category — itselects the family's ``*ModelNotFoundError`` class, which ``pipe_operator.py``@@ -31,7 +36,13 @@ re-raises as a ``PipeOperatorModelAvailabilityError`` carrying the model handle.``pig-01`` is literally "this deployment does not know that model", seen from thegateway. For ``pig-05`` and ``pig-06`` the model exists and *is* served — it justcannot do what was asked, or was asked over the wrong protocol — so claiming it-was not found would be false.+was not found would be false. The same holds for ``model_not_allowed_error``: the+model exists and an integration serves it, only not for this caller.++**``model_not_allowed_error`` is the one member whose advice names the model.**+The rest defer every specific to the gateway's own message, but that message names+only the backend's wire id, which the method's author never wrote. So the advice+names the model handle the deck resolved the pipe's model to.**``pig-02`` is the family's ``CONTACT_SUPPORT`` arm.** A switched-off integrationis the gateway operator's fact: nothing about the request causes it, none of the@@ -65,7 +76,7 @@ the trap it is.from __future__ import annotationsimport json-from typing import Any+from typing import TYPE_CHECKING, Anyimport anthropicimport httpx@@ -90,8 +101,12 @@ from pipelex.cogt.inference.error_classify import classify_inference_errorfrom pipelex.cogt.inference.error_render import InferenceErrorFamily, render_inference_errorfrom pipelex.cogt.inference.provider_name import ProviderNamefrom pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError+from pipelex.pipe_run.exceptions import PipeRouterErrorfrom pipelex.system.pipe_run_mode import PipeRunMode+if TYPE_CHECKING:+ from collections.abc import Callable+_ORIGIN = "https://manifold.example.com"# Every code in the family, paired with the member it must reach. Reused by the@@ -102,11 +117,16 @@ _EVERY_CODE_AND_MEMBER: list[tuple[str, GatewayRoutingRefusal]] = [("pig-02", GatewayRoutingRefusal.DISABLED_INTEGRATION),("pig-05", GatewayRoutingRefusal.WRONG_PROTOCOL),("pig-06", GatewayRoutingRefusal.UNSERVED_CAPABILITY),+ ("model_not_allowed_error", GatewayRoutingRefusal.MODEL_NOT_ALLOWED),]# The same codes as a flat list, for the cases that only need the wire code._EVERY_CODE: list[str] = [pair[0] for pair in _EVERY_CODE_AND_MEMBER]+# The members the gateway renders in its own fail-closed envelope, the ``pig-`` ones.+_FAIL_CLOSED_CODES_AND_MEMBERS: list[tuple[str, GatewayRoutingRefusal]] = [pair for pair in _EVERY_CODE_AND_MEMBER if pair[0].startswith("pig-")]+_FAIL_CLOSED_CODES: list[str] = [pair[0] for pair in _FAIL_CLOSED_CODES_AND_MEMBERS]+# Every member whose remedy is picking a different model, i.e. all of them but the# one that means an operator never enabled the integration._CHANGE_MODEL_CODES: list[str] = [code for code, member in _EVERY_CODE_AND_MEMBER if member is not GatewayRoutingRefusal.DISABLED_INTEGRATION]@@ -134,6 +154,23 @@ def _fail_closed_refusal_body(message: str, code: str) -> dict[str, Any]:_UNKNOWN_MODEL_BODY = _fail_closed_refusal_body("No model found for the request", "pig-01")+# The backend's id for the model, which is all the gateway's own message names, and+# the handle the deck resolved the method's model to, which is the deck's own name for it.+_MODEL_NOT_ALLOWED_WIRE_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"+_MODEL_NOT_ALLOWED_HANDLE = "claude-4.5-sonnet"++# The body the Portkey middleware answers with, at HTTP 412, verbatim as a run on the+# hosted API received it. Unlike the fail-closed envelope, the code sits in ``type``+# and ``code`` is null.+_MODEL_NOT_ALLOWED_BODY: dict[str, Any] = {+ "error": {+ "message": f"Model {_MODEL_NOT_ALLOWED_WIRE_ID} is not allowed for this integration",+ "type": "model_not_allowed_error",+ "param": None,+ "code": None,+ }+}+def _as_the_portkey_sdk_raises_it(*, status_code: int, body: dict[str, Any]) -> BaseException:"""Build the exception through Portkey's own factory rather than by hand.@@ -172,6 +209,30 @@ def _as_the_openai_sdk_raises_it(*, status_code: int, body: dict[str, Any]) -> Breturn client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+def _as_the_anthropic_sdk_raises_it(*, status_code: int, body: dict[str, Any]) -> BaseException:+ """Build the exception through the Anthropic SDK's own factory, the hop Claude travels on.++ For a status it has no subclass for, 412 among them, the factory returns a plain+ ``APIStatusError``, which a hand-built ``BadRequestError`` would not show.+ """+ request = httpx.Request("POST", f"{_ORIGIN}/v1/messages")+ response = httpx.Response(+ status_code=status_code,+ request=request,+ content=json.dumps(body).encode(),+ headers={"content-type": "application/json"},+ )+ client = anthropic.AsyncAnthropic(api_key="unused-in-this-test", base_url=_ORIGIN)+ return client._make_status_error_from_response(response) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]+++def _as_plain_httpx_raises_it(*, status_code: int, body: dict[str, Any]) -> BaseException:+ """The manifold's native ``/v1/pipelex/*`` routes, which the runtime calls over plain ``httpx``."""+ request = httpx.Request("POST", f"{_ORIGIN}/v1/pipelex/extract")+ response = httpx.Response(status_code=status_code, request=request, json=body)+ return httpx.HTTPStatusError(f"Client error '{status_code}'", request=request, response=response)++def _envelope(code: str | None, *, status_code: int = 400, provider: ProviderName = ProviderName.GATEWAY) -> ProviderErrorMetadata:return ProviderErrorMetadata(provider=provider,@@ -182,18 +243,28 @@ def _envelope(code: str | None, *, status_code: int = 400, provider: ProviderNam)-def _rendered_error(metadata: ProviderErrorMetadata, *, family: InferenceErrorFamily = InferenceErrorFamily.LLM) -> CogtError:+def _rendered_error(+ metadata: ProviderErrorMetadata,+ *,+ family: InferenceErrorFamily = InferenceErrorFamily.LLM,+ model_handle: str = "fake-handle",+) -> CogtError:return render_inference_error(metadata=metadata,classification=classify_inference_error(metadata),family=family,- model_desc="claude-fake",- model_handle="fake-handle",+ model_desc=f"{model_handle} → Model[{_MODEL_NOT_ALLOWED_WIRE_ID}]",+ model_handle=model_handle,)-def _rendered_detail(metadata: ProviderErrorMetadata, *, family: InferenceErrorFamily = InferenceErrorFamily.LLM) -> str:- rendered = _rendered_error(metadata, family=family)+def _rendered_detail(+ metadata: ProviderErrorMetadata,+ *,+ family: InferenceErrorFamily = InferenceErrorFamily.LLM,+ model_handle: str = "fake-handle",+) -> str:+ rendered = _rendered_error(metadata, family=family, model_handle=model_handle)assert rendered.user_action is not Nonereturn rendered.user_action.detail@@ -221,18 +292,33 @@ class TestTheCodeIsRecognized:@pytest.mark.parametrize("code",- [None, "pig-07", "pig-09", "pig-11", "invalid_request_error", "PIG-01", "pipelex_storage_unreadable", "model_not_found", ""],+ [+ None,+ "pig-07",+ "pig-09",+ "pig-11",+ "invalid_request_error",+ "PIG-01",+ "pipelex_storage_unreadable",+ "model_not_found",+ "model_not_allowed",+ "MODEL_NOT_ALLOWED_ERROR",+ "",+ ],)def test_any_other_code_is_not_a_routing_refusal(self, code: str | None) -> None:"""The request-shape limits, the unresolvable references, and a vendor's own code are all other families."""assert _envelope(code).gateway_routing_refusal is None@pytest.mark.parametrize("provider", list(ProviderName))- def test_the_code_is_read_whichever_provider_reported_it(self, provider: ProviderName) -> None:+ @pytest.mark.parametrize(+ ("code", "expected"), [("pig-01", GatewayRoutingRefusal.UNKNOWN_MODEL), ("model_not_allowed_error", GatewayRoutingRefusal.MODEL_NOT_ALLOWED)]+ )+ def test_the_code_is_read_whichever_provider_reported_it(self, provider: ProviderName, code: str, expected: GatewayRoutingRefusal) -> None:"""Claude reaches the gateway on the Anthropic driver, so the refusal is not always reported as GATEWAY."""- metadata = _envelope("pig-01", provider=provider)+ metadata = _envelope(code, provider=provider)- assert metadata.gateway_routing_refusal == GatewayRoutingRefusal.UNKNOWN_MODEL+ assert metadata.gateway_routing_refusal == expectedclass TestTheCodeSurvivesEveryExtractHop:@@ -259,7 +345,7 @@ class TestTheCodeSurvivesEveryExtractHop:assert isinstance(getattr(exc, "body", None), str)assert extract_gateway_metadata(exc).gateway_routing_refusal == GatewayRoutingRefusal.UNKNOWN_MODEL- @pytest.mark.parametrize(("code", "expected"), _EVERY_CODE_AND_MEMBER)+ @pytest.mark.parametrize(("code", "expected"), _FAIL_CLOSED_CODES_AND_MEMBERS)def test_through_the_openai_substrate_that_carries_every_chat_call(self, code: str, expected: GatewayRoutingRefusal) -> None:"""The hop a gateway or manifold LLM call actually takes.@@ -299,6 +385,35 @@ class TestTheCodeSurvivesEveryExtractHop:assert metadata.provider_error_code == "pig-01"assert metadata.gateway_routing_refusal == GatewayRoutingRefusal.UNKNOWN_MODEL+ @pytest.mark.parametrize(+ ("raise_it", "extract"),+ [+ pytest.param(_as_the_openai_sdk_raises_it, extract_openai_metadata, id="openai-substrate"),+ pytest.param(_as_the_anthropic_sdk_raises_it, extract_anthropic_metadata, id="anthropic-driver"),+ pytest.param(_as_the_portkey_sdk_raises_it, extract_gateway_metadata, id="portkey-substrate"),+ pytest.param(_as_plain_httpx_raises_it, extract_manifold_metadata, id="native-routes-httpx"),+ ],+ )+ def test_the_model_not_allowed_refusal_through_every_hop(+ self,+ raise_it: Callable[..., BaseException],+ extract: Callable[[BaseException], ProviderErrorMetadata],+ ) -> None:+ """The substrate's own envelope, at 412, through each SDK's own factory.++ Its code sits in ``type`` with ``code`` null, the inverse of the fail-closed+ envelope. The two vendor-facing hops read ``type`` first; the two+ Pipelex-service hops read ``code`` first and fall back to ``type``, which is+ the fallback this case exercises.+ """+ exc = raise_it(status_code=412, body=_MODEL_NOT_ALLOWED_BODY)++ metadata = extract(exc)++ assert metadata.status_code == 412+ assert metadata.provider_error_code == "model_not_allowed_error"+ assert metadata.gateway_routing_refusal == GatewayRoutingRefusal.MODEL_NOT_ALLOWED+def test_the_google_hop_would_not_carry_a_gateway_code_and_nothing_reaches_it_today(self) -> None:"""``pig-05``'s hop, pinned as the trap it is rather than as a passing path.@@ -368,6 +483,28 @@ class TestClassification:assert result.is_model_not_found is (code == "pig-01")+ def test_a_model_not_allowed_refusal_asks_for_another_model_and_is_not_a_missing_model(self) -> None:+ """The model exists and an integration serves it, only not for this caller.++ So ``CHANGE_MODEL`` is what the caller can do, and the flag stays unset for+ the reason it does on ``pig-05`` and ``pig-06``: "not found" would be false.+ """+ result = classify_inference_error(_envelope("model_not_allowed_error", status_code=412))++ assert result.gateway_routing_refusal == GatewayRoutingRefusal.MODEL_NOT_ALLOWED+ assert result.category == InferenceErrorCategory.CONFIGURATION+ assert result.category.is_retryable is False+ assert result.user_action_kind == UserActionKind.CHANGE_MODEL+ assert result.is_model_not_found is False++ def test_a_412_without_a_recognized_code_still_takes_the_status_ladder(self) -> None:+ """The contrast: the ladder's generic 4xx arm, which is what the refusal used to read as."""+ result = classify_inference_error(_envelope("something-else", status_code=412))++ assert result.gateway_routing_refusal is None+ assert result.category == InferenceErrorCategory.CONFIGURATION+ assert result.user_action_kind == UserActionKind.CHANGE_INPUT+def test_a_plain_400_without_a_recognized_code_still_takes_the_status_ladder(self) -> None:"""Only the gateway's own codes name a routing decision; a bare 400 from a vendor does not."""result = classify_inference_error(_envelope("something-else"))@@ -423,6 +560,32 @@ class TestEndToEnd:assert rendered.user_action is not Noneassert "not found" not in rendered.user_action.detail+ def test_a_model_not_allowed_refusal_reaches_the_caller_as_a_change_model_naming_the_handle(self) -> None:+ """The refusal a run on the hosted API received, through the hop an LLM call on the gateway takes.++ It used to read "review the prompt, parameters, and inputs". It now asks for+ another model, names it by its handle, and stays the family's+ failure class, since the model was found.+ """+ exc = _as_the_openai_sdk_raises_it(status_code=412, body=_MODEL_NOT_ALLOWED_BODY)+ metadata = extract_openai_metadata(exc)++ rendered = render_inference_error(+ metadata=metadata,+ classification=classify_inference_error(metadata),+ family=InferenceErrorFamily.LLM,+ model_desc=f"{_MODEL_NOT_ALLOWED_HANDLE} → Model[{_MODEL_NOT_ALLOWED_WIRE_ID}]",+ model_handle=_MODEL_NOT_ALLOWED_HANDLE,+ )++ assert isinstance(rendered, LLMCompletionError)+ assert not isinstance(rendered, LLMModelNotFoundError)+ assert rendered.error_category == InferenceErrorCategory.CONFIGURATION+ assert rendered.user_action is not None+ assert rendered.user_action.kind == UserActionKind.CHANGE_MODEL+ assert f"'{_MODEL_NOT_ALLOWED_HANDLE}'" in rendered.user_action.detail+ assert rendered.user_action.detail != _GENERIC_ADVICE+class TestRenderedAdvice:"""Each member gets advice naming what has to change, not a generic rejection line."""@@ -437,7 +600,7 @@ class TestRenderedAdvice:"""Branching ahead of the action kind is what this asserts.``CHANGE_MODEL`` alone would render "the requested model was not found" for- three members whose model exists, and ``CONTACT_SUPPORT`` alone would render+ every member but ``pig-01``, whose model exists, and ``CONTACT_SUPPORT`` alone would render"the error could not be classified" for a refusal that named itself exactly."""detail = _rendered_detail(_envelope(code))@@ -452,11 +615,49 @@ class TestRenderedAdvice:("pig-02", "has not enabled"),("pig-05", "not over the protocol"),("pig-06", "does not serve that capability"),+ ("model_not_allowed_error", "does not allow the model"),],)def test_the_detail_names_the_remedy(self, code: str, expected_phrase: str) -> None:assert expected_phrase in _rendered_detail(_envelope(code))+ def test_the_model_not_allowed_advice_names_the_handle_and_not_the_wire_id(self) -> None:+ """The gateway's message names only the backend's id, which the method's author never wrote."""+ detail = _rendered_detail(_envelope("model_not_allowed_error", status_code=412), model_handle=_MODEL_NOT_ALLOWED_HANDLE)++ assert f"'{_MODEL_NOT_ALLOWED_HANDLE}'" in detail+ assert _MODEL_NOT_ALLOWED_WIRE_ID not in detail+ assert "pick another model" in detail+ assert "prompt" not in detail++ def test_the_model_not_allowed_advice_offers_the_default_only_when_the_pipe_named_the_model(self) -> None:+ """The Render step cannot tell a model the pipe named from the deck's default.++ When the default is the refused model, an unconditional "leave the pipe's+ model unset" would tell the caller to do what they already did.+ """+ detail = _rendered_detail(_envelope("model_not_allowed_error", status_code=412), model_handle=_MODEL_NOT_ALLOWED_HANDLE)++ assert "if the pipe named this one, leaving its model unset uses the default" in detail++ def test_the_model_not_allowed_advice_says_who_settles_a_deck_disagreement_on_either_gateway(self) -> None:+ """The same code comes from the Pipelex gateway and from a user's own Portkey workspace.++ Nothing on the wire tells them apart, so advice that only said "contact+ support" would send the owner of a Portkey workspace to Pipelex for a+ setting in their own dashboard.+ """+ detail = _rendered_detail(_envelope("model_not_allowed_error", status_code=412), model_handle=_MODEL_NOT_ALLOWED_HANDLE)++ assert "model deck" in detail+ assert "on the Pipelex gateway, contact support" in detail+ assert "on a Portkey workspace of your own, allow the model in the integration that serves it" in detail++ @pytest.mark.parametrize("code", [code for code in _EVERY_CODE if code != "model_not_allowed_error"])+ def test_every_other_member_leaves_the_handle_to_the_message(self, code: str) -> None:+ """Only the member whose gateway message names no handle the author wrote names one itself."""+ assert "some-distinctive-handle" not in _rendered_detail(_envelope(code), model_handle="some-distinctive-handle")+def test_the_unserved_model_and_the_wrong_protocol_are_told_apart(self) -> None:"""A model the gateway does not serve and one it serves differently need different moves."""unknown = _rendered_detail(_envelope("pig-01"))@@ -559,22 +760,36 @@ class TestWhatTheFamilyAnswersOverHTTP:"""The category is not only advice — it decides the status an HTTP surface returns.``CONFIGURATION`` implies ``ErrorDomain.CONFIG``, which- ``error_domain_to_http_status`` renders as 500; the ``CONTENT`` these four used- to receive implies ``INPUT`` and renders as 422. So classifying the family- moved its HTTP answer as surely as it moved its wording, and that is the half a+ ``error_domain_to_http_status`` renders as 500; the ``CONTENT`` the ``pig-``+ members used to receive on their 400 implies ``INPUT`` and renders as 422. So+ classifying the family moved their HTTP answer as surely as it moved its wording, and that is the half aclassification test cannot see. 500 is also the status outer clients retry,which is worth pinning precisely because "never retried" elsewhere in thismodule is a statement about ``InferenceErrorCategory.is_retryable`` inside thisprocess and about nothing else."""- @pytest.mark.parametrize("code", _EVERY_CODE)+ @pytest.mark.parametrize("code", _FAIL_CLOSED_CODES)def test_every_member_answers_500_rather_than_the_422_it_used_to(self, code: str) -> None:report = _rendered_error(_envelope(code)).to_error_report()assert report.error_domain == ErrorDomain.CONFIGassert report.http_status == 500+ def test_the_model_not_allowed_refusal_answers_500_as_it_did_before(self) -> None:+ """At its real status the member's answer does not move: only its advice does.++ The ladder's generic 4xx arm already read a 412 as ``CONFIGURATION``, so the+ family changes this refusal's ``user_action`` and nothing about its status.+ """+ report = _rendered_error(_envelope("model_not_allowed_error", status_code=412)).to_error_report()+ unrecognized_report = _rendered_error(_envelope("something-else", status_code=412)).to_error_report()++ assert report.error_domain == ErrorDomain.CONFIG+ assert report.http_status == 500+ assert unrecognized_report.error_domain == ErrorDomain.CONFIG+ assert unrecognized_report.http_status == 500+def test_an_unrecognized_400_still_answers_422(self) -> None:"""The contrast that makes the case above a change rather than a constant."""report = _rendered_error(_envelope("something-else")).to_error_report()@@ -593,6 +808,10 @@ class TestTheAdviceSurvivesThePipeBoundary:``error_domain`` off the ``__cause__`` chain. That inheritance is what carriesevery word of this family's most-argued detail through the pipe layer, andnothing else in this module crosses that boundary to see it.++ Every other member keeps the family's failure class, which the pipe operator+ lets through for the pipe router to locate. The last case pins that route, for+ the member a run on the hosted API reached."""def _re_raised_as_the_pipe_operator_does(self, cause: LLMModelNotFoundError) -> PipeOperatorModelAvailabilityError:@@ -626,3 +845,33 @@ class TestTheAdviceSurvivesThePipeBoundary:assert report.user_action_detail() == _rendered_detail(_envelope("pig-01"))assert report.error_domain == ErrorDomain.CONFIGassert report.http_status == 500++ def test_the_model_not_allowed_advice_reaches_the_caller_through_the_located_failure(self) -> None:+ """The failure class is not re-raised by the pipe operator, so the router's location is the boundary.++ ``PipeRouterError`` reports the root fault located at the failing pipe, with+ the classification its cause chain carries. When nothing on the chain+ carried a user action, that report fell back to "the message gives the+ cause", which is what a run on the hosted API read before this member.+ """+ rendered = _rendered_error(_envelope("model_not_allowed_error", status_code=412), model_handle=_MODEL_NOT_ALLOWED_HANDLE)+ located = PipeRouterError.make_located(+ failure=rendered,+ run_mode=PipeRunMode.LIVE,+ pipe_code="score_lead",+ output_name=None,+ pipe_stack=["score_lead"],+ )+ located.__cause__ = rendered++ report = located.to_error_report()++ assert report.error_type == "LLMCompletionError"+ assert report.error_domain == ErrorDomain.CONFIG+ assert report.error_category == InferenceErrorCategory.CONFIGURATION+ assert report.retryable is False+ assert report.user_action is not None+ assert report.user_action.kind == UserActionKind.CHANGE_MODEL+ assert report.user_action_detail() == _rendered_detail(+ _envelope("model_not_allowed_error", status_code=412), model_handle=_MODEL_NOT_ALLOWED_HANDLE+ )diff --git a/tests/unit/pipelex/providers/test_instructor_retry_shapes.py b/tests/unit/pipelex/providers/test_instructor_retry_shapes.pynew file mode 100644index 000000000..3c37f91dd--- /dev/null+++ b/tests/unit/pipelex/providers/test_instructor_retry_shapes.py@@ -0,0 +1,304 @@+"""What ``instructor``'s real retry loop raises, pinned against the recovery and the test helper.++Every structured-output call goes through ``instructor``, which raises an+``InstructorRetryException`` from whatever ended its loop, and the workers recover+the SDK exception from that wrapper to classify it. The worker tests build the+wrapper by hand with ``wrap_in_instructor_retry``, and a hand-built wrapper is how+an earlier ``instructor`` release changed that shape with every worker test still+green. So this module drives the real loop instead: real ``from_openai`` and+``from_anthropic`` factories, over real SDK clients whose transport is an+``httpx.MockTransport``, with the schema-only retrying the workers pass.++Three shapes end the loop: an API error on the first attempt, an invalid output+re-asked and then an API error, and a re-ask budget spent on invalid outputs. For+each, the recovery must return the exception that ended the loop, and the two API+error shapes must be the shape the helper builds.++The Anthropic SDK release locked here takes an ``httpx`` client; later releases take+``httpx2``, so a lock move there changes how the transport is handed over.+"""++from __future__ import annotations++import json+from typing import TYPE_CHECKING, Any++import anthropic+import httpx+import openai+import pytest+from instructor import Mode, from_anthropic, from_openai+from instructor.core import FailedAttempt, InstructorRetryException+from pydantic import ValidationError+from tenacity import RetryError++from pipelex.base_exceptions import ErrorDomain+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, LLMModelNotFoundError+from pipelex.cogt.inference.error_classification import GatewayRoutingRefusal, UserActionKind, extract_underlying_sdk_exception+from pipelex.cogt.llm.instructor_retry import make_instructor_schema_retrying+from pipelex.pipe_run.exceptions import PipeRouterError+from pipelex.providers.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker+from pipelex.system.pipe_run_mode import PipeRunMode+from tests.helpers.instructor_test_utils import DummySchema, make_llm_job, wrap_in_instructor_retry++if TYPE_CHECKING:+ from collections.abc import Awaitable, Callable++ from pytest_mock import MockerFixture++_ORIGIN = "https://gateway.example.com"+_MAX_ATTEMPTS = 3++# A scripted HTTP answer: the status and the JSON body.+_ScriptedResponse = tuple[int, dict[str, Any]]++# The refusal a run on the hosted API received, verbatim, and the handle its method named.+_WIRE_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"+_MODEL_HANDLE = "claude-4.5-sonnet"+_MODEL_NOT_ALLOWED: _ScriptedResponse = (+ 412,+ {"error": {"message": f"Model {_WIRE_ID} is not allowed for this integration", "type": "model_not_allowed_error", "param": None, "code": None}},+)++_OPENAI_RATE_LIMIT: _ScriptedResponse = (429, {"error": {"message": "Rate limit reached", "type": "rate_limit_error", "code": "rate_limit_exceeded"}})+_ANTHROPIC_RATE_LIMIT: _ScriptedResponse = (429, {"type": "error", "error": {"type": "rate_limit_error", "message": "Rate limit reached"}})++# A tool call whose arguments fail ``DummySchema``, which requires ``text``.+_INVALID_ARGUMENTS: dict[str, Any] = {"wrong": 1}++_OPENAI_INVALID_OUTPUT: _ScriptedResponse = (+ 200,+ {+ "id": "chatcmpl-invalid",+ "object": "chat.completion",+ "created": 0,+ "model": "some-model",+ "choices": [+ {+ "index": 0,+ "finish_reason": "tool_calls",+ "message": {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {"id": "call_1", "type": "function", "function": {"name": "DummySchema", "arguments": json.dumps(_INVALID_ARGUMENTS)}}+ ],+ },+ }+ ],+ "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},+ },+)+_ANTHROPIC_INVALID_OUTPUT: _ScriptedResponse = (+ 200,+ {+ "id": "msg_invalid",+ "type": "message",+ "role": "assistant",+ "model": "some-model",+ "content": [{"type": "tool_use", "id": "toolu_1", "name": "DummySchema", "input": _INVALID_ARGUMENTS}],+ "stop_reason": "tool_use",+ "stop_sequence": None,+ "usage": {"input_tokens": 1, "output_tokens": 1},+ },+)+++class _ScriptedTransport:+ """Answers each request with the next scripted response, and counts the requests."""++ def __init__(self, responses: list[_ScriptedResponse]) -> None:+ self.responses = list(responses)+ self.nb_requests = 0++ def handle(self, request: httpx.Request) -> httpx.Response:+ scripted_response = self.responses[self.nb_requests]+ self.nb_requests += 1+ status_code, body = scripted_response+ return httpx.Response(status_code=status_code, json=body, request=request)++ def make_http_client(self) -> httpx.AsyncClient:+ return httpx.AsyncClient(transport=httpx.MockTransport(self.handle))+++def _failed_attempts_of(raised: InstructorRetryException) -> list[FailedAttempt]:+ """The attempts instructor recorded as failed, which the real loop always sets, empty or not."""+ failed_attempts = raised.failed_attempts+ assert failed_attempts is not None+ return failed_attempts+++async def _raised_by_openai(transport: _ScriptedTransport) -> InstructorRetryException:+ """Drive ``from_openai``'s loop to its end, the way the OpenAI completions worker calls it."""+ async with transport.make_http_client() as http_client:+ client = openai.AsyncOpenAI(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1", max_retries=0, http_client=http_client)+ structured = from_openai(client, mode=Mode.TOOLS)+ with pytest.raises(InstructorRetryException) as exc_info:+ await structured.chat.completions.create_with_completion(+ model="some-model",+ messages=[{"role": "user", "content": "Generate a structured object."}],+ response_model=DummySchema,+ max_retries=make_instructor_schema_retrying(max_attempts=_MAX_ATTEMPTS),+ )+ return exc_info.value+++async def _raised_by_anthropic(transport: _ScriptedTransport) -> InstructorRetryException:+ """Drive ``from_anthropic``'s loop to its end, the way the Anthropic worker calls it."""+ async with transport.make_http_client() as http_client:+ client = anthropic.AsyncAnthropic(api_key="unused-in-this-test", base_url=_ORIGIN, max_retries=0, http_client=http_client)+ structured = from_anthropic(client)+ with pytest.raises(InstructorRetryException) as exc_info:+ await structured.messages.create_with_completion(+ model="some-model",+ max_tokens=64,+ messages=[{"role": "user", "content": "Generate a structured object."}],+ response_model=DummySchema,+ max_retries=make_instructor_schema_retrying(max_attempts=_MAX_ATTEMPTS),+ )+ return exc_info.value+++# The two shapes an API error ends: on the first attempt, and after a re-ask. Each+# names the driver, the script, the SDK exception class that must end the loop, and+# how many parse failures instructor must have recorded before it.+_API_ERROR_SHAPES = [+ pytest.param(_raised_by_openai, [_MODEL_NOT_ALLOWED], openai.APIStatusError, 0, id="openai-api-error-first"),+ pytest.param(_raised_by_openai, [_OPENAI_INVALID_OUTPUT, _OPENAI_RATE_LIMIT], openai.RateLimitError, 1, id="openai-invalid-then-api-error"),+ pytest.param(_raised_by_anthropic, [_ANTHROPIC_RATE_LIMIT], anthropic.RateLimitError, 0, id="anthropic-api-error-first"),+ pytest.param(+ _raised_by_anthropic, [_ANTHROPIC_INVALID_OUTPUT, _ANTHROPIC_RATE_LIMIT], anthropic.RateLimitError, 1, id="anthropic-invalid-then-api-error"+ ),+]++_SPENT_BUDGET_SHAPES = [+ pytest.param(_raised_by_openai, [_OPENAI_INVALID_OUTPUT] * _MAX_ATTEMPTS, id="openai-budget-spent"),+ pytest.param(_raised_by_anthropic, [_ANTHROPIC_INVALID_OUTPUT] * _MAX_ATTEMPTS, id="anthropic-budget-spent"),+]++[email protected](loop_scope="class")+class TestInstructorRetryShapes:+ @pytest.mark.parametrize(("raised_by", "responses", "sdk_exception_class", "nb_parse_failures"), _API_ERROR_SHAPES)+ async def test_an_api_error_ends_the_loop_on_the_cause_and_is_what_the_recovery_returns(+ self,+ raised_by: Callable[[_ScriptedTransport], Awaitable[InstructorRetryException]],+ responses: list[_ScriptedResponse],+ sdk_exception_class: type[Exception],+ nb_parse_failures: int,+ ) -> None:+ """The SDK exception sits on ``__cause__`` alone; ``failed_attempts`` lists only the parse failures before it."""+ transport = _ScriptedTransport(responses)++ raised = await raised_by(transport)++ assert transport.nb_requests == len(responses)+ assert isinstance(raised.__cause__, sdk_exception_class)+ assert [type(failed_attempt.exception) for failed_attempt in _failed_attempts_of(raised)] == [ValidationError] * nb_parse_failures+ assert extract_underlying_sdk_exception(raised) is raised.__cause__++ @pytest.mark.parametrize(("raised_by", "responses", "sdk_exception_class", "nb_parse_failures"), _API_ERROR_SHAPES)+ async def test_the_helper_builds_the_shape_the_real_loop_raises(+ self,+ raised_by: Callable[[_ScriptedTransport], Awaitable[InstructorRetryException]],+ responses: list[_ScriptedResponse],+ sdk_exception_class: type[Exception],+ nb_parse_failures: int,+ ) -> None:+ """``wrap_in_instructor_retry``, fed what the real loop saw, must build what the real loop raised.++ Every worker test builds its wrapper with that helper, so this is what keeps+ those tests honest across an ``instructor`` release that changes the shape.+ """+ raised = await raised_by(_ScriptedTransport(responses))+ sdk_exc = raised.__cause__+ assert isinstance(sdk_exc, sdk_exception_class)+ parse_failures = [failed_attempt.exception for failed_attempt in _failed_attempts_of(raised)]+ assert len(parse_failures) == nb_parse_failures++ built = wrap_in_instructor_retry(sdk_exc, earlier_parse_failures=parse_failures)++ assert built.__cause__ is raised.__cause__+ assert [(attempt.attempt_number, attempt.exception) for attempt in _failed_attempts_of(built)] == [+ (attempt.attempt_number, attempt.exception) for attempt in _failed_attempts_of(raised)+ ]+ assert built.n_attempts == raised.n_attempts+ assert extract_underlying_sdk_exception(built) is extract_underlying_sdk_exception(raised)++ @pytest.mark.parametrize(("raised_by", "responses"), _SPENT_BUDGET_SHAPES)+ async def test_a_spent_budget_recovers_the_last_parse_failure(+ self,+ raised_by: Callable[[_ScriptedTransport], Awaitable[InstructorRetryException]],+ responses: list[_ScriptedResponse],+ ) -> None:+ """Once the re-ask budget is spent the cause is tenacity's ``RetryError``, whose last attempt holds the final parse failure."""+ transport = _ScriptedTransport(responses)++ raised = await raised_by(transport)++ assert transport.nb_requests == _MAX_ATTEMPTS+ assert isinstance(raised.__cause__, RetryError)+ failed_attempts = _failed_attempts_of(raised)+ assert len(failed_attempts) == _MAX_ATTEMPTS+ recovered = extract_underlying_sdk_exception(raised)+ assert isinstance(recovered, ValidationError)+ assert recovered is failed_attempts[-1].exception++ async def test_a_gateway_model_not_allowed_refusal_reaches_the_run_report_as_change_model(self, mocker: MockerFixture) -> None:+ """The run the refusal was first seen on, end to end: the worker over real instructor, then the pipe's location.++ That run read ``error_category: unknown`` and a user action saying only that+ the message gives the cause. It must read as a configuration error whose+ next step is another model, named as the method named it, and it must not+ be retried or re-asked.+ """+ transport = _ScriptedTransport([_MODEL_NOT_ALLOWED])+ worker = object.__new__(OpenAICompletionsLLMWorker)+ inference_model = mocker.MagicMock()+ inference_model.name = _MODEL_HANDLE+ inference_model.model_id = _WIRE_ID+ inference_model.desc = f"{_MODEL_HANDLE} → SDK[gateway_completions]•Backend[pipelex_gateway]•Model[{_WIRE_ID}]"+ inference_model.thinking_mode = None+ inference_model.listed_constraints = []+ worker.inference_model = inference_model+ completions_factory = mocker.MagicMock()+ completions_factory.make_simple_messages = mocker.AsyncMock(return_value=[{"role": "user", "content": "Score this lead."}])+ completions_factory.make_extras = mocker.MagicMock(return_value=({}, {}))+ worker.openai_completions_factory = completions_factory+ llm_job = make_llm_job(mocker)+ llm_job.job_params.seed = None+ llm_job.job_config.schema_reask_max_attempts = _MAX_ATTEMPTS++ async with transport.make_http_client() as http_client:+ client = openai.AsyncOpenAI(api_key="unused-in-this-test", base_url=f"{_ORIGIN}/v1", max_retries=0, http_client=http_client)+ worker.instructor_for_objects = from_openai(client, mode=Mode.TOOLS)+ with pytest.raises(LLMCompletionError) as exc_info:+ await worker._gen_object(llm_job=llm_job, schema=DummySchema) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]++ failure = exc_info.value+ assert transport.nb_requests == 1+ assert not isinstance(failure, LLMModelNotFoundError)+ assert failure.provider_metadata is not None+ assert failure.provider_metadata.status_code == 412+ assert failure.provider_metadata.gateway_routing_refusal == GatewayRoutingRefusal.MODEL_NOT_ALLOWED++ located = PipeRouterError.make_located(+ failure=failure,+ run_mode=PipeRunMode.LIVE,+ pipe_code="score_lead",+ output_name=None,+ pipe_stack=["score_lead"],+ )+ located.__cause__ = failure+ report = located.to_error_report()++ assert report.error_type == "LLMCompletionError"+ assert report.error_domain == ErrorDomain.CONFIG+ assert report.error_category == InferenceErrorCategory.CONFIGURATION+ assert report.retryable is False+ assert report.http_status == 500+ assert report.user_action is not None+ assert report.user_action.kind == UserActionKind.CHANGE_MODEL+ assert f"'{_MODEL_HANDLE}'" in report.user_action.detail+ assert _WIRE_ID not in report.user_action.detail+ assert report.message.startswith("Pipe 'score_lead' failed: ")diff --git a/wip/structured-error-classification/design.md b/wip/structured-error-classification/design.mdnew file mode 100644index 000000000..b0d21f8e0--- /dev/null+++ b/wip/structured-error-classification/design.md@@ -0,0 +1,79 @@+---+status: landed+item: L-260927-086ea4+---++# A gateway's model_not_allowed refusal says to change the model++## The bug, verified++The item reports that a run whose model the gateway refuses reaches the user as an unknown runtime failure. Verification on 2026-09-27 found two independent causes. The first has since been fixed by the instructor upgrade (pipelex#1289, merged as `85729dfd3`). The second is what remains of this item.++### Cause 1, fixed: instructor's new retry loop hid the SDK exception++The campaign's failure message was the fallback text of `OpenAICompletionsLLMWorker._gen_object`. That text is produced when `extract_underlying_sdk_exception` cannot find the SDK exception inside an `InstructorRetryException`, and it carries `error_category=UNKNOWN` and no user action. From instructor 1.15.3 on, every `from_*` factory runs through `instructor/v2/core/retry.py`, which raises `InstructorRetryException` from any exception that ends the loop. It records only parse failures in `failed_attempts`, so an API error sits on `__cause__` alone. The recovery only looked in `failed_attempts` and in a tenacity `RetryError`. As a result, every provider error on a structured-output call was reported as unclassified and never retried, on every LLM worker that uses instructor. This was bisected over `httpx.MockTransport`: 1.15.1 raised the SDK exception raw, while 1.15.3, 1.15.4 and 1.17.0 wrapped it.++pipelex#1289 moved the lock to instructor 1.17.0 and the floor to `>=1.17.0,<2.0.0`, and it reordered the recovery to read `__cause__` first, unwrapping a tenacity `RetryError`, with `failed_attempts` as the fallback. That is the order the first draft of this design proposed. It also rewrote `wrap_in_instructor_retry` to build the new shape, and the "`instructor` Unwrap" section of `docs/under-the-hood/error-model.md`.++Re-verified at `85729dfd3`, by driving the real instructor factories over `httpx.MockTransport` with `make_instructor_schema_retrying(max_attempts=3)`, then running the recovery, classify and render:++| Scenario | Recovered | Report |+|---|---|---|+| OpenAI, gateway 412 `model_not_allowed_error` | `APIStatusError` | `configuration` / `config` / 500 / `change_input`: "The provider rejected the request — review the prompt, parameters, and inputs." |+| OpenAI, invalid output then 429 | `RateLimitError` | `transient` / `runtime` / 429 / `wait_and_retry`, retryable |+| OpenAI, re-ask budget spent | `ValidationError` | `content` / `input` / 422 / `change_input` |+| Anthropic, 429 | `RateLimitError` | `transient` / `runtime` / 429 / `wait_and_retry`, retryable |++The hosted plane still runs this bug: `pipelex-server` pins `pipelex==0.68.0`, which was released before `85729dfd3`, and its lock resolves instructor 1.15.4. It gets the fix with the next pipelex release and the pin move that follows, so nothing needs filing.++### Cause 2, open: a 412 lands on the status ladder's generic 4xx arm++Now that the SDK exception is recovered, the 412 is classified. `extract_openai_metadata` reads `provider_error_code = "model_not_allowed_error"` off `exc.type`, but no code map knows that code. So `classify_inference_error` falls through to its unrecognized-4xx arm (`pipelex/cogt/inference/error_classify.py`, the `if status_code >= 400:` branch), which returns `CONFIGURATION` / `CHANGE_INPUT`. The domain is right. The action is wrong: nothing in the prompt, parameters or inputs causes this refusal, and only another model avoids it. A caller branching on `user_action.kind` is sent to edit its inputs.++### Where the refusal comes from++`model_not_allowed_error` is the Portkey gateway's own code, not a model vendor's. Both raise sites are in the middleware the manifold vendors from upstream (`pipelex-manifold/src/middlewares/portkey/utils.ts`). They answer HTTP 412 with `{error: {message: "Model <wire id> is not allowed for this integration", type: "model_not_allowed_error", param: null, code: null}}` in two cases: the integration has `allow_all_models` off and does not list the model, or it lists the model as `archived`.++`pipelex_gateway` points at Portkey's cloud by default (`pipelex/providers/gateway/gateway_factory.py`, `get_endpoint`), where Pipelex configures the integrations. The manifold would raise the same code if one of its integrations turned `allow_all_models` off. So does Portkey's cloud for a user's own workspace reached through the `portkey` backend, which runs on the same OpenAI completions worker and Extract hop and reports the same provider; there the allow-list is the user's own. Either way, the model exists, and the integration that would serve it refuses it for this caller. The caller can pick another model. For whoever operates the gateway, the refusal means the model deck and the integration's allow-list disagree, which is what `L-260923-2c7f6b` tracks for api-dev.++## Decisions++### D1. `model_not_allowed_error` joins the gateway's routing refusals as `MODEL_NOT_ALLOWED`++A new member, `GatewayRoutingRefusal.MODEL_NOT_ALLOWED`, is mapped from the code `"model_not_allowed_error"` in `_GATEWAY_ROUTING_REFUSAL_BY_CODE`. It classifies as `CONFIGURATION` / `CHANGE_MODEL`, is never retried, and leaves `is_model_not_found` unset.++- **Why this family.** The gateway raises it before any provider sees the request, and it concerns which model may be used. That is the family's definition: the refusal is not an inference failure and not the prompt's fault, and a retry earns the same answer. The family exists to keep such refusals away from the status-ladder arm.+- **Why the flag stays unset.** The model exists and an integration serves it; this caller may not use it there. Saying "not found" would be false, the same reasoning `WRONG_PROTOCOL` and `UNSERVED_CAPABILITY` follow. The error stays the family's failure class and carries the distinction in its advice.+- **Why match on the code alone.** Every other routing code is in the `pig-` namespace, which no vendor emits into. This one is Portkey's code, and no model vendor uses it. Portkey's cloud emits it for the integrations Pipelex configures and for a user's own workspace behind the `portkey` backend alike, and the manifold's vendored middleware emits it too. The first draft said only gateways Pipelex configures emit it, which review round 1 refuted; the conclusion stands, because the refusal means the same thing and calls for the same move from any of them, and only the advice has to hold for both (D2). The enum docstring and the map comment are rewritten to say that this member is keyed on the gateway substrate's code rather than on a `pig-NN`, and that it arrives on 412 rather than 400. Also checking the status would add nothing: both raise sites answer 412, and the remedy would be the same under any status.+- **Every Extract hop already reads the code.** The code sits in `type`, with `code: null`. The OpenAI hop reads `exc.type`, the Anthropic hop reads the body's `error.type`, and the two Pipelex-service hops read `code` and then fall back to `type`. This was probed at `85729dfd3` with the campaign's 412 body, each exception built the way production builds it: Portkey's and OpenAI's `_make_status_error_from_response` (through the test module's `_as_the_portkey_sdk_raises_it` and `_as_the_openai_sdk_raises_it`), an `httpx.HTTPStatusError` for the manifold's native routes, and `AsyncAnthropic._make_status_error_from_response`, which returns a plain `APIStatusError` for a 412. All four returned `provider_error_code = "model_not_allowed_error"`, so the change touches no Extract hop. The tests pin every hop anyway, as `TestTheCodeSurvivesEveryExtractHop` does for the other members.++### D2. The advice names the model handle++The family's advice deliberately names no model, because the gateway's own message states every specific. That reasoning fails here. The gateway's message names only the backend's wire id (`us.anthropic.claude-sonnet-4-5-20250929-v1:0`), which the method's author never wrote. `render_inference_error` already receives `model_handle`, so it threads the handle through `_render_detail` into `_render_gateway_routing_refusal_detail`, and only this member uses it. The handle is the one the model deck resolved the pipe's model to, so a pipe that named a preset or an alias reads the handle behind it. The advice reads:++> The inference gateway does not allow the model 'claude-4.5-sonnet' for this account — pick another model for the pipe; if the pipe named this one, leaving its model unset uses the default instead. If your model deck lists it as available, the deck and the gateway's allow-list disagree: on the Pipelex gateway, contact support; on a Portkey workspace of your own, allow the model in the integration that serves it.++Both hints are conditional, which review round 1 asked for, because the Render step cannot tell the cases apart. The handle may be one the pipe named or the deck's default, and when it is the default, telling the caller to leave the model unset would tell them to do what they already did. And a deck disagreement is Pipelex's to settle on the Pipelex gateway and the user's own on their Portkey workspace, which nothing on the wire distinguishes.++The report's `model` field already carries the handle, through `fill_model_and_provider` in `LLMWorkerAbstract`. So the change is to the advice, not to attribution.++### D3. The HTTP answer does not move++At `85729dfd3` the refusal already answers 500 (`CONFIGURATION` implies `ErrorDomain.CONFIG`), and it still does after this change. Only `user_action` changes. The changelog says so, because a classification change usually moves the status.++### D4. One test pins the hand-built wrapper to what instructor really raises++pipelex#1289 rewrote `wrap_in_instructor_retry` to build the new shape by hand. `test_instructor_retry.py` pins the retry predicate against instructor's `_RETRYABLE_PARSE_ERRORS`, but nothing pins the wrapper's shape against the real loop. A hand-built exception is how cause 1 stayed green on the old lock, so the helper would drift silently at the next instructor release that changes the shape.++One test module fixes that by driving `from_openai` and `from_anthropic` over `httpx.MockTransport` with `make_instructor_schema_retrying`, in three shapes: an API error first, an invalid output followed by an API error, and a spent budget. For each shape it asserts that `extract_underlying_sdk_exception` returns the exception that ended the loop, and that the exception instructor raised matches the shape the helper builds: the SDK exception as `__cause__`, and only parse failures in `failed_attempts`. The same module drives the campaign's scenario end to end for D1.++### D5. Two statements #1289 left inaccurate are corrected++- The "`instructor` Unwrap" section of `docs/under-the-hood/error-model.md` still says that a `pydantic.ValidationError` from a schema mismatch "lands in `UNKNOWN`". It does not. `_STATUSLESS_BY_TYPE_NAME` maps it to `CONTENT` / `CHANGE_INPUT`, and the spent-budget row above shows it answering 422.+- The Unreleased changelog entry "A transport error inside a structured-output call is classified again" dates the wrapping to "`instructor` 1.16 and later". The bisection shows it started in 1.15.3. The correct range tells a reader on an older pipelex whether they were affected, since pipelex before this release accepted `instructor>=1.13`.++## What does not change++`located_failure.py` stays as it is. Its floor and fallback are right for a failure nothing classified, which a gateway refusal no longer is. Refusing, at validation time, a model the account cannot use is `L-260927-2a587b` (`pipelex-server`), and resolving every model reference to a served handle at validation is `L-260925-cfb6f3`. Both would prevent the run; this item classifies the failure when it happens anyway.++`pipelex-server`'s fixture `_MODEL_REFUSED_REPORT` (`platform/tests/unit/platform/routers/v1/test_runs.py`) records the `change_input` shape. It tests the platform's route rather than the runner's classification, so it stays green, but it will read as out of date once the server moves its `pipelex` pin.diff --git a/wip/structured-error-classification/plan.md b/wip/structured-error-classification/plan.mdnew file mode 100644index 000000000..bae8c9942--- /dev/null+++ b/wip/structured-error-classification/plan.md@@ -0,0 +1,56 @@+---+status: landed+item: L-260927-086ea4+---++# Plan: a gateway's model_not_allowed refusal says to change the model++The design is `design.md` beside this file. The work happens in the worktree `_pipelex--structured-error-classification`, on the branch `fix/Structured-error-classification`, based on `85729dfd3`. Tests come first: write the failing tests before touching the code they cover. Louis approved this plan for implementation on 2026-09-28.++## Picking it up++- Enter the worktree with `wt open --for L-260927-086ea4`, then run `ledger claim L-260927-086ea4 --renew` from inside it.+- The helpers the tests need already exist in `tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.py`: `_as_the_portkey_sdk_raises_it`, `_as_the_openai_sdk_raises_it` and `_ORIGIN`. For the Anthropic hop, `anthropic.AsyncAnthropic(api_key=…, base_url=_ORIGIN)._make_status_error_from_response(response)` builds what the SDK raises; for a 412 that is a plain `APIStatusError`.+- For the real-loop module, the earlier probe is a working sketch. Build a real `openai.AsyncOpenAI` or `anthropic.AsyncAnthropic` with `http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler))` and `max_retries=0`, wrap it with `from_openai(client, mode=Mode.TOOLS)` or `from_anthropic(client)`, and call `create_with_completion(…, response_model=…, max_retries=make_instructor_schema_retrying(max_attempts=3))`. For an invalid output, return a chat completion whose tool call's arguments fail the schema. The Anthropic SDK release locked here takes an `httpx` client. More recent Anthropic releases insist on `httpx2`, so a future lock move will need the transport changed.+- The campaign's 412 body is `{"error": {"message": "Model us.anthropic.claude-sonnet-4-5-20250929-v1:0 is not allowed for this integration", "type": "model_not_allowed_error", "param": null, "code": null}}`, and its model handle is `claude-4.5-sonnet`.+- `_render_detail` has a single caller, `render_inference_error`, so adding `model_handle` to its signature touches nothing else. It is keyword-only after the existing subject.++## What already landed++The instructor half of the first draft (reading `__cause__` first in `extract_underlying_sdk_exception`, moving the lock, raising the floor, and rewriting the helper and the unwrap docs) landed in pipelex#1289 as `85729dfd3`. It was re-verified against the real instructor 1.17.0 loop; the table is in the design.++## Classify model_not_allowed and pin the instructor wrapper to the real loop++- [x] Write the failing tests first.+ - Extend `tests/unit/pipelex/cogt/inference/test_gateway_routing_refusals.py`:+ - `model_not_allowed_error` maps to `MODEL_NOT_ALLOWED`;+ - it survives every Extract hop, each exception built through its SDK's own factory: the OpenAI hop through `_make_status_error_from_response(response)`, which takes no `request` argument, the Anthropic hop, the Portkey substrate, and the manifold's httpx hop;+ - it classifies as `CONFIGURATION` / `CHANGE_MODEL`, not retried, with `is_model_not_found` unset;+ - the advice names the model handle and not the wire id;+ - the HTTP status stays 500;+ - the advice survives the pipe boundary.+ - Add the real-loop module of design D4, for example `tests/unit/pipelex/providers/test_instructor_retry_shapes.py`. It drives `from_openai` and `from_anthropic` over `httpx.MockTransport` for the three shapes, and asserts that the recovery returns the exception that ended the loop and that the raised shape is the one `wrap_in_instructor_retry` builds. In the same module, drive the campaign's scenario end to end: the OpenAI completions worker's `_gen_object` over real instructor, the gateway answering 412 with the campaign's body, and the located failure report showing `error_domain: config`, `error_category: configuration` and `user_action.kind: change_model`.+- [x] Add `GatewayRoutingRefusal.MODEL_NOT_ALLOWED` and its map entry. Rewrite the enum docstring and the map comment per design D1: this member is keyed on the gateway substrate's code and arrives on 412, it has two raise sites (not in the allow-list, or archived), and no vendor collision is possible.+- [x] Add its arm to `_classify_gateway_routing_refusal`, with a comment tying it to `WRONG_PROTOCOL` and `UNSERVED_CAPABILITY` on why the flag stays unset.+- [x] Thread `model_handle` from `render_inference_error` through `_render_detail` into `_render_gateway_routing_refusal_detail`, and write this member's advice (design D2). The other members keep ignoring the handle. Update that function's docstring, which says the family names no model, to state this exception and its reason.+- [x] In `docs/under-the-hood/error-model.md`, add the row to the "When the model cannot be routed" table and a bullet on the unset flag and the handle in the advice. In the "`instructor` Unwrap" section, correct the sentence claiming a `pydantic.ValidationError` lands in `UNKNOWN` (design D5).+- [x] In `CHANGELOG.md` under Unreleased, add a Fixed entry: a gateway's `model_not_allowed_error` refusal now reads as a configuration error with a `change_model` action naming the model handle, instead of "review the prompt, parameters, and inputs", and the HTTP status is unchanged at 500. Correct the existing "A transport error inside a structured-output call is classified again" entry from "1.16 and later" to "1.15.3 and later" (design D5).+- [x] Add a note to `L-260927-24b43d` saying the lock moved from instructor 1.15.1 to 1.17.0 in `85729dfd3`, since that item's evidence cites 1.15.1 internals and needs re-reading at the new version.+- [x] Run `make agent-check`, then `make agent-test`.++### Checkpoint++- [x] Record here the final wording of the advice and anything review changed in the design.+ - The advice reads: "The inference gateway does not allow the model '<handle>' for this account — pick another model for the pipe; if the pipe named this one, leaving its model unset uses the default instead. If your model deck lists it as available, the deck and the gateway's allow-list disagree: on the Pipelex gateway, contact support; on a Portkey workspace of your own, allow the model in the integration that serves it." The design's draft said "for this access"; "account" is the word a caller recognizes.+ - Review round 1 (profile 3, cubic, Codex, code-review) changed two things in the design, both recorded there. Portkey's cloud also emits `model_not_allowed_error` for a user's own workspace behind the `portkey` backend, so the claim that only Pipelex-configured gateways emit it was corrected in the code comment, the error-model doc and D1, and the advice now says who settles a deck disagreement on each gateway. The hint to leave the pipe's model unset is now conditional, since the refused model may be the default. The round also removed the family counts from the doc and the test docstring. A finding that the `wip/` documents name private repositories and paths was refuted: the public repository's source and other campaign documents already do, and nothing named is a secret.+ - The real-loop module is `tests/unit/pipelex/providers/test_instructor_retry_shapes.py`. Beyond the design, it pins that a spent budget recovers the very `ValidationError` object instructor recorded last in `failed_attempts`.+ - Beyond D5, the "Behavior Summary" table of `docs/under-the-hood/error-model.md` made the same false claim as the unwrap section (a spent re-ask budget reading `UNKNOWN`, HTTP 500). It reads `CONTENT` / `CHANGE_INPUT`, HTTP 422, verified by rendering a recovered `ValidationError`, and the row now says so; a row for this refusal was added beside it.+- [x] Run `/rev`, then open the pull request with `Closes L-260927-086ea4`.+ - Review converged at round 2, which fixed nothing. Its one confirmed finding, that STRICT disclosure drops every inference error's `user_action` (this refusal's included), was already true before this branch and covers the whole family, so it went to the ledger.++## Out of scope, with where it lives++- Refusing a model the account cannot use at validation: `L-260927-2a587b` (`pipelex-server`). Resolving every model reference to a served handle: `L-260925-cfb6f3`.+- The api-dev gateway's allow-list disagreeing with the remote config: `L-260923-2c7f6b` (`pipelex-remote-config`).+- Usage accounting across re-asked attempts: `L-260927-24b43d`.+- Keeping an inference error's authored `user_action` under STRICT disclosure, where it is dropped with the provider-bearing message: `L-260928-69b18f`.