Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
fix(routing): advance the chain on an upstream 502, 503 or 504
Jack Rudenko5e839b511c78
diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.mdindex a07dc9c3..c76a17c9 100644--- a/ai-docs/architecture/adapters.md+++ b/ai-docs/architecture/adapters.md@@ -247,7 +247,7 @@ Every retry hook in claudish keys off the HTTP **status** (`anthropic-compat.ts``sniffResponsesStreamHead()` peeks at the stream head in `composed-handler.ts` step **7b**, which is the only window where the status code is still ours to choose (once Hono flushes the 200, a 503 is no longer expressible):- **Retryable** (`server_is_overloaded`, `server_error`, `service_unavailable_error`, prose "overloaded"/"try again later") → re-issue upstream with **progressive** backoff `3s → 15s → 30s` (`STREAM_RETRY_DELAYS_MS`). Progressive, not tight: the outage that motivated this ran ~6.5 minutes, so only the late attempts recover anything.-- **All retries exhausted** → HTTP **503** `overloaded_error`. Safe specifically because `fallback-handler.ts`'s `isRetryableError` does NOT list 503 — it cannot silently switch the user off a pinned model, it reaches Claude Code, which runs its own retry loop.+- **All retries exhausted** → HTTP **503** `overloaded_error`. For a pinned `provider@model` (no `FallbackHandler`) it reaches Claude Code, which runs its own retry loop. Inside a bare-name chain, `isRetryableError` advances on it to the next candidate (since 2026-09-24, see "An unavailable endpoint advances the chain" below).- **Terminal** in-stream errors (`context_length_exceeded`, `invalid_request_error`) are NOT retried — they keep the existing inline-text treatment, which is the actionable path for them.- **Anything else** (any content event) → `clean`, and the consumed bytes are **replayed byte-identically** so the real parser sees an unchanged stream.@@ -370,8 +370,10 @@ There is now one response `FallbackHandler` must treat specially **before it looat a status or a body at all**: the 503 an exhausted tier-1 connection hold handsback so Claude Code will re-POST (`handlers/shared/recovery-marker.ts`).-A 503 already stops the chain — `isRetryableError` has no 503 branch. That is not-enough, twice:+The status cannot hold the chain: since 2026-09-24 `isRetryableError` ADVANCES on an+upstream 502/503/504 (next section), so the marker is the only thing that keeps a+recovery 503 in place. Even before that, when a 503 stopped the chain on status+alone, the status was not enough, twice:- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**,status-agnostic on purpose (it exists *because* of the remap above), and its list@@ -403,6 +405,31 @@ on a non-ok path, the marker must be stripped there.**Evidence, including the live chain runs and the mutation set:`ai-docs/reports/network-recovery-phase4-status-flip-20260911.md`.+### An unavailable endpoint advances the chain (2026-09-24)++A bare `grok-4.7` failed on every request while OpenCode Zen Go, its first candidate,+answered `503 {"type":"server_error","message":"Upstream request failed: Endpoint is+unavailable."}`. `--probe grok-4.7` in the same minute showed the five candidates behind it+answering (Grok Build, xAI, OpenRouter, OpenCode Zen, Poe). `isRetryableError` had no+502/503/504 branch, so the first attempt's 503 was returned as-is with no `[Fallback]` line,+and Claude Code's retry loop re-sent the same chain to the same dead hop, ten times.++The exclusion rested on "a 503 cannot silently switch the user off a pinned model". A pinned+`provider@model` builds no `FallbackHandler`, so inside a chain that protected nothing. Now:++- `isRetryableError` returns `true` for 502, 503 and 504. A plain 500 still stops the chain+ unless it carries billing wording: it can carry account state, and it is the one 5xx a+ request's own payload can provoke on every provider alike.+- `exhaustedChainStatus` counts 502 and 504 as transient beside 429 and 503, so a chain in+ which every hop was unavailable still ends retryable (503), as the single 502 that used to be+ returned as-is did.+- Unchanged, and the reason this is safe: claudish's own connection verdicts+ (`x-claudish-recovery` on the 503 arm, `x-claudish-connection-error` on the 400 arm) are+ checked before any status, in `handle()` and on `isRetryableError`'s first line. A network+ outage on this machine still holds the chain.+- Changed on purpose: the stream sniffers' exhaustion 503 (overloaded after 3s → 15s → 30s of+ retries) now advances a bare-name chain instead of returning to Claude Code.+## The catalog's endpoint contract has two halves (v9.0.7)`gpt-6-astra` was in the catalog and could not run. The child exited 1 ondiff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.mdindex a4ccf975..805b5ab5 100644--- a/ai-docs/architecture/network-recovery.md+++ b/ai-docs/architecture/network-recovery.md@@ -268,9 +268,10 @@ every candidate — each paying its own multi-minute budget — and per CLAUDE.man advance off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money for a faultno provider caused.-503 does not have this problem: `isRetryableError` has **no 503 branch**. It is the status the house-already owns for "transient after our own retries, do not switch the user's provider" — the stream-sniffer's exhaustion arm uses it for exactly that reason (`adapters.md`).+503 was chosen because, at the time, `isRetryableError` had **no 503 branch**. Since 2026-09-24 it+has one: an upstream 502/503/504 advances a bare-name chain (`adapters.md`, "An unavailable endpoint+advances the chain"). The recovery 503 is unaffected, because the status was never its guarantee:+the `x-claudish-recovery` marker below is checked before any status is read.529 was considered and rejected: it is absent from `exhaustedChainStatus`'s transient set and wouldadd a status this codebase has never carried, re-opening every `status ===` under `handlers/`.diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex e881bd25..d6046649 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -2100,10 +2100,10 @@ export class ComposedHandler implements ModelHandler {* tail is where recovery actually happens.** Returns `exhausted` when every attempt failed — the caller turns that into a- * 503 so Claude Code runs its own retry loop against the same pinned model.- * (A 503 is safe here specifically because fallback-handler's isRetryableError- * does NOT list 503, so this cannot silently switch the user off the model they- * pinned; it reaches Claude Code untouched.)+ * 503. For a pinned `provider@model` there is no FallbackHandler, so the 503+ * reaches Claude Code, which runs its own retry loop against the same model.+ * Inside a bare-name chain, fallback-handler's isRetryableError advances on it+ * to the next candidate: this provider stayed overloaded through every retry.*/private async settleResponsesStreamHead(initial: Response,diff --git a/packages/cli/src/handlers/fallback-handler.test.ts b/packages/cli/src/handlers/fallback-handler.test.tsindex 0cbcbc98..02d39db9 100644--- a/packages/cli/src/handlers/fallback-handler.test.ts+++ b/packages/cli/src/handlers/fallback-handler.test.ts@@ -359,7 +359,12 @@ describe("Group 5: isRetryableError — unit tests via FallbackHandler behavior"// whether FallbackHandler tries the next candidate or stops.const { Hono } = require("hono");- const { FallbackHandler, isProvider, isRetryableError } = require("./fallback-handler.js");+ const {+ FallbackHandler,+ exhaustedChainStatus,+ isProvider,+ isRetryableError,+ } = require("./fallback-handler.js");const { extractUpstreamStatus } = require("./shared/anthropic-error.js");function remappedErrorBody(upstreamStatus?: unknown): string {@@ -485,9 +490,108 @@ describe("Group 5: isRetryableError — unit tests via FallbackHandler behavior"expect(result.usedFallback).toBe(true);});- test("503 service unavailable is NOT retryable — stops immediately", async () => {+ test("503 service unavailable advances to the next candidate", async () => {const result = await runFallback(503, '{"error":"service unavailable"}');- expect(result.usedFallback).toBe(false);+ expect(result.usedFallback).toBe(true);+ });++ describe("an unavailable endpoint advances the chain", () => {+ for (const status of [502, 503, 504]) {+ test(`HTTP ${status} with a plain body is retryable`, () => {+ expect(isRetryableError(status, "{}")).toBe(true);+ });+ }++ test("plain HTTP 500 remains non-retryable", () => {+ expect(isRetryableError(500, '{"error":"internal server error"}')).toBe(false);+ });++ test("OpenCode Zen Go endpoint-unavailable 503 reaches the second candidate", async () => {+ const result = await runFallbackForProvider(+ "OpenCode Zen Go",+ 503,+ '{"type":"error","error":{"type":"server_error","message":"Upstream request failed: Endpoint is unavailable."}}'+ );+ expect(result.usedFallback).toBe(true);+ expect(result.status).toBe(200);+ });++ test("502 then 504 reaches 200 and calls each candidate exactly once", async () => {+ const calls = [0, 0, 0];+ const handler = new FallbackHandler(+ [502, 504, 200].map((status, index) => {+ const candidate = mockHandler(status, "{}");+ return {+ name: `provider-${index}`,+ handler: {+ ...candidate,+ handle: async () => {+ calls[index]++;+ return candidate.handle();+ },+ },+ };+ })+ );+ const app = new Hono();+ app.post("/test", async (c: any) => handler.handle(c, { model: "test-model" }));+ const response = await app.request("/test", { method: "POST", body: "{}" });+ expect(response.status).toBe(200);+ expect(calls).toEqual([1, 1, 1]);+ });++ test("all candidates unavailable: 502 then 504 ends with HTTP 503", async () => {+ const handler = new FallbackHandler([+ { name: "a", handler: mockHandler(502, "{}") },+ { name: "b", handler: mockHandler(504, "{}") },+ ]);+ const app = new Hono();+ app.post("/test", async (c: any) => handler.handle(c, { model: "test-model" }));+ const response = await app.request("/test", { method: "POST", body: "{}" });+ expect(response.status).toBe(503);+ });+ });++ describe("exhaustedChainStatus counts 502 and 504 as transient", () => {+ test("502 and 504 exhaust to 503", () => {+ expect(+ exhaustedChainStatus([+ { provider: "a", status: 502, message: "{}" },+ { provider: "b", status: 504, message: "{}" },+ ])+ ).toBe(503);+ });++ for (const upstreamStatus of [502, 504]) {+ test(`remapped 400 with upstream_status ${upstreamStatus} and 503 exhaust to 503`, () => {+ expect(+ exhaustedChainStatus([+ {+ provider: "a",+ status: 400,+ message: JSON.stringify({+ type: "error",+ error: {+ type: "invalid_request_error",+ message: "x",+ upstream_status: upstreamStatus,+ },+ }),+ },+ { provider: "b", status: 503, message: "{}" },+ ])+ ).toBe(503);+ });+ }++ test("terminal 401 mixed with 502 exhausts to 400", () => {+ expect(+ exhaustedChainStatus([+ { provider: "a", status: 401, message: "{}" },+ { provider: "b", status: 502, message: "{}" },+ ])+ ).toBe(400);+ });});test("401 authentication_error (refreshAuth failure) is retryable — falls through to next provider", async () => {diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.tsindex 7897e040..93bc2a27 100644--- a/packages/cli/src/handlers/fallback-handler.ts+++ b/packages/cli/src/handlers/fallback-handler.ts@@ -203,9 +203,10 @@ export class FallbackHandler implements ModelHandler {/*** Determine if an HTTP error is retryable (should try next provider).- * Auth errors, billing errors, rate limits, and model-not-found errors- * warrant trying a different provider. True server errors (500 without- * billing context) do NOT — they'd likely fail on any provider.+ * Auth errors, billing errors, rate limits, model-not-found errors and an+ * unavailable endpoint (502/503/504) warrant trying a different provider. A+ * plain 500 does NOT: it may carry account state, and it is the one 5xx a+ * request's own payload can provoke on every provider alike.*/export function isRetryableError(status: number,@@ -309,6 +310,20 @@ export function isRetryableError(// Rate limited — per-provider limit, a different provider may have capacityif (status === 429) return true;+ // Unavailable — a gateway or overload answer is about THIS endpoint, not about+ // the model anywhere else. A bare `grok-4.7` stopped at its first candidate on+ // Zen Go's `503 Upstream request failed: Endpoint is unavailable.` while the+ // five candidates behind it answered (`--probe grok-4.7`, 2026-09-24), and+ // Claude Code's retry loop re-sent the same chain to the same dead hop.+ //+ // The old exclusion argued a 503 "cannot silently switch the user off the model+ // they pinned". A pinned `provider@model` builds no FallbackHandler at all+ // (proxy-server: one candidate → its handler directly), so inside a chain that+ // protected nothing. claudish's OWN "cannot reach the host" 503 still holds the+ // chain: the connection-verdict check at the top of this function answers first,+ // and `handle()` returns a recovery-hold response before it gets here.+ if (status === 502 || status === 503 || status === 504) return true;+const lower = errorBody.toLowerCase();// Unprocessable (422) — some providers (OpenRouter) use this for model unavailability@@ -455,9 +470,9 @@ export function isRetryableError(* candidate, so a full chain of them never reached this function at all. Making* the chain advance is what surfaced the second half of the same defect.*- * Scoped to 429 and 503 deliberately: exactly the set already treated as transient- * for un-remapped statuses, so the rule becomes independent of whether a remap- * happened rather than gaining a new one. A remapped 401/403/402 stays terminal+ * Scoped to 429, 502, 503 and 504 deliberately: exactly the set treated as+ * transient for un-remapped statuses, so the rule stays independent of whether a+ * remap happened rather than gaining a new one. A remapped 401/403/402 stays terminal* and still surfaces inline, which is the whole point of the 400 doctrine.** ── A RECOVERY 503 CAN NEVER REACH THIS FUNCTION, BY CONSTRUCTION ────────────@@ -483,11 +498,15 @@ export function exhaustedChainStatus(errors: Array<{ provider: string; status: number; message: string }>): number {if (errors.length === 0) return 400;+ // 502 and 504 joined 429/503 when `isRetryableError` began advancing on them:+ // a chain in which every hop was unavailable must still end retryable, as the+ // single 502 that used to be returned as-is did.+ const TRANSIENT = new Set([429, 502, 503, 504]);const isTransient = (e: { status: number; message: string }): boolean => {- if (e.status === 429 || e.status === 503) return true;+ if (TRANSIENT.has(e.status)) return true;if (hasQuotaExhaustionWording(e.message)) return true;const upstream = e.status === 400 ? extractUpstreamStatus(e.message) : undefined;- return upstream === 429 || upstream === 503;+ return upstream !== undefined && TRANSIENT.has(upstream);};return errors.every(isTransient) ? 503 : 400;}diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.tsindex 9a5f5290..5420541a 100644--- a/packages/cli/src/handlers/recovery-status.test.ts+++ b/packages/cli/src/handlers/recovery-status.test.ts@@ -448,6 +448,7 @@ describe("exhaustedChainStatus cannot fold a recovery 503 into a terminal 400",});describe("the marker is checked before the phrase list, inside isRetryableError too", () => {+ const { CONNECTION_FAULT_HEADER, RECOVERY_MARKER_VALUE } = require("./shared/recovery-marker.js");const quotaBody = JSON.stringify({ error: { message: "You have exceeded your quota" } });const marked = new Headers({ [RECOVERY_MARKER_HEADER]: "1" });@@ -460,12 +461,17 @@ describe("the marker is checked before the phrase list, inside isRetryableError});test("the marker also beats the statuses that are otherwise retryable", () => {- for (const status of [401, 402, 403, 404, 429]) {+ for (const status of [401, 402, 403, 404, 429, 502, 503, 504]) {expect(isRetryableError(status, "{}", "Held Probe", marked)).toBe(false);expect(isRetryableError(status, "{}", "Held Probe")).toBe(true);}});+ test("the connection-fault marker also blocks an unavailable endpoint", () => {+ const headers = new Headers({ [CONNECTION_FAULT_HEADER]: RECOVERY_MARKER_VALUE });+ expect(isRetryableError(503, "{}", "x", headers)).toBe(false);+ });+test("a wrong value is not the marker", () => {expect(isRetryableError(401, "{}", "x", new Headers({ [RECOVERY_MARKER_HEADER]: "0" }))).toBe(truediff --git a/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts b/packages/cli/src/handlers/shared/devin-stream-head-sniffer.tsindex 3f6a3ded..4f670a25 100644--- a/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts+++ b/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts@@ -31,10 +31,10 @@* carries information the user needs (an unserved uid, a revoked entitlement),* so it is surfaced as an HTTP **400 `invalid_request_error`** — a status Claude* Code renders verbatim and inline — rather than buried under "API error ·- * Retrying · attempt N/10". Retryable-after-our-own-retries becomes **503**,- * which is safe specifically because `fallback-handler.isRetryableError` does- * NOT list 503: it cannot silently switch the user off a pinned model, and it- * reaches Claude Code, which runs its own retry loop.+ * Retrying · attempt N/10". Retryable-after-our-own-retries becomes **503**.+ * For a pinned `dv@` spec that reaches Claude Code, which runs its own retry+ * loop; inside a bare-name chain `fallback-handler.isRetryableError` advances on+ * it to the next candidate.** `permission_denied` deserves a note. It was caused, in every reproduction* during protocol work, SOLELY by encoding a message with the wrong role enumdiff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.tsindex 75177c33..f2de46c8 100644--- a/packages/cli/src/handlers/shared/recovery-marker.ts+++ b/packages/cli/src/handlers/shared/recovery-marker.ts@@ -15,8 +15,11 @@** ## Why a header and not a status, and not a wording rule*- * `fallback-handler.ts`'s `isRetryableError` has no 503 branch, so on the- * status alone a 503 already stops the chain. That is NOT enough, twice over:+ * The status cannot carry it: `fallback-handler.ts`'s `isRetryableError`+ * ADVANCES the chain on a 503 (an upstream "endpoint unavailable" is about one+ * endpoint), so this marker is the only thing that holds the chain for a+ * recovery 503. Before that branch existed a 503 stopped the chain on status+ * alone, and even then that was NOT enough, twice over:** 1. **The FIRST statement in its body is `hasQuotaExhaustionWording(errorBody)`,* which is deliberately status-agnostic** — it exists because the transport