Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

fix(routing): advance the chain on an upstream 502, 503 or 504

Jack Rudenko5e839b511c78
diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.md
index a07dc9c3..c76a17c9 100644
--- a/ai-docs/architecture/adapters.md
+++ b/ai-docs/architecture/adapters.md
@@ -247,7 +247,7 @@ Every retry hook in claudish keys off the HTTP **status** (`anthropic-compat.ts`
`sniffResponsesStreamHead()` peeks at the stream head in `composed-handler.ts` step **7b**, which is the only window where the status code is still ours to choose (once Hono flushes the 200, a 503 is no longer expressible):
- **Retryable** (`server_is_overloaded`, `server_error`, `service_unavailable_error`, prose "overloaded"/"try again later") → re-issue upstream with **progressive** backoff `3s → 15s → 30s` (`STREAM_RETRY_DELAYS_MS`). Progressive, not tight: the outage that motivated this ran ~6.5 minutes, so only the late attempts recover anything.
-- **All retries exhausted** → HTTP **503** `overloaded_error`. Safe specifically because `fallback-handler.ts`'s `isRetryableError` does NOT list 503 — it cannot silently switch the user off a pinned model, it reaches Claude Code, which runs its own retry loop.
+- **All retries exhausted** → HTTP **503** `overloaded_error`. For a pinned `provider@model` (no `FallbackHandler`) it reaches Claude Code, which runs its own retry loop. Inside a bare-name chain, `isRetryableError` advances on it to the next candidate (since 2026-09-24, see "An unavailable endpoint advances the chain" below).
- **Terminal** in-stream errors (`context_length_exceeded`, `invalid_request_error`) are NOT retried — they keep the existing inline-text treatment, which is the actionable path for them.
- **Anything else** (any content event) → `clean`, and the consumed bytes are **replayed byte-identically** so the real parser sees an unchanged stream.
@@ -370,8 +370,10 @@ There is now one response `FallbackHandler` must treat specially **before it loo
at a status or a body at all**: the 503 an exhausted tier-1 connection hold hands
back so Claude Code will re-POST (`handlers/shared/recovery-marker.ts`).
-A 503 already stops the chain — `isRetryableError` has no 503 branch. That is not
-enough, twice:
+The status cannot hold the chain: since 2026-09-24 `isRetryableError` ADVANCES on an
+upstream 502/503/504 (next section), so the marker is the only thing that keeps a
+recovery 503 in place. Even before that, when a 503 stopped the chain on status
+alone, the status was not enough, twice:
- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**,
status-agnostic on purpose (it exists *because* of the remap above), and its list
@@ -403,6 +405,31 @@ on a non-ok path, the marker must be stripped there.**
Evidence, including the live chain runs and the mutation set:
`ai-docs/reports/network-recovery-phase4-status-flip-20260911.md`.
+### An unavailable endpoint advances the chain (2026-09-24)
+
+A bare `grok-4.7` failed on every request while OpenCode Zen Go, its first candidate,
+answered `503 {"type":"server_error","message":"Upstream request failed: Endpoint is
+unavailable."}`. `--probe grok-4.7` in the same minute showed the five candidates behind it
+answering (Grok Build, xAI, OpenRouter, OpenCode Zen, Poe). `isRetryableError` had no
+502/503/504 branch, so the first attempt's 503 was returned as-is with no `[Fallback]` line,
+and Claude Code's retry loop re-sent the same chain to the same dead hop, ten times.
+
+The exclusion rested on "a 503 cannot silently switch the user off a pinned model". A pinned
+`provider@model` builds no `FallbackHandler`, so inside a chain that protected nothing. Now:
+
+- `isRetryableError` returns `true` for 502, 503 and 504. A plain 500 still stops the chain
+ unless it carries billing wording: it can carry account state, and it is the one 5xx a
+ request's own payload can provoke on every provider alike.
+- `exhaustedChainStatus` counts 502 and 504 as transient beside 429 and 503, so a chain in
+ which every hop was unavailable still ends retryable (503), as the single 502 that used to be
+ returned as-is did.
+- Unchanged, and the reason this is safe: claudish's own connection verdicts
+ (`x-claudish-recovery` on the 503 arm, `x-claudish-connection-error` on the 400 arm) are
+ checked before any status, in `handle()` and on `isRetryableError`'s first line. A network
+ outage on this machine still holds the chain.
+- Changed on purpose: the stream sniffers' exhaustion 503 (overloaded after 3s → 15s → 30s of
+ retries) now advances a bare-name chain instead of returning to Claude Code.
+
## The catalog's endpoint contract has two halves (v9.0.7)
`gpt-6-astra` was in the catalog and could not run. The child exited 1 on
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.md
index a4ccf975..805b5ab5 100644
--- a/ai-docs/architecture/network-recovery.md
+++ b/ai-docs/architecture/network-recovery.md
@@ -268,9 +268,10 @@ every candidate — each paying its own multi-minute budget — and per CLAUDE.m
an advance off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money for a fault
no provider caused.
-503 does not have this problem: `isRetryableError` has **no 503 branch**. It is the status the house
-already owns for "transient after our own retries, do not switch the user's provider" — the stream
-sniffer's exhaustion arm uses it for exactly that reason (`adapters.md`).
+503 was chosen because, at the time, `isRetryableError` had **no 503 branch**. Since 2026-09-24 it
+has one: an upstream 502/503/504 advances a bare-name chain (`adapters.md`, "An unavailable endpoint
+advances the chain"). The recovery 503 is unaffected, because the status was never its guarantee:
+the `x-claudish-recovery` marker below is checked before any status is read.
529 was considered and rejected: it is absent from `exhaustedChainStatus`'s transient set and would
add a status this codebase has never carried, re-opening every `status ===` under `handlers/`.
diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.ts
index e881bd25..d6046649 100644
--- a/packages/cli/src/handlers/composed-handler.ts
+++ b/packages/cli/src/handlers/composed-handler.ts
@@ -2100,10 +2100,10 @@ export class ComposedHandler implements ModelHandler {
* tail is where recovery actually happens.
*
* Returns `exhausted` when every attempt failed — the caller turns that into a
- * 503 so Claude Code runs its own retry loop against the same pinned model.
- * (A 503 is safe here specifically because fallback-handler's isRetryableError
- * does NOT list 503, so this cannot silently switch the user off the model they
- * pinned; it reaches Claude Code untouched.)
+ * 503. For a pinned `provider@model` there is no FallbackHandler, so the 503
+ * reaches Claude Code, which runs its own retry loop against the same model.
+ * Inside a bare-name chain, fallback-handler's isRetryableError advances on it
+ * to the next candidate: this provider stayed overloaded through every retry.
*/
private async settleResponsesStreamHead(
initial: Response,
diff --git a/packages/cli/src/handlers/fallback-handler.test.ts b/packages/cli/src/handlers/fallback-handler.test.ts
index 0cbcbc98..02d39db9 100644
--- a/packages/cli/src/handlers/fallback-handler.test.ts
+++ b/packages/cli/src/handlers/fallback-handler.test.ts
@@ -359,7 +359,12 @@ describe("Group 5: isRetryableError — unit tests via FallbackHandler behavior"
// whether FallbackHandler tries the next candidate or stops.
const { Hono } = require("hono");
- const { FallbackHandler, isProvider, isRetryableError } = require("./fallback-handler.js");
+ const {
+ FallbackHandler,
+ exhaustedChainStatus,
+ isProvider,
+ isRetryableError,
+ } = require("./fallback-handler.js");
const { extractUpstreamStatus } = require("./shared/anthropic-error.js");
function remappedErrorBody(upstreamStatus?: unknown): string {
@@ -485,9 +490,108 @@ describe("Group 5: isRetryableError — unit tests via FallbackHandler behavior"
expect(result.usedFallback).toBe(true);
});
- test("503 service unavailable is NOT retryable — stops immediately", async () => {
+ test("503 service unavailable advances to the next candidate", async () => {
const result = await runFallback(503, '{"error":"service unavailable"}');
- expect(result.usedFallback).toBe(false);
+ expect(result.usedFallback).toBe(true);
+ });
+
+ describe("an unavailable endpoint advances the chain", () => {
+ for (const status of [502, 503, 504]) {
+ test(`HTTP ${status} with a plain body is retryable`, () => {
+ expect(isRetryableError(status, "{}")).toBe(true);
+ });
+ }
+
+ test("plain HTTP 500 remains non-retryable", () => {
+ expect(isRetryableError(500, '{"error":"internal server error"}')).toBe(false);
+ });
+
+ test("OpenCode Zen Go endpoint-unavailable 503 reaches the second candidate", async () => {
+ const result = await runFallbackForProvider(
+ "OpenCode Zen Go",
+ 503,
+ '{"type":"error","error":{"type":"server_error","message":"Upstream request failed: Endpoint is unavailable."}}'
+ );
+ expect(result.usedFallback).toBe(true);
+ expect(result.status).toBe(200);
+ });
+
+ test("502 then 504 reaches 200 and calls each candidate exactly once", async () => {
+ const calls = [0, 0, 0];
+ const handler = new FallbackHandler(
+ [502, 504, 200].map((status, index) => {
+ const candidate = mockHandler(status, "{}");
+ return {
+ name: `provider-${index}`,
+ handler: {
+ ...candidate,
+ handle: async () => {
+ calls[index]++;
+ return candidate.handle();
+ },
+ },
+ };
+ })
+ );
+ const app = new Hono();
+ app.post("/test", async (c: any) => handler.handle(c, { model: "test-model" }));
+ const response = await app.request("/test", { method: "POST", body: "{}" });
+ expect(response.status).toBe(200);
+ expect(calls).toEqual([1, 1, 1]);
+ });
+
+ test("all candidates unavailable: 502 then 504 ends with HTTP 503", async () => {
+ const handler = new FallbackHandler([
+ { name: "a", handler: mockHandler(502, "{}") },
+ { name: "b", handler: mockHandler(504, "{}") },
+ ]);
+ const app = new Hono();
+ app.post("/test", async (c: any) => handler.handle(c, { model: "test-model" }));
+ const response = await app.request("/test", { method: "POST", body: "{}" });
+ expect(response.status).toBe(503);
+ });
+ });
+
+ describe("exhaustedChainStatus counts 502 and 504 as transient", () => {
+ test("502 and 504 exhaust to 503", () => {
+ expect(
+ exhaustedChainStatus([
+ { provider: "a", status: 502, message: "{}" },
+ { provider: "b", status: 504, message: "{}" },
+ ])
+ ).toBe(503);
+ });
+
+ for (const upstreamStatus of [502, 504]) {
+ test(`remapped 400 with upstream_status ${upstreamStatus} and 503 exhaust to 503`, () => {
+ expect(
+ exhaustedChainStatus([
+ {
+ provider: "a",
+ status: 400,
+ message: JSON.stringify({
+ type: "error",
+ error: {
+ type: "invalid_request_error",
+ message: "x",
+ upstream_status: upstreamStatus,
+ },
+ }),
+ },
+ { provider: "b", status: 503, message: "{}" },
+ ])
+ ).toBe(503);
+ });
+ }
+
+ test("terminal 401 mixed with 502 exhausts to 400", () => {
+ expect(
+ exhaustedChainStatus([
+ { provider: "a", status: 401, message: "{}" },
+ { provider: "b", status: 502, message: "{}" },
+ ])
+ ).toBe(400);
+ });
});
test("401 authentication_error (refreshAuth failure) is retryable — falls through to next provider", async () => {
diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.ts
index 7897e040..93bc2a27 100644
--- a/packages/cli/src/handlers/fallback-handler.ts
+++ b/packages/cli/src/handlers/fallback-handler.ts
@@ -203,9 +203,10 @@ export class FallbackHandler implements ModelHandler {
/**
* Determine if an HTTP error is retryable (should try next provider).
- * Auth errors, billing errors, rate limits, and model-not-found errors
- * warrant trying a different provider. True server errors (500 without
- * billing context) do NOT — they'd likely fail on any provider.
+ * Auth errors, billing errors, rate limits, model-not-found errors and an
+ * unavailable endpoint (502/503/504) warrant trying a different provider. A
+ * plain 500 does NOT: it may carry account state, and it is the one 5xx a
+ * request's own payload can provoke on every provider alike.
*/
export function isRetryableError(
status: number,
@@ -309,6 +310,20 @@ export function isRetryableError(
// Rate limited — per-provider limit, a different provider may have capacity
if (status === 429) return true;
+ // Unavailable — a gateway or overload answer is about THIS endpoint, not about
+ // the model anywhere else. A bare `grok-4.7` stopped at its first candidate on
+ // Zen Go's `503 Upstream request failed: Endpoint is unavailable.` while the
+ // five candidates behind it answered (`--probe grok-4.7`, 2026-09-24), and
+ // Claude Code's retry loop re-sent the same chain to the same dead hop.
+ //
+ // The old exclusion argued a 503 "cannot silently switch the user off the model
+ // they pinned". A pinned `provider@model` builds no FallbackHandler at all
+ // (proxy-server: one candidate → its handler directly), so inside a chain that
+ // protected nothing. claudish's OWN "cannot reach the host" 503 still holds the
+ // chain: the connection-verdict check at the top of this function answers first,
+ // and `handle()` returns a recovery-hold response before it gets here.
+ if (status === 502 || status === 503 || status === 504) return true;
+
const lower = errorBody.toLowerCase();
// Unprocessable (422) — some providers (OpenRouter) use this for model unavailability
@@ -455,9 +470,9 @@ export function isRetryableError(
* candidate, so a full chain of them never reached this function at all. Making
* the chain advance is what surfaced the second half of the same defect.
*
- * Scoped to 429 and 503 deliberately: exactly the set already treated as transient
- * for un-remapped statuses, so the rule becomes independent of whether a remap
- * happened rather than gaining a new one. A remapped 401/403/402 stays terminal
+ * Scoped to 429, 502, 503 and 504 deliberately: exactly the set treated as
+ * transient for un-remapped statuses, so the rule stays independent of whether a
+ * remap happened rather than gaining a new one. A remapped 401/403/402 stays terminal
* and still surfaces inline, which is the whole point of the 400 doctrine.
*
* ── A RECOVERY 503 CAN NEVER REACH THIS FUNCTION, BY CONSTRUCTION ────────────
@@ -483,11 +498,15 @@ export function exhaustedChainStatus(
errors: Array<{ provider: string; status: number; message: string }>
): number {
if (errors.length === 0) return 400;
+ // 502 and 504 joined 429/503 when `isRetryableError` began advancing on them:
+ // a chain in which every hop was unavailable must still end retryable, as the
+ // single 502 that used to be returned as-is did.
+ const TRANSIENT = new Set([429, 502, 503, 504]);
const isTransient = (e: { status: number; message: string }): boolean => {
- if (e.status === 429 || e.status === 503) return true;
+ if (TRANSIENT.has(e.status)) return true;
if (hasQuotaExhaustionWording(e.message)) return true;
const upstream = e.status === 400 ? extractUpstreamStatus(e.message) : undefined;
- return upstream === 429 || upstream === 503;
+ return upstream !== undefined && TRANSIENT.has(upstream);
};
return errors.every(isTransient) ? 503 : 400;
}
diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.ts
index 9a5f5290..5420541a 100644
--- a/packages/cli/src/handlers/recovery-status.test.ts
+++ b/packages/cli/src/handlers/recovery-status.test.ts
@@ -448,6 +448,7 @@ describe("exhaustedChainStatus cannot fold a recovery 503 into a terminal 400",
});
describe("the marker is checked before the phrase list, inside isRetryableError too", () => {
+ const { CONNECTION_FAULT_HEADER, RECOVERY_MARKER_VALUE } = require("./shared/recovery-marker.js");
const quotaBody = JSON.stringify({ error: { message: "You have exceeded your quota" } });
const marked = new Headers({ [RECOVERY_MARKER_HEADER]: "1" });
@@ -460,12 +461,17 @@ describe("the marker is checked before the phrase list, inside isRetryableError
});
test("the marker also beats the statuses that are otherwise retryable", () => {
- for (const status of [401, 402, 403, 404, 429]) {
+ for (const status of [401, 402, 403, 404, 429, 502, 503, 504]) {
expect(isRetryableError(status, "{}", "Held Probe", marked)).toBe(false);
expect(isRetryableError(status, "{}", "Held Probe")).toBe(true);
}
});
+ test("the connection-fault marker also blocks an unavailable endpoint", () => {
+ const headers = new Headers({ [CONNECTION_FAULT_HEADER]: RECOVERY_MARKER_VALUE });
+ expect(isRetryableError(503, "{}", "x", headers)).toBe(false);
+ });
+
test("a wrong value is not the marker", () => {
expect(isRetryableError(401, "{}", "x", new Headers({ [RECOVERY_MARKER_HEADER]: "0" }))).toBe(
true
diff --git a/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts b/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts
index 3f6a3ded..4f670a25 100644
--- a/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts
+++ b/packages/cli/src/handlers/shared/devin-stream-head-sniffer.ts
@@ -31,10 +31,10 @@
* carries information the user needs (an unserved uid, a revoked entitlement),
* so it is surfaced as an HTTP **400 `invalid_request_error`** — a status Claude
* Code renders verbatim and inline — rather than buried under "API error ·
- * Retrying · attempt N/10". Retryable-after-our-own-retries becomes **503**,
- * which is safe specifically because `fallback-handler.isRetryableError` does
- * NOT list 503: it cannot silently switch the user off a pinned model, and it
- * reaches Claude Code, which runs its own retry loop.
+ * Retrying · attempt N/10". Retryable-after-our-own-retries becomes **503**.
+ * For a pinned `dv@` spec that reaches Claude Code, which runs its own retry
+ * loop; inside a bare-name chain `fallback-handler.isRetryableError` advances on
+ * it to the next candidate.
*
* `permission_denied` deserves a note. It was caused, in every reproduction
* during protocol work, SOLELY by encoding a message with the wrong role enum
diff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.ts
index 75177c33..f2de46c8 100644
--- a/packages/cli/src/handlers/shared/recovery-marker.ts
+++ b/packages/cli/src/handlers/shared/recovery-marker.ts
@@ -15,8 +15,11 @@
*
* ## Why a header and not a status, and not a wording rule
*
- * `fallback-handler.ts`'s `isRetryableError` has no 503 branch, so on the
- * status alone a 503 already stops the chain. That is NOT enough, twice over:
+ * The status cannot carry it: `fallback-handler.ts`'s `isRetryableError`
+ * ADVANCES the chain on a 503 (an upstream "endpoint unavailable" is about one
+ * endpoint), so this marker is the only thing that holds the chain for a
+ * recovery 503. Before that branch existed a 503 stopped the chain on status
+ * alone, and even then that was NOT enough, twice over:
*
* 1. **The FIRST statement in its body is `hasQuotaExhaustionWording(errorBody)`,
* which is deliberately status-agnostic** — it exists because the transport