Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

feat(recovery): hand the retry back to the client when the reason is legible

Jack Rudenko7a9f4c5b1a25
diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.md
index bf7e01eb..7af0fa2a 100644
--- a/ai-docs/architecture/adapters.md
+++ b/ai-docs/architecture/adapters.md
@@ -360,6 +360,45 @@ The general lesson: **any code that branches on an HTTP status downstream of the
remap is suspect.** Grep for `status ===` under `handlers/` before assuming a new
one is safe.
+### The status is not always enough: `x-claudish-recovery` (network recovery, Tier 2)
+
+There is now one response `FallbackHandler` must treat specially **before it looks
+at a status or a body at all**: the 503 an exhausted tier-1 connection hold hands
+back so Claude Code will re-POST (`handlers/shared/recovery-marker.ts`).
+
+A 503 already stops the chain — `isRetryableError` has no 503 branch. That is not
+enough, twice:
+
+- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**,
+ status-agnostic on purpose (it exists *because* of the remap above), and its list
+ carries the bare substring `"quota"`. A 503 whose MESSAGE happened to contain that
+ word advanced the chain — and per CLAUDE.md's standing invariant, advancing off a
+ `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money, during an
+ outage, for a fault no provider caused.
+- **`exhaustedChainStatus` could demote it.** With an earlier candidate already
+ failed, a non-retryable response goes to `formatCombinedError`, whose status is 503
+ only if EVERY accumulated error is transient. One earlier auth/404 turns our 503
+ into a terminal 400 and the client never re-POSTs.
+
+So the marker is a **header**, checked in two independent places: `handle()` returns
+a marked response VERBATIM before reading the body (which is what defeats the
+combining), and `isRetryableError` returns `false` on it ABOVE the quota match
+(which is what defeats the wording). The second is unreachable in the shipped call
+graph and is kept anyway, so the guarantee belongs to the decision rather than to one
+call site.
+
+**Order is the property, not presence.** Moving the marker check below the quota
+match — present, but late — is a live billing bug, and is mutation-covered as one.
+
+It cannot be forged: every non-ok exit from `ComposedHandler` is `c.json(...)`, which
+builds headers from nothing, and the only path that copies upstream headers verbatim
+(`stream-head-sniffer.ts`'s `replayResponse()`) runs after `!response.ok` has already
+returned, i.e. on a 200. **If a future edit ever returns an upstream `Response` object
+on a non-ok path, the marker must be stripped there.**
+
+Evidence, including the live chain runs and the mutation set:
+`ai-docs/reports/network-recovery-phase4-status-flip-20260911.md`.
+
## The catalog's endpoint contract has two halves (v9.0.7)
`gpt-6-astra` was in the catalog and could not run. The child exited 1 on
diff --git a/ai-docs/reports/network-recovery-phase4-status-flip-20260911.md b/ai-docs/reports/network-recovery-phase4-status-flip-20260911.md
new file mode 100644
index 00000000..c66155c6
--- /dev/null
+++ b/ai-docs/reports/network-recovery-phase4-status-flip-20260911.md
@@ -0,0 +1,375 @@
+# Network recovery, design-Phase 4 — the status flip, measured
+
+> Promoted out of the gitignored session directory deliberately, as
+> `network-recovery-phase0-measurements.md`, `-phase2-verification-` and `-phase3-pane-` were.
+> CLAUDE.md records three write-ups already lost because they lived in `ai-docs/sessions/`, which
+> does not survive a fresh clone or `git worktree remove`.
+>
+> Session `dev-feature-network-recovery-20260910-110514-0ad0a5a0` · measured 2026-09-11 on
+> macOS 25.6.0, bun 1.4.0, magmux 0.11.0, Claude Code 2.1.268. The phase narrative and the per-file
+> inventory stay in that session's `implementation-log-phase4.md`; what is here is the evidence.
+
+Phase 4 is the only phase where a mistake costs the user **money** rather than time. At the end of a
+tier-1 hold the handler now chooses between a retryable **503** and today's terminal **400**, and
+that 503 has to cross `FallbackHandler` without advancing the chain — because advancing off a
+`SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes a real per-token price for a fault that
+had nothing to do with any provider.
+
+Suite: **3331 → 3357 pass**, 18 skip, **2 → 2 fail** (the two pre-existing `displayWidth`
+Unicode-oracle cases, red on clean main under bun 1.4.0 while CI pins 1.3.10). +26 tests, zero new
+failures. `typecheck`, `lint` and `format` exit 0.
+
+---
+
+## 1. C-1 and C-6's second run — one real interactive session
+
+A real `claudish -i --model [email protected]:0.5b`, a real Claude Code 2.1.268, a real magmux wrapper,
+a real recovery pane, a real `ollama serve` with a real 0.5B model. The only harness is a PTY:
+`pty-host.py` allocates one of a known size and execs claudish on the slave side, which is what a
+terminal does. macOS `script(1)` cannot be used — it calls `tcgetattr` on its own stdin and dies
+when that is a pipe, which it is for every process an agent session spawns.
+
+Fault: ollama stopped, so `127.0.0.1:11434` refuses in ~1 ms. Server restarted **after the handoff**.
+
+### C-1 — the turn does not end
+
+`API Error: 400` never appeared on screen or in the byte stream, polled every 5 s for the whole run.
+What appeared instead, at t+61 s, with the session still live:
+
+```
+ ▐▛███▛█ Claude Code v2.1.268
+▝▜██████▀ [email protected]:0.5b with xhigh effort · API Usage Billing
+❯ Reply with exactly one word: banana
+✶ Wibbling… (57s)
+────────────────────────────────────────────────────────────────────────────────
+ * [email protected]:0.5b | $0.00 | 1s | 󰍛 294M | ░░░░░░ 0% • $0.000 • N/A
+────────────────────────────────────────────────────────────────────────────────
+ ██ NETWORK · Ollama refused
+ Cannot connect to Ollama at http://localhost:11434/v1/models. Make sure the server is running.
+ host localhost:11434 · ConnectionRefused · local
+ next attempt in 48s · attempt 8 · 57s in recovery
+ last: ConnectionRefused · 2 requests held
+ click here or Ctrl-G Tab, then [r] try now · [q] give up
+```
+
+`2 requests held` is not decoration: Claude Code issues the main-loop request and the small
+title/summary request concurrently, both fail within milliseconds, and both are parked on **one**
+episode with **one** timer. That is the coordinator's whole reason to exist, visible.
+
+### C-6, second run — the two tiers are ONE episode
+
+The decisive sequence, verbatim from the session's own debug log:
+
+```
+13:53:05.646 [Recovery] Ollama exhausted after 12 attempts in 165009ms
+ (episode f4d31fb2-…, ui_lease=true) — handing the retry back to the client (503, tier 2)
+13:53:05.646 [Recovery] episode f4d31fb2-… handed off after 165009ms, 12 attempts — holding 120s for a client retry
+13:53:06.192 [Recovery] Ollama rejoined episode f4d31fb2-… (client retry 1, ladder 5, attempts 12)
+13:54:06.285 [Recovery] episode f4d31fb2-… handed off after 225647ms, 16 attempts — holding 120s for a client retry
+13:54:08.534 [Recovery] Ollama answered on the client's own retry — closing episode f4d31fb2-… instead of waiting out its grace
+```
+
+- Claude Code re-POSTed **546 ms** after the 503 — well inside the measured 38.4 s backoff cap and
+ 3.1× inside `EPISODE_GRACE_MS`.
+- The rejoin carried **the same episode id**, `ladder 5` (not reset to 0) and `attempts 12`. One
+ banner, one attempt counter, one episode id across both tiers.
+- There were **two** handoffs and three client requests. `distinct episode ids in the whole log: 1`.
+
+And the turn finished, on a model that was not running when it started:
+
+```
+❯ Reply with exactly one word: banana
+⏺ banana
+✻ Cooked for 4m 1s · done 11:54 pm
+ 28772 tokens
+────────────────────────────────────────────────────────────────────────
+ * [email protected]:0.5b | $0.15 | 4m2s | 󰍛 375M | ░░░░░░ 14% • Ollama • 12% (28k/32k)
+────────────────────────────────────────────────────────────────────────
+ ██ NETWORK · Ollama recovered
+ 16 attempts over 4m 02s
+ this pane closes on its own
+```
+
+| | |
+|---|---|
+| handoff observed | t+171 s |
+| `ollama serve` started | t+191 s — **after** the handoff |
+| turn answered | t+246 s |
+| upstream 200s after recovery | 2 |
+| distinct episode ids | **1** |
+| `API Error: 400` ever on screen | **false** |
+
+**Recovery took ~43 s longer than the outage did**, and that is by design: `ladderIndex` is
+episode-scoped and does not reset on re-entry, so the rung after the rejoin was 60 s. `[r] try now`
+exists for exactly that impatience.
+
+Harness: `phase4/c1-c6-live.ts`. Evidence: `steps.txt`, `timeline.txt`, `C-6-final.txt`,
+`recovery-lines.txt`.
+
+---
+
+## 2. C-18 — Tier 2 engages for a SLOW connection failure
+
+The round-2 CRITICAL's regression test, and the only criterion in the file that is not a ~1 ms
+loopback refusal. Two runs against **F-SLOW** (`https://192.0.2.1`, TEST-NET-1), with a real magmux
+and a real `claudish recovery-pane`.
+
+**Fidelity check first**, because `192.0.2.1` is unrouted by convention and a captive portal can
+answer it: measured **75 006 ms**, code `ConnectionRefused`. Phase 0 measured 75 004 ms and Phase 2
+measured 75 005 ms on the same machine. ADD-2 confirmed a third time — *slow and fast connect
+failures are indistinguishable by code on macOS*, only elapsed time separates them.
+
+### RUN 1 — the pane alive and painting
+
+```
+RUN 1 status : 503
+RUN 1 x-should-retry : true
+RUN 1 x-claudish-recovery : 1
+RUN 1 body: {"type":"error","error":{"type":"overloaded_error","message":
+ "Cannot reach Slow Probe at https://192.0.2.1/v1/chat/completions. Check your network connection.
+ claudish retried 5× over 4m 30s without reaching it — still trying, watch the recovery pane."}}
+RUN 1 banner still painted at exhaustion : true
+RUN 1 dead-proxy notice ever shown : false
+```
+
+The attempts, which is how you confirm the fault really was slow rather than refused — they end at
+the **45 s clamp**, not at the ladder's nominal gaps:
+
+| attempt | at | code |
+|---|---|---|
+| 1 | 13:40:14 | `ConnectionRefused` (unclamped, 75 s) |
+| 2 | 13:41:04 | `TimeoutError` — 5 s gap + 45 s clamp |
+| 3 | 13:41:59 | `TimeoutError` — 10 s + 45 s |
+| 4 | 13:43:14 | `TimeoutError` — 30 s + 45 s |
+| 5 | 13:44:44 | `TimeoutError` — 60 s + 45 s |
+
+**The lease, sampled every 10 s for the whole 420-second run**, including deep inside each 45-second
+clamped attempt — the exact window in which the superseded frame-renewed lease died with the banner
+still painted:
+
+```
+[151.4s] lease={"paneOpen":true,"lastAckAgoMs":139,"valid":true} painting=true
+[211.4s] lease={"paneOpen":true,"lastAckAgoMs":9, "valid":true} painting=true
+[341.4s] lease={"paneOpen":true,"lastAckAgoMs":143,"valid":true} painting=true
+[411.4s] lease={"paneOpen":true,"lastAckAgoMs":105,"valid":true} painting=true
+```
+
+`lastAckAgoMs` never exceeded ~210 ms against a `UI_LEASE_MS` of 10 000. The heartbeat is on the
+pane's own 1 Hz timer, so it is indifferent to the proxy having nothing to say for 45 seconds. Under
+the superseded design this column would have read `valid:false` for most of the run and RUN 1 would
+have answered 400.
+
+The screen at the moment of exhaustion, same second as the 503:
+
+```
+ ██ NETWORK · Slow Probe refused
+ Cannot reach 192.0.2.1 for Slow Probe. This is a network problem on your machine — check your
+ internet connection, VPN, or DNS resolver (e.g. Tailscale MagicDNS) — not Slow Probe.
+ host 192.0.2.1 · ConnectionRefused
+ connecting to 192.0.2.1… · attempt 4 · 4m 30s in recovery
+ last: TimeoutError · 1 request held
+ click here or Ctrl-G Tab, then [r] try now · [q] give up
+```
+
+Two things to read carefully here. The banner says **`refused`** because macOS reports
+`ECONNREFUSED` for an unrouted remote address — ADD-2 again, and the *remote* wording
+(`Cannot reach … for …`, network/DNS advice) is correct precisely because `isLoopback` decides the
+sentence rather than the code. And it says `attempt 4` where the log says 5: the pane is one 1 Hz
+tick behind at the instant of capture, which is a render lag and not a state disagreement.
+
+### RUN 2 — the pane closed before exhaustion
+
+```
+RUN 2 status : 400
+RUN 2 x-claudish-recovery : (absent)
+RUN 2 body: {"type":"error","error":{"type":"connection_error","message":
+ "Cannot reach Slow Probe at https://192.0.2.1/v1/chat/completions. Check your network connection."}}
+```
+
+Byte-identical to the pre-recovery behaviour, including the `connection_error` type
+`probe-live.ts:334` keys off.
+
+Harness: `phase4/c18-live.ts`. Evidence: `steps.txt`, `C-18-run1-lease-samples.txt`,
+`C-18-run1-at-exhaustion.txt`, `C-18-recovery-log.txt`.
+
+---
+
+## 3. C-12 — the chain does not advance. THE ONE THAT COSTS MONEY.
+
+A real `Bun.serve` + Hono app, a real `FallbackHandler` over two real `ComposedHandler`s, two really
+refused sockets, the real ladder on the **real clock**, a real magmux with a real recovery pane
+holding a real lease, and a real HTTP POST — so the status and headers below are the ones a client
+actually receives, which also proves they survive `Bun.serve` rather than only existing on the
+`Response` object.
+
+**One substitution, and it is forced rather than convenient:** the candidate list is constructed
+directly instead of by `route()` on a bare model name. A real bare-name chain resolves to real
+vendor endpoints, and taking the network down to those needs privileges this environment does not
+have. Everything the criterion is about — the chain, the advance decision, the marker, the log line
+— is downstream of routing and runs unchanged.
+
+The deadline was shortened through `API_TIMEOUT_MS=60000`, exactly as a user would shorten it, so
+each run takes 15 s rather than 4 minutes. The ladder's gaps are unaffected; only how many fit is.
+
+### Variant A — the ordinary case
+
+```
+held for 15.0 s
+status: 503 x-should-retry: true x-claudish-recovery: 1
+body: {"error":{"type":"overloaded_error","message":"Cannot connect to Sakana Fugu at
+ http://127.0.0.1:64490/v1/chat/completions. Make sure the server is running.
+ claudish retried 3× over 15s without reaching it — still trying, watch the recovery pane."}}
+
+[Fallback] … trying next provider : 0 (MUST be 0)
+[Fallback] … falling through (cost) : 0 (MUST be 0)
+any mention of candidate B at all : 0
+the line that replaced it : 1
+ [Fallback] Sakana Fugu (subscription) is unreachable and claudish is still retrying it —
+ holding the chain here rather than switching providers mid-outage.
+VERDICT: PASS
+```
+
+### Variant B — the 503 message deliberately contains "quota"
+
+This is the structural guarantee, and a wording-based fix fails it. `isRetryableError`'s FIRST
+statement is `hasQuotaExhaustionWording(errorBody)` — status-agnostic by design, because the
+transport has already remapped terminal errors to 400 — and its phrase list contains the **bare
+substring `"quota"`**. A provider display name of `Quota Cloud` puts that word in the 503's message
+for free.
+
+```
+variant: quota — candidate 1 display name "Quota Cloud"
+status: 503 x-claudish-recovery: 1
+body: … "Cannot connect to Quota Cloud at http://127.0.0.1:64995/… claudish retried 3× over 15s …"
+
+message contains "quota" : true
+hasQuotaExhaustionWording(message) : true ← THE HAZARD, confirmed on this exact body
+[Fallback] … trying next provider : 0
+any mention of candidate B at all : 0
+VERDICT: PASS
+```
+
+The hazard is asserted first, on the real body, because a variant that did not actually trip the
+phrase list would prove nothing.
+
+Harness: `phase4/c12-live.ts` (`plain` | `quota`). Evidence: `steps.txt`, `C-12-log.txt`.
+
+---
+
+## 4. `exhaustedChainStatus`, decided
+
+A recovery 503 **cannot reach it**, by construction rather than by luck. `errors.push` happens in
+exactly three places in `FallbackHandler.handle()`; the verbatim marker return precedes all of them,
+and the only other push is the `catch`, which takes a thrown error rather than a `Response`.
+
+Widening `isTransient` to look for the marker would therefore be dead code that reads like a live
+guarantee, so nothing was changed there. What pins the property is a test driving the real shape —
+candidate 1 failing with a **retryable 401**, candidate 2 answering the marked 503 — and asserting
+the client receives 503 with the marker intact rather than the combined terminal 400 that
+`exhaustedChainStatus([401, 503])` would have produced. The function's doc comment now names the
+trap rather than the fix.
+
+## 5. Forgeability, re-verified at source
+
+Both round-3 facts still hold, checked 2026-09-11:
+
+- every non-ok exit from `ComposedHandler` is `c.json(...)`, which builds headers from nothing; the
+ file's single `c.header()` (`X-Dropped-Params`) sits **after** the `!response.ok` early returns;
+- the only path that copies upstream headers verbatim — `stream-head-sniffer.ts`'s
+ `replayResponse()` — is reached from step 7b, which runs after `!response.ok` has already
+ returned, i.e. on a 200. `FallbackHandler` returns an ok response as success before it ever looks
+ at the marker.
+
+**The first forgery mutation was itself a finding.** Mutating the `ensureAnthropicErrorFormat` exit
+to `return response` SURVIVED — because an upstream **401** does not take that branch. It takes the
+terminal-error 400 remap (`wrapAnthropicError(400, surfaced, "invalid_request_error",
+response.status)`). A mutation on a branch the path does not execute proves nothing about the path.
+Both exits now have a mutation and a test.
+
+## 6. Mutation proof — 17 mutations, 17 killed, 0 survivors
+
+Runner: `phase4/mutate.ts`. Each reverts ONE property, runs the tests that claim to pin it, and
+restores **by file copy — never git**: the stash stack and the index are shared with the main
+checkout and every sibling worktree.
+
+Killed: the lease gating the 503 · `gave_up` excluded from the 503 arm · the 503 itself ·
+`x-should-retry` · the marker header's wire NAME · the recovery clause in the message · **the marker
+early return in `isRetryableError`** · **the marker's POSITION relative to the quota phrase match** ·
+the verbatim return in `handle()` · the marker's exact VALUE · forgery via the remapped terminal exit ·
+forgery via the non-terminal exit · the watchdog being set at all · the watchdog following the UI
+switch · the rejoin inside the grace window · closing the handoff on the client's own success ·
+closing ONLY a handoff episode.
+
+The two in bold are the billing pair. The second is the one worth naming: the marker check *present
+but late* — moved below `hasQuotaExhaustionWording` — is killed only by the C-12 quota variant.
+Presence is not the property; **order** is.
+
+Two survivors on the first pass, both real test gaps, both closed:
+
+| survivor | why | what closed it |
+|---|---|---|
+| the marker header's wire NAME | both sides import the constant, so a rename moves them together and nothing notices | an assertion on the literal `"x-claudish-recovery"`, plus one reading the header off the response with no constant |
+| FORGERY | the mutated exit was the wrong branch (§5) | the mutation moved to the branch that executes, plus a second test for the other exit |
+
+---
+
+## 7. Three things found while building, none of them in the plan
+
+### 7.1 A handoff that ends in SUCCESS is not a rejoin, and left the banner lying
+
+The design says the client's re-POST rejoins the episode. That is what happens while the network is
+still down. When the network has **come back**, the re-POST's first attempt is `ComposedHandler`'s
+byte-identical primary fetch, which runs *before* the coordinator sees the request — so it simply
+succeeds. No `catch`, no `joinEpisode`, no rejoin. The episode then sat in `handoff` until its
+120-second grace expired, with the pane painting *"waiting for Claude Code to retry"* over a session
+that was already working.
+
+That is not cosmetic. The pane's legibility is the entire justification for the retryable status; a
+banner that lies about a live outage teaches the user to ignore the next real one.
+
+`noteTargetReachable()` closes a `handoff` episode when any request reaches the same target, guarded
+by `episodeCount() > 0` — a `Map.size` read that is 0 on every machine that has never had an outage,
+so the healthy path pays one integer comparison. Only `handoff` is closed: an episode in
+`attempting`/`waiting` still has parked waiters whose ladder it is, and one request's success says
+nothing about another's.
+
+Found by a failing assertion, not by review. Confirmed live in §1 — `13:54:08.534 [Recovery] Ollama
+answered on the client's own retry — closing episode … instead of waiting out its grace` — and the
+banner flipped to `██ NETWORK · Ollama recovered` in the same second.
+
+### 7.2 `gave_up` must be excluded from the 503 arm, and the lease cannot do it
+
+The exhaustion branch covers both `exhausted` and `gave_up`. Gating the flip on the lease alone
+answers a retryable 503 for `[q] give up` — and the pane that took the keystroke is by definition
+alive, so the lease IS valid at that instant. Claude Code would re-ask immediately and the give-up
+key would become a no-op with the banner still on screen. The guard is `result.kind === "exhausted"
+&& leased`; row 9 of the status table stays 400.
+
+### 7.3 The watchdog's side effect has no narrower lever
+
+`CLAUDE_CODE_RETRY_WATCHDOG=1` is global to the child, so it turns the stream-head sniffer's
+exhaustion 503 and `exhaustedChainStatus`'s all-transient 503 into ~300-attempt retries too. Neither
+opens an episode or a pane, so while the client loops on them the reason is legible **nowhere** —
+they degrade to "API error · Retrying" for far longer than before. More retry is the intended remedy
+for both, but the visibility argument that earns the recovery 503 its retryable status does not
+extend to them. The variable is fixed at spawn and cannot key on whether a banner exists, so this is
+the honest price of one process-wide switch rather than something a better gate could avoid.
+
+It is gated on `resolveRecoveryUi()` so surface and reach move together: `--no-recovery-ui` means no
+pane, no lease, an inline 400 at exhaustion, nothing handed back — and therefore no reason to extend
+the client's budget either.
+
+---
+
+## 8. What is NOT verified, and why
+
+- **C-13's attribution across a handoff.** `retry_attempts` / `recovery_episode_id` are §3.4 stats
+ fields and do not exist yet; the 503 arm records a stats event with `http_status: 503` but carries
+ no episode id. C-13 is a later phase's criterion.
+- **C-19** (a network drop during a *forced auth* retry). The auth sites were fixed in Phase 1 and
+ route through the same `recoverConnection`, but the 401-then-drop sequence was not driven live.
+- **The watchdog's ~300-attempt reach, end to end.** `retryWatchdogEnv()` is unit-tested and the
+ variable is verified to reach the child's env, but observing 300 client attempts would take a day
+ of wall clock. The measured facts it rests on are Phase 0's (`~11` attempts without it; the
+ `retry-after` honouring; the 38.4 s backoff cap).
+- **A bare auto-routing name through `route()`** for C-12 (§3).
diff --git a/packages/cli/src/claude-runner.ts b/packages/cli/src/claude-runner.ts
index e6b9a36e..e8b1966a 100644
--- a/packages/cli/src/claude-runner.ts
+++ b/packages/cli/src/claude-runner.ts
@@ -27,7 +27,7 @@ import { parseModelSpec } from "./providers/model-parser.js";
import { getProviderByName } from "./providers/provider-definitions.js";
import { route } from "./providers/routing-rules.js";
import { installRecoveryUi, shutdownRecoveryUi } from "./recovery/magmux-ui.js";
-import { resolveRecoveryUi } from "./recovery/settings.js";
+import { resolveRecoveryUi, retryWatchdogEnv } from "./recovery/settings.js";
import { setClaudeCodeRunning } from "./telemetry.js";
import { beginTerminalIsolation } from "./terminal-isolation.js";
import { getThemeMode } from "./theme/theme-mode.js";
@@ -1634,6 +1634,13 @@ export async function runClaudeWithProxy(
...advisorToolEnv.vars,
};
+ // The client half of "retry forever". Tier 1 holds one request for the
+ // derived deadline and then hands the retry back as a 503; how far recovery
+ // actually reaches is decided by Claude Code's willingness to re-ask, which
+ // this sets. The arithmetic, the accepted cost and the side effect on
+ // claudish's OTHER 503s are all stated at `retryWatchdogEnv()`.
+ Object.assign(env, retryWatchdogEnv());
+
// Provider display name, best-effort and FREE. Only an explicit `provider@model`
// spec names its provider without routing, and route() would touch credentials /
// 1Password — an unacceptable cost on the spawn path. A bare model name therefore
diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.ts
index fc6d8bfa..0d2a62ef 100644
--- a/packages/cli/src/handlers/composed-handler.ts
+++ b/packages/cli/src/handlers/composed-handler.ts
@@ -39,7 +39,7 @@ import { deepMergeParams } from "../model-params.js";
import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";
import { isTerminal429 } from "../providers/transport/openai.js";
import { recoveryClock } from "../recovery/clock.js";
-import { uiLeaseValid } from "../recovery/coordinator.js";
+import { episodeCount, noteTargetReachable, uiLeaseValid } from "../recovery/coordinator.js";
import { resolveRecoveryEnabled } from "../recovery/settings.js";
import {
type OpenAIImageBlock,
@@ -60,11 +60,16 @@ import {
} from "./shared/anthropic-error.js";
import { sseResponseToJson } from "./shared/collect-sse-message.js";
import type { ConnectionErrorKind } from "./shared/connection-error.js";
-import { buildConnectionErrorMessage, classifyConnectionError } from "./shared/connection-error.js";
+import {
+ buildConnectionErrorMessage,
+ buildRecoveryHoldMessage,
+ classifyConnectionError,
+} from "./shared/connection-error.js";
import { sniffDevinStreamHead } from "./shared/devin-stream-head-sniffer.js";
import { hasActionableLink, hasModelUnsupportedWording } from "./shared/model-unsupported.js";
import { filterIdentity } from "./shared/openai-compat.js";
import { hasPlanLimitWording, isQuotaExhaustionError } from "./shared/quota-exhaustion.js";
+import { recoveryHoldHeaders } from "./shared/recovery-marker.js";
import {
CONTEXT_OVERFLOW_PHRASE,
isContextOverflowError,
@@ -421,6 +426,85 @@ export class ComposedHandler implements ModelHandler {
return c.json(wrapAnthropicError(400, msg, "connection_error"), 400 as any);
}
+ /**
+ * The OTHER response shape: "claudish could not reach the host YET, and a
+ * surface is painting that fact, so the retry is handed back to the client."
+ *
+ * This is tier 2. The episode stays open for its grace window, Claude Code's
+ * own backoff elapses, it re-POSTs, and `joinEpisode` finds the open episode —
+ * same `episodeId`, same ladder position, `clientRetries` incremented. One
+ * banner, one attempt counter, one episode across both tiers.
+ *
+ * ── WHY `new Response` AND NEVER `c.header()` + `c.json()` ──────────────────
+ *
+ * `fallback-handler.ts` carries an explicit invariant: each candidate handler
+ * must NOT mutate the Hono `Context` (e.g. `c.header()`) before returning a
+ * non-ok Response. `c` is SHARED across every candidate in a chain, so a
+ * sticky `x-should-retry` set here would ride out on
+ * `formatCombinedError`'s `c.json(..., exhaustedChainStatus(errors))` — a
+ * terminal 400 telling Claude Code to retry it, which reintroduces exactly
+ * the buried-reason failure the whole 400-not-503 doctrine exists to prevent.
+ * A standalone `Response` cannot do that. The file's one `c.header()` call
+ * (`X-Dropped-Params`) is safe only because it sits after the `!response.ok`
+ * early returns; this arm has no such luck and must not acquire any.
+ *
+ * ── WHY 503 AND NOT 529 ─────────────────────────────────────────────────────
+ *
+ * `exhaustedChainStatus`'s transient set already contains 503 and does not
+ * contain 529, and `isRetryableError`'s remap-recovery check already scopes
+ * to 429/503. 529 would add a status this codebase has never carried and
+ * re-open every `status ===` under `handlers/`. 503 is the status the house
+ * already owns for "transient after our own retries, do not switch the user's
+ * provider".
+ *
+ * No `reportError` and no `logStderr("Error: …")` on this arm, deliberately.
+ * This is a handoff, not a verdict: with the retry watchdog enabled the
+ * client may re-ask hundreds of times during one outage, and a terminal-shaped
+ * error line per handoff would drown the pane's own account of the same
+ * fault. The 400 arm — lease gone, or the user pressed give-up — is where the
+ * failure is finally reported, once.
+ */
+ private respondRecoveryHold(
+ error: unknown,
+ conn: { kind: ConnectionErrorKind; code: string },
+ endpoint: string,
+ ctx: {
+ startTime: number;
+ attempts: number;
+ recoveryMs: number;
+ fallbackMeta?: { chain: string[]; attempts: number };
+ }
+ ): Response {
+ const reason = buildConnectionErrorMessage(conn.kind, this.provider.displayName, endpoint);
+ const msg = buildRecoveryHoldMessage(reason, ctx.attempts, ctx.recoveryMs);
+ try {
+ const { error_class, error_code } = classifyError(error, undefined);
+ recordStats({
+ model_id: this.targetModel,
+ provider_name: this.provider.name,
+ stream_format: this.provider.streamFormat,
+ latency_ms: Math.round(performance.now() - ctx.startTime),
+ success: false,
+ http_status: 503,
+ error_class,
+ error_code,
+ token_strategy: this.options.tokenStrategy ?? "standard",
+ adapter_name: this.getActiveAdapterName(),
+ middleware_names: this.middlewareManager.getActiveNames(this.bareModelName),
+ fallback_used: ctx.fallbackMeta !== undefined,
+ fallback_chain: ctx.fallbackMeta?.chain,
+ fallback_attempts: ctx.fallbackMeta?.attempts,
+ invocation_mode: this.options.invocationMode ?? "auto-route",
+ });
+ } catch {
+ // Stats must never crash claudish
+ }
+ return new Response(JSON.stringify(wrapAnthropicError(503, msg, "overloaded_error")), {
+ status: 503,
+ headers: recoveryHoldHeaders(),
+ });
+ }
+
/**
* The inbound client's abort signal, or one that never fires.
*
@@ -551,15 +635,41 @@ export class ComposedHandler implements ModelHandler {
default: {
// THE LEASE IS READ HERE AND NOWHERE ELSE, at the instant the status is
// chosen — never cached off the result object, which would be stale by
- // the time it was destructured.
+ // the time it was destructured. That staleness is exactly how the
+ // superseded `uiAttached` latch could report a banner that had been
+ // killed, closed or frozen minutes earlier, and then answer a retryable
+ // status with the reason visible nowhere: the one outcome strictly
+ // worse than the terminal 400 this feature exists to remove.
//
- // In THIS phase it only goes into the log. That is deliberate: shipping
- // the 503 flip before the banner exists recreates the buried-reason bug
- // this whole feature exists to kill. What the log buys now is the
- // ability to prove, one phase early, that the lease is still valid at
- // exhaustion after a 45-second attempt — the defect that made every
- // loopback-only test pass while the feature did not work.
+ // THE RULE THE TWO ARMS ENCODE. A retryable status is permissible
+ // exactly when claudish still has a surface on which the reason is
+ // legible. Absent such a surface, the reason must ride the status,
+ // which means 400.
const leased = uiLeaseValid(result.episodeId);
+
+ // `gave_up` is NOT eligible, and the guard is `result.kind` rather than
+ // the lease. `[q] give up` is the user saying stop; a lease may well
+ // still be valid at that instant (the pane that took the keystroke is
+ // by definition alive), so a lease-only test would answer a retryable
+ // 503 and Claude Code would immediately re-ask — turning the give-up
+ // key into a no-op with a banner still on screen.
+ if (result.kind === "exhausted" && leased) {
+ log(
+ `[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +
+ `in ${result.recoveryMs}ms (episode ${result.episodeId}, ui_lease=true) — ` +
+ "handing the retry back to the client (503, tier 2)"
+ );
+ return {
+ kind: "respond",
+ response: this.respondRecoveryHold(result.error, result.conn, result.endpoint, {
+ startTime: ctx.startTime,
+ attempts: result.attempts,
+ recoveryMs: result.recoveryMs,
+ fallbackMeta: ctx.fallbackMeta,
+ }),
+ };
+ }
+
log(
`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +
`in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}, ` +
@@ -1081,6 +1191,22 @@ export class ComposedHandler implements ModelHandler {
response = outcome.value;
}
+ // We reached the host. ANY status proves that — a 401 is a conversation, a
+ // refused socket is not — so this is the moment an episode parked in
+ // `handoff` for this target learns that its outage is over.
+ //
+ // It has to be said here rather than left to the grace timer because tier
+ // 2's rejoin is not what happens when the network comes back. Claude Code's
+ // re-POST enters the byte-identical primary fetch ABOVE, which succeeds
+ // outright: no catch, no `joinEpisode`, no rejoin — and the pane would go
+ // on painting "waiting for Claude Code to retry" over a working session for
+ // the rest of the 120-second grace.
+ //
+ // `episodeCount()` is a `Map.size` read and is 0 on every machine that has
+ // never had an outage, so the healthy path pays one integer comparison for
+ // a banner that stops lying.
+ if (episodeCount() > 0) noteTargetReachable(this.provider.name, endpoint);
+
// Check if the transport fell back to a different model (e.g., capacity exhaustion)
if (this.provider.getActiveModelName?.()) {
const activeModel = this.provider.getActiveModelName()!;
diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.ts
index 1a8596f6..69899eee 100644
--- a/packages/cli/src/handlers/fallback-handler.ts
+++ b/packages/cli/src/handlers/fallback-handler.ts
@@ -14,6 +14,7 @@ import { logStderr } from "../logger.js";
import { ComposedHandler } from "./composed-handler.js";
import { extractUpstreamStatus } from "./shared/anthropic-error.js";
import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";
+import { hasRecoveryMarker, isRecoveryHoldResponse } from "./shared/recovery-marker.js";
import type { ModelHandler } from "./types.js";
export interface FallbackCandidate {
@@ -75,11 +76,48 @@ export class FallbackHandler implements ModelHandler {
return response;
}
+ // ── THE RECOVERY MARKER, CHECKED BEFORE THE BODY IS EVEN READ ─────────
+ //
+ // A recovery-exhaustion 503 is OUR status, not an upstream one. It says
+ // "claudish held this request for its whole deadline, could not reach
+ // the host, and a surface is painting that fact" — so it must leave
+ // this handler UNTOUCHED, for two separate reasons:
+ //
+ // 1. It is not a reason to advance the chain. During a network outage
+ // the next candidate is unreachable for the same reason, so the
+ // chain would burn every candidate — and per the standing CLAUDE.md
+ // invariant, advancing off a SUBSCRIPTION_PROVIDERS candidate onto
+ // a metered one quotes the user a real per-token price for a fault
+ // that had nothing to do with any provider.
+ // 2. It must not be FOLDED into a combined error either. Below,
+ // `formatCombinedError` emits `exhaustedChainStatus(errors)`, which
+ // returns 503 only if EVERY accumulated error is transient — so one
+ // earlier auth/404 failure would turn our 503 into a terminal 400,
+ // Claude Code would never re-POST, and tier 2 would never begin.
+ //
+ // Returning it verbatim answers both at once. This is a HEADER and not a
+ // status, so the "read the upstream status via `extractUpstreamStatus`,
+ // never the wire status" doctrine is untouched. It cannot be forged by
+ // an upstream — see `recovery-marker.ts` for the two source facts that
+ // make that structural rather than hopeful.
+ //
+ // NOT a wording rule, and that is the whole point: `isRetryableError`'s
+ // first statement is a status-agnostic phrase match whose list contains
+ // the bare substring "quota", so a 503 whose message merely MENTIONED a
+ // quota would have advanced the chain and spent the user's money.
+ if (isRecoveryHoldResponse(response)) {
+ logStderr(
+ `[Fallback] ${name} is unreachable and claudish is still retrying it — ` +
+ "holding the chain here rather than switching providers mid-outage."
+ );
+ return response;
+ }
+
// Clone before reading body so we can still return the original if needed
const errorBody = await response.clone().text();
// Non-retryable error (rate limit, server error, bad format) — stop trying
- if (!isRetryableError(response.status, errorBody, name)) {
+ if (!isRetryableError(response.status, errorBody, name, response.headers)) {
if (errors.length > 0) {
// We had previous fallback attempts; show combined error
errors.push({ provider: name, status: response.status, message: errorBody });
@@ -169,7 +207,30 @@ export class FallbackHandler implements ModelHandler {
* warrant trying a different provider. True server errors (500 without
* billing context) do NOT — they'd likely fail on any provider.
*/
-export function isRetryableError(status: number, errorBody: string, provider?: string): boolean {
+export function isRetryableError(
+ status: number,
+ errorBody: string,
+ provider?: string,
+ headers?: Headers
+): boolean {
+ // ── CLAUDISH'S OWN RECOVERY 503 IS NEVER A REASON TO ADVANCE ───────────────
+ //
+ // FIRST, above the quota wording check, and the ORDER is the whole point.
+ // The next statement is `hasQuotaExhaustionWording(errorBody)`, which is
+ // deliberately status-agnostic and whose phrase list contains the bare
+ // substring "quota" — so a recovery 503 that merely mentioned one would be
+ // read as a spent subscription and walk the user onto metered billing in the
+ // middle of a network outage.
+ //
+ // `handle()` already returns a marked response verbatim before this function
+ // is consulted, so in the shipped call graph this branch is unreachable.
+ // It is kept anyway, and it is not belt-and-braces decoration: it makes the
+ // guarantee a property of the DECISION rather than of one call site, so a
+ // future refactor that moves, inlines or reorders the check in `handle()`
+ // cannot silently re-open the billing hole. Two independent mechanisms, one
+ // marker.
+ if (hasRecoveryMarker(headers)) return false;
+
// A spent subscription allowance is retryable AT THE CHAIN LEVEL: this
// provider cannot serve, but the next one can.
//
@@ -395,6 +456,25 @@ export function isRetryableError(status: number, errorBody: string, provider?: s
* for un-remapped statuses, so the rule becomes independent of whether a remap
* happened rather than gaining a new one. A remapped 401/403/402 stays terminal
* and still surfaces inline, which is the whole point of the 400 doctrine.
+ *
+ * ── A RECOVERY 503 CAN NEVER REACH THIS FUNCTION, BY CONSTRUCTION ────────────
+ *
+ * This was the sharper half of the marker-header finding, and it is worth
+ * stating where the trap is rather than where the fix is. If claudish's tier-1
+ * exhaustion 503 were pushed into `errors`, one EARLIER candidate failing with
+ * auth/404 would make `errors.every(isTransient)` false and this function would
+ * answer a terminal **400** — Claude Code would render the reason inline and
+ * never re-POST, so tier 2 would never begin and the feature would fail exactly
+ * in the case it was built for (an auto-routed chain during an outage).
+ *
+ * Nothing here is fixed to prevent that, because the upstream guard already
+ * makes it unreachable: `handle()` returns a marked response VERBATIM before
+ * any `errors.push`, and the only other push is the `catch`, which takes a
+ * thrown error rather than a Response. Widening `isTransient` to look for the
+ * marker would therefore be dead code that reads like a live guarantee. What
+ * pins it instead is a test that drives the real shape — candidate 1 failing
+ * with a retryable auth error, candidate 2 answering the marked 503 — and
+ * asserts the client receives 503 with the marker intact, never a combined 400.
*/
export function exhaustedChainStatus(
errors: Array<{ provider: string; status: number; message: string }>
diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.ts
new file mode 100644
index 00000000..7ccab88d
--- /dev/null
+++ b/packages/cli/src/handlers/recovery-status.test.ts
@@ -0,0 +1,686 @@
+/**
+ * Tier 2 — the status flip, and the one thing in this feature that costs money
+ * rather than time.
+ *
+ * At the end of a tier-1 hold `ComposedHandler` chooses between two answers:
+ *
+ * lease valid → **503 `overloaded_error`**, `x-should-retry: true`,
+ * `x-claudish-recovery: 1`. Claude Code re-POSTs, rejoins the
+ * same episode, and the turn survives.
+ * no lease → **400 `connection_error`**, byte-identical to the pre-recovery
+ * behaviour, including the type `probe-live.ts` keys off.
+ *
+ * The rule underneath: *a retryable status is permissible exactly while
+ * claudish still has a surface on which the reason is legible.* Answer 503 with
+ * no banner and the user gets "API error · Retrying" with the reason nowhere —
+ * strictly worse than the bug this feature removes.
+ *
+ * ── WHY THE CHAIN ASSERTIONS ARE THE EXPENSIVE ONES ─────────────────────────
+ *
+ * A 503 crossing `FallbackHandler` must not advance the chain: during an outage
+ * the next candidate is unreachable for the same reason, and per the standing
+ * CLAUDE.md invariant, advancing off a `SUBSCRIPTION_PROVIDERS` candidate onto a
+ * metered one quotes the user a real per-token price. `isRetryableError` has no
+ * 503 branch, but its FIRST statement is a status-agnostic phrase match whose
+ * list contains the bare substring `"quota"` — so a 503 whose MESSAGE mentioned
+ * one would have advanced. That is why the guarantee is a marker header rather
+ * than a wording rule, and why one test below deliberately builds a 503 message
+ * containing the word "quota" and proves the chain still holds. A wording-based
+ * fix fails that test; this one passes it.
+ *
+ * ── ON SETUP FIDELITY ───────────────────────────────────────────────────────
+ *
+ * Every assertion drives the real `ComposedHandler.handle()` through its real
+ * fetch catch, the real ladder and the real coordinator. Only three things are
+ * substituted, each at a seam production already has: the clock (injected, so a
+ * 270-second deadline costs no real time), `globalThis.fetch` (restored in
+ * `afterEach`), and the UI hooks (`registerRecoveryUi`, the same call
+ * `installRecoveryUi` makes). No `mock.module()` — Bun's registry bleeds those
+ * into sibling test files.
+ */
+
+import { afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test";
+import type { Context } from "hono";
+import { resetRecoveryClock, setRecoveryClock } from "../recovery/clock.js";
+import {
+ type RecoveryUiHooks,
+ closeAllEpisodes,
+ describeEpisode,
+ registerRecoveryUi,
+ renderableEpisodeFrame,
+} from "../recovery/coordinator.js";
+import { retryWatchdogEnv } from "../recovery/settings.js";
+import {
+ capturedLines,
+ startLogCapture,
+ stopLogCapture,
+} from "../recovery/test-helpers/capture-log.js";
+import { FakeClock, advanceUntilSettled } from "../recovery/test-helpers/fake-clock.js";
+import { ComposedHandler } from "./composed-handler.js";
+import { FallbackHandler, isRetryableError } from "./fallback-handler.js";
+import { buildRecoveryHoldMessage, formatHoldDuration } from "./shared/connection-error.js";
+import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";
+import { RECOVERY_MARKER_HEADER, isRecoveryHoldResponse } from "./shared/recovery-marker.js";
+import type { ModelHandler } from "./types.js";
+
+// ───────────────────────────────────────────────────────────────────────────
+// Fixture — PROVOKED, never hand-written
+// ───────────────────────────────────────────────────────────────────────────
+
+const realFetch = globalThis.fetch;
+let CONNECT_FAILURE: unknown;
+
+beforeAll(async () => {
+ // Bun's fetch uses neither Node's errno names nor `.cause`; it throws a flat
+ // TypeError carrying its own `code`. A hand-written `{ code: "ECONNREFUSED" }`
+ // would test the classifier's table rather than the runtime's behaviour.
+ try {
+ await realFetch("http://127.0.0.1:1/refused-on-purpose");
+ throw new Error("127.0.0.1:1 accepted a connection — this fixture needs a closed port");
+ } catch (e) {
+ CONNECT_FAILURE = e;
+ }
+ expect((CONNECT_FAILURE as { code?: string }).code).toBe("ConnectionRefused");
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Harness
+// ───────────────────────────────────────────────────────────────────────────
+
+let clock: FakeClock;
+
+/**
+ * A lease that answers for whichever episode is live, switchable mid-test.
+ *
+ * Registered through `registerRecoveryUi` — the exact call `installRecoveryUi`
+ * makes in production — so `composed-handler` reads the lease through the same
+ * `uiLeaseValid` indirection it uses with a real pane attached. What is NOT
+ * covered here is magmux-ui's own heartbeat accounting; that has its own file
+ * (`recovery/lease.test.ts`), and pretending otherwise here would test the
+ * double rather than the seam.
+ */
+let leaseAnswer = false;
+const leasedEpisodeIds: string[] = [];
+const hooks: RecoveryUiHooks = {
+ onEpisodeOpened: (episodeId) => {
+ leasedEpisodeIds.push(episodeId);
+ },
+ onEpisodeClosed: () => {},
+ leaseValid: () => leaseAnswer,
+};
+
+beforeEach(() => {
+ clock = new FakeClock(0);
+ setRecoveryClock(clock);
+ leaseAnswer = false;
+ leasedEpisodeIds.length = 0;
+ registerRecoveryUi(hooks);
+});
+
+afterEach(() => {
+ registerRecoveryUi(null);
+ closeAllEpisodes();
+ resetRecoveryClock();
+ globalThis.fetch = realFetch;
+});
+
+/** Every outbound fetch fails the way a dead socket does. */
+function breakTheNetwork(): void {
+ globalThis.fetch = (async () => {
+ throw CONNECT_FAILURE;
+ }) as unknown as typeof fetch;
+}
+
+interface ProbeOptions {
+ name?: string;
+ displayName?: string;
+ endpoint?: string;
+}
+
+function makeTransport(opts: ProbeOptions = {}) {
+ return {
+ // Unique per handler by default: the episode key is `${provider}|${host}`,
+ // and two tests sharing one key would share an episode.
+ name: opts.name ?? `probe-${Math.random().toString(36).slice(2)}`,
+ displayName: opts.displayName ?? "Held Probe",
+ streamFormat: "openai-sse",
+ getEndpoint: () => opts.endpoint ?? "http://127.0.0.1:1/v1/chat/completions",
+ getHeaders: async () => ({}),
+ } as unknown as Parameters<typeof makeHandler>[0];
+}
+
+function makeHandler(transport: unknown): ComposedHandler {
+ return new ComposedHandler(transport as never, "held-model", "held-model", 8080, {});
+}
+
+const PAYLOAD = {
+ model: "held-model",
+ max_tokens: 16,
+ messages: [{ role: "user", content: "hi" }],
+};
+
+/**
+ * A Context carrying a REAL `Request`, because two of the properties under test
+ * are keyed on one: the inbound deadline anchor (a `WeakMap` on `c.req.raw`) and
+ * the client-disconnect signal. A context without one silently gets a fresh
+ * anchor per read, which would make the rejoin test pass for the wrong reason.
+ */
+function makeContext(): { c: Context; jsonCalls: number } {
+ const state = { jsonCalls: 0 };
+ const raw = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });
+ const c = {
+ req: { raw, header: () => undefined },
+ header: () => {},
+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),
+ json: (body: unknown, status?: number) => {
+ state.jsonCalls++;
+ return new Response(JSON.stringify(body), { status: status ?? 200 });
+ },
+ } as unknown as Context;
+ return {
+ c,
+ get jsonCalls() {
+ return state.jsonCalls;
+ },
+ } as { c: Context; jsonCalls: number };
+}
+
+/** Run one held request to exhaustion on the fake clock. */
+async function runToExhaustion(handler: ComposedHandler, c: Context): Promise<Response> {
+ return advanceUntilSettled(clock, handler.handle(c, PAYLOAD), 400_000);
+}
+
+async function bodyOf(response: Response): Promise<{
+ type?: string;
+ error?: { type?: string; message?: string };
+}> {
+ return JSON.parse(await response.clone().text());
+}
+
+// ───────────────────────────────────────────────────────────────────────────
+// The flip itself — C-18's unit half
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-18 — exhaustion with the lease HELD answers 503, not 400", () => {
+ test("503 overloaded_error, x-should-retry and the recovery marker", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+
+ expect(response.status).toBe(503);
+ expect(response.headers.get("x-should-retry")).toBe("true");
+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBe("1");
+ expect(response.headers.get("content-type")).toContain("application/json");
+
+ const body = await bodyOf(response);
+ expect(body.type).toBe("error");
+ expect(body.error?.type).toBe("overloaded_error");
+
+ // The surface was ASKED for, through the same hook `installRecoveryUi`
+ // registers. A 503 granted by a lease nobody ever requested a pane for
+ // would be the forbidden state.
+ expect(leasedEpisodeIds).toHaveLength(1);
+ });
+
+ test("the message is the SAME sentence the 400 carries, plus a recovery clause", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const held = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+
+ leaseAnswer = false;
+ const inline = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+
+ const heldMsg = (await bodyOf(held)).error?.message ?? "";
+ const inlineMsg = (await bodyOf(inline)).error?.message ?? "";
+
+ // One fault, one sentence. If these two ever diverge the user is being told
+ // two different stories about the same failure.
+ expect(heldMsg.startsWith(inlineMsg)).toBe(true);
+ expect(heldMsg).toContain("claudish retried");
+ expect(heldMsg).toContain("still trying, watch the recovery pane");
+ // The clause names BOTH numbers the criterion asks for: how many attempts,
+ // and how long.
+ expect(heldMsg).toMatch(/claudish retried \d+× over (\d+m )?\d+s without reaching it/);
+ });
+
+ test("the 503 arm does not go through c.json — the Hono context is never mutated", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const ctx = makeContext();
+ const response = await runToExhaustion(makeHandler(makeTransport()), ctx.c);
+
+ // `c` is SHARED across every candidate in a chain, so a header set on it
+ // rides out on `formatCombinedError`'s terminal 400 — a 400 telling Claude
+ // Code to retry it, which is the buried-reason failure the whole doctrine
+ // exists to prevent. A standalone Response cannot do that.
+ expect(response.status).toBe(503);
+ expect(ctx.jsonCalls).toBe(0);
+ });
+});
+
+describe("no lease — today's 400, byte for byte", () => {
+ test("400 connection_error, no marker, no x-should-retry", async () => {
+ breakTheNetwork();
+ leaseAnswer = false;
+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+
+ expect(response.status).toBe(400);
+ const body = await bodyOf(response);
+ // `probe-live.ts` classifies on the TYPE, not the status. Changing it would
+ // silently reclassify every unreachable provider in the config TUI.
+ expect(body.error?.type).toBe("connection_error");
+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBeNull();
+ expect(response.headers.get("x-should-retry")).toBeNull();
+ expect(body.error?.message).not.toContain("claudish retried");
+ });
+
+ test("a lease that THROWS reads as no surface, never as yes", async () => {
+ breakTheNetwork();
+ registerRecoveryUi({
+ onEpisodeOpened: () => {},
+ onEpisodeClosed: () => {},
+ leaseValid: () => {
+ throw new Error("the pane process died mid-probe");
+ },
+ });
+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+
+ // The asymmetry is the point: false costs a legible inline error, true
+ // costs the turn.
+ expect(response.status).toBe(400);
+ });
+});
+
+describe("[q] give up is 400 even with a live lease", () => {
+ test("the give-up key is not turned into a no-op by the lease", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const handler = makeHandler(makeTransport());
+ const { c } = makeContext();
+ const inFlight = handler.handle(c, PAYLOAD);
+
+ // Let the episode open and park, then press give-up the way the pane does.
+ await advanceUntilSettled(clock, Promise.resolve(), 1);
+ await clock.advance(100);
+ const frame = renderableEpisodeFrame();
+ expect(frame).not.toBeNull();
+ const { giveUp } = await import("../recovery/coordinator.js");
+ giveUp((frame as { episodeId: string }).episodeId);
+
+ const response = await advanceUntilSettled(clock, inFlight, 400_000);
+
+ // The pane that took the keystroke is by definition alive, so the lease is
+ // valid at this instant. A lease-only test would answer 503, Claude Code
+ // would immediately re-ask, and "give up" would do nothing at all.
+ expect(leaseAnswer).toBe(true);
+ expect(response.status).toBe(400);
+ expect((await bodyOf(response)).error?.type).toBe("connection_error");
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-12 — the chain does not advance. THE ONE THAT COSTS MONEY.
+// ───────────────────────────────────────────────────────────────────────────
+
+/**
+ * Run `first` as candidate 1 of a two-candidate chain and report whether
+ * candidate 2 — the metered provider a subscription user gets silently moved
+ * onto — was ever reached.
+ */
+async function runInChain(
+ first: ModelHandler,
+ opts: { firstName?: string } = {}
+): Promise<{ status: number; advanced: boolean; response: Response }> {
+ let advanced = false;
+ const second: ModelHandler = {
+ async handle() {
+ advanced = true;
+ return new Response(JSON.stringify({ ok: true }), { status: 200 });
+ },
+ } as unknown as ModelHandler;
+
+ const chain = new FallbackHandler([
+ { name: opts.firstName ?? "Held Probe", handler: first },
+ { name: "Metered Fallback", handler: second },
+ ]);
+ const { c } = makeContext();
+ const response = await advanceUntilSettled(clock, chain.handle(c, PAYLOAD), 400_000);
+ return { status: response.status, advanced, response };
+}
+
+describe("C-12 — a recovery 503 does not advance the fallback chain", () => {
+ test("ABSENCE: no `trying next provider`, and candidate 2 is never reached", async () => {
+ startLogCapture();
+ try {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const outcome = await runInChain(makeHandler(makeTransport()));
+
+ expect(outcome.advanced).toBe(false);
+ expect(outcome.status).toBe(503);
+ expect(isRecoveryHoldResponse(outcome.response)).toBe(true);
+
+ const lines = await capturedLines();
+ // The literal string C-12 names. Asserted as an ABSENCE, because a test
+ // that only checked the status would pass against the bug.
+ expect(lines.some((l) => l.includes("trying next provider"))).toBe(false);
+ } finally {
+ stopLogCapture();
+ }
+ });
+
+ test("STRUCTURAL: a 503 whose message contains the word `quota` still holds the chain", async () => {
+ startLogCapture();
+ try {
+ breakTheNetwork();
+ leaseAnswer = true;
+ // A provider whose own display name carries the word. Nothing exotic —
+ // the sentence is built from the provider and endpoint the user
+ // configured, so a vendor called "Quota Cloud" produces this for free.
+ const handler = makeHandler(makeTransport({ displayName: "Quota Cloud" }));
+ const outcome = await runInChain(handler, { firstName: "Quota Cloud" });
+
+ // FIRST prove the hazard is real on this very body: the phrase match that
+ // runs first inside `isRetryableError` DOES fire on it.
+ const message = (await bodyOf(outcome.response)).error?.message ?? "";
+ expect(message.toLowerCase()).toContain("quota");
+ expect(hasQuotaExhaustionWording(message)).toBe(true);
+
+ // And the chain still does not move. A wording-based fix fails here.
+ expect(outcome.advanced).toBe(false);
+ expect(outcome.status).toBe(503);
+
+ const lines = await capturedLines();
+ expect(lines.some((l) => l.includes("trying next provider"))).toBe(false);
+ // Nor the quieter advance, which is the same event with a cost warning.
+ expect(lines.some((l) => l.includes("falling through to the next provider"))).toBe(false);
+ } finally {
+ stopLogCapture();
+ }
+ });
+
+ test("the 503 the SHIPPED wording produces does not trip the phrase list", async () => {
+ // The belt behind the marker's brace. The marker is what makes chain-safety
+ // structural; this keeps the default sentence from relying on it.
+ const msg = buildRecoveryHoldMessage(
+ "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses. " +
+ "Check your network connection.",
+ 7,
+ 225_000
+ );
+ expect(hasQuotaExhaustionWording(msg)).toBe(false);
+ expect(msg).toContain("claudish retried 7× over 3m 45s without reaching it");
+ });
+});
+
+describe("exhaustedChainStatus cannot fold a recovery 503 into a terminal 400", () => {
+ test("an EARLIER candidate's retryable auth failure does not demote our 503", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+
+ // Candidate 1 fails 401 — retryable, so the chain advances and pushes an
+ // error that is NOT transient. Candidate 2 is the held one. Without the
+ // verbatim return, `formatCombinedError` would run `exhaustedChainStatus`
+ // over [401, 503], find one non-transient, and answer a terminal 400 —
+ // Claude Code would never re-POST and tier 2 would never begin.
+ const authFailure: ModelHandler = {
+ async handle() {
+ return new Response(JSON.stringify({ error: { message: "Invalid API key" } }), {
+ status: 401,
+ });
+ },
+ } as unknown as ModelHandler;
+
+ const chain = new FallbackHandler([
+ { name: "Revoked Key", handler: authFailure },
+ { name: "Held Probe", handler: makeHandler(makeTransport()) },
+ ]);
+ const { c } = makeContext();
+ const response = await advanceUntilSettled(clock, chain.handle(c, PAYLOAD), 400_000);
+
+ expect(response.status).toBe(503);
+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBe("1");
+ expect(response.headers.get("x-should-retry")).toBe("true");
+ const body = await bodyOf(response);
+ // Not `all_providers_failed`, which is what a combined error would say.
+ expect(body.error?.type).toBe("overloaded_error");
+ });
+});
+
+describe("the marker is checked before the phrase list, inside isRetryableError too", () => {
+ const quotaBody = JSON.stringify({ error: { message: "You have exceeded your quota" } });
+ const marked = new Headers({ [RECOVERY_MARKER_HEADER]: "1" });
+
+ test("without the marker the phrase list wins — the hazard, demonstrated", () => {
+ expect(isRetryableError(503, quotaBody, "Held Probe")).toBe(true);
+ });
+
+ test("with the marker the answer is false, so the ordering is the guarantee", () => {
+ expect(isRetryableError(503, quotaBody, "Held Probe", marked)).toBe(false);
+ });
+
+ test("the marker also beats the statuses that are otherwise retryable", () => {
+ for (const status of [401, 402, 403, 404, 429]) {
+ expect(isRetryableError(status, "{}", "Held Probe", marked)).toBe(false);
+ expect(isRetryableError(status, "{}", "Held Probe")).toBe(true);
+ }
+ });
+
+ test("a wrong value is not the marker", () => {
+ expect(isRetryableError(401, "{}", "x", new Headers({ [RECOVERY_MARKER_HEADER]: "0" }))).toBe(
+ true
+ );
+ expect(isRetryableError(401, "{}", "x", new Headers())).toBe(true);
+ });
+});
+
+describe("the marker's wire name is a contract, not an implementation detail", () => {
+ test("the literal header name, pinned where a rename cannot move both sides together", () => {
+ // `recovery-marker.ts` mints it and `fallback-handler.ts` reads it, and both
+ // import the constant — so renaming the constant moves both sides at once
+ // and no behavioural test notices. The name is on the wire between a
+ // response and the handler that decides whether to spend the user's money;
+ // pin the string itself.
+ expect(RECOVERY_MARKER_HEADER).toBe("x-claudish-recovery");
+ });
+
+ test("the 503 carries that literal name, read off the response with no constant", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);
+ expect(response.headers.get("x-claudish-recovery")).toBe("1");
+ });
+});
+
+describe("the marker cannot be forged by an upstream provider", () => {
+ test("an upstream error response is rebuilt, so its headers never reach the chain", async () => {
+ // The general property, asserted on the response the client receives: an
+ // upstream's headers do not survive `ComposedHandler`'s non-ok exits. This
+ // is what makes "cannot be forged" structural rather than a claim about one
+ // status code.
+ globalThis.fetch = (async () =>
+ new Response(JSON.stringify({ error: { message: "upstream is busy" } }), {
+ status: 503,
+ headers: { [RECOVERY_MARKER_HEADER]: "1", "x-should-retry": "true" },
+ })) as unknown as typeof fetch;
+
+ const response = await makeHandler(makeTransport()).handle(makeContext().c, PAYLOAD);
+
+ expect(response.status).toBe(503);
+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBeNull();
+ expect(isRecoveryHoldResponse(response)).toBe(false);
+ });
+
+ test("an upstream 401 carrying the header still advances the chain", async () => {
+ // The discriminating shape. If ComposedHandler ever copied upstream headers
+ // onto a non-ok response, a hostile or merely eccentric provider could pin
+ // every user to itself by emitting this header — and the chain would stop
+ // where it should advance. It must NOT be observable.
+ globalThis.fetch = (async () =>
+ new Response(JSON.stringify({ error: { message: "Invalid API key" } }), {
+ status: 401,
+ headers: { [RECOVERY_MARKER_HEADER]: "1", "x-should-retry": "true" },
+ })) as unknown as typeof fetch;
+
+ const outcome = await runInChain(makeHandler(makeTransport()));
+
+ expect(outcome.advanced).toBe(true);
+ expect(isRecoveryHoldResponse(outcome.response)).toBe(false);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-6's tier-2 half — one episode across the handoff
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("the client's re-POST rejoins the SAME episode", () => {
+ test("same episode id, clientRetries++, tier 2, ladder NOT reset", async () => {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const transport = makeTransport({ name: "rejoin-probe" });
+
+ const first = await runToExhaustion(makeHandler(transport), makeContext().c);
+ expect(first.status).toBe(503);
+
+ const afterHandoff = renderableEpisodeFrame();
+ expect(afterHandoff?.state).toBe("handoff");
+ const episodeId = afterHandoff?.episodeId as string;
+ const attemptsAfterTier1 = afterHandoff?.attempts as number;
+ const ladderAfterTier1 = describeEpisode(episodeId)?.ladderIndex as number;
+ expect(attemptsAfterTier1).toBeGreaterThan(1);
+
+ // Claude Code's own backoff, measured at ≤38.4 s and honoured verbatim —
+ // comfortably inside EPISODE_GRACE_MS, which is why the episode is still
+ // there to rejoin.
+ await clock.advance(10_000);
+
+ // The re-POST. A NEW inbound request, so it gets a fresh per-request
+ // deadline — the deadline belongs to the socket and the socket is new.
+ const second = await runToExhaustion(makeHandler(transport), makeContext().c);
+ expect(second.status).toBe(503);
+
+ const rejoined = renderableEpisodeFrame();
+ // ONE episode id across both tiers: one banner, one attempt counter.
+ expect(rejoined?.episodeId).toBe(episodeId);
+ expect(rejoined?.clientRetries).toBe(1);
+ expect(rejoined?.tier).toBe(2);
+ // The counters continue rather than restarting, which is what makes the
+ // pane read as one continuous recovery.
+ expect(rejoined?.attempts).toBeGreaterThan(attemptsAfterTier1);
+ // And the ladder resumes where it left off instead of dropping back to 5 s
+ // — a reset would hammer a dead host harder the longer the outage lasted.
+ expect(describeEpisode(episodeId)?.ladderIndex).toBeGreaterThanOrEqual(ladderAfterTier1);
+ });
+
+ test("the turn completes on the client's re-POST, and the banner stops at once", async () => {
+ startLogCapture();
+ try {
+ await recoversAfterHandoff();
+ } finally {
+ stopLogCapture();
+ }
+ });
+
+ async function recoversAfterHandoff(): Promise<void> {
+ breakTheNetwork();
+ leaseAnswer = true;
+ const transport = makeTransport({ name: "recover-after-handoff" });
+
+ const first = await runToExhaustion(makeHandler(transport), makeContext().c);
+ expect(first.status).toBe(503);
+ const episodeId = renderableEpisodeFrame()?.episodeId as string;
+ expect(describeEpisode(episodeId)?.state).toBe("handoff");
+
+ await clock.advance(5_000);
+
+ // The network comes back AFTER the handoff. This is C-6's second run in
+ // miniature: the proof that the two tiers are one episode is that the turn
+ // completes, not that the banner looked continuous.
+ // A real `openai-sse` body, which is what this transport declares and what
+ // the handler's parser expects. A JSON completion would be parsed as an
+ // empty stream and the assertion would pass or fail for reasons that have
+ // nothing to do with recovery.
+ const sse =
+ `data: ${JSON.stringify({
+ id: "chatcmpl-1",
+ object: "chat.completion.chunk",
+ choices: [{ index: 0, delta: { role: "assistant", content: "back" } }],
+ })}\n\n` +
+ `data: ${JSON.stringify({
+ id: "chatcmpl-1",
+ object: "chat.completion.chunk",
+ choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
+ })}\n\n` +
+ "data: [DONE]\n\n";
+ globalThis.fetch = (async () =>
+ new Response(sse, {
+ status: 200,
+ headers: { "content-type": "text/event-stream" },
+ })) as unknown as typeof fetch;
+
+ const second = await runToExhaustion(makeHandler(transport), makeContext().c);
+
+ expect(second.status).toBe(200);
+ const text = await second.clone().text();
+ expect(text).toContain("back");
+
+ // ── THE PART THAT IS NOT OBVIOUS ────────────────────────────────────────
+ //
+ // The re-POST did NOT rejoin the episode. Its first attempt is the
+ // handler's byte-identical primary fetch, which runs before the coordinator
+ // sees the request — and with the network back it simply succeeded. Nothing
+ // in the ladder ever ran.
+ //
+ // So the episode has to be closed by the SUCCESS, not by a rejoin. Left to
+ // the grace timer it would sit in `handoff` for two more minutes with the
+ // pane painting "waiting for Claude Code to retry" over a working session.
+ expect(describeEpisode(episodeId)).toBeNull();
+ expect(renderableEpisodeFrame()).toBeNull();
+
+ const lines = await capturedLines();
+ expect(lines.some((l) => l.includes("answered on the client's own retry"))).toBe(true);
+ expect(lines.some((l) => l.includes(`episode ${episodeId} closed: recovered`))).toBe(true);
+ }
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The client's retry budget
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("CLAUDE_CODE_RETRY_WATCHDOG follows the recovery UI switch", () => {
+ const KEY = "CLAUDISH_RECOVERY_UI";
+ let saved: string | undefined;
+
+ beforeEach(() => {
+ saved = process.env[KEY];
+ });
+
+ afterEach(() => {
+ if (saved === undefined) delete process.env[KEY];
+ else process.env[KEY] = saved;
+ });
+
+ test("set when the UI is enabled — this is what makes the reach ~a day", () => {
+ delete process.env[KEY];
+ expect(retryWatchdogEnv()).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });
+ });
+
+ test("absent when the user turned the UI off", () => {
+ // No pane means no lease means an inline 400 at exhaustion means nothing to
+ // hand back — so extending the client's budget would buy only the side
+ // effect on claudish's other 503s.
+ process.env[KEY] = "0";
+ expect(retryWatchdogEnv()).toEqual({});
+ expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv()).toBe(false);
+ });
+});
+
+describe("formatHoldDuration", () => {
+ test("minutes appear only when there are minutes, and it never goes negative", () => {
+ expect(formatHoldDuration(0)).toBe("0s");
+ expect(formatHoldDuration(-5_000)).toBe("0s");
+ expect(formatHoldDuration(45_000)).toBe("45s");
+ expect(formatHoldDuration(225_000)).toBe("3m 45s");
+ expect(formatHoldDuration(120_000)).toBe("2m 0s");
+ });
+});
diff --git a/packages/cli/src/handlers/shared/connection-error.ts b/packages/cli/src/handlers/shared/connection-error.ts
index f20408db..b04f08f9 100644
--- a/packages/cli/src/handlers/shared/connection-error.ts
+++ b/packages/cli/src/handlers/shared/connection-error.ts
@@ -161,3 +161,54 @@ export function buildConnectionErrorMessage(
return `Cannot reach ${displayName} at ${endpoint}. Check your network connection.`;
}
}
+
+/**
+ * `3m 45s`, `45s`, `0s`. Never negative, never fractional.
+ *
+ * Deliberately a separate implementation from the pane's formatter of the same
+ * shape: the pane runs in ANOTHER PROCESS and shares no module with this one,
+ * and importing across that boundary is what the recovery wire protocol exists
+ * to avoid.
+ */
+export function formatHoldDuration(ms: number): string {
+ const total = Math.max(0, Math.round(ms / 1000));
+ const minutes = Math.floor(total / 60);
+ const seconds = total % 60;
+ return minutes > 0 ? `${minutes}m ${seconds}s` : `${seconds}s`;
+}
+
+/**
+ * The sentence an EXHAUSTED tier-1 hold carries on its retryable 503.
+ *
+ * It is `buildConnectionErrorMessage`'s own sentence plus a recovery clause,
+ * and it has to stay built that way: the inline 400 and the handed-back 503
+ * describe ONE fault, and two sentences that drift tell the user two different
+ * stories about it.
+ *
+ * ── WHAT THIS SENTENCE MUST NEVER SAY ───────────────────────────────────────
+ *
+ * `fallback-handler.ts`'s `hasQuotaExhaustionWording` is the FIRST statement in
+ * `isRetryableError`'s body and is status-agnostic on purpose. Its phrase list
+ * contains the bare substring `"quota"`, plus `"usage limit"`, `"plan limit"`,
+ * `"daily limit"`, `"billing cycle"`, `"credit balance"`, `"out of credits"`
+ * and `"exceeded your current"`. A 503 whose MESSAGE trips that list advances
+ * the fallback chain and moves a subscription user onto metered billing during
+ * an outage.
+ *
+ * The structural defence is the `x-claudish-recovery` marker
+ * (`recovery-marker.ts`), which takes the message out of that decision
+ * entirely. This wording rule is the belt behind that brace, and it is asserted
+ * directly on the composed sentence in `recovery-status.test.ts` — so a future
+ * edit that reaches for "retry quota" or "attempt limit" fails a test rather
+ * than a user's invoice.
+ */
+export function buildRecoveryHoldMessage(
+ reason: string,
+ attempts: number,
+ elapsedMs: number
+): string {
+ return (
+ `${reason} claudish retried ${attempts}× over ${formatHoldDuration(elapsedMs)} ` +
+ "without reaching it — still trying, watch the recovery pane."
+ );
+}
diff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.ts
new file mode 100644
index 00000000..648288bb
--- /dev/null
+++ b/packages/cli/src/handlers/shared/recovery-marker.ts
@@ -0,0 +1,93 @@
+/**
+ * The one header that makes recovery chain-safety STRUCTURAL rather than a
+ * wording rule.
+ *
+ * ## What it is for
+ *
+ * When a tier-1 hold exhausts its deadline with a surface still painting the
+ * reason, `ComposedHandler` answers **503 `overloaded_error`** instead of
+ * today's terminal 400, so Claude Code re-POSTs and the turn survives into
+ * tier 2. That status then has to travel past `FallbackHandler`, and the
+ * fallback chain is where a mistake here costs the user real money rather than
+ * time: advancing off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one
+ * quotes a per-token price for a fault that had nothing to do with the
+ * provider.
+ *
+ * ## Why a header and not a status, and not a wording rule
+ *
+ * `fallback-handler.ts`'s `isRetryableError` has no 503 branch, so on the
+ * status alone a 503 already stops the chain. That is NOT enough, twice over:
+ *
+ * 1. **The FIRST statement in its body is `hasQuotaExhaustionWording(errorBody)`,
+ * which is deliberately status-agnostic** — it exists because the transport
+ * has already remapped terminal errors to 400, so a status-gated test there
+ * would miss a spent plan. Its phrase list contains the **bare substring
+ * `"quota"`**, plus `"usage limit"`, `"plan limit"`, `"daily limit"`,
+ * `"billing cycle"`, `"credit balance"`, `"out of credits"` and
+ * `"exceeded your current"`. A 503 whose MESSAGE happens to trip that list
+ * advances the chain and spends the user's money. Checking the marker FIRST
+ * removes the message from the decision entirely.
+ * 2. **A 503 does not necessarily reach the client.** When a candidate's error
+ * is not retryable and earlier candidates already failed, `handle()` pushes
+ * it and calls `formatCombinedError`, whose status comes from
+ * `exhaustedChainStatus(errors)` — 503 only if EVERY accumulated error is
+ * transient. One earlier auth/404 failure turns our 503 into a terminal 400
+ * and tier 2 never begins. The marker makes the response return VERBATIM
+ * before it can be folded into a combined one.
+ *
+ * ## Why it cannot be forged by an upstream provider
+ *
+ * The marker is only ever minted on a `Response` this process constructs, and
+ * verified at source (2026-09-11):
+ *
+ * - every non-ok exit from `ComposedHandler` is `c.json(...)` — Hono builds
+ * those headers from nothing, so an upstream's headers are never copied onto
+ * a failure response. The single `c.header()` call in that file
+ * (`X-Dropped-Params`) sits AFTER the `!response.ok` early returns;
+ * - the one path that DOES copy upstream headers verbatim,
+ * `stream-head-sniffer.ts`'s `replayResponse()`, is reachable only from step
+ * 7b, which runs after `!response.ok` has already returned — i.e. on a 200.
+ * `FallbackHandler` returns an ok response as success before it ever looks at
+ * the marker, so a forged one on a 200 changes nothing.
+ *
+ * Keep both properties true. If a future edit returns an upstream `Response`
+ * object on a non-ok path, the marker must be stripped there.
+ */
+
+/** The header name. Referenced by name in no other file — import this. */
+export const RECOVERY_MARKER_HEADER = "x-claudish-recovery";
+
+/** The only value that counts. Anything else is not our response. */
+export const RECOVERY_MARKER_VALUE = "1";
+
+/**
+ * The headers a tier-1 exhaustion 503 carries.
+ *
+ * `x-should-retry` is read by the Anthropic TS SDK's `shouldRetry` before it
+ * looks at the status at all, so it survives a future change to Claude Code's
+ * status classification. It is set ONLY on this arm.
+ */
+export function recoveryHoldHeaders(): Record<string, string> {
+ return {
+ "content-type": "application/json",
+ "x-should-retry": "true",
+ [RECOVERY_MARKER_HEADER]: RECOVERY_MARKER_VALUE,
+ };
+}
+
+/** Does this header bag carry the marker? Tolerates `undefined` for callers that have none. */
+export function hasRecoveryMarker(headers: Headers | undefined | null): boolean {
+ try {
+ return headers?.get(RECOVERY_MARKER_HEADER) === RECOVERY_MARKER_VALUE;
+ } catch {
+ // A synthesised response with a non-Headers `headers` field. Not ours.
+ return false;
+ }
+}
+
+/** Is this response claudish's own recovery-exhaustion 503? */
+export function isRecoveryHoldResponse(
+ response: { headers?: Headers } | null | undefined
+): boolean {
+ return hasRecoveryMarker(response?.headers);
+}
diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.ts
index 3ccd21f7..f58c4b2a 100644
--- a/packages/cli/src/recovery/coordinator.test.ts
+++ b/packages/cli/src/recovery/coordinator.test.ts
@@ -32,6 +32,7 @@ import {
episodeCount,
giveUp,
joinEpisode,
+ noteTargetReachable,
tryNow,
uiLeaseValid,
} from "./coordinator.js";
@@ -431,6 +432,72 @@ describe("an episode rejoined within EPISODE_GRACE_MS continues", () => {
});
});
+// ───────────────────────────────────────────────────────────────────────────
+// 4b. The OTHER way a handoff ends: the client's own retry simply succeeds
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("noteTargetReachable — the handoff that ends without a rejoin", () => {
+ test("a reachable target closes the parked episode as recovered", async () => {
+ useFakeClock();
+ const s = seed({ providerName: "reach-1" });
+ const h = joinEpisode(s);
+ const id = h.episodeId;
+ h.recordAttemptResult("ConnectionRefused", false);
+ h.handoff();
+ h.leave();
+ expect(describeEpisode(id)?.state).toBe("handoff");
+
+ // Tier 2's rejoin is NOT what happens when the network comes back. The
+ // client's re-POST enters the handler's byte-identical primary fetch, which
+ // succeeds outright — no catch, no `joinEpisode`. Without this call the
+ // episode would sit here for the rest of its 120-second grace with the pane
+ // painting "waiting for Claude Code to retry" over a working session.
+ noteTargetReachable("reach-1", "http://127.0.0.1:9/v1/chat/completions");
+
+ expect(describeEpisode(id)).toBeNull();
+ expect(episodeCount()).toBe(0);
+ });
+
+ test("a DIFFERENT host does not close it — the key is provider|host", async () => {
+ useFakeClock();
+ const s = seed({ providerName: "reach-2" });
+ const h = joinEpisode(s);
+ h.recordAttemptResult("ConnectionRefused", false);
+ h.handoff();
+ h.leave();
+
+ noteTargetReachable("reach-2", "http://127.0.0.1:10/v1/chat/completions");
+ expect(describeEpisode(h.episodeId)?.state).toBe("handoff");
+ noteTargetReachable("other-provider", "http://127.0.0.1:9/v1/chat/completions");
+ expect(describeEpisode(h.episodeId)?.state).toBe("handoff");
+ });
+
+ test("a LIVE episode with parked waiters is never closed underneath them", async () => {
+ useFakeClock();
+ const s = seed({ providerName: "reach-3" });
+ const h = joinEpisode(s);
+ h.recordAttemptResult("ConnectionRefused", false);
+ const woken: WaitOutcome[] = [];
+ const parked = h.waitForNextAttempt(never(), 30_000).then((o) => {
+ woken.push(o);
+ });
+ await drain();
+
+ // One request's success says nothing about another's. Closing here would
+ // wake this waiter with `gave_up` and answer an inline error for an outage
+ // that its own ladder was still working on.
+ noteTargetReachable("reach-3", "http://127.0.0.1:9/v1/chat/completions");
+
+ expect(describeEpisode(h.episodeId)?.state).toBe("waiting");
+ expect(woken).toHaveLength(0);
+
+ await clock.advance(30_000);
+ await parked;
+ expect(woken).toEqual([{ kind: "attempt" }]);
+ h.leave();
+ });
+});
+
// ───────────────────────────────────────────────────────────────────────────
// Success, and the seams that are deliberately inert in this phase
// ───────────────────────────────────────────────────────────────────────────
diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.ts
index 1bec8921..4876c849 100644
--- a/packages/cli/src/recovery/coordinator.ts
+++ b/packages/cli/src/recovery/coordinator.ts
@@ -617,6 +617,49 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {
}
}
+/**
+ * A request reached this target WITHOUT the ladder's help. Close any episode
+ * parked in `handoff` for it.
+ *
+ * ── THE CASE THIS EXISTS FOR, WHICH THE DESIGN DID NOT ANTICIPATE ───────────
+ *
+ * Tier 2 hands the retry back as a 503 and leaves the episode `handoff` for its
+ * grace window. Claude Code's backoff elapses and it re-POSTs — and the
+ * re-POST's FIRST attempt is the handler's byte-identical primary fetch, which
+ * runs BEFORE the coordinator ever sees the request. When the network has come
+ * back in the meantime, that attempt simply SUCCEEDS: no `catch`, no
+ * `joinEpisode`, no rejoin. The turn completes, and the episode sits in
+ * `handoff` until its 120-second grace expires.
+ *
+ * That leaves the pane painting *"waiting for Claude Code to retry"* over a
+ * session that is already working — for up to two minutes, on the one surface
+ * whose legibility is what earns the 503 its retryable status in the first
+ * place. A banner that lies is worse than no banner, because the next real
+ * outage is read as more of the same.
+ *
+ * ── WHY THE HEALTHY PATH CAN AFFORD TO ASK ──────────────────────────────────
+ *
+ * The caller guards this with `episodeCount() > 0`, which is one `Map.size`
+ * read. On a machine that has never had an outage the map is empty and nothing
+ * below runs — no URL parse, no string build. That is the whole cost NFR-1
+ * pays for the banner telling the truth.
+ *
+ * ONLY `handoff` is closed, and the restriction is load-bearing: an episode in
+ * `attempting`/`waiting` still has parked waiters whose ladder this is, and
+ * closing it underneath them would strand or gave-up requests that another
+ * request's success says nothing about. A `handoff` episode has no waiters by
+ * construction — that is what `handoff` means.
+ */
+export function noteTargetReachable(providerName: string, endpoint: string): void {
+ const ep = episodes.get(`${providerName}|${hostOf(endpoint)}`);
+ if (!ep || ep.state !== "handoff") return;
+ log(
+ `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +
+ `closing episode ${ep.episodeId} instead of waiting out its grace`
+ );
+ closeEpisode(ep, "recovered");
+}
+
/**
* Collapse the wait for an episode and attempt now. The pane's `[r] try now`
* key routes here in a later phase; nothing calls it in this one, and it is
diff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.ts
index e894fc84..f0373466 100644
--- a/packages/cli/src/recovery/settings.ts
+++ b/packages/cli/src/recovery/settings.ts
@@ -186,3 +186,58 @@ export function logDeadlineIfShortened(deadlineMs: number): void {
export function resetDeadlineNotice(): void {
deadlineNoticeLogged = false;
}
+
+// ─── The client's retry budget ───────────────────────────────────────────────
+
+/**
+ * The environment that decides how far recovery REACHES, handed to the Claude
+ * Code child.
+ *
+ * Tier 1 holds one inbound request for the derived deadline (~270 s at the
+ * default) and then answers a retryable 503, which the client re-POSTs into the
+ * SAME episode. So the outer bound on recovery is not ours at all — it is
+ * Claude Code's willingness to keep asking, and `CLAUDE_CODE_RETRY_WATCHDOG`
+ * is the variable that sets it.
+ *
+ * ── WHAT IT COSTS, STATED PLAINLY ───────────────────────────────────────────
+ *
+ * | | unset | set (what ships) |
+ * |----------------------------|----------|------------------|
+ * | client retry budget | ~11 | **~300** |
+ * | unattended reach | ~66 min | **~a day** |
+ * | worst-case attempts / turn | ~46 | **~2,100** |
+ * | duplicate-charge exposure | accepted | **~30× that** |
+ *
+ * The duplicate-charge exposure is real and is the reason this is worth a
+ * paragraph rather than a line: a request the provider ACCEPTED and began
+ * billing can still fail on `ECONNRESET`/`EPIPE`, and every re-ask pays for it
+ * again — including on paths where inference had already started. That
+ * arithmetic was put to the user with these numbers in front of them and the
+ * larger magnitude was chosen deliberately. It is an informed acceptance, not
+ * an oversight; do not quietly re-litigate it here.
+ *
+ * ── THE SIDE EFFECT, WHICH IS NOT OPTIONAL ──────────────────────────────────
+ *
+ * The watchdog is GLOBAL to the child, so it amplifies every 503 claudish can
+ * emit and not only the recovery one: the stream-head sniffer's exhaustion and
+ * `exhaustedChainStatus`'s all-transient chain also become ~300-attempt
+ * retries. Neither of those opens an episode or a pane, so while the client
+ * loops on them the reason is legible NOWHERE — they degrade to "API error ·
+ * Retrying" for far longer than before. More retry is the intended remedy for
+ * both (an upstream overload is genuinely transient, which is why they are
+ * 503s at all), but the visibility argument that earns the recovery 503 its
+ * retryable status does not extend to them. That gap is the honest price of
+ * one process-wide switch, and there is no narrower lever: the variable is
+ * fixed at spawn and cannot key on whether a banner exists.
+ *
+ * ── WHY IT IS GATED ON THE UI SWITCH ────────────────────────────────────────
+ *
+ * `resolveRecoveryUi()` is the user's configuration — flag > env > project >
+ * global > true. Off means no pane, therefore no lease, therefore an inline
+ * 400 at exhaustion, therefore nothing handed back for the client to re-ask —
+ * so extending its budget would buy only the side effect above. Surface and
+ * reach move together or not at all.
+ */
+export function retryWatchdogEnv(): Record<string, string> {
+ return resolveRecoveryUi() ? { CLAUDE_CODE_RETRY_WATCHDOG: "1" } : {};
+}