Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
feat(recovery): hand the retry back to the client when the reason is legible
Jack Rudenko7a9f4c5b1a25
diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.mdindex bf7e01eb..7af0fa2a 100644--- a/ai-docs/architecture/adapters.md+++ b/ai-docs/architecture/adapters.md@@ -360,6 +360,45 @@ The general lesson: **any code that branches on an HTTP status downstream of theremap is suspect.** Grep for `status ===` under `handlers/` before assuming a newone is safe.+### The status is not always enough: `x-claudish-recovery` (network recovery, Tier 2)++There is now one response `FallbackHandler` must treat specially **before it looks+at a status or a body at all**: the 503 an exhausted tier-1 connection hold hands+back so Claude Code will re-POST (`handlers/shared/recovery-marker.ts`).++A 503 already stops the chain — `isRetryableError` has no 503 branch. That is not+enough, twice:++- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**,+ status-agnostic on purpose (it exists *because* of the remap above), and its list+ carries the bare substring `"quota"`. A 503 whose MESSAGE happened to contain that+ word advanced the chain — and per CLAUDE.md's standing invariant, advancing off a+ `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money, during an+ outage, for a fault no provider caused.+- **`exhaustedChainStatus` could demote it.** With an earlier candidate already+ failed, a non-retryable response goes to `formatCombinedError`, whose status is 503+ only if EVERY accumulated error is transient. One earlier auth/404 turns our 503+ into a terminal 400 and the client never re-POSTs.++So the marker is a **header**, checked in two independent places: `handle()` returns+a marked response VERBATIM before reading the body (which is what defeats the+combining), and `isRetryableError` returns `false` on it ABOVE the quota match+(which is what defeats the wording). The second is unreachable in the shipped call+graph and is kept anyway, so the guarantee belongs to the decision rather than to one+call site.++**Order is the property, not presence.** Moving the marker check below the quota+match — present, but late — is a live billing bug, and is mutation-covered as one.++It cannot be forged: every non-ok exit from `ComposedHandler` is `c.json(...)`, which+builds headers from nothing, and the only path that copies upstream headers verbatim+(`stream-head-sniffer.ts`'s `replayResponse()`) runs after `!response.ok` has already+returned, i.e. on a 200. **If a future edit ever returns an upstream `Response` object+on a non-ok path, the marker must be stripped there.**++Evidence, including the live chain runs and the mutation set:+`ai-docs/reports/network-recovery-phase4-status-flip-20260911.md`.+## The catalog's endpoint contract has two halves (v9.0.7)`gpt-6-astra` was in the catalog and could not run. The child exited 1 ondiff --git a/ai-docs/reports/network-recovery-phase4-status-flip-20260911.md b/ai-docs/reports/network-recovery-phase4-status-flip-20260911.mdnew file mode 100644index 00000000..c66155c6--- /dev/null+++ b/ai-docs/reports/network-recovery-phase4-status-flip-20260911.md@@ -0,0 +1,375 @@+# Network recovery, design-Phase 4 — the status flip, measured++> Promoted out of the gitignored session directory deliberately, as+> `network-recovery-phase0-measurements.md`, `-phase2-verification-` and `-phase3-pane-` were.+> CLAUDE.md records three write-ups already lost because they lived in `ai-docs/sessions/`, which+> does not survive a fresh clone or `git worktree remove`.+>+> Session `dev-feature-network-recovery-20260910-110514-0ad0a5a0` · measured 2026-09-11 on+> macOS 25.6.0, bun 1.4.0, magmux 0.11.0, Claude Code 2.1.268. The phase narrative and the per-file+> inventory stay in that session's `implementation-log-phase4.md`; what is here is the evidence.++Phase 4 is the only phase where a mistake costs the user **money** rather than time. At the end of a+tier-1 hold the handler now chooses between a retryable **503** and today's terminal **400**, and+that 503 has to cross `FallbackHandler` without advancing the chain — because advancing off a+`SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes a real per-token price for a fault that+had nothing to do with any provider.++Suite: **3331 → 3357 pass**, 18 skip, **2 → 2 fail** (the two pre-existing `displayWidth`+Unicode-oracle cases, red on clean main under bun 1.4.0 while CI pins 1.3.10). +26 tests, zero new+failures. `typecheck`, `lint` and `format` exit 0.++---++## 1. C-1 and C-6's second run — one real interactive session++A real `claudish -i --model [email protected]:0.5b`, a real Claude Code 2.1.268, a real magmux wrapper,+a real recovery pane, a real `ollama serve` with a real 0.5B model. The only harness is a PTY:+`pty-host.py` allocates one of a known size and execs claudish on the slave side, which is what a+terminal does. macOS `script(1)` cannot be used — it calls `tcgetattr` on its own stdin and dies+when that is a pipe, which it is for every process an agent session spawns.++Fault: ollama stopped, so `127.0.0.1:11434` refuses in ~1 ms. Server restarted **after the handoff**.++### C-1 — the turn does not end++`API Error: 400` never appeared on screen or in the byte stream, polled every 5 s for the whole run.+What appeared instead, at t+61 s, with the session still live:++```+ ▐▛███▛█ Claude Code v2.1.268+▝▜██████▀ [email protected]:0.5b with xhigh effort · API Usage Billing+❯ Reply with exactly one word: banana+✶ Wibbling… (57s)+────────────────────────────────────────────────────────────────────────────────+ * [email protected]:0.5b | $0.00 | 1s | 294M | ░░░░░░ 0% • $0.000 • N/A+────────────────────────────────────────────────────────────────────────────────+ ██ NETWORK · Ollama refused+ Cannot connect to Ollama at http://localhost:11434/v1/models. Make sure the server is running.+ host localhost:11434 · ConnectionRefused · local+ next attempt in 48s · attempt 8 · 57s in recovery+ last: ConnectionRefused · 2 requests held+ click here or Ctrl-G Tab, then [r] try now · [q] give up+```++`2 requests held` is not decoration: Claude Code issues the main-loop request and the small+title/summary request concurrently, both fail within milliseconds, and both are parked on **one**+episode with **one** timer. That is the coordinator's whole reason to exist, visible.++### C-6, second run — the two tiers are ONE episode++The decisive sequence, verbatim from the session's own debug log:++```+13:53:05.646 [Recovery] Ollama exhausted after 12 attempts in 165009ms+ (episode f4d31fb2-…, ui_lease=true) — handing the retry back to the client (503, tier 2)+13:53:05.646 [Recovery] episode f4d31fb2-… handed off after 165009ms, 12 attempts — holding 120s for a client retry+13:53:06.192 [Recovery] Ollama rejoined episode f4d31fb2-… (client retry 1, ladder 5, attempts 12)+13:54:06.285 [Recovery] episode f4d31fb2-… handed off after 225647ms, 16 attempts — holding 120s for a client retry+13:54:08.534 [Recovery] Ollama answered on the client's own retry — closing episode f4d31fb2-… instead of waiting out its grace+```++- Claude Code re-POSTed **546 ms** after the 503 — well inside the measured 38.4 s backoff cap and+ 3.1× inside `EPISODE_GRACE_MS`.+- The rejoin carried **the same episode id**, `ladder 5` (not reset to 0) and `attempts 12`. One+ banner, one attempt counter, one episode id across both tiers.+- There were **two** handoffs and three client requests. `distinct episode ids in the whole log: 1`.++And the turn finished, on a model that was not running when it started:++```+❯ Reply with exactly one word: banana+⏺ banana+✻ Cooked for 4m 1s · done 11:54 pm+ 28772 tokens+────────────────────────────────────────────────────────────────────────+ * [email protected]:0.5b | $0.15 | 4m2s | 375M | ░░░░░░ 14% • Ollama • 12% (28k/32k)+────────────────────────────────────────────────────────────────────────+ ██ NETWORK · Ollama recovered+ 16 attempts over 4m 02s+ this pane closes on its own+```++| | |+|---|---|+| handoff observed | t+171 s |+| `ollama serve` started | t+191 s — **after** the handoff |+| turn answered | t+246 s |+| upstream 200s after recovery | 2 |+| distinct episode ids | **1** |+| `API Error: 400` ever on screen | **false** |++**Recovery took ~43 s longer than the outage did**, and that is by design: `ladderIndex` is+episode-scoped and does not reset on re-entry, so the rung after the rejoin was 60 s. `[r] try now`+exists for exactly that impatience.++Harness: `phase4/c1-c6-live.ts`. Evidence: `steps.txt`, `timeline.txt`, `C-6-final.txt`,+`recovery-lines.txt`.++---++## 2. C-18 — Tier 2 engages for a SLOW connection failure++The round-2 CRITICAL's regression test, and the only criterion in the file that is not a ~1 ms+loopback refusal. Two runs against **F-SLOW** (`https://192.0.2.1`, TEST-NET-1), with a real magmux+and a real `claudish recovery-pane`.++**Fidelity check first**, because `192.0.2.1` is unrouted by convention and a captive portal can+answer it: measured **75 006 ms**, code `ConnectionRefused`. Phase 0 measured 75 004 ms and Phase 2+measured 75 005 ms on the same machine. ADD-2 confirmed a third time — *slow and fast connect+failures are indistinguishable by code on macOS*, only elapsed time separates them.++### RUN 1 — the pane alive and painting++```+RUN 1 status : 503+RUN 1 x-should-retry : true+RUN 1 x-claudish-recovery : 1+RUN 1 body: {"type":"error","error":{"type":"overloaded_error","message":+ "Cannot reach Slow Probe at https://192.0.2.1/v1/chat/completions. Check your network connection.+ claudish retried 5× over 4m 30s without reaching it — still trying, watch the recovery pane."}}+RUN 1 banner still painted at exhaustion : true+RUN 1 dead-proxy notice ever shown : false+```++The attempts, which is how you confirm the fault really was slow rather than refused — they end at+the **45 s clamp**, not at the ladder's nominal gaps:++| attempt | at | code |+|---|---|---|+| 1 | 13:40:14 | `ConnectionRefused` (unclamped, 75 s) |+| 2 | 13:41:04 | `TimeoutError` — 5 s gap + 45 s clamp |+| 3 | 13:41:59 | `TimeoutError` — 10 s + 45 s |+| 4 | 13:43:14 | `TimeoutError` — 30 s + 45 s |+| 5 | 13:44:44 | `TimeoutError` — 60 s + 45 s |++**The lease, sampled every 10 s for the whole 420-second run**, including deep inside each 45-second+clamped attempt — the exact window in which the superseded frame-renewed lease died with the banner+still painted:++```+[151.4s] lease={"paneOpen":true,"lastAckAgoMs":139,"valid":true} painting=true+[211.4s] lease={"paneOpen":true,"lastAckAgoMs":9, "valid":true} painting=true+[341.4s] lease={"paneOpen":true,"lastAckAgoMs":143,"valid":true} painting=true+[411.4s] lease={"paneOpen":true,"lastAckAgoMs":105,"valid":true} painting=true+```++`lastAckAgoMs` never exceeded ~210 ms against a `UI_LEASE_MS` of 10 000. The heartbeat is on the+pane's own 1 Hz timer, so it is indifferent to the proxy having nothing to say for 45 seconds. Under+the superseded design this column would have read `valid:false` for most of the run and RUN 1 would+have answered 400.++The screen at the moment of exhaustion, same second as the 503:++```+ ██ NETWORK · Slow Probe refused+ Cannot reach 192.0.2.1 for Slow Probe. This is a network problem on your machine — check your+ internet connection, VPN, or DNS resolver (e.g. Tailscale MagicDNS) — not Slow Probe.+ host 192.0.2.1 · ConnectionRefused+ connecting to 192.0.2.1… · attempt 4 · 4m 30s in recovery+ last: TimeoutError · 1 request held+ click here or Ctrl-G Tab, then [r] try now · [q] give up+```++Two things to read carefully here. The banner says **`refused`** because macOS reports+`ECONNREFUSED` for an unrouted remote address — ADD-2 again, and the *remote* wording+(`Cannot reach … for …`, network/DNS advice) is correct precisely because `isLoopback` decides the+sentence rather than the code. And it says `attempt 4` where the log says 5: the pane is one 1 Hz+tick behind at the instant of capture, which is a render lag and not a state disagreement.++### RUN 2 — the pane closed before exhaustion++```+RUN 2 status : 400+RUN 2 x-claudish-recovery : (absent)+RUN 2 body: {"type":"error","error":{"type":"connection_error","message":+ "Cannot reach Slow Probe at https://192.0.2.1/v1/chat/completions. Check your network connection."}}+```++Byte-identical to the pre-recovery behaviour, including the `connection_error` type+`probe-live.ts:334` keys off.++Harness: `phase4/c18-live.ts`. Evidence: `steps.txt`, `C-18-run1-lease-samples.txt`,+`C-18-run1-at-exhaustion.txt`, `C-18-recovery-log.txt`.++---++## 3. C-12 — the chain does not advance. THE ONE THAT COSTS MONEY.++A real `Bun.serve` + Hono app, a real `FallbackHandler` over two real `ComposedHandler`s, two really+refused sockets, the real ladder on the **real clock**, a real magmux with a real recovery pane+holding a real lease, and a real HTTP POST — so the status and headers below are the ones a client+actually receives, which also proves they survive `Bun.serve` rather than only existing on the+`Response` object.++**One substitution, and it is forced rather than convenient:** the candidate list is constructed+directly instead of by `route()` on a bare model name. A real bare-name chain resolves to real+vendor endpoints, and taking the network down to those needs privileges this environment does not+have. Everything the criterion is about — the chain, the advance decision, the marker, the log line+— is downstream of routing and runs unchanged.++The deadline was shortened through `API_TIMEOUT_MS=60000`, exactly as a user would shorten it, so+each run takes 15 s rather than 4 minutes. The ladder's gaps are unaffected; only how many fit is.++### Variant A — the ordinary case++```+held for 15.0 s+status: 503 x-should-retry: true x-claudish-recovery: 1+body: {"error":{"type":"overloaded_error","message":"Cannot connect to Sakana Fugu at+ http://127.0.0.1:64490/v1/chat/completions. Make sure the server is running.+ claudish retried 3× over 15s without reaching it — still trying, watch the recovery pane."}}++[Fallback] … trying next provider : 0 (MUST be 0)+[Fallback] … falling through (cost) : 0 (MUST be 0)+any mention of candidate B at all : 0+the line that replaced it : 1+ [Fallback] Sakana Fugu (subscription) is unreachable and claudish is still retrying it —+ holding the chain here rather than switching providers mid-outage.+VERDICT: PASS+```++### Variant B — the 503 message deliberately contains "quota"++This is the structural guarantee, and a wording-based fix fails it. `isRetryableError`'s FIRST+statement is `hasQuotaExhaustionWording(errorBody)` — status-agnostic by design, because the+transport has already remapped terminal errors to 400 — and its phrase list contains the **bare+substring `"quota"`**. A provider display name of `Quota Cloud` puts that word in the 503's message+for free.++```+variant: quota — candidate 1 display name "Quota Cloud"+status: 503 x-claudish-recovery: 1+body: … "Cannot connect to Quota Cloud at http://127.0.0.1:64995/… claudish retried 3× over 15s …"++message contains "quota" : true+hasQuotaExhaustionWording(message) : true ← THE HAZARD, confirmed on this exact body+[Fallback] … trying next provider : 0+any mention of candidate B at all : 0+VERDICT: PASS+```++The hazard is asserted first, on the real body, because a variant that did not actually trip the+phrase list would prove nothing.++Harness: `phase4/c12-live.ts` (`plain` | `quota`). Evidence: `steps.txt`, `C-12-log.txt`.++---++## 4. `exhaustedChainStatus`, decided++A recovery 503 **cannot reach it**, by construction rather than by luck. `errors.push` happens in+exactly three places in `FallbackHandler.handle()`; the verbatim marker return precedes all of them,+and the only other push is the `catch`, which takes a thrown error rather than a `Response`.++Widening `isTransient` to look for the marker would therefore be dead code that reads like a live+guarantee, so nothing was changed there. What pins the property is a test driving the real shape —+candidate 1 failing with a **retryable 401**, candidate 2 answering the marked 503 — and asserting+the client receives 503 with the marker intact rather than the combined terminal 400 that+`exhaustedChainStatus([401, 503])` would have produced. The function's doc comment now names the+trap rather than the fix.++## 5. Forgeability, re-verified at source++Both round-3 facts still hold, checked 2026-09-11:++- every non-ok exit from `ComposedHandler` is `c.json(...)`, which builds headers from nothing; the+ file's single `c.header()` (`X-Dropped-Params`) sits **after** the `!response.ok` early returns;+- the only path that copies upstream headers verbatim — `stream-head-sniffer.ts`'s+ `replayResponse()` — is reached from step 7b, which runs after `!response.ok` has already+ returned, i.e. on a 200. `FallbackHandler` returns an ok response as success before it ever looks+ at the marker.++**The first forgery mutation was itself a finding.** Mutating the `ensureAnthropicErrorFormat` exit+to `return response` SURVIVED — because an upstream **401** does not take that branch. It takes the+terminal-error 400 remap (`wrapAnthropicError(400, surfaced, "invalid_request_error",+response.status)`). A mutation on a branch the path does not execute proves nothing about the path.+Both exits now have a mutation and a test.++## 6. Mutation proof — 17 mutations, 17 killed, 0 survivors++Runner: `phase4/mutate.ts`. Each reverts ONE property, runs the tests that claim to pin it, and+restores **by file copy — never git**: the stash stack and the index are shared with the main+checkout and every sibling worktree.++Killed: the lease gating the 503 · `gave_up` excluded from the 503 arm · the 503 itself ·+`x-should-retry` · the marker header's wire NAME · the recovery clause in the message · **the marker+early return in `isRetryableError`** · **the marker's POSITION relative to the quota phrase match** ·+the verbatim return in `handle()` · the marker's exact VALUE · forgery via the remapped terminal exit ·+forgery via the non-terminal exit · the watchdog being set at all · the watchdog following the UI+switch · the rejoin inside the grace window · closing the handoff on the client's own success ·+closing ONLY a handoff episode.++The two in bold are the billing pair. The second is the one worth naming: the marker check *present+but late* — moved below `hasQuotaExhaustionWording` — is killed only by the C-12 quota variant.+Presence is not the property; **order** is.++Two survivors on the first pass, both real test gaps, both closed:++| survivor | why | what closed it |+|---|---|---|+| the marker header's wire NAME | both sides import the constant, so a rename moves them together and nothing notices | an assertion on the literal `"x-claudish-recovery"`, plus one reading the header off the response with no constant |+| FORGERY | the mutated exit was the wrong branch (§5) | the mutation moved to the branch that executes, plus a second test for the other exit |++---++## 7. Three things found while building, none of them in the plan++### 7.1 A handoff that ends in SUCCESS is not a rejoin, and left the banner lying++The design says the client's re-POST rejoins the episode. That is what happens while the network is+still down. When the network has **come back**, the re-POST's first attempt is `ComposedHandler`'s+byte-identical primary fetch, which runs *before* the coordinator sees the request — so it simply+succeeds. No `catch`, no `joinEpisode`, no rejoin. The episode then sat in `handoff` until its+120-second grace expired, with the pane painting *"waiting for Claude Code to retry"* over a session+that was already working.++That is not cosmetic. The pane's legibility is the entire justification for the retryable status; a+banner that lies about a live outage teaches the user to ignore the next real one.++`noteTargetReachable()` closes a `handoff` episode when any request reaches the same target, guarded+by `episodeCount() > 0` — a `Map.size` read that is 0 on every machine that has never had an outage,+so the healthy path pays one integer comparison. Only `handoff` is closed: an episode in+`attempting`/`waiting` still has parked waiters whose ladder it is, and one request's success says+nothing about another's.++Found by a failing assertion, not by review. Confirmed live in §1 — `13:54:08.534 [Recovery] Ollama+answered on the client's own retry — closing episode … instead of waiting out its grace` — and the+banner flipped to `██ NETWORK · Ollama recovered` in the same second.++### 7.2 `gave_up` must be excluded from the 503 arm, and the lease cannot do it++The exhaustion branch covers both `exhausted` and `gave_up`. Gating the flip on the lease alone+answers a retryable 503 for `[q] give up` — and the pane that took the keystroke is by definition+alive, so the lease IS valid at that instant. Claude Code would re-ask immediately and the give-up+key would become a no-op with the banner still on screen. The guard is `result.kind === "exhausted"+&& leased`; row 9 of the status table stays 400.++### 7.3 The watchdog's side effect has no narrower lever++`CLAUDE_CODE_RETRY_WATCHDOG=1` is global to the child, so it turns the stream-head sniffer's+exhaustion 503 and `exhaustedChainStatus`'s all-transient 503 into ~300-attempt retries too. Neither+opens an episode or a pane, so while the client loops on them the reason is legible **nowhere** —+they degrade to "API error · Retrying" for far longer than before. More retry is the intended remedy+for both, but the visibility argument that earns the recovery 503 its retryable status does not+extend to them. The variable is fixed at spawn and cannot key on whether a banner exists, so this is+the honest price of one process-wide switch rather than something a better gate could avoid.++It is gated on `resolveRecoveryUi()` so surface and reach move together: `--no-recovery-ui` means no+pane, no lease, an inline 400 at exhaustion, nothing handed back — and therefore no reason to extend+the client's budget either.++---++## 8. What is NOT verified, and why++- **C-13's attribution across a handoff.** `retry_attempts` / `recovery_episode_id` are §3.4 stats+ fields and do not exist yet; the 503 arm records a stats event with `http_status: 503` but carries+ no episode id. C-13 is a later phase's criterion.+- **C-19** (a network drop during a *forced auth* retry). The auth sites were fixed in Phase 1 and+ route through the same `recoverConnection`, but the 401-then-drop sequence was not driven live.+- **The watchdog's ~300-attempt reach, end to end.** `retryWatchdogEnv()` is unit-tested and the+ variable is verified to reach the child's env, but observing 300 client attempts would take a day+ of wall clock. The measured facts it rests on are Phase 0's (`~11` attempts without it; the+ `retry-after` honouring; the 38.4 s backoff cap).+- **A bare auto-routing name through `route()`** for C-12 (§3).diff --git a/packages/cli/src/claude-runner.ts b/packages/cli/src/claude-runner.tsindex e6b9a36e..e8b1966a 100644--- a/packages/cli/src/claude-runner.ts+++ b/packages/cli/src/claude-runner.ts@@ -27,7 +27,7 @@ import { parseModelSpec } from "./providers/model-parser.js";import { getProviderByName } from "./providers/provider-definitions.js";import { route } from "./providers/routing-rules.js";import { installRecoveryUi, shutdownRecoveryUi } from "./recovery/magmux-ui.js";-import { resolveRecoveryUi } from "./recovery/settings.js";+import { resolveRecoveryUi, retryWatchdogEnv } from "./recovery/settings.js";import { setClaudeCodeRunning } from "./telemetry.js";import { beginTerminalIsolation } from "./terminal-isolation.js";import { getThemeMode } from "./theme/theme-mode.js";@@ -1634,6 +1634,13 @@ export async function runClaudeWithProxy(...advisorToolEnv.vars,};+ // The client half of "retry forever". Tier 1 holds one request for the+ // derived deadline and then hands the retry back as a 503; how far recovery+ // actually reaches is decided by Claude Code's willingness to re-ask, which+ // this sets. The arithmetic, the accepted cost and the side effect on+ // claudish's OTHER 503s are all stated at `retryWatchdogEnv()`.+ Object.assign(env, retryWatchdogEnv());+// Provider display name, best-effort and FREE. Only an explicit `provider@model`// spec names its provider without routing, and route() would touch credentials /// 1Password — an unacceptable cost on the spawn path. A bare model name thereforediff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex fc6d8bfa..0d2a62ef 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -39,7 +39,7 @@ import { deepMergeParams } from "../model-params.js";import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";import { isTerminal429 } from "../providers/transport/openai.js";import { recoveryClock } from "../recovery/clock.js";-import { uiLeaseValid } from "../recovery/coordinator.js";+import { episodeCount, noteTargetReachable, uiLeaseValid } from "../recovery/coordinator.js";import { resolveRecoveryEnabled } from "../recovery/settings.js";import {type OpenAIImageBlock,@@ -60,11 +60,16 @@ import {} from "./shared/anthropic-error.js";import { sseResponseToJson } from "./shared/collect-sse-message.js";import type { ConnectionErrorKind } from "./shared/connection-error.js";-import { buildConnectionErrorMessage, classifyConnectionError } from "./shared/connection-error.js";+import {+ buildConnectionErrorMessage,+ buildRecoveryHoldMessage,+ classifyConnectionError,+} from "./shared/connection-error.js";import { sniffDevinStreamHead } from "./shared/devin-stream-head-sniffer.js";import { hasActionableLink, hasModelUnsupportedWording } from "./shared/model-unsupported.js";import { filterIdentity } from "./shared/openai-compat.js";import { hasPlanLimitWording, isQuotaExhaustionError } from "./shared/quota-exhaustion.js";+import { recoveryHoldHeaders } from "./shared/recovery-marker.js";import {CONTEXT_OVERFLOW_PHRASE,isContextOverflowError,@@ -421,6 +426,85 @@ export class ComposedHandler implements ModelHandler {return c.json(wrapAnthropicError(400, msg, "connection_error"), 400 as any);}+ /**+ * The OTHER response shape: "claudish could not reach the host YET, and a+ * surface is painting that fact, so the retry is handed back to the client."+ *+ * This is tier 2. The episode stays open for its grace window, Claude Code's+ * own backoff elapses, it re-POSTs, and `joinEpisode` finds the open episode —+ * same `episodeId`, same ladder position, `clientRetries` incremented. One+ * banner, one attempt counter, one episode across both tiers.+ *+ * ── WHY `new Response` AND NEVER `c.header()` + `c.json()` ──────────────────+ *+ * `fallback-handler.ts` carries an explicit invariant: each candidate handler+ * must NOT mutate the Hono `Context` (e.g. `c.header()`) before returning a+ * non-ok Response. `c` is SHARED across every candidate in a chain, so a+ * sticky `x-should-retry` set here would ride out on+ * `formatCombinedError`'s `c.json(..., exhaustedChainStatus(errors))` — a+ * terminal 400 telling Claude Code to retry it, which reintroduces exactly+ * the buried-reason failure the whole 400-not-503 doctrine exists to prevent.+ * A standalone `Response` cannot do that. The file's one `c.header()` call+ * (`X-Dropped-Params`) is safe only because it sits after the `!response.ok`+ * early returns; this arm has no such luck and must not acquire any.+ *+ * ── WHY 503 AND NOT 529 ─────────────────────────────────────────────────────+ *+ * `exhaustedChainStatus`'s transient set already contains 503 and does not+ * contain 529, and `isRetryableError`'s remap-recovery check already scopes+ * to 429/503. 529 would add a status this codebase has never carried and+ * re-open every `status ===` under `handlers/`. 503 is the status the house+ * already owns for "transient after our own retries, do not switch the user's+ * provider".+ *+ * No `reportError` and no `logStderr("Error: …")` on this arm, deliberately.+ * This is a handoff, not a verdict: with the retry watchdog enabled the+ * client may re-ask hundreds of times during one outage, and a terminal-shaped+ * error line per handoff would drown the pane's own account of the same+ * fault. The 400 arm — lease gone, or the user pressed give-up — is where the+ * failure is finally reported, once.+ */+ private respondRecoveryHold(+ error: unknown,+ conn: { kind: ConnectionErrorKind; code: string },+ endpoint: string,+ ctx: {+ startTime: number;+ attempts: number;+ recoveryMs: number;+ fallbackMeta?: { chain: string[]; attempts: number };+ }+ ): Response {+ const reason = buildConnectionErrorMessage(conn.kind, this.provider.displayName, endpoint);+ const msg = buildRecoveryHoldMessage(reason, ctx.attempts, ctx.recoveryMs);+ try {+ const { error_class, error_code } = classifyError(error, undefined);+ recordStats({+ model_id: this.targetModel,+ provider_name: this.provider.name,+ stream_format: this.provider.streamFormat,+ latency_ms: Math.round(performance.now() - ctx.startTime),+ success: false,+ http_status: 503,+ error_class,+ error_code,+ token_strategy: this.options.tokenStrategy ?? "standard",+ adapter_name: this.getActiveAdapterName(),+ middleware_names: this.middlewareManager.getActiveNames(this.bareModelName),+ fallback_used: ctx.fallbackMeta !== undefined,+ fallback_chain: ctx.fallbackMeta?.chain,+ fallback_attempts: ctx.fallbackMeta?.attempts,+ invocation_mode: this.options.invocationMode ?? "auto-route",+ });+ } catch {+ // Stats must never crash claudish+ }+ return new Response(JSON.stringify(wrapAnthropicError(503, msg, "overloaded_error")), {+ status: 503,+ headers: recoveryHoldHeaders(),+ });+ }+/*** The inbound client's abort signal, or one that never fires.*@@ -551,15 +635,41 @@ export class ComposedHandler implements ModelHandler {default: {// THE LEASE IS READ HERE AND NOWHERE ELSE, at the instant the status is// chosen — never cached off the result object, which would be stale by- // the time it was destructured.+ // the time it was destructured. That staleness is exactly how the+ // superseded `uiAttached` latch could report a banner that had been+ // killed, closed or frozen minutes earlier, and then answer a retryable+ // status with the reason visible nowhere: the one outcome strictly+ // worse than the terminal 400 this feature exists to remove.//- // In THIS phase it only goes into the log. That is deliberate: shipping- // the 503 flip before the banner exists recreates the buried-reason bug- // this whole feature exists to kill. What the log buys now is the- // ability to prove, one phase early, that the lease is still valid at- // exhaustion after a 45-second attempt — the defect that made every- // loopback-only test pass while the feature did not work.+ // THE RULE THE TWO ARMS ENCODE. A retryable status is permissible+ // exactly when claudish still has a surface on which the reason is+ // legible. Absent such a surface, the reason must ride the status,+ // which means 400.const leased = uiLeaseValid(result.episodeId);++ // `gave_up` is NOT eligible, and the guard is `result.kind` rather than+ // the lease. `[q] give up` is the user saying stop; a lease may well+ // still be valid at that instant (the pane that took the keystroke is+ // by definition alive), so a lease-only test would answer a retryable+ // 503 and Claude Code would immediately re-ask — turning the give-up+ // key into a no-op with a banner still on screen.+ if (result.kind === "exhausted" && leased) {+ log(+ `[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` ++ `in ${result.recoveryMs}ms (episode ${result.episodeId}, ui_lease=true) — ` ++ "handing the retry back to the client (503, tier 2)"+ );+ return {+ kind: "respond",+ response: this.respondRecoveryHold(result.error, result.conn, result.endpoint, {+ startTime: ctx.startTime,+ attempts: result.attempts,+ recoveryMs: result.recoveryMs,+ fallbackMeta: ctx.fallbackMeta,+ }),+ };+ }+log(`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +`in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}, ` +@@ -1081,6 +1191,22 @@ export class ComposedHandler implements ModelHandler {response = outcome.value;}+ // We reached the host. ANY status proves that — a 401 is a conversation, a+ // refused socket is not — so this is the moment an episode parked in+ // `handoff` for this target learns that its outage is over.+ //+ // It has to be said here rather than left to the grace timer because tier+ // 2's rejoin is not what happens when the network comes back. Claude Code's+ // re-POST enters the byte-identical primary fetch ABOVE, which succeeds+ // outright: no catch, no `joinEpisode`, no rejoin — and the pane would go+ // on painting "waiting for Claude Code to retry" over a working session for+ // the rest of the 120-second grace.+ //+ // `episodeCount()` is a `Map.size` read and is 0 on every machine that has+ // never had an outage, so the healthy path pays one integer comparison for+ // a banner that stops lying.+ if (episodeCount() > 0) noteTargetReachable(this.provider.name, endpoint);+// Check if the transport fell back to a different model (e.g., capacity exhaustion)if (this.provider.getActiveModelName?.()) {const activeModel = this.provider.getActiveModelName()!;diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.tsindex 1a8596f6..69899eee 100644--- a/packages/cli/src/handlers/fallback-handler.ts+++ b/packages/cli/src/handlers/fallback-handler.ts@@ -14,6 +14,7 @@ import { logStderr } from "../logger.js";import { ComposedHandler } from "./composed-handler.js";import { extractUpstreamStatus } from "./shared/anthropic-error.js";import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";+import { hasRecoveryMarker, isRecoveryHoldResponse } from "./shared/recovery-marker.js";import type { ModelHandler } from "./types.js";export interface FallbackCandidate {@@ -75,11 +76,48 @@ export class FallbackHandler implements ModelHandler {return response;}+ // ── THE RECOVERY MARKER, CHECKED BEFORE THE BODY IS EVEN READ ─────────+ //+ // A recovery-exhaustion 503 is OUR status, not an upstream one. It says+ // "claudish held this request for its whole deadline, could not reach+ // the host, and a surface is painting that fact" — so it must leave+ // this handler UNTOUCHED, for two separate reasons:+ //+ // 1. It is not a reason to advance the chain. During a network outage+ // the next candidate is unreachable for the same reason, so the+ // chain would burn every candidate — and per the standing CLAUDE.md+ // invariant, advancing off a SUBSCRIPTION_PROVIDERS candidate onto+ // a metered one quotes the user a real per-token price for a fault+ // that had nothing to do with any provider.+ // 2. It must not be FOLDED into a combined error either. Below,+ // `formatCombinedError` emits `exhaustedChainStatus(errors)`, which+ // returns 503 only if EVERY accumulated error is transient — so one+ // earlier auth/404 failure would turn our 503 into a terminal 400,+ // Claude Code would never re-POST, and tier 2 would never begin.+ //+ // Returning it verbatim answers both at once. This is a HEADER and not a+ // status, so the "read the upstream status via `extractUpstreamStatus`,+ // never the wire status" doctrine is untouched. It cannot be forged by+ // an upstream — see `recovery-marker.ts` for the two source facts that+ // make that structural rather than hopeful.+ //+ // NOT a wording rule, and that is the whole point: `isRetryableError`'s+ // first statement is a status-agnostic phrase match whose list contains+ // the bare substring "quota", so a 503 whose message merely MENTIONED a+ // quota would have advanced the chain and spent the user's money.+ if (isRecoveryHoldResponse(response)) {+ logStderr(+ `[Fallback] ${name} is unreachable and claudish is still retrying it — ` ++ "holding the chain here rather than switching providers mid-outage."+ );+ return response;+ }+// Clone before reading body so we can still return the original if neededconst errorBody = await response.clone().text();// Non-retryable error (rate limit, server error, bad format) — stop trying- if (!isRetryableError(response.status, errorBody, name)) {+ if (!isRetryableError(response.status, errorBody, name, response.headers)) {if (errors.length > 0) {// We had previous fallback attempts; show combined errorerrors.push({ provider: name, status: response.status, message: errorBody });@@ -169,7 +207,30 @@ export class FallbackHandler implements ModelHandler {* warrant trying a different provider. True server errors (500 without* billing context) do NOT — they'd likely fail on any provider.*/-export function isRetryableError(status: number, errorBody: string, provider?: string): boolean {+export function isRetryableError(+ status: number,+ errorBody: string,+ provider?: string,+ headers?: Headers+): boolean {+ // ── CLAUDISH'S OWN RECOVERY 503 IS NEVER A REASON TO ADVANCE ───────────────+ //+ // FIRST, above the quota wording check, and the ORDER is the whole point.+ // The next statement is `hasQuotaExhaustionWording(errorBody)`, which is+ // deliberately status-agnostic and whose phrase list contains the bare+ // substring "quota" — so a recovery 503 that merely mentioned one would be+ // read as a spent subscription and walk the user onto metered billing in the+ // middle of a network outage.+ //+ // `handle()` already returns a marked response verbatim before this function+ // is consulted, so in the shipped call graph this branch is unreachable.+ // It is kept anyway, and it is not belt-and-braces decoration: it makes the+ // guarantee a property of the DECISION rather than of one call site, so a+ // future refactor that moves, inlines or reorders the check in `handle()`+ // cannot silently re-open the billing hole. Two independent mechanisms, one+ // marker.+ if (hasRecoveryMarker(headers)) return false;+// A spent subscription allowance is retryable AT THE CHAIN LEVEL: this// provider cannot serve, but the next one can.//@@ -395,6 +456,25 @@ export function isRetryableError(status: number, errorBody: string, provider?: s* for un-remapped statuses, so the rule becomes independent of whether a remap* happened rather than gaining a new one. A remapped 401/403/402 stays terminal* and still surfaces inline, which is the whole point of the 400 doctrine.+ *+ * ── A RECOVERY 503 CAN NEVER REACH THIS FUNCTION, BY CONSTRUCTION ────────────+ *+ * This was the sharper half of the marker-header finding, and it is worth+ * stating where the trap is rather than where the fix is. If claudish's tier-1+ * exhaustion 503 were pushed into `errors`, one EARLIER candidate failing with+ * auth/404 would make `errors.every(isTransient)` false and this function would+ * answer a terminal **400** — Claude Code would render the reason inline and+ * never re-POST, so tier 2 would never begin and the feature would fail exactly+ * in the case it was built for (an auto-routed chain during an outage).+ *+ * Nothing here is fixed to prevent that, because the upstream guard already+ * makes it unreachable: `handle()` returns a marked response VERBATIM before+ * any `errors.push`, and the only other push is the `catch`, which takes a+ * thrown error rather than a Response. Widening `isTransient` to look for the+ * marker would therefore be dead code that reads like a live guarantee. What+ * pins it instead is a test that drives the real shape — candidate 1 failing+ * with a retryable auth error, candidate 2 answering the marked 503 — and+ * asserts the client receives 503 with the marker intact, never a combined 400.*/export function exhaustedChainStatus(errors: Array<{ provider: string; status: number; message: string }>diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.tsnew file mode 100644index 00000000..7ccab88d--- /dev/null+++ b/packages/cli/src/handlers/recovery-status.test.ts@@ -0,0 +1,686 @@+/**+ * Tier 2 — the status flip, and the one thing in this feature that costs money+ * rather than time.+ *+ * At the end of a tier-1 hold `ComposedHandler` chooses between two answers:+ *+ * lease valid → **503 `overloaded_error`**, `x-should-retry: true`,+ * `x-claudish-recovery: 1`. Claude Code re-POSTs, rejoins the+ * same episode, and the turn survives.+ * no lease → **400 `connection_error`**, byte-identical to the pre-recovery+ * behaviour, including the type `probe-live.ts` keys off.+ *+ * The rule underneath: *a retryable status is permissible exactly while+ * claudish still has a surface on which the reason is legible.* Answer 503 with+ * no banner and the user gets "API error · Retrying" with the reason nowhere —+ * strictly worse than the bug this feature removes.+ *+ * ── WHY THE CHAIN ASSERTIONS ARE THE EXPENSIVE ONES ─────────────────────────+ *+ * A 503 crossing `FallbackHandler` must not advance the chain: during an outage+ * the next candidate is unreachable for the same reason, and per the standing+ * CLAUDE.md invariant, advancing off a `SUBSCRIPTION_PROVIDERS` candidate onto a+ * metered one quotes the user a real per-token price. `isRetryableError` has no+ * 503 branch, but its FIRST statement is a status-agnostic phrase match whose+ * list contains the bare substring `"quota"` — so a 503 whose MESSAGE mentioned+ * one would have advanced. That is why the guarantee is a marker header rather+ * than a wording rule, and why one test below deliberately builds a 503 message+ * containing the word "quota" and proves the chain still holds. A wording-based+ * fix fails that test; this one passes it.+ *+ * ── ON SETUP FIDELITY ───────────────────────────────────────────────────────+ *+ * Every assertion drives the real `ComposedHandler.handle()` through its real+ * fetch catch, the real ladder and the real coordinator. Only three things are+ * substituted, each at a seam production already has: the clock (injected, so a+ * 270-second deadline costs no real time), `globalThis.fetch` (restored in+ * `afterEach`), and the UI hooks (`registerRecoveryUi`, the same call+ * `installRecoveryUi` makes). No `mock.module()` — Bun's registry bleeds those+ * into sibling test files.+ */++import { afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test";+import type { Context } from "hono";+import { resetRecoveryClock, setRecoveryClock } from "../recovery/clock.js";+import {+ type RecoveryUiHooks,+ closeAllEpisodes,+ describeEpisode,+ registerRecoveryUi,+ renderableEpisodeFrame,+} from "../recovery/coordinator.js";+import { retryWatchdogEnv } from "../recovery/settings.js";+import {+ capturedLines,+ startLogCapture,+ stopLogCapture,+} from "../recovery/test-helpers/capture-log.js";+import { FakeClock, advanceUntilSettled } from "../recovery/test-helpers/fake-clock.js";+import { ComposedHandler } from "./composed-handler.js";+import { FallbackHandler, isRetryableError } from "./fallback-handler.js";+import { buildRecoveryHoldMessage, formatHoldDuration } from "./shared/connection-error.js";+import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";+import { RECOVERY_MARKER_HEADER, isRecoveryHoldResponse } from "./shared/recovery-marker.js";+import type { ModelHandler } from "./types.js";++// ───────────────────────────────────────────────────────────────────────────+// Fixture — PROVOKED, never hand-written+// ───────────────────────────────────────────────────────────────────────────++const realFetch = globalThis.fetch;+let CONNECT_FAILURE: unknown;++beforeAll(async () => {+ // Bun's fetch uses neither Node's errno names nor `.cause`; it throws a flat+ // TypeError carrying its own `code`. A hand-written `{ code: "ECONNREFUSED" }`+ // would test the classifier's table rather than the runtime's behaviour.+ try {+ await realFetch("http://127.0.0.1:1/refused-on-purpose");+ throw new Error("127.0.0.1:1 accepted a connection — this fixture needs a closed port");+ } catch (e) {+ CONNECT_FAILURE = e;+ }+ expect((CONNECT_FAILURE as { code?: string }).code).toBe("ConnectionRefused");+});++// ───────────────────────────────────────────────────────────────────────────+// Harness+// ───────────────────────────────────────────────────────────────────────────++let clock: FakeClock;++/**+ * A lease that answers for whichever episode is live, switchable mid-test.+ *+ * Registered through `registerRecoveryUi` — the exact call `installRecoveryUi`+ * makes in production — so `composed-handler` reads the lease through the same+ * `uiLeaseValid` indirection it uses with a real pane attached. What is NOT+ * covered here is magmux-ui's own heartbeat accounting; that has its own file+ * (`recovery/lease.test.ts`), and pretending otherwise here would test the+ * double rather than the seam.+ */+let leaseAnswer = false;+const leasedEpisodeIds: string[] = [];+const hooks: RecoveryUiHooks = {+ onEpisodeOpened: (episodeId) => {+ leasedEpisodeIds.push(episodeId);+ },+ onEpisodeClosed: () => {},+ leaseValid: () => leaseAnswer,+};++beforeEach(() => {+ clock = new FakeClock(0);+ setRecoveryClock(clock);+ leaseAnswer = false;+ leasedEpisodeIds.length = 0;+ registerRecoveryUi(hooks);+});++afterEach(() => {+ registerRecoveryUi(null);+ closeAllEpisodes();+ resetRecoveryClock();+ globalThis.fetch = realFetch;+});++/** Every outbound fetch fails the way a dead socket does. */+function breakTheNetwork(): void {+ globalThis.fetch = (async () => {+ throw CONNECT_FAILURE;+ }) as unknown as typeof fetch;+}++interface ProbeOptions {+ name?: string;+ displayName?: string;+ endpoint?: string;+}++function makeTransport(opts: ProbeOptions = {}) {+ return {+ // Unique per handler by default: the episode key is `${provider}|${host}`,+ // and two tests sharing one key would share an episode.+ name: opts.name ?? `probe-${Math.random().toString(36).slice(2)}`,+ displayName: opts.displayName ?? "Held Probe",+ streamFormat: "openai-sse",+ getEndpoint: () => opts.endpoint ?? "http://127.0.0.1:1/v1/chat/completions",+ getHeaders: async () => ({}),+ } as unknown as Parameters<typeof makeHandler>[0];+}++function makeHandler(transport: unknown): ComposedHandler {+ return new ComposedHandler(transport as never, "held-model", "held-model", 8080, {});+}++const PAYLOAD = {+ model: "held-model",+ max_tokens: 16,+ messages: [{ role: "user", content: "hi" }],+};++/**+ * A Context carrying a REAL `Request`, because two of the properties under test+ * are keyed on one: the inbound deadline anchor (a `WeakMap` on `c.req.raw`) and+ * the client-disconnect signal. A context without one silently gets a fresh+ * anchor per read, which would make the rejoin test pass for the wrong reason.+ */+function makeContext(): { c: Context; jsonCalls: number } {+ const state = { jsonCalls: 0 };+ const raw = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });+ const c = {+ req: { raw, header: () => undefined },+ header: () => {},+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),+ json: (body: unknown, status?: number) => {+ state.jsonCalls++;+ return new Response(JSON.stringify(body), { status: status ?? 200 });+ },+ } as unknown as Context;+ return {+ c,+ get jsonCalls() {+ return state.jsonCalls;+ },+ } as { c: Context; jsonCalls: number };+}++/** Run one held request to exhaustion on the fake clock. */+async function runToExhaustion(handler: ComposedHandler, c: Context): Promise<Response> {+ return advanceUntilSettled(clock, handler.handle(c, PAYLOAD), 400_000);+}++async function bodyOf(response: Response): Promise<{+ type?: string;+ error?: { type?: string; message?: string };+}> {+ return JSON.parse(await response.clone().text());+}++// ───────────────────────────────────────────────────────────────────────────+// The flip itself — C-18's unit half+// ───────────────────────────────────────────────────────────────────────────++describe("C-18 — exhaustion with the lease HELD answers 503, not 400", () => {+ test("503 overloaded_error, x-should-retry and the recovery marker", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);++ expect(response.status).toBe(503);+ expect(response.headers.get("x-should-retry")).toBe("true");+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBe("1");+ expect(response.headers.get("content-type")).toContain("application/json");++ const body = await bodyOf(response);+ expect(body.type).toBe("error");+ expect(body.error?.type).toBe("overloaded_error");++ // The surface was ASKED for, through the same hook `installRecoveryUi`+ // registers. A 503 granted by a lease nobody ever requested a pane for+ // would be the forbidden state.+ expect(leasedEpisodeIds).toHaveLength(1);+ });++ test("the message is the SAME sentence the 400 carries, plus a recovery clause", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const held = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);++ leaseAnswer = false;+ const inline = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);++ const heldMsg = (await bodyOf(held)).error?.message ?? "";+ const inlineMsg = (await bodyOf(inline)).error?.message ?? "";++ // One fault, one sentence. If these two ever diverge the user is being told+ // two different stories about the same failure.+ expect(heldMsg.startsWith(inlineMsg)).toBe(true);+ expect(heldMsg).toContain("claudish retried");+ expect(heldMsg).toContain("still trying, watch the recovery pane");+ // The clause names BOTH numbers the criterion asks for: how many attempts,+ // and how long.+ expect(heldMsg).toMatch(/claudish retried \d+× over (\d+m )?\d+s without reaching it/);+ });++ test("the 503 arm does not go through c.json — the Hono context is never mutated", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const ctx = makeContext();+ const response = await runToExhaustion(makeHandler(makeTransport()), ctx.c);++ // `c` is SHARED across every candidate in a chain, so a header set on it+ // rides out on `formatCombinedError`'s terminal 400 — a 400 telling Claude+ // Code to retry it, which is the buried-reason failure the whole doctrine+ // exists to prevent. A standalone Response cannot do that.+ expect(response.status).toBe(503);+ expect(ctx.jsonCalls).toBe(0);+ });+});++describe("no lease — today's 400, byte for byte", () => {+ test("400 connection_error, no marker, no x-should-retry", async () => {+ breakTheNetwork();+ leaseAnswer = false;+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);++ expect(response.status).toBe(400);+ const body = await bodyOf(response);+ // `probe-live.ts` classifies on the TYPE, not the status. Changing it would+ // silently reclassify every unreachable provider in the config TUI.+ expect(body.error?.type).toBe("connection_error");+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBeNull();+ expect(response.headers.get("x-should-retry")).toBeNull();+ expect(body.error?.message).not.toContain("claudish retried");+ });++ test("a lease that THROWS reads as no surface, never as yes", async () => {+ breakTheNetwork();+ registerRecoveryUi({+ onEpisodeOpened: () => {},+ onEpisodeClosed: () => {},+ leaseValid: () => {+ throw new Error("the pane process died mid-probe");+ },+ });+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);++ // The asymmetry is the point: false costs a legible inline error, true+ // costs the turn.+ expect(response.status).toBe(400);+ });+});++describe("[q] give up is 400 even with a live lease", () => {+ test("the give-up key is not turned into a no-op by the lease", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const handler = makeHandler(makeTransport());+ const { c } = makeContext();+ const inFlight = handler.handle(c, PAYLOAD);++ // Let the episode open and park, then press give-up the way the pane does.+ await advanceUntilSettled(clock, Promise.resolve(), 1);+ await clock.advance(100);+ const frame = renderableEpisodeFrame();+ expect(frame).not.toBeNull();+ const { giveUp } = await import("../recovery/coordinator.js");+ giveUp((frame as { episodeId: string }).episodeId);++ const response = await advanceUntilSettled(clock, inFlight, 400_000);++ // The pane that took the keystroke is by definition alive, so the lease is+ // valid at this instant. A lease-only test would answer 503, Claude Code+ // would immediately re-ask, and "give up" would do nothing at all.+ expect(leaseAnswer).toBe(true);+ expect(response.status).toBe(400);+ expect((await bodyOf(response)).error?.type).toBe("connection_error");+ });+});++// ───────────────────────────────────────────────────────────────────────────+// C-12 — the chain does not advance. THE ONE THAT COSTS MONEY.+// ───────────────────────────────────────────────────────────────────────────++/**+ * Run `first` as candidate 1 of a two-candidate chain and report whether+ * candidate 2 — the metered provider a subscription user gets silently moved+ * onto — was ever reached.+ */+async function runInChain(+ first: ModelHandler,+ opts: { firstName?: string } = {}+): Promise<{ status: number; advanced: boolean; response: Response }> {+ let advanced = false;+ const second: ModelHandler = {+ async handle() {+ advanced = true;+ return new Response(JSON.stringify({ ok: true }), { status: 200 });+ },+ } as unknown as ModelHandler;++ const chain = new FallbackHandler([+ { name: opts.firstName ?? "Held Probe", handler: first },+ { name: "Metered Fallback", handler: second },+ ]);+ const { c } = makeContext();+ const response = await advanceUntilSettled(clock, chain.handle(c, PAYLOAD), 400_000);+ return { status: response.status, advanced, response };+}++describe("C-12 — a recovery 503 does not advance the fallback chain", () => {+ test("ABSENCE: no `trying next provider`, and candidate 2 is never reached", async () => {+ startLogCapture();+ try {+ breakTheNetwork();+ leaseAnswer = true;+ const outcome = await runInChain(makeHandler(makeTransport()));++ expect(outcome.advanced).toBe(false);+ expect(outcome.status).toBe(503);+ expect(isRecoveryHoldResponse(outcome.response)).toBe(true);++ const lines = await capturedLines();+ // The literal string C-12 names. Asserted as an ABSENCE, because a test+ // that only checked the status would pass against the bug.+ expect(lines.some((l) => l.includes("trying next provider"))).toBe(false);+ } finally {+ stopLogCapture();+ }+ });++ test("STRUCTURAL: a 503 whose message contains the word `quota` still holds the chain", async () => {+ startLogCapture();+ try {+ breakTheNetwork();+ leaseAnswer = true;+ // A provider whose own display name carries the word. Nothing exotic —+ // the sentence is built from the provider and endpoint the user+ // configured, so a vendor called "Quota Cloud" produces this for free.+ const handler = makeHandler(makeTransport({ displayName: "Quota Cloud" }));+ const outcome = await runInChain(handler, { firstName: "Quota Cloud" });++ // FIRST prove the hazard is real on this very body: the phrase match that+ // runs first inside `isRetryableError` DOES fire on it.+ const message = (await bodyOf(outcome.response)).error?.message ?? "";+ expect(message.toLowerCase()).toContain("quota");+ expect(hasQuotaExhaustionWording(message)).toBe(true);++ // And the chain still does not move. A wording-based fix fails here.+ expect(outcome.advanced).toBe(false);+ expect(outcome.status).toBe(503);++ const lines = await capturedLines();+ expect(lines.some((l) => l.includes("trying next provider"))).toBe(false);+ // Nor the quieter advance, which is the same event with a cost warning.+ expect(lines.some((l) => l.includes("falling through to the next provider"))).toBe(false);+ } finally {+ stopLogCapture();+ }+ });++ test("the 503 the SHIPPED wording produces does not trip the phrase list", async () => {+ // The belt behind the marker's brace. The marker is what makes chain-safety+ // structural; this keeps the default sentence from relying on it.+ const msg = buildRecoveryHoldMessage(+ "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses. " ++ "Check your network connection.",+ 7,+ 225_000+ );+ expect(hasQuotaExhaustionWording(msg)).toBe(false);+ expect(msg).toContain("claudish retried 7× over 3m 45s without reaching it");+ });+});++describe("exhaustedChainStatus cannot fold a recovery 503 into a terminal 400", () => {+ test("an EARLIER candidate's retryable auth failure does not demote our 503", async () => {+ breakTheNetwork();+ leaseAnswer = true;++ // Candidate 1 fails 401 — retryable, so the chain advances and pushes an+ // error that is NOT transient. Candidate 2 is the held one. Without the+ // verbatim return, `formatCombinedError` would run `exhaustedChainStatus`+ // over [401, 503], find one non-transient, and answer a terminal 400 —+ // Claude Code would never re-POST and tier 2 would never begin.+ const authFailure: ModelHandler = {+ async handle() {+ return new Response(JSON.stringify({ error: { message: "Invalid API key" } }), {+ status: 401,+ });+ },+ } as unknown as ModelHandler;++ const chain = new FallbackHandler([+ { name: "Revoked Key", handler: authFailure },+ { name: "Held Probe", handler: makeHandler(makeTransport()) },+ ]);+ const { c } = makeContext();+ const response = await advanceUntilSettled(clock, chain.handle(c, PAYLOAD), 400_000);++ expect(response.status).toBe(503);+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBe("1");+ expect(response.headers.get("x-should-retry")).toBe("true");+ const body = await bodyOf(response);+ // Not `all_providers_failed`, which is what a combined error would say.+ expect(body.error?.type).toBe("overloaded_error");+ });+});++describe("the marker is checked before the phrase list, inside isRetryableError too", () => {+ const quotaBody = JSON.stringify({ error: { message: "You have exceeded your quota" } });+ const marked = new Headers({ [RECOVERY_MARKER_HEADER]: "1" });++ test("without the marker the phrase list wins — the hazard, demonstrated", () => {+ expect(isRetryableError(503, quotaBody, "Held Probe")).toBe(true);+ });++ test("with the marker the answer is false, so the ordering is the guarantee", () => {+ expect(isRetryableError(503, quotaBody, "Held Probe", marked)).toBe(false);+ });++ test("the marker also beats the statuses that are otherwise retryable", () => {+ for (const status of [401, 402, 403, 404, 429]) {+ expect(isRetryableError(status, "{}", "Held Probe", marked)).toBe(false);+ expect(isRetryableError(status, "{}", "Held Probe")).toBe(true);+ }+ });++ test("a wrong value is not the marker", () => {+ expect(isRetryableError(401, "{}", "x", new Headers({ [RECOVERY_MARKER_HEADER]: "0" }))).toBe(+ true+ );+ expect(isRetryableError(401, "{}", "x", new Headers())).toBe(true);+ });+});++describe("the marker's wire name is a contract, not an implementation detail", () => {+ test("the literal header name, pinned where a rename cannot move both sides together", () => {+ // `recovery-marker.ts` mints it and `fallback-handler.ts` reads it, and both+ // import the constant — so renaming the constant moves both sides at once+ // and no behavioural test notices. The name is on the wire between a+ // response and the handler that decides whether to spend the user's money;+ // pin the string itself.+ expect(RECOVERY_MARKER_HEADER).toBe("x-claudish-recovery");+ });++ test("the 503 carries that literal name, read off the response with no constant", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const response = await runToExhaustion(makeHandler(makeTransport()), makeContext().c);+ expect(response.headers.get("x-claudish-recovery")).toBe("1");+ });+});++describe("the marker cannot be forged by an upstream provider", () => {+ test("an upstream error response is rebuilt, so its headers never reach the chain", async () => {+ // The general property, asserted on the response the client receives: an+ // upstream's headers do not survive `ComposedHandler`'s non-ok exits. This+ // is what makes "cannot be forged" structural rather than a claim about one+ // status code.+ globalThis.fetch = (async () =>+ new Response(JSON.stringify({ error: { message: "upstream is busy" } }), {+ status: 503,+ headers: { [RECOVERY_MARKER_HEADER]: "1", "x-should-retry": "true" },+ })) as unknown as typeof fetch;++ const response = await makeHandler(makeTransport()).handle(makeContext().c, PAYLOAD);++ expect(response.status).toBe(503);+ expect(response.headers.get(RECOVERY_MARKER_HEADER)).toBeNull();+ expect(isRecoveryHoldResponse(response)).toBe(false);+ });++ test("an upstream 401 carrying the header still advances the chain", async () => {+ // The discriminating shape. If ComposedHandler ever copied upstream headers+ // onto a non-ok response, a hostile or merely eccentric provider could pin+ // every user to itself by emitting this header — and the chain would stop+ // where it should advance. It must NOT be observable.+ globalThis.fetch = (async () =>+ new Response(JSON.stringify({ error: { message: "Invalid API key" } }), {+ status: 401,+ headers: { [RECOVERY_MARKER_HEADER]: "1", "x-should-retry": "true" },+ })) as unknown as typeof fetch;++ const outcome = await runInChain(makeHandler(makeTransport()));++ expect(outcome.advanced).toBe(true);+ expect(isRecoveryHoldResponse(outcome.response)).toBe(false);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// C-6's tier-2 half — one episode across the handoff+// ───────────────────────────────────────────────────────────────────────────++describe("the client's re-POST rejoins the SAME episode", () => {+ test("same episode id, clientRetries++, tier 2, ladder NOT reset", async () => {+ breakTheNetwork();+ leaseAnswer = true;+ const transport = makeTransport({ name: "rejoin-probe" });++ const first = await runToExhaustion(makeHandler(transport), makeContext().c);+ expect(first.status).toBe(503);++ const afterHandoff = renderableEpisodeFrame();+ expect(afterHandoff?.state).toBe("handoff");+ const episodeId = afterHandoff?.episodeId as string;+ const attemptsAfterTier1 = afterHandoff?.attempts as number;+ const ladderAfterTier1 = describeEpisode(episodeId)?.ladderIndex as number;+ expect(attemptsAfterTier1).toBeGreaterThan(1);++ // Claude Code's own backoff, measured at ≤38.4 s and honoured verbatim —+ // comfortably inside EPISODE_GRACE_MS, which is why the episode is still+ // there to rejoin.+ await clock.advance(10_000);++ // The re-POST. A NEW inbound request, so it gets a fresh per-request+ // deadline — the deadline belongs to the socket and the socket is new.+ const second = await runToExhaustion(makeHandler(transport), makeContext().c);+ expect(second.status).toBe(503);++ const rejoined = renderableEpisodeFrame();+ // ONE episode id across both tiers: one banner, one attempt counter.+ expect(rejoined?.episodeId).toBe(episodeId);+ expect(rejoined?.clientRetries).toBe(1);+ expect(rejoined?.tier).toBe(2);+ // The counters continue rather than restarting, which is what makes the+ // pane read as one continuous recovery.+ expect(rejoined?.attempts).toBeGreaterThan(attemptsAfterTier1);+ // And the ladder resumes where it left off instead of dropping back to 5 s+ // — a reset would hammer a dead host harder the longer the outage lasted.+ expect(describeEpisode(episodeId)?.ladderIndex).toBeGreaterThanOrEqual(ladderAfterTier1);+ });++ test("the turn completes on the client's re-POST, and the banner stops at once", async () => {+ startLogCapture();+ try {+ await recoversAfterHandoff();+ } finally {+ stopLogCapture();+ }+ });++ async function recoversAfterHandoff(): Promise<void> {+ breakTheNetwork();+ leaseAnswer = true;+ const transport = makeTransport({ name: "recover-after-handoff" });++ const first = await runToExhaustion(makeHandler(transport), makeContext().c);+ expect(first.status).toBe(503);+ const episodeId = renderableEpisodeFrame()?.episodeId as string;+ expect(describeEpisode(episodeId)?.state).toBe("handoff");++ await clock.advance(5_000);++ // The network comes back AFTER the handoff. This is C-6's second run in+ // miniature: the proof that the two tiers are one episode is that the turn+ // completes, not that the banner looked continuous.+ // A real `openai-sse` body, which is what this transport declares and what+ // the handler's parser expects. A JSON completion would be parsed as an+ // empty stream and the assertion would pass or fail for reasons that have+ // nothing to do with recovery.+ const sse =+ `data: ${JSON.stringify({+ id: "chatcmpl-1",+ object: "chat.completion.chunk",+ choices: [{ index: 0, delta: { role: "assistant", content: "back" } }],+ })}\n\n` ++ `data: ${JSON.stringify({+ id: "chatcmpl-1",+ object: "chat.completion.chunk",+ choices: [{ index: 0, delta: {}, finish_reason: "stop" }],+ })}\n\n` ++ "data: [DONE]\n\n";+ globalThis.fetch = (async () =>+ new Response(sse, {+ status: 200,+ headers: { "content-type": "text/event-stream" },+ })) as unknown as typeof fetch;++ const second = await runToExhaustion(makeHandler(transport), makeContext().c);++ expect(second.status).toBe(200);+ const text = await second.clone().text();+ expect(text).toContain("back");++ // ── THE PART THAT IS NOT OBVIOUS ────────────────────────────────────────+ //+ // The re-POST did NOT rejoin the episode. Its first attempt is the+ // handler's byte-identical primary fetch, which runs before the coordinator+ // sees the request — and with the network back it simply succeeded. Nothing+ // in the ladder ever ran.+ //+ // So the episode has to be closed by the SUCCESS, not by a rejoin. Left to+ // the grace timer it would sit in `handoff` for two more minutes with the+ // pane painting "waiting for Claude Code to retry" over a working session.+ expect(describeEpisode(episodeId)).toBeNull();+ expect(renderableEpisodeFrame()).toBeNull();++ const lines = await capturedLines();+ expect(lines.some((l) => l.includes("answered on the client's own retry"))).toBe(true);+ expect(lines.some((l) => l.includes(`episode ${episodeId} closed: recovered`))).toBe(true);+ }+});++// ───────────────────────────────────────────────────────────────────────────+// The client's retry budget+// ───────────────────────────────────────────────────────────────────────────++describe("CLAUDE_CODE_RETRY_WATCHDOG follows the recovery UI switch", () => {+ const KEY = "CLAUDISH_RECOVERY_UI";+ let saved: string | undefined;++ beforeEach(() => {+ saved = process.env[KEY];+ });++ afterEach(() => {+ if (saved === undefined) delete process.env[KEY];+ else process.env[KEY] = saved;+ });++ test("set when the UI is enabled — this is what makes the reach ~a day", () => {+ delete process.env[KEY];+ expect(retryWatchdogEnv()).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });+ });++ test("absent when the user turned the UI off", () => {+ // No pane means no lease means an inline 400 at exhaustion means nothing to+ // hand back — so extending the client's budget would buy only the side+ // effect on claudish's other 503s.+ process.env[KEY] = "0";+ expect(retryWatchdogEnv()).toEqual({});+ expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv()).toBe(false);+ });+});++describe("formatHoldDuration", () => {+ test("minutes appear only when there are minutes, and it never goes negative", () => {+ expect(formatHoldDuration(0)).toBe("0s");+ expect(formatHoldDuration(-5_000)).toBe("0s");+ expect(formatHoldDuration(45_000)).toBe("45s");+ expect(formatHoldDuration(225_000)).toBe("3m 45s");+ expect(formatHoldDuration(120_000)).toBe("2m 0s");+ });+});diff --git a/packages/cli/src/handlers/shared/connection-error.ts b/packages/cli/src/handlers/shared/connection-error.tsindex f20408db..b04f08f9 100644--- a/packages/cli/src/handlers/shared/connection-error.ts+++ b/packages/cli/src/handlers/shared/connection-error.ts@@ -161,3 +161,54 @@ export function buildConnectionErrorMessage(return `Cannot reach ${displayName} at ${endpoint}. Check your network connection.`;}}++/**+ * `3m 45s`, `45s`, `0s`. Never negative, never fractional.+ *+ * Deliberately a separate implementation from the pane's formatter of the same+ * shape: the pane runs in ANOTHER PROCESS and shares no module with this one,+ * and importing across that boundary is what the recovery wire protocol exists+ * to avoid.+ */+export function formatHoldDuration(ms: number): string {+ const total = Math.max(0, Math.round(ms / 1000));+ const minutes = Math.floor(total / 60);+ const seconds = total % 60;+ return minutes > 0 ? `${minutes}m ${seconds}s` : `${seconds}s`;+}++/**+ * The sentence an EXHAUSTED tier-1 hold carries on its retryable 503.+ *+ * It is `buildConnectionErrorMessage`'s own sentence plus a recovery clause,+ * and it has to stay built that way: the inline 400 and the handed-back 503+ * describe ONE fault, and two sentences that drift tell the user two different+ * stories about it.+ *+ * ── WHAT THIS SENTENCE MUST NEVER SAY ───────────────────────────────────────+ *+ * `fallback-handler.ts`'s `hasQuotaExhaustionWording` is the FIRST statement in+ * `isRetryableError`'s body and is status-agnostic on purpose. Its phrase list+ * contains the bare substring `"quota"`, plus `"usage limit"`, `"plan limit"`,+ * `"daily limit"`, `"billing cycle"`, `"credit balance"`, `"out of credits"`+ * and `"exceeded your current"`. A 503 whose MESSAGE trips that list advances+ * the fallback chain and moves a subscription user onto metered billing during+ * an outage.+ *+ * The structural defence is the `x-claudish-recovery` marker+ * (`recovery-marker.ts`), which takes the message out of that decision+ * entirely. This wording rule is the belt behind that brace, and it is asserted+ * directly on the composed sentence in `recovery-status.test.ts` — so a future+ * edit that reaches for "retry quota" or "attempt limit" fails a test rather+ * than a user's invoice.+ */+export function buildRecoveryHoldMessage(+ reason: string,+ attempts: number,+ elapsedMs: number+): string {+ return (+ `${reason} claudish retried ${attempts}× over ${formatHoldDuration(elapsedMs)} ` ++ "without reaching it — still trying, watch the recovery pane."+ );+}diff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.tsnew file mode 100644index 00000000..648288bb--- /dev/null+++ b/packages/cli/src/handlers/shared/recovery-marker.ts@@ -0,0 +1,93 @@+/**+ * The one header that makes recovery chain-safety STRUCTURAL rather than a+ * wording rule.+ *+ * ## What it is for+ *+ * When a tier-1 hold exhausts its deadline with a surface still painting the+ * reason, `ComposedHandler` answers **503 `overloaded_error`** instead of+ * today's terminal 400, so Claude Code re-POSTs and the turn survives into+ * tier 2. That status then has to travel past `FallbackHandler`, and the+ * fallback chain is where a mistake here costs the user real money rather than+ * time: advancing off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one+ * quotes a per-token price for a fault that had nothing to do with the+ * provider.+ *+ * ## Why a header and not a status, and not a wording rule+ *+ * `fallback-handler.ts`'s `isRetryableError` has no 503 branch, so on the+ * status alone a 503 already stops the chain. That is NOT enough, twice over:+ *+ * 1. **The FIRST statement in its body is `hasQuotaExhaustionWording(errorBody)`,+ * which is deliberately status-agnostic** — it exists because the transport+ * has already remapped terminal errors to 400, so a status-gated test there+ * would miss a spent plan. Its phrase list contains the **bare substring+ * `"quota"`**, plus `"usage limit"`, `"plan limit"`, `"daily limit"`,+ * `"billing cycle"`, `"credit balance"`, `"out of credits"` and+ * `"exceeded your current"`. A 503 whose MESSAGE happens to trip that list+ * advances the chain and spends the user's money. Checking the marker FIRST+ * removes the message from the decision entirely.+ * 2. **A 503 does not necessarily reach the client.** When a candidate's error+ * is not retryable and earlier candidates already failed, `handle()` pushes+ * it and calls `formatCombinedError`, whose status comes from+ * `exhaustedChainStatus(errors)` — 503 only if EVERY accumulated error is+ * transient. One earlier auth/404 failure turns our 503 into a terminal 400+ * and tier 2 never begins. The marker makes the response return VERBATIM+ * before it can be folded into a combined one.+ *+ * ## Why it cannot be forged by an upstream provider+ *+ * The marker is only ever minted on a `Response` this process constructs, and+ * verified at source (2026-09-11):+ *+ * - every non-ok exit from `ComposedHandler` is `c.json(...)` — Hono builds+ * those headers from nothing, so an upstream's headers are never copied onto+ * a failure response. The single `c.header()` call in that file+ * (`X-Dropped-Params`) sits AFTER the `!response.ok` early returns;+ * - the one path that DOES copy upstream headers verbatim,+ * `stream-head-sniffer.ts`'s `replayResponse()`, is reachable only from step+ * 7b, which runs after `!response.ok` has already returned — i.e. on a 200.+ * `FallbackHandler` returns an ok response as success before it ever looks at+ * the marker, so a forged one on a 200 changes nothing.+ *+ * Keep both properties true. If a future edit returns an upstream `Response`+ * object on a non-ok path, the marker must be stripped there.+ */++/** The header name. Referenced by name in no other file — import this. */+export const RECOVERY_MARKER_HEADER = "x-claudish-recovery";++/** The only value that counts. Anything else is not our response. */+export const RECOVERY_MARKER_VALUE = "1";++/**+ * The headers a tier-1 exhaustion 503 carries.+ *+ * `x-should-retry` is read by the Anthropic TS SDK's `shouldRetry` before it+ * looks at the status at all, so it survives a future change to Claude Code's+ * status classification. It is set ONLY on this arm.+ */+export function recoveryHoldHeaders(): Record<string, string> {+ return {+ "content-type": "application/json",+ "x-should-retry": "true",+ [RECOVERY_MARKER_HEADER]: RECOVERY_MARKER_VALUE,+ };+}++/** Does this header bag carry the marker? Tolerates `undefined` for callers that have none. */+export function hasRecoveryMarker(headers: Headers | undefined | null): boolean {+ try {+ return headers?.get(RECOVERY_MARKER_HEADER) === RECOVERY_MARKER_VALUE;+ } catch {+ // A synthesised response with a non-Headers `headers` field. Not ours.+ return false;+ }+}++/** Is this response claudish's own recovery-exhaustion 503? */+export function isRecoveryHoldResponse(+ response: { headers?: Headers } | null | undefined+): boolean {+ return hasRecoveryMarker(response?.headers);+}diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.tsindex 3ccd21f7..f58c4b2a 100644--- a/packages/cli/src/recovery/coordinator.test.ts+++ b/packages/cli/src/recovery/coordinator.test.ts@@ -32,6 +32,7 @@ import {episodeCount,giveUp,joinEpisode,+ noteTargetReachable,tryNow,uiLeaseValid,} from "./coordinator.js";@@ -431,6 +432,72 @@ describe("an episode rejoined within EPISODE_GRACE_MS continues", () => {});});+// ───────────────────────────────────────────────────────────────────────────+// 4b. The OTHER way a handoff ends: the client's own retry simply succeeds+// ───────────────────────────────────────────────────────────────────────────++describe("noteTargetReachable — the handoff that ends without a rejoin", () => {+ test("a reachable target closes the parked episode as recovered", async () => {+ useFakeClock();+ const s = seed({ providerName: "reach-1" });+ const h = joinEpisode(s);+ const id = h.episodeId;+ h.recordAttemptResult("ConnectionRefused", false);+ h.handoff();+ h.leave();+ expect(describeEpisode(id)?.state).toBe("handoff");++ // Tier 2's rejoin is NOT what happens when the network comes back. The+ // client's re-POST enters the handler's byte-identical primary fetch, which+ // succeeds outright — no catch, no `joinEpisode`. Without this call the+ // episode would sit here for the rest of its 120-second grace with the pane+ // painting "waiting for Claude Code to retry" over a working session.+ noteTargetReachable("reach-1", "http://127.0.0.1:9/v1/chat/completions");++ expect(describeEpisode(id)).toBeNull();+ expect(episodeCount()).toBe(0);+ });++ test("a DIFFERENT host does not close it — the key is provider|host", async () => {+ useFakeClock();+ const s = seed({ providerName: "reach-2" });+ const h = joinEpisode(s);+ h.recordAttemptResult("ConnectionRefused", false);+ h.handoff();+ h.leave();++ noteTargetReachable("reach-2", "http://127.0.0.1:10/v1/chat/completions");+ expect(describeEpisode(h.episodeId)?.state).toBe("handoff");+ noteTargetReachable("other-provider", "http://127.0.0.1:9/v1/chat/completions");+ expect(describeEpisode(h.episodeId)?.state).toBe("handoff");+ });++ test("a LIVE episode with parked waiters is never closed underneath them", async () => {+ useFakeClock();+ const s = seed({ providerName: "reach-3" });+ const h = joinEpisode(s);+ h.recordAttemptResult("ConnectionRefused", false);+ const woken: WaitOutcome[] = [];+ const parked = h.waitForNextAttempt(never(), 30_000).then((o) => {+ woken.push(o);+ });+ await drain();++ // One request's success says nothing about another's. Closing here would+ // wake this waiter with `gave_up` and answer an inline error for an outage+ // that its own ladder was still working on.+ noteTargetReachable("reach-3", "http://127.0.0.1:9/v1/chat/completions");++ expect(describeEpisode(h.episodeId)?.state).toBe("waiting");+ expect(woken).toHaveLength(0);++ await clock.advance(30_000);+ await parked;+ expect(woken).toEqual([{ kind: "attempt" }]);+ h.leave();+ });+});+// ───────────────────────────────────────────────────────────────────────────// Success, and the seams that are deliberately inert in this phase// ───────────────────────────────────────────────────────────────────────────diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 1bec8921..4876c849 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -617,6 +617,49 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {}}+/**+ * A request reached this target WITHOUT the ladder's help. Close any episode+ * parked in `handoff` for it.+ *+ * ── THE CASE THIS EXISTS FOR, WHICH THE DESIGN DID NOT ANTICIPATE ───────────+ *+ * Tier 2 hands the retry back as a 503 and leaves the episode `handoff` for its+ * grace window. Claude Code's backoff elapses and it re-POSTs — and the+ * re-POST's FIRST attempt is the handler's byte-identical primary fetch, which+ * runs BEFORE the coordinator ever sees the request. When the network has come+ * back in the meantime, that attempt simply SUCCEEDS: no `catch`, no+ * `joinEpisode`, no rejoin. The turn completes, and the episode sits in+ * `handoff` until its 120-second grace expires.+ *+ * That leaves the pane painting *"waiting for Claude Code to retry"* over a+ * session that is already working — for up to two minutes, on the one surface+ * whose legibility is what earns the 503 its retryable status in the first+ * place. A banner that lies is worse than no banner, because the next real+ * outage is read as more of the same.+ *+ * ── WHY THE HEALTHY PATH CAN AFFORD TO ASK ──────────────────────────────────+ *+ * The caller guards this with `episodeCount() > 0`, which is one `Map.size`+ * read. On a machine that has never had an outage the map is empty and nothing+ * below runs — no URL parse, no string build. That is the whole cost NFR-1+ * pays for the banner telling the truth.+ *+ * ONLY `handoff` is closed, and the restriction is load-bearing: an episode in+ * `attempting`/`waiting` still has parked waiters whose ladder this is, and+ * closing it underneath them would strand or gave-up requests that another+ * request's success says nothing about. A `handoff` episode has no waiters by+ * construction — that is what `handoff` means.+ */+export function noteTargetReachable(providerName: string, endpoint: string): void {+ const ep = episodes.get(`${providerName}|${hostOf(endpoint)}`);+ if (!ep || ep.state !== "handoff") return;+ log(+ `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` ++ `closing episode ${ep.episodeId} instead of waiting out its grace`+ );+ closeEpisode(ep, "recovered");+}+/*** Collapse the wait for an episode and attempt now. The pane's `[r] try now`* key routes here in a later phase; nothing calls it in this one, and it isdiff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.tsindex e894fc84..f0373466 100644--- a/packages/cli/src/recovery/settings.ts+++ b/packages/cli/src/recovery/settings.ts@@ -186,3 +186,58 @@ export function logDeadlineIfShortened(deadlineMs: number): void {export function resetDeadlineNotice(): void {deadlineNoticeLogged = false;}++// ─── The client's retry budget ───────────────────────────────────────────────++/**+ * The environment that decides how far recovery REACHES, handed to the Claude+ * Code child.+ *+ * Tier 1 holds one inbound request for the derived deadline (~270 s at the+ * default) and then answers a retryable 503, which the client re-POSTs into the+ * SAME episode. So the outer bound on recovery is not ours at all — it is+ * Claude Code's willingness to keep asking, and `CLAUDE_CODE_RETRY_WATCHDOG`+ * is the variable that sets it.+ *+ * ── WHAT IT COSTS, STATED PLAINLY ───────────────────────────────────────────+ *+ * | | unset | set (what ships) |+ * |----------------------------|----------|------------------|+ * | client retry budget | ~11 | **~300** |+ * | unattended reach | ~66 min | **~a day** |+ * | worst-case attempts / turn | ~46 | **~2,100** |+ * | duplicate-charge exposure | accepted | **~30× that** |+ *+ * The duplicate-charge exposure is real and is the reason this is worth a+ * paragraph rather than a line: a request the provider ACCEPTED and began+ * billing can still fail on `ECONNRESET`/`EPIPE`, and every re-ask pays for it+ * again — including on paths where inference had already started. That+ * arithmetic was put to the user with these numbers in front of them and the+ * larger magnitude was chosen deliberately. It is an informed acceptance, not+ * an oversight; do not quietly re-litigate it here.+ *+ * ── THE SIDE EFFECT, WHICH IS NOT OPTIONAL ──────────────────────────────────+ *+ * The watchdog is GLOBAL to the child, so it amplifies every 503 claudish can+ * emit and not only the recovery one: the stream-head sniffer's exhaustion and+ * `exhaustedChainStatus`'s all-transient chain also become ~300-attempt+ * retries. Neither of those opens an episode or a pane, so while the client+ * loops on them the reason is legible NOWHERE — they degrade to "API error ·+ * Retrying" for far longer than before. More retry is the intended remedy for+ * both (an upstream overload is genuinely transient, which is why they are+ * 503s at all), but the visibility argument that earns the recovery 503 its+ * retryable status does not extend to them. That gap is the honest price of+ * one process-wide switch, and there is no narrower lever: the variable is+ * fixed at spawn and cannot key on whether a banner exists.+ *+ * ── WHY IT IS GATED ON THE UI SWITCH ────────────────────────────────────────+ *+ * `resolveRecoveryUi()` is the user's configuration — flag > env > project >+ * global > true. Off means no pane, therefore no lease, therefore an inline+ * 400 at exhaustion, therefore nothing handed back for the client to re-ask —+ * so extending its budget would buy only the side effect above. Surface and+ * reach move together or not at all.+ */+export function retryWatchdogEnv(): Record<string, string> {+ return resolveRecoveryUi() ? { CLAUDE_CODE_RETRY_WATCHDOG: "1" } : {};+}