Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
fix(recovery): stop the ladder faking attempts, billing latency, and ignoring [q]
Jack Rudenko4e45c0cdb4c0
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.mdindex dd5792ee..d4d79e0b 100644--- a/ai-docs/architecture/network-recovery.md+++ b/ai-docs/architecture/network-recovery.md@@ -59,8 +59,10 @@ The second tier hands the retry back to the client. What the client will then sp- **~11 attempts** on its default budget, spanning **~174 s** for a 503 chain (backoff caps at38.4 s per gap, and `retry-after` is honoured verbatim);-- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets when the recovery UI is- enabled — reaching **~a day** of unattended recovery on a hung connection.+- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets only when recovery is+ enabled, the UI is allowed, AND this launch can actually obtain a pane — reaching **~a day** of+ unattended recovery on a hung connection. All three gates, because the watchdog amplifies EVERY+ 503 the session sees and is worth its cost only where a surface can exist (§7).It is a count, not a clock. That is why `EPISODE_GRACE_MS` is 120 s: the client's worst gap is38.4 s, so 120 s is 3.1× inside it and a returning client always finds its episode still open.@@ -79,7 +81,8 @@ POST /v1/messages ──► ComposedHandler.handle() startTime = performance.nclassifyConnectionError() ── null ──► rethrow, unchanged│ non-null▼- shouldSkipTier1()? probe header · recovery disabled · no budget left+ shouldSkipTier1()? probe header · recovery disabled ·+ user pressed [q] · no budget left│ no┌───────────────────▼──────────────────────────────────────────┐│ TIER 1 — in-request, PRE-FLUSH, deadline derived from §1 │@@ -165,9 +168,28 @@ is the only honest discriminator, and it reads the address, not the error.| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the pane, so the retry may be handed back || exhausted, **no lease** | **400** `connection_error` | — | nothing can display the reason, so it must ride the status || `[q] give up` | **400** `connection_error` | — | guarded on `result.kind`, never on the lease — see below |+| **a request arriving within 60 s of `[q]`** | **400** `connection_error` | — | the ladder is SKIPPED outright: the user said stop || client disconnected | 499 (unread) | — | the socket is already gone || unclassifiable throw on any attempt | rethrown unchanged | — | recovery must not widen what "transient" means |+### `[q] give up` suppresses the HOLD, not only the banner++`bye` originally did two things, both inside the UI manager: it suppressed re-opening the pane for+60 s, and it called `giveUpAll()` — which iterates the episodes alive *at that instant*. Nothing+recorded that the user had asked recovery to stop, and `shouldSkipTier1` had no gate that could+notice. So the next request against the same dead target opened a NEW episode, found the pane+suppressed, got no pane and no lease, and then held its socket for the full ~4.5-minute deadline+before answering 400 — **a long hold with no surface, which is the state this design calls "strictly+worse than the bug this feature exists to remove", reached from the one affordance whose entire+purpose is to end it.** Before recovery existed those requests failed in milliseconds.++It is not an edge case. Claude Code issues concurrent requests during an outage (main loop, title+model, subagents), so a request arriving inside the suppression window is the EXPECTED one.++The fact is therefore process-level — `recovery/settings.ts`'s `recoveryGiveUpActive()` — and is read+by **both** `shouldSkipTier1` (the hold) and `ensureRecoveryUi` (the surface). One number, one+window, two readers. A control that appears to stop something and does not is worse than no control.+### 429 would have been a live billing bug`fallback-handler.ts` contains, verbatim:@@ -320,6 +342,28 @@ common local failure — never reached tier 1 at all. Same class of fix in `vert`healthChecked` latches on **success only**, so a retried `refreshAuth()` re-probes instead ofreturning a false success to attempt two.+### Three things the auth path needs that the fetch path gets for free++- **The clamp has to be ENFORCED, not handed over.** `op` takes the per-attempt signal, and the fetch+ path threads it into `fetch`. `refreshAuth()` and `getHeaders()` take no arguments, so+ `() => this.provider.refreshAuth!()` discards it — and Grok's token exchange then performs an+ unbounded `fetch(auth.x.ai/oauth2/token)`. A swallowed connection there outlived both the 45 s cap+ and the client's own disconnect while `withConnectionRetry` awaited a promise nothing could settle:+ the unbounded hold this feature exists to remove, reached from inside the machinery that removes+ it. `untilAborted` now enforces the ceiling at the one place that owns it. The operation is+ *abandoned*, not cancelled — threading a real signal through every transport's auth call is the+ deeper fix and is still worth doing.+- **The error must name the host that actually failed.** `connectionEndpointFor` falls back to the+ MODEL endpoint when the error carries no `claudishEndpoint`, so an `auth.x.ai` outage was reported+ — in the banner, in the log and in the episode key — as `api.x.ai`. Grok's refresh wrapper now+ attaches it, as `local.ts` already did.+- **`noteTargetReachable` closes by PROVIDER, not by host.** Because an auth episode can be keyed on+ a different host than the one a later success reaches, a strict key lookup missed it and the pane+ painted "waiting for Claude Code to retry" over a working session for the full 120 s grace. A+ request that reached the model endpoint had to authenticate first, so it has proved every host it+ touched is answering — and a `handoff` episode holds no waiters, so nothing is closed from under+ anyone.+---## 6. Stats — and the one step whose omission is silent@@ -398,6 +442,38 @@ ladder, and then accepted it again, with the arithmetic in front of them, at the`recovery_episode_id` is what makes the exposure *attributable* — without it, attribution breaksexactly across a tier-2 handoff, which is the highest-exposure path.+**What RISK-6 does NOT extend to, and the two ways it leaked past its own boundary.** Both were+found in review and closed; both are the same mistake — applying the accepted arithmetic to a case+the user was never shown.++1. **A LATENCY event is not a network fault.** `classifyConnectionError` mapped the NAME+ `TimeoutError` to `unreachable`. That name is also what `AbortSignal.timeout` on a transport's own+ *inference* request rejects with — `vertex-oauth.ts` puts a 30 s ceiling on the model call itself.+ A `vx@` turn whose time-to-first-byte exceeded 30 s (ordinary for a thinking model) was therefore+ classified as "the host is unreachable", entered the ladder, took the tier-2 handoff, and was+ re-POSTed ~300 times — **each one running one more real billed inference against a host that had+ answered the TCP connect and was already generating.** The discriminator is now the SIGNAL'S+ ORIGIN, not the name: our per-attempt clamp and our reachability probes carry a `markOwnTimeout`+ own-property and classify; everything else keeps its pre-recovery route out, unclassified and+ rethrown. The user-facing sentence was wrong too — it told the user to check VPN and DNS for a+ host that was mid-generation.+2. **The watchdog needs three gates, not one.** `CLAUDE_CODE_RETRY_WATCHDOG=1` was exported from the+ UI preference alone, so `-p`, `--no-recovery`, a pipe and a machine without magmux all expanded+ every UNRELATED 503 to ~300 client attempts while being structurally incapable of holding the+ lease a recovery 503 requires. It now requires `resolveRecoveryEnabled()` **and**+ `resolveRecoveryUi()` **and** `magmuxPaneCapability() !== none` — the last asked BEFORE the child+ environment is finalised, which is why that predicate is side-effect free and is the same one+ `planMagmuxWrap` consumes.++**The transport's `getRequestInit()` is re-minted per attempt.** It was hoisted once, before the+primary fetch, and spread into every re-issue. A transport that returns a one-shot+`AbortSignal.timeout` therefore poisoned the whole ladder the moment it fired:+`AbortSignal.any([fired, live])` is ALREADY ABORTED, so every later attempt rejected without opening+a socket. For `vx@` that made tier 1 silently dead past t+30 s — the request held the full deadline+making **zero** real connect attempts while the log and the pane counted attempts that never left the+process. `mergeSignalIntoInit` now also drops an `own` that is already aborted, as the belt behind+that brace.+**RISK-7 — a CI run holding a dead endpoint for the full deadline.** `--no-recovery` /`CLAUDISH_RECOVERY=0` restores today's immediate 400 everywhere, byte for byte. `--no-recovery-ui` /`CLAUDISH_RECOVERY_UI=0` keeps the retries and drops only the pane (and with it the 503 arm, sincediff --git a/packages/cli/src/claude-runner.ts b/packages/cli/src/claude-runner.tsindex e8b1966a..ee3c8e10 100644--- a/packages/cli/src/claude-runner.ts+++ b/packages/cli/src/claude-runner.ts@@ -17,7 +17,7 @@ import { isatty } from "node:tty";import { lookupModelForProvider } from "./adapters/model-catalog.js";import { classifierPassthroughEnabled } from "./classifier-passthrough.js";import { ENV } from "./config.js";-import { planMagmuxWrap } from "./launcher/magmux-wrapper.js";+import { magmuxPaneCapability, planMagmuxWrap } from "./launcher/magmux-wrapper.js";// Aliased: runClaudeWithProxy declares its own local `log` (a quiet-aware// console printer), and an unaliased import would be shadowed inside it.import { log as debugLog, logStderr } from "./logger.js";@@ -1634,12 +1634,22 @@ export async function runClaudeWithProxy(...advisorToolEnv.vars,};+ // Can this launch put a recovery banner on screen at all? Asked HERE, before+ // the child environment is finalised, because the answer gates the watchdog+ // below — and asked through a side-effect-free function so asking is free and+ // repeatable. `planMagmuxWrap` further down consumes the same predicate, so+ // "may we wrap" and "may we amplify the client's retries" cannot drift.+ const paneCapability = magmuxPaneCapability({+ interactive: Boolean(config.interactive),+ stdoutIsTty: Boolean(process.stdout.isTTY),+ });+// The client half of "retry forever". Tier 1 holds one request for the// derived deadline and then hands the retry back as a 503; how far recovery// actually reaches is decided by Claude Code's willingness to re-ask, which- // this sets. The arithmetic, the accepted cost and the side effect on- // claudish's OTHER 503s are all stated at `retryWatchdogEnv()`.- Object.assign(env, retryWatchdogEnv());+ // this sets. The arithmetic, the accepted cost, the side effect on claudish's+ // OTHER 503s and all three gates are stated at `retryWatchdogEnv()`.+ Object.assign(env, retryWatchdogEnv({ paneEligible: paneCapability.kind !== "none" }));// Provider display name, best-effort and FREE. Only an explicit `provider@model`// spec names its provider without routing, and route() would touch credentials /@@ -1958,16 +1968,22 @@ export async function runClaudeWithProxy(// decides whether a retry happens. Off means no pane, therefore no lease,// therefore an inline error at exhaustion — which is the 503-vs-400 decision// made structurally rather than by a second predicate somewhere else.- const wrap = resolveRecoveryUi()- ? planMagmuxWrap({- claudeBinary,- claudeArgs,- childEnv: env,- cwd: process.cwd(),- interactive: Boolean(config.interactive),- stdoutIsTty: Boolean(process.stdout.isTTY),- })- : null;+ //+ // `paneCapability` was resolved before the environment was finalised, and is+ // re-used rather than re-derived: the watchdog above is exported on exactly+ // the launches that reach `planMagmuxWrap` or the ambient branch below.+ const wrap =+ resolveRecoveryUi() && paneCapability.kind === "wrap"+ ? planMagmuxWrap({+ claudeBinary,+ claudeArgs,+ childEnv: env,+ cwd: process.cwd(),+ interactive: Boolean(config.interactive),+ stdoutIsTty: Boolean(process.stdout.isTTY),+ magmuxBinary: paneCapability.magmux,+ })+ : null;const proc = wrap? spawn(wrap.command, wrap.args, { env: wrap.env, stdio, shell: false })diff --git a/packages/cli/src/handlers/composed-handler-recovery.test.ts b/packages/cli/src/handlers/composed-handler-recovery.test.tsindex 856cd33e..f412275b 100644--- a/packages/cli/src/handlers/composed-handler-recovery.test.ts+++ b/packages/cli/src/handlers/composed-handler-recovery.test.ts@@ -22,15 +22,24 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";import type { Context } from "hono";import { probeLink } from "../providers/probe-live.js";import type { ProviderTransport } from "../providers/transport/types.js";-import { resetRecoveryClock } from "../recovery/clock.js";+import { resetRecoveryClock, setRecoveryClock } from "../recovery/clock.js";import { closeAllEpisodes, episodeCount } from "../recovery/coordinator.js";-import { resetRecoveryFlagOverrides, setRecoveryFlagOverrides } from "../recovery/settings.js";+import { UI_SUPPRESS_AFTER_BYE_MS, __handleCommandForTests } from "../recovery/magmux-ui.js";+import {+ recoveryGiveUpActive,+ resetRecoveryFlagOverrides,+ resetRecoveryGiveUp,+ setRecoveryFlagOverrides,+} from "../recovery/settings.js";import {capturedRecoveryLines,startLogCapture,stopLogCapture,} from "../recovery/test-helpers/capture-log.js";+import { FakeClock } from "../recovery/test-helpers/fake-clock.js";+import { RECOVERY_PROTOCOL_VERSION } from "../recovery/types.js";import { ComposedHandler } from "./composed-handler.js";+import { classifyConnectionError, markOwnTimeout } from "./shared/connection-error.js";/** A port nothing is listening on, taken and released so it is VERIFIED free. */function freePort(): number {@@ -52,6 +61,9 @@ afterEach(() => {closeAllEpisodes();resetRecoveryClock();resetRecoveryFlagOverrides();+ // PROCESS state, shared with every sibling test file in this Bun process. A+ // give-up left set would skip the ladder in tests that never pressed a key.+ resetRecoveryGiveUp();});let seq = 0;@@ -302,3 +314,246 @@ describe("shouldSkipTier1 — three gates, and no loopback carve-out", () => {await p;});});++// ───────────────────────────────────────────────────────────────────────────+// The transport's own signal, per attempt+// ───────────────────────────────────────────────────────────────────────────++describe("every ladder attempt gets a FRESH transport init", () => {+ /**+ * The defect this pins, in one sentence: `getRequestInit()` was called once,+ * before the primary fetch, and the same object was spread into every+ * re-issue — so a transport that returns a ONE-SHOT `AbortSignal.timeout`+ * (`vertex-oauth.ts` returns a 30-second one) poisoned the whole ladder the+ * moment that signal fired. `AbortSignal.any([fired, live])` is ALREADY+ * ABORTED, so the attempt rejected before it opened a socket.+ *+ * The visible cost was not a slow retry but a FAKE one: tier 1 held the+ * request for the full ~270 s deadline making ZERO real connect attempts,+ * while the `[Recovery]` log and the pane both counted attempts that never+ * left the process.+ *+ * So the assertion is about what the NETWORK saw, and it needs the real+ * runtime: a real `AbortSignal.timeout`, real `fetch`, and a real refused+ * port. A stubbed `globalThis.fetch` cannot fail this test, because the+ * rejection being tested happens inside fetch itself.+ */+ test("a retry after the transport's own signal has fired still reaches the socket", async () => {+ const clock = new FakeClock(0);+ setRecoveryClock(clock);++ // Shorter than the gap the test then waits, so by attempt 2 the FIRST+ // signal minted here is certainly dead.+ const CEILING_MS = 40;+ let initCalls = 0;+ const transport = {+ ...(deadTransport(deadPort) as object),+ getRequestInit: () => {+ initCalls++;+ return { signal: AbortSignal.timeout(CEILING_MS) };+ },+ } as unknown as ProviderTransport;++ const handler = new ComposedHandler(transport, "dead-model", "dead-model", 8080, {});+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ const { c } = contextFor(req);+ const p = handler.handle(c, PAYLOAD);++ // Attempt 1 fails on a refused socket and the ladder parks on its 5 s rung.+ // The wait is REAL time and is what kills the first signal.+ await new Promise((r) => setTimeout(r, 150));+ expect(episodeCount()).toBe(1);+ expect(initCalls).toBe(1);++ // Fire the rung. Attempt 2 runs here.+ await clock.advance(5_000);+ await new Promise((r) => setTimeout(r, 150));++ const lines = await capturedRecoveryLines();+ const attempt2 = lines.find((l) => /attempt 2 failed/.test(l)) ?? "";++ // THE ASSERTION. `ConnectionRefused` is the runtime saying it reached the+ // network and the port said no. `TimeoutError` here would mean the attempt+ // was refused by an expired signal without a socket ever being opened —+ // and with a signal whose origin is not ours it is not even classifiable,+ // so the ladder would break out entirely.+ expect(attempt2).toContain("ConnectionRefused");+ expect(attempt2).not.toContain("TimeoutError");+ // Re-minted, not re-used: once for the primary fetch, once per re-issue.+ expect(initCalls).toBeGreaterThanOrEqual(2);++ ac.abort(new DOMException("done", "AbortError"));+ await p.catch(() => {});+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Whose timeout was it?+// ───────────────────────────────────────────────────────────────────────────++describe("a transport's own request timeout is NOT a connection failure", () => {+ /**+ * `TimeoutError` used to mean `unreachable` whatever raised it, and+ * `AbortSignal.timeout` on a transport's INFERENCE request raises exactly+ * that name. So a turn whose time-to-first-byte exceeded the transport's own+ * ceiling — ordinary for a thinking model on a streaming endpoint — was+ * classified as "the host is unreachable", entered the ladder, took the+ * tier-2 handoff, and with the retry watchdog on was re-POSTed ~300 times.+ * Each of those re-POSTs runs ONE MORE REAL BILLED INFERENCE against a host+ * that answered the TCP connect and was already generating.+ *+ * RISK-6 was accepted for `ECONNRESET`/`EPIPE` on a genuine network fault.+ * A latency event is neither a network fault nor transient, and was not+ * priced. The discriminator is therefore the SIGNAL'S ORIGIN, not the name.+ */+ test("a hung upstream that trips the TRANSPORT's ceiling never opens an episode", async () => {+ // A server that accepts the connection and then never answers: the exact+ // shape a slow model presents, and the one a network classifier must not+ // confuse with an unreachable host.+ const hung = Bun.serve({+ port: 0,+ fetch: () => new Promise<Response>(() => {}),+ });+ try {+ const transport = {+ name: `hung-${Date.now()}`,+ displayName: "Slow Model",+ streamFormat: "openai-sse",+ getEndpoint: () => `http://127.0.0.1:${hung.port}/v1/chat/completions`,+ getHeaders: async () => ({}),+ getRequestInit: () => ({ signal: AbortSignal.timeout(60) }),+ } as unknown as ProviderTransport;++ const handler = new ComposedHandler(transport, "slow-model", "slow-model", 8080, {});+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ const { c } = contextFor(req);++ // Raced explicitly rather than simply awaited, because the defect's+ // signature is a HOLD: misclassified, this request parks on the ladder+ // for the full derived deadline (~270 s), which a bare `await` would+ // report as a hung test run rather than as a failure.+ const p = handler.handle(c, PAYLOAD);+ const outcome = await Promise.race([+ p.then(+ () => "resolved" as const,+ (e: unknown) => e+ ),+ new Promise<"still-holding">((r) => setTimeout(() => r("still-holding"), 2_000)),+ ]);++ // Unclassified means rethrown unchanged — the pre-recovery route, which+ // is the whole point. No ladder, no hold, no 300 re-POSTs.+ expect(outcome).not.toBe("still-holding");+ expect(outcome).not.toBe("resolved");+ expect((outcome as Error).name).toBe("TimeoutError");+ expect(episodeCount()).toBe(0);+ expect((await capturedRecoveryLines()).some((l) => l.includes("opened for"))).toBe(false);++ ac.abort(new DOMException("done", "AbortError"));+ await p.catch(() => {});+ } finally {+ hung.stop(true);+ }+ }, 15_000);++ test("but OUR OWN clamp still is one — the ladder can retry what it gave up on", async () => {+ // The other half, and the reason the discriminator cannot simply be "drop+ // TimeoutError": the per-attempt clamp aborts with that same name, and an+ // attempt WE cut short is a failed attempt to be retried, not an error+ // that escapes the ladder. It carries `markOwnTimeout`, so it classifies.+ const clampAbort = markOwnTimeout(new DOMException("attempt cap", "TimeoutError"));+ expect(classifyConnectionError(clampAbort)).toEqual({+ kind: "unreachable",+ code: "TimeoutError",+ });+ // Provoked, not hand-written: this is what a real `AbortSignal.timeout`+ // rejection looks like, and it must NOT classify.+ const hung = Bun.serve({ port: 0, fetch: () => new Promise<Response>(() => {}) });+ try {+ let raised: unknown;+ try {+ await fetch(`http://127.0.0.1:${hung.port}/`, { signal: AbortSignal.timeout(40) });+ } catch (e) {+ raised = e;+ }+ expect((raised as Error).name).toBe("TimeoutError");+ expect(classifyConnectionError(raised)).toBeNull();+ } finally {+ hung.stop(true);+ }+ });+});++// ───────────────────────────────────────────────────────────────────────────+// `[q] give up`+// ───────────────────────────────────────────────────────────────────────────++describe("[q] give up stops the HOLD, not only the banner", () => {+ /**+ * `bye` suppressed the pane for 60 s and gave up on the episodes alive at+ * that instant — and nothing else. The next request against the same dead+ * target opened a NEW episode, found the pane suppressed, got no pane and no+ * lease, and then held its socket for the full ~4.5-minute deadline with the+ * reason legible NOWHERE before answering 400.+ *+ * Claude Code issues concurrent requests during an outage — the main loop,+ * the title model, subagents — so a request arriving inside the suppression+ * window is the EXPECTED case. Before this feature existed those requests+ * failed in milliseconds.+ */+ test("a request arriving after the key does not hang — it fails fast, as it used to", async () => {+ // The real command, through the real handler the socket server calls.+ __handleCommandForTests({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "bye",+ reason: "user_quit",+ });++ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const req = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });+ const { c, captured } = contextFor(req);++ const started = performance.now();+ await handler.handle(c, PAYLOAD);+ const elapsed = performance.now() - started;++ expect(elapsed).toBeLessThan(1_000);+ expect(captured.status).toBe(400);+ expect(captured.body?.error?.type).toBe("connection_error");+ // No episode: the hold is what the user asked to stop.+ expect(episodeCount()).toBe(0);+ expect((await capturedRecoveryLines()).some((l) => l.includes("skipped (gave-up)"))).toBe(true);+ });++ test("and the suppression expires with the banner's, not before or after it", async () => {+ const clock = new FakeClock(0);+ setRecoveryClock(clock);+ __handleCommandForTests({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "bye",+ reason: "user_quit",+ });+ expect(recoveryGiveUpActive()).toBe(true);++ // One number, read by the hold and by the surface. A minute later the+ // ladder is available again — the key is a pause, not an off switch.+ await clock.advance(UI_SUPPRESS_AFTER_BYE_MS - 1);+ expect(recoveryGiveUpActive()).toBe(true);+ await clock.advance(2);+ expect(recoveryGiveUpActive()).toBe(false);+ });+});diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex 17049fdb..53e7b623 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -45,7 +45,7 @@ import {noteTargetReachable,uiLeaseValid,} from "../recovery/coordinator.js";-import { resolveRecoveryEnabled } from "../recovery/settings.js";+import { recoveryGiveUpActive, resolveRecoveryEnabled } from "../recovery/settings.js";import {type OpenAIImageBlock,type VisionProxyAuthHeaders,@@ -624,8 +624,9 @@ export class ComposedHandler implements ModelHandler {* Should the retry ladder be skipped entirely, answering today's immediate* 400? Returns the REASON, or null to proceed.*- * THREE GATES, AND DELIBERATELY NO FOURTH. Each is a fact available at this- * instant — a request header, configuration, and the clock.+ * FOUR GATES, AND DELIBERATELY NO FIFTH. Each is a fact available at this+ * instant — a request header, configuration, the user's explicit give-up,+ * and the clock.** There is NO special case for a refused loopback address, and that absence* is a decision rather than an omission. An earlier design skipped the ladder@@ -649,6 +650,12 @@ export class ComposedHandler implements ModelHandler {// The CI / scripted-`-p` master switch. Off restores today's behaviour// everywhere, byte for byte.if (!resolveRecoveryEnabled()) return "recovery-disabled";+ // The user pressed `[q] give up`. It has to be read HERE and not only in+ // the UI manager: suppressing the banner alone left the NEXT request —+ // and during an outage Claude Code always has one — holding its socket for+ // the full deadline with the reason legible nowhere. The key says stop, so+ // the hold stops too, for the same window the surface does.+ if (recoveryGiveUpActive()) return "gave-up";// Attempt 1 already spent the budget. A guard, not a hope.if (recoveryClock().now() + MIN_ATTEMPT_SLOT_MS > deadlineAt) return "no-budget";return null;@@ -1290,6 +1297,20 @@ export class ComposedHandler implements ModelHandler {// that finally CONNECTS behave differently from the one that failed —// and at the moment a network returns, N woken waiters would stampede// unqueued into a provider that has just come back.+ //+ // `getRequestInit()` IS CALLED AGAIN, PER ATTEMPT, and that is not a+ // tidiness preference. A transport may return a ONE-SHOT signal from it —+ // `vertex-oauth.ts` returns `AbortSignal.timeout(30000)`, `local.ts` a+ // ten-minute one — and `mergeSignalIntoInit` composes it with the clamp+ // via `AbortSignal.any`. Re-using the hoisted object meant that from 30 s+ // after the FIRST call, every ladder attempt was handed an+ // already-aborted signal and rejected instantly without touching the+ // network: tier 1 silently dead past t+30 s, the request held for the+ // full ~270 s deadline making ZERO real connect attempts, while the+ // `[Recovery]` log and the pane both reported attempts that never left+ // the process. A transport ceiling that must survive across attempts has+ // to be RE-MINTED, never re-used. `doParamRetry` and `doAuthRetry` below+ // already call it freshly per attempt; this is the same rule.const doFetchWith = (sig: AbortSignal) =>fetch(endpoint,@@ -1298,7 +1319,7 @@ export class ComposedHandler implements ModelHandler {method: "POST",headers,body: serialized?.body ?? JSON.stringify(requestPayload),- ...requestInit,+ ...(this.provider.getRequestInit?.() || {}),},sig)diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.tsindex 7ccab88d..9a5f5290 100644--- a/packages/cli/src/handlers/recovery-status.test.ts+++ b/packages/cli/src/handlers/recovery-status.test.ts@@ -647,31 +647,60 @@ describe("the client's re-POST rejoins the SAME episode", () => {// The client's retry budget// ───────────────────────────────────────────────────────────────────────────-describe("CLAUDE_CODE_RETRY_WATCHDOG follows the recovery UI switch", () => {- const KEY = "CLAUDISH_RECOVERY_UI";- let saved: string | undefined;+describe("CLAUDE_CODE_RETRY_WATCHDOG needs all three gates", () => {+ const UI = "CLAUDISH_RECOVERY_UI";+ const ON = "CLAUDISH_RECOVERY";+ let savedUi: string | undefined;+ let savedOn: string | undefined;beforeEach(() => {- saved = process.env[KEY];+ savedUi = process.env[UI];+ savedOn = process.env[ON];});afterEach(() => {- if (saved === undefined) delete process.env[KEY];- else process.env[KEY] = saved;+ if (savedUi === undefined) delete process.env[UI];+ else process.env[UI] = savedUi;+ if (savedOn === undefined) delete process.env[ON];+ else process.env[ON] = savedOn;});- test("set when the UI is enabled — this is what makes the reach ~a day", () => {- delete process.env[KEY];- expect(retryWatchdogEnv()).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });+ test("set when the UI is enabled and a pane is reachable — the reach is ~a day", () => {+ delete process.env[UI];+ delete process.env[ON];+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });});test("absent when the user turned the UI off", () => {// No pane means no lease means an inline 400 at exhaustion means nothing to// hand back — so extending the client's budget would buy only the side// effect on claudish's other 503s.- process.env[KEY] = "0";- expect(retryWatchdogEnv()).toEqual({});- expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv()).toBe(false);+ delete process.env[ON];+ process.env[UI] = "0";+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});+ expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv({ paneEligible: true })).toBe(false);+ });++ test("absent when THIS LAUNCH can never obtain a pane — `-p`, no TTY, no magmux", () => {+ // The gate that was missing. Those launches cannot hold the lease a+ // recovery 503 requires, so the watchdog would apply the accepted+ // duplicate-request exposure to every UNRELATED 503 the session sees and+ // return nothing for it: ~300 attempts on an upstream overload, for hours,+ // in exactly the configurations that opted out.+ delete process.env[UI];+ delete process.env[ON];+ expect(retryWatchdogEnv({ paneEligible: false })).toEqual({});+ });++ test("absent under --no-recovery, however the UI is configured", () => {+ // RISK-7's promise is that `--no-recovery` restores pre-recovery behaviour+ // byte for byte. A ladder that never runs hands nothing back, so a client+ // looping 300 times is pure cost — and CI is where this switch lives.+ process.env[ON] = "0";+ delete process.env[UI];+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});+ process.env[UI] = "1";+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});});});diff --git a/packages/cli/src/handlers/shared/connection-error.ts b/packages/cli/src/handlers/shared/connection-error.tsindex b04f08f9..eaf91bfc 100644--- a/packages/cli/src/handlers/shared/connection-error.ts+++ b/packages/cli/src/handlers/shared/connection-error.ts@@ -49,20 +49,43 @@ const BUN_CONNECT_MESSAGE = /unable to connect\. is the computer able to access/*** Error NAMES that mean the same thing as a connect code, matched separately- * because they do not arrive as one.+ * because they do not arrive as one — AND ONLY WHEN THE SIGNAL WAS OURS.** `AbortSignal.timeout(...)` rejects with a `DOMException` whose `name` is* `TimeoutError` and whose `code` is the NUMBER 23 — so `findConnectionCode`'s* `typeof e.code === "string"` test skipped it, its message ("The operation* timed out.") matched no fallback, and classification returned **null**. A- * provider that accepts the TCP connection and then never answers therefore- * fell straight through into a bare 500, with none of the actionable wording- * every other unreachable-host failure gets.+ * REACHABILITY PROBE that hung therefore fell straight through into a bare+ * 500, with none of the actionable wording every other unreachable-host+ * failure gets.*- * "Refused is covered, hung is not" was the whole of the gap. It bites hardest- * on local providers, whose health probes carry an `AbortSignal.timeout` and- * whose inference requests carry a ten-minute one — a wedged Ollama is exactly- * the fault a retry ladder is for.+ * ── WHY THE NAME ALONE IS NOT THE DISCRIMINATOR ─────────────────────────────+ *+ * `TimeoutError` is also what a TRANSPORT'S OWN request ceiling rejects with.+ * `vertex-oauth.ts` puts `AbortSignal.timeout(30000)` on the *inference* call+ * and `local.ts` a ten-minute one. Classifying by name alone turned "the model+ * took longer than its transport's ceiling" — a LATENCY event, ordinary for a+ * thinking model on a streaming endpoint — into `unreachable`: it entered the+ * retry ladder, took the tier-2 handoff, and with `CLAUDE_CODE_RETRY_WATCHDOG`+ * on it was re-POSTed ~300 times, each one running ONE MORE REAL BILLED+ * INFERENCE against a host that had answered the TCP connect and was already+ * generating. RISK-6 (`network-recovery.md` §7) was accepted for+ * `ECONNRESET`/`EPIPE` on a genuine network fault. It was never accepted for a+ * latency event, which is neither a network fault nor transient — and+ * `buildConnectionErrorMessage` then told the user to check their VPN and DNS+ * for a host that was mid-generation.+ *+ * So the discriminator is the SIGNAL'S ORIGIN, carried on the error as an own+ * property, not the name:+ *+ * - the ladder's per-attempt clamp (`transient-retry.ts`) — WE gave up on+ * this attempt, and re-issuing is the entire point;+ * - a reachability PROBE we issued with our own short timeout (`local.ts`'s+ * 5 s `/api/tags`) — WE asked "is anything there" and got no answer.+ *+ * Both mean "we could not get a response out of this host inside a window WE+ * chose". A transport's inference ceiling means something else, so it keeps+ * its pre-recovery route out of here: unclassified, rethrown, untouched.** ONLY `TimeoutError`. `AbortError` is deliberately absent and must stay* absent: it is how a client says it has gone away and how our own per-attempt@@ -73,6 +96,43 @@ const NAME_KIND: Record<string, ConnectionErrorKind> = {TimeoutError: "unreachable",};+/**+ * The own-property that says "this timeout is CLAUDISH'S OWN".+ *+ * Non-enumerable on purpose: it must not change how an error serialises into a+ * log, a stats record or an upstream error body.+ */+const OWN_TIMEOUT_FLAG = "claudishOwnTimeout";++/**+ * Tag an error as raised by a timeout CLAUDISH set on a reachability question+ * of its own — the retry ladder's per-attempt clamp, or a health probe.+ *+ * Returns the same object, so it composes inside an `abort(...)` call or an+ * assignment. Defensive about frozen errors: a failure to tag degrades to+ * "unclassified", which is the safe direction (no ladder, no re-issue).+ */+export function markOwnTimeout<T>(error: T): T {+ if (error && typeof error === "object") {+ try {+ Object.defineProperty(error, OWN_TIMEOUT_FLAG, {+ value: true,+ enumerable: false,+ configurable: true,+ writable: true,+ });+ } catch {+ /* frozen — falls through to unclassified, which is the safe direction */+ }+ }+ return error;+}++/** Was this error raised by one of OUR timeouts? See `markOwnTimeout`. */+function isOwnTimeout(error: unknown): boolean {+ return Boolean((error as Record<string, unknown> | null | undefined)?.[OWN_TIMEOUT_FLAG]);+}+/*** Walk an error and its `cause` chain (undici's `TypeError: fetch failed` wraps* the real syscall error in `.cause`) and return the first known connection@@ -89,7 +149,11 @@ function findConnectionCode(error: unknown): string | null {// on it. Walked at the same depth as `.code`, because a transport that// rethrows with `{ cause }` buries a hung-probe timeout just as deeply as// it buries a refusal.- if (typeof e.name === "string" && e.name in NAME_KIND) return e.name;+ //+ // `isOwnTimeout(e)` is the load-bearing half: a `TimeoutError` raised by a+ // transport's own INFERENCE ceiling is not a reachability fact and must not+ // enter the ladder. See `NAME_KIND`'s header for what that cost.+ if (typeof e.name === "string" && e.name in NAME_KIND && isOwnTimeout(e)) return e.name;e = e.cause;}const msg = String((error as any)?.message ?? error ?? "");diff --git a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.tsindex 5d29e155..9353031a 100644--- a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts+++ b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts@@ -40,7 +40,7 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";import { closeAllEpisodes } from "../../recovery/coordinator.js";import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";-import { classifyConnectionError } from "./connection-error.js";+import { classifyConnectionError, markOwnTimeout } from "./connection-error.js";import {PER_ATTEMPT_CONNECT_CAP_MS,mergeSignalIntoInit,@@ -167,9 +167,12 @@ describe("F-SLOW — a host that swallows the connection", () => {// Why the ladder can treat its own timeout as a retryable network fault:// `AbortSignal.timeout` rejects with a DOMException whose `name` is// `TimeoutError` and whose `code` is the NUMBER 23, so a classifier keyed- // only on a string `code` returned null and a hung upstream fell through- // into a bare 500.- const clampAbort = new DOMException("attempt cap", "TimeoutError");+ // only on a string `code` returned null and a hung probe fell through into+ // a bare 500.+ //+ // `markOwnTimeout` is what the clamp adds — see the sibling test below for+ // why the NAME alone must not be enough.+ const clampAbort = markOwnTimeout(new DOMException("attempt cap", "TimeoutError"));const conn = classifyConnectionError(clampAbort);expect(conn).not.toBeNull();expect(conn?.kind).toBe("unreachable");diff --git a/packages/cli/src/handlers/shared/transient-retry.test.ts b/packages/cli/src/handlers/shared/transient-retry.test.tsindex addcbbcc..a024b81c 100644--- a/packages/cli/src/handlers/shared/transient-retry.test.ts+++ b/packages/cli/src/handlers/shared/transient-retry.test.ts@@ -568,3 +568,76 @@ describe("mergeSignalIntoInit composes rather than replaces", () => {expect(mergeSignalIntoInit(bare, undefined)).toBe(bare as unknown as RequestInit);});});++// ───────────────────────────────────────────────────────────────────────────+// The clamp binds operations that cannot take a signal+// ───────────────────────────────────────────────────────────────────────────++describe("the per-attempt clamp bounds an op that IGNORES its signal", () => {+ /**+ * The fetch path threads the signal into `fetch`. THE AUTH PATH CANNOT:+ * `refreshAuth()` and `getHeaders()` take no arguments, so the handler's+ * `() => this.provider.refreshAuth!()` discards the signal, and Grok's token+ * exchange then performs an unbounded `fetch(auth.x.ai/oauth2/token)`. A+ * swallowed connection there outlived the 45-second cap AND the client's own+ * disconnect, while this function sat awaiting a promise nothing could+ * settle — the unbounded hold this whole feature exists to remove, reached+ * from inside the machinery that removes it.+ *+ * A deadline one path can ignore is not a deadline, so it is enforced at the+ * one place that owns it.+ */+ test("an attempt that never settles ends at the cap and the ladder goes on", async () => {+ useFakeClock();+ const started: number[] = [];+ let settled = 0;+ // Signal ignored entirely, exactly like `() => provider.refreshAuth!()`.+ const deafOp = async () => {+ started.push(clock.now());+ if (started.length === 1) {+ await new Promise(() => {}); // never settles, never observes the signal+ }+ settled++;+ throw CONNECT_FAILURE;+ };++ const result = await advanceUntilSettled(+ clock,+ withConnectionRetry(deafOp, CONNECT_FAILURE, ctx({ deadlineAt: 200_000 })),+ 400_000+ );++ // Attempt 1 was abandoned at the cap rather than awaited forever, so the+ // ladder reached its later rungs and the request answered inside its+ // budget. Without the enforcement this promise never resolves at all.+ expect(started.length).toBeGreaterThan(1);+ // Attempt 1 ended AT THE CAP — not at the deadline, and not never — and the+ // next rung's own gap follows it.+ expect((started[1] as number) - (started[0] as number)).toBe(+ PER_ATTEMPT_CONNECT_CAP_MS + connectionRetryDelayMs(1)+ );+ expect(result.kind).toBe("exhausted");+ expect(settled).toBeGreaterThan(0);+ }, 15_000);++ test("a client disconnect releases such an op too, not only a fetch", async () => {+ useFakeClock();+ const ac = new AbortController();+ const deafOp = async () => {+ ac.abort(new DOMException("client gone", "AbortError"));+ await new Promise(() => {});+ return "never" as unknown;+ };++ const result = await advanceUntilSettled(+ clock,+ withConnectionRetry(deafOp, CONNECT_FAILURE, ctx({ signal: ac.signal })),+ 400_000+ );++ // NFR-2 for the auth path: the socket is already gone, so the waiter must+ // unwind now rather than when an unbounded auth fetch eventually gives up.+ expect(result.kind).toBe("client_gone");+ expect(episodeCount()).toBe(0);+ }, 15_000);+});diff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.tsindex 5c6043ba..9cc7bb36 100644--- a/packages/cli/src/handlers/shared/transient-retry.ts+++ b/packages/cli/src/handlers/shared/transient-retry.ts@@ -33,6 +33,7 @@ import {type ConnectionErrorKind,buildConnectionErrorMessage,classifyConnectionError,+ markOwnTimeout,} from "./connection-error.js";export { EPISODE_GRACE_MS } from "../../recovery/coordinator.js";@@ -142,6 +143,19 @@ export interface ConnectionErrorInfo {* overwriting is a ten-minute `AbortSignal.timeout`. Compose rather than* replace: the per-attempt clamp then bounds the attempt while the transport's* own ceiling still applies, which is what both of them were for.+ *+ * ── AN ALREADY-FIRED `own` IS DROPPED, NOT COMPOSED ─────────────────────────+ *+ * `AbortSignal.any([fired, live])` returns a signal that is ALREADY ABORTED, so+ * the fetch rejects before it opens a socket — measured on this machine's Bun.+ * A transport's `getRequestInit()` signal is one-shot, so a caller that hoists+ * the init out of its retry loop poisons every later attempt with a ceiling+ * that expired during attempt 1: real connect attempts stop happening while the+ * ladder, the log and the pane all go on reporting them.+ *+ * Callers must re-mint the init per attempt (`composed-handler.ts` does), and+ * this is the belt behind that brace: an expired ceiling has already had its+ * say, and carrying it forward can only convert a live attempt into a no-op.*/export function mergeSignalIntoInit(init: Record<string, unknown>,@@ -149,7 +163,8 @@ export function mergeSignalIntoInit(): RequestInit {if (!signal) return init as RequestInit;const own = init.signal as AbortSignal | undefined;- return { ...init, signal: own ? AbortSignal.any([own, signal]) : signal } as RequestInit;+ if (!own || own.aborted) return { ...init, signal } as RequestInit;+ return { ...init, signal: AbortSignal.any([own, signal]) } as RequestInit;}export interface ConnectionRetryContext {@@ -234,6 +249,54 @@ function isClientAbort(signal: AbortSignal, err: unknown): boolean {return (err as { name?: string })?.name === "AbortError";}+/**+ * Await `p`, but stop waiting when `signal` fires.+ *+ * ── WHY THE CLAMP CANNOT JUST BE HANDED OVER AND TRUSTED ────────────────────+ *+ * `op` takes the signal, and the fetch path threads it all the way to `fetch`.+ * THE AUTH PATH CANNOT: `refreshAuth()` and `getHeaders()` take no arguments on+ * the transport interface, so `() => this.provider.refreshAuth!()` closes over+ * nothing and discards the signal entirely. Grok's token exchange then performs+ * an UNBOUNDED `fetch(auth.x.ai/oauth2/token)` — so a swallowed connection+ * outlived both the 45-second attempt cap and the client's own disconnect,+ * while `withConnectionRetry` sat awaiting a promise nothing could settle. A+ * deadline that one path can ignore is not a deadline.+ *+ * So the clamp is enforced HERE, at the one place that owns it, rather than in+ * N transports that would each have to remember. The operation is abandoned,+ * not cancelled — it may still be running — but the waiter unwinds, the ladder+ * advances, and the request answers inside its budget, which is what the budget+ * was for. Threading a real signal into every auth call is the deeper fix and+ * is worth doing; this is what makes the ceiling true in the meantime.+ */+function untilAborted<T>(p: Promise<T>, signal: AbortSignal): Promise<T> {+ if (signal.aborted) {+ // Nothing else will ever observe `p`, so claim its rejection here or Bun+ // reports an unhandled one.+ void p.catch(() => {});+ return Promise.reject(signal.reason ?? new DOMException("aborted", "AbortError"));+ }+ return new Promise<T>((resolve, reject) => {+ const onAbort = () => {+ reject(signal.reason ?? new DOMException("aborted", "AbortError"));+ };+ signal.addEventListener("abort", onAbort, { once: true });+ // Registered unconditionally, so a late settle of an abandoned operation is+ // always handled — `resolve`/`reject` after settlement are no-ops.+ p.then(+ (v) => {+ signal.removeEventListener("abort", onAbort);+ resolve(v);+ },+ (e) => {+ signal.removeEventListener("abort", onAbort);+ reject(e);+ }+ );+ });+}+/*** Re-issue `op` on a classified connection error, on the shared episode clock.*@@ -241,6 +304,8 @@ function isClientAbort(signal: AbortSignal, err: unknown): boolean {* single contract cover both the fetch path and the auth path: the per-attempt* clamp is built here, merged with the client signal here, and handed in, so* the clamp bounds the OPERATION rather than only the `fetch` buried inside it.+ * `untilAborted` then makes that binding rather than advisory, for the auth+ * closures that structurally cannot accept a signal at all.** Classification runs on EVERY attempt, not just the first. A retry that throws* an auth failure, a transport bug or a programming error must not be@@ -348,15 +413,19 @@ export async function withConnectionRetry<T>(MIN_ATTEMPT_SLOT_MS,Math.min(PER_ATTEMPT_CONNECT_CAP_MS, remaining));+ // `markOwnTimeout` is what makes this abort re-classifiable at all.+ // `classifyConnectionError` keys `TimeoutError` on the SIGNAL'S ORIGIN+ // rather than on the name, precisely so a transport's own inference+ // ceiling stays out of the ladder — so the clamp has to say it is ours.const clampTimer = clock.setTimeout(- () => attemptAc.abort(new DOMException("attempt cap", "TimeoutError")),+ () => attemptAc.abort(markOwnTimeout(new DOMException("attempt cap", "TimeoutError"))),clampMs);const merged = AbortSignal.any([ctx.signal, attemptAc.signal]);try {requestRetries++;- const value = await op(merged);+ const value = await untilAborted(op(merged), merged);handle.recordAttemptResult("ok", true);log(`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +diff --git a/packages/cli/src/launcher/magmux-wrapper.test.ts b/packages/cli/src/launcher/magmux-wrapper.test.tsindex a7e05d3b..65527496 100644--- a/packages/cli/src/launcher/magmux-wrapper.test.ts+++ b/packages/cli/src/launcher/magmux-wrapper.test.ts@@ -10,13 +10,14 @@*/import { afterEach, describe, expect, test } from "bun:test";-import { mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";+import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import {STRIPPED_CHILD_VARS,buildLauncherScript,envDeltas,+ magmuxPaneCapability,planMagmuxWrap,} from "./magmux-wrapper.js";@@ -195,7 +196,7 @@ describe("planMagmuxWrap", () => {expect(script).not.toContain("sk-inherited-secret");expect(script).not.toContain("PATH=");expect(statSync(plan?.scriptPath as string).mode & 0o777).toBe(0o600);- expect(statSync(join(root, "claudish-launch-4242")).mode & 0o777).toBe(0o700);+ expect(statSync(dirname(plan?.scriptPath as string)).mode & 0o777).toBe(0o700);});test("MAGMUX_SCROLLBACK defaults to 10000 but never overrides the user's own", () => {@@ -238,3 +239,141 @@ describe("planMagmuxWrap", () => {expect(plan?.paneExitCode()).toBeNull();});});++// ───────────────────────────────────────────────────────────────────────────+// The launcher script is a SECRET on disk+// ───────────────────────────────────────────────────────────────────────────++describe("the script's directory is unguessable and cannot be pre-created", () => {+ /**+ * The file this guards is sourced by the LOGIN SHELL immediately before+ * `exec`, and it carries claudish's env deltas — including, on the proxy+ * path, the `ANTHROPIC_API_KEY` Claude Code authenticates to us with.+ *+ * It used to live at `<tmp>/claudish-launch-<pid>`, created with+ * `mkdirSync(..., { recursive: true })`. `recursive: true` neither throws on+ * `EEXIST` nor applies `mode` to a directory that already exists, and the+ * name contains only the pid, which is enumerable — so on a multi-user host+ * another uid could pre-create (or symlink) that path and have it adopted+ * without a word, then read the session's secrets or swap the script between+ * the write and the `exec`. CWE-377 / CWE-59.+ *+ * The rest of this change already knew the rule: `socket-server.ts` uses+ * `randomBytes(12)` for its directory and `O_EXCL` for the pane lock.+ */+ test("the path does not contain the pid, and two plans never collide", () => {+ const root = scratch();+ const a = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: root });+ const b = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: root });++ // Same pid, same tmpRoot, different directory: the name is random, so it+ // cannot be pre-created by someone who knows the pid.+ expect(dirname(a?.scriptPath as string)).not.toBe(dirname(b?.scriptPath as string));+ expect(a?.scriptPath).not.toContain(String(BASE.pid));+ expect(statSync(dirname(a?.scriptPath as string)).mode & 0o777).toBe(0o700);+ expect(statSync(a?.scriptPath as string).mode & 0o777).toBe(0o600);+ });++ test("a directory planted at the OLD predictable path is never adopted", () => {+ // The attack, verbatim: another uid creates the path first — world-writable+ // or a symlink into its own tree — and waits for claudish to write the+ // session's environment into it. `recursive: true` adopted it silently and+ // left its mode alone, and `writeFileSync` follows whatever is already+ // there. `mkdtempSync` + `wx` makes both impossible.+ const root = scratch();+ const planted = join(root, `claudish-launch-${BASE.pid}`);+ mkdirSync(planted, { recursive: true, mode: 0o777 });+ const decoy = join(planted, "claude-launch.sh");+ writeFileSync(decoy, "# planted");++ const plan = planMagmuxWrap({+ ...BASE,+ childEnv: { ANTHROPIC_API_KEY: "sk-session-secret" },+ tmpRoot: root,+ });++ expect(dirname(plan?.scriptPath as string)).not.toBe(planted);+ // The secret did not land anywhere the planter can read.+ expect(readFileSync(decoy, "utf-8")).toBe("# planted");+ expect(readFileSync(plan?.scriptPath as string, "utf-8")).toContain("sk-session-secret");+ expect(statSync(dirname(plan?.scriptPath as string)).mode & 0o777).toBe(0o700);+ });++ test("a plan that is never watched still takes its secret off disk at exit", () => {+ // `watch()` cleans up on the CHILD's exit, which covers the ordinary path+ // only. A SIGTERM to claudish, or a launch that never spawns, would+ // otherwise leave the key in /tmp.+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ expect(process.listeners("exit").length).toBeGreaterThan(0);+ plan?.cleanup();+ expect(() => statSync(plan?.scriptPath as string)).toThrow();+ // Idempotent, and it unhooks itself: a long-lived host must not accumulate+ // one exit listener per launch.+ plan?.cleanup();+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Eligibility, asked separately because the watchdog has to ask it EARLY+// ───────────────────────────────────────────────────────────────────────────++describe("magmuxPaneCapability", () => {+ test("wrap when interactive, on a TTY, with a binary and no ambient socket", () => {+ expect(+ magmuxPaneCapability({+ interactive: true,+ stdoutIsTty: true,+ magmuxBinary: "/opt/homebrew/bin/magmux",+ parentEnv: {},+ })+ ).toEqual({ kind: "wrap", magmux: "/opt/homebrew/bin/magmux" });+ });++ test("ambient inside someone else's magmux — a pane is reachable without wrapping", () => {+ // `team --grid --mode interactive`, or a user who launched claudish in a+ // pane by hand. Nothing to wrap, but there IS a multiplexer to ask.+ expect(+ magmuxPaneCapability({+ interactive: true,+ stdoutIsTty: true,+ magmuxBinary: "/opt/homebrew/bin/magmux",+ parentEnv: { MAGMUX_SOCK: "/tmp/magmux-someone-else.sock" },+ })+ ).toEqual({ kind: "ambient", sock: "/tmp/magmux-someone-else.sock" });+ });++ test("none for `-p`, for a pipe, and for a machine without magmux", () => {+ const env = {};+ expect(+ magmuxPaneCapability({ interactive: false, stdoutIsTty: true, parentEnv: env }).kind+ ).toBe("none");+ expect(+ magmuxPaneCapability({ interactive: true, stdoutIsTty: false, parentEnv: env }).kind+ ).toBe("none");+ expect(+ magmuxPaneCapability({+ interactive: true,+ stdoutIsTty: true,+ magmuxBinary: null,+ parentEnv: env,+ }).kind+ ).toBe("none");+ });++ test("it is the SAME predicate planMagmuxWrap uses — they cannot drift", () => {+ // The watchdog is decided from `magmuxPaneCapability` before the child+ // environment is finalised; the wrap is decided from `planMagmuxWrap`+ // afterwards. Two independent gates would mean a launch that exports the+ // ~300-attempt retry budget and then turns out to have no surface for it.+ for (const input of [+ { interactive: false, stdoutIsTty: true },+ { interactive: true, stdoutIsTty: false },+ { interactive: true, stdoutIsTty: true, magmuxBinary: null },+ { interactive: true, stdoutIsTty: true },+ ] as const) {+ const shape = { ...BASE, ...input, childEnv: {}, tmpRoot: scratch(), parentEnv: {} };+ const wrapped = planMagmuxWrap(shape) !== null;+ expect(wrapped).toBe(magmuxPaneCapability(shape).kind === "wrap");+ }+ });+});diff --git a/packages/cli/src/launcher/magmux-wrapper.ts b/packages/cli/src/launcher/magmux-wrapper.tsindex a1c6da55..0c4b5a5b 100644--- a/packages/cli/src/launcher/magmux-wrapper.ts+++ b/packages/cli/src/launcher/magmux-wrapper.ts@@ -35,7 +35,16 @@*/import type { ChildProcess } from "node:child_process";-import { chmodSync, existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import {+ chmodSync,+ closeSync,+ existsSync,+ mkdirSync,+ mkdtempSync,+ openSync,+ rmSync,+ writeFileSync,+} from "node:fs";import { type Socket, connect as netConnect } from "node:net";import { join } from "node:path";import { findMagmuxBinaryOrNull } from "./magmux-binary.js";@@ -140,14 +149,34 @@ export function buildLauncherScript(}/**- * Build the plan, or null when this launch must not be wrapped.+ * Can this launch obtain a recovery pane at all, and how?*- * Returns null — never throws — for every "not applicable" case, because a- * missing multiplexer means the session should launch exactly as it did before- * this feature existed.+ * SIDE-EFFECT FREE — it writes nothing, spawns nothing, and may therefore be+ * asked BEFORE the child environment is finalised. That ordering is the whole+ * reason it exists as its own function: `CLAUDE_CODE_RETRY_WATCHDOG` expands+ * Claude Code's retry budget from ~11 attempts to ~300 for EVERY 503 the+ * session sees, so it may only be exported by a launch that can actually put+ * the reason on a surface. Exporting it from the UI preference alone applied+ * the accepted duplicate-request exposure (RISK-6) in `-p`, under+ * `--no-recovery`, and on machines without magmux — the configurations that+ * opted out — where it buys nothing at all.+ *+ * `wrap` — we will start magmux ourselves and own the control socket.+ * `ambient` — already inside someone else's magmux (`team --grid`, or a user+ * who launched claudish in a pane): nothing to wrap, but there IS a+ * multiplexer to ask for a pane.+ * `none` — no surface is reachable from this launch.*/-export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {- if (!input.interactive || !input.stdoutIsTty) return null;+export type MagmuxPaneCapability =+ | { kind: "wrap"; magmux: string }+ | { kind: "ambient"; sock: string }+ | { kind: "none"; reason: "not-interactive" | "no-tty" | "no-magmux" };++export function magmuxPaneCapability(+ input: Pick<MagmuxWrapInput, "interactive" | "stdoutIsTty" | "parentEnv" | "magmuxBinary">+): MagmuxPaneCapability {+ const parentEnv = input.parentEnv ?? process.env;+ if (!input.interactive) return { kind: "none", reason: "not-interactive" };// ALREADY INSIDE A MULTIPLEXER — do not nest one.//// `team --grid --mode interactive` launches one `claudish --model X -i` per@@ -160,32 +189,72 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {// installed against that AMBIENT socket instead, and the cross-process// `O_EXCL` lock in `magmux-ui.ts` is what keeps the grid to ONE banner rather// than N.- if ((input.parentEnv ?? process.env).MAGMUX_SOCK) return null;+ //+ // Checked BEFORE the TTY gate on purpose, to match `claude-runner.ts`'s+ // ambient branch, which asks only for `interactive && MAGMUX_SOCK`.+ const ambient = parentEnv.MAGMUX_SOCK;+ if (ambient) return { kind: "ambient", sock: ambient };+ if (!input.stdoutIsTty) return { kind: "none", reason: "no-tty" };const magmux = input.magmuxBinary === undefined ? findMagmuxBinaryOrNull() : input.magmuxBinary;- if (!magmux) return null;+ if (!magmux) return { kind: "none", reason: "no-magmux" };+ return { kind: "wrap", magmux };+}++/**+ * Build the plan, or null when this launch must not be wrapped.+ *+ * Returns null — never throws — for every "not applicable" case, because a+ * missing multiplexer means the session should launch exactly as it did before+ * this feature existed. The "not applicable" set is `magmuxPaneCapability`'s,+ * so the gate that decides the watchdog and the gate that decides the wrap+ * cannot drift apart.+ */+export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {+ const capability = magmuxPaneCapability(input);+ if (capability.kind !== "wrap") return null;+ const magmux = capability.magmux;const pid = input.pid ?? process.pid;const id = `claudish-${pid}`;const controlSocket = `/tmp/magmux-${id}.sock`;- // 0700 directory, 0600 file. The script carries claudish's deltas, which on- // the proxy path include the key Claude Code authenticates to us with.+ // ── 0700 UNGUESSABLE directory, 0600 O_EXCL file ──────────────────────────+ //+ // The file below is SOURCED by the login shell immediately before `exec`, and+ // it carries claudish's env deltas — on the proxy path that includes the+ // `ANTHROPIC_API_KEY` Claude Code authenticates to us with. So both halves of+ // this matter, and the previous shape got both wrong:+ //+ // - the name was `claudish-launch-<pid>`, which is ENUMERABLE, and+ // `mkdirSync(..., { recursive: true })` neither throws on `EEXIST` nor+ // applies `mode` to a directory that already exists — so a directory+ // another uid had pre-created (or symlinked elsewhere) was adopted+ // silently, and `writeFileSync` follows an existing path. That is+ // CWE-377/CWE-59: read the session's secrets, or swap the script between+ // the write and the `exec`, and get code execution as the claudish user.+ // - `writeFileSync`'s `mode` is a CREATION mode; on an existing file it is+ // ignored, so the `chmod` was load-bearing and silent when it failed.+ //+ // `mkdtempSync` gives an unguessable name AND fails rather than adopting, and+ // `wx` is `O_CREAT | O_EXCL`, so the file cannot pre-exist as a symlink to+ // somewhere else. The same rule the rest of this change already follows:+ // `socket-server.ts` uses `randomBytes(12)` for its directory and `O_EXCL`+ // for the pane lock.const tmpRoot = input.tmpRoot ?? "/tmp";- const scriptDir = join(tmpRoot, `claudish-launch-${pid}`);- mkdirSync(scriptDir, { mode: 0o700, recursive: true });+ mkdirSync(tmpRoot, { recursive: true });+ const scriptDir = mkdtempSync(join(tmpRoot, "claudish-launch-"));+ chmodSync(scriptDir, 0o700);// Named for Claude Code on purpose: magmux attaches its controller by// spotting `claude`/`claudish` in the pane's command.const scriptPath = join(scriptDir, "claude-launch.sh");const parentEnv = input.parentEnv ?? process.env;const deltas = envDeltas(parentEnv, input.childEnv);- writeFileSync(scriptPath, buildLauncherScript(input.claudeBinary, input.claudeArgs, deltas), {- mode: 0o600,- });+ const fd = openSync(scriptPath, "wx", 0o600);try {- chmodSync(scriptPath, 0o600);- } catch {- /* the 0700 directory is the real gate */+ writeFileSync(fd, buildLauncherScript(input.claudeBinary, input.claudeArgs, deltas));+ } finally {+ closeSync(fd);}const env: Record<string, string> = { ...input.childEnv };@@ -214,8 +283,12 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {let paneExit: number | null = null;const exitListeners: Array<(code: number | null) => void> = [];let control: Socket | null = null;+ let cleaned = false;const cleanup = () => {+ if (cleaned) return;+ cleaned = true;+ process.removeListener("exit", cleanup);try {control?.destroy();} catch {@@ -236,6 +309,13 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {}};+ // A file holding the session's `ANTHROPIC_API_KEY` must not outlive the+ // process on a SIGTERM or a crash-out — `watch()`'s `proc.exit` hook only+ // covers the ordinary path, and claudish is not always the one that dies+ // last. Same discipline as `socket-server.ts`'s socket unlink; the listener+ // removes itself in `cleanup` so a long-lived host cannot accumulate them.+ process.on("exit", cleanup);+return {command: magmux,args,diff --git a/packages/cli/src/profile-config.ts b/packages/cli/src/profile-config.tsindex cb8bcedc..924b6c45 100644--- a/packages/cli/src/profile-config.ts+++ b/packages/cli/src/profile-config.ts@@ -264,10 +264,22 @@ export interface ClaudishProfileConfig {* Whether claudish may own a terminal surface on which a recovery episode's* reason is legible. Default ON.*- * In this phase it decides one thing only: a REFUSED connection to a- * loopback endpoint (a stopped Ollama) skips the ladder entirely when the- * user has turned this off, because nothing could ever display why the- * request is being held and they have said they do not want it to be.+ * What it actually decides, and the blast radius is larger than "a banner":+ *+ * - whether the interactive session is WRAPPED in a magmux pane at launch+ * (`claude-runner.ts` → `planMagmuxWrap`);+ * - whether `CLAUDE_CODE_RETRY_WATCHDOG=1` is exported to Claude Code,+ * which moves its retry budget from ~11 attempts to ~300 for every 503+ * the session sees (it is one of three gates — see `retryWatchdogEnv`);+ * - and through those, whether an exhausted hold can ever answer a+ * retryable 503 rather than an inline 400, since the 503 requires a live+ * pane lease.+ *+ * It never decides whether a RETRY happens. There is no loopback carve-out+ * and its absence is deliberate (`composed-handler.ts`'s `shouldSkipTier1`);+ * an earlier draft of this comment described one, and it had already been+ * deleted.+ ** Precedence: --recovery-ui/--no-recovery-ui flag > CLAUDISH_RECOVERY_UI env* > project `.claudish.json` > this field > true. The scoped read is* `readRecoveryUi()`, NOT this allowlist — see its doc comment.diff --git a/packages/cli/src/providers/grok/grok-credentials.test.ts b/packages/cli/src/providers/grok/grok-credentials.test.tsindex 25284a72..62cdf449 100644--- a/packages/cli/src/providers/grok/grok-credentials.test.ts+++ b/packages/cli/src/providers/grok/grok-credentials.test.ts@@ -3,6 +3,7 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";import { GROK_PUBLIC_CLIENT_ID } from "../../auth/grok-oauth.js";+import { classifyConnectionError } from "../../handlers/shared/connection-error.js";import {DEFAULT_GROK_PROXY_URL,FALLBACK_GROK_CLIENT_VERSION,@@ -599,3 +600,38 @@ describe("forceRefreshGrokAccessToken", () => {expect(successfulFetch).toHaveBeenCalledTimes(1);});});++describe("a refused token exchange names the AUTH host, not the inference host", () => {+ test("the error carries claudishEndpoint = the token endpoint", async () => {+ // `getHeaders()` is a `gk@` request's FIRST network touch and it reaches+ // `auth.x.ai` — never `api.x.ai`. `ComposedHandler` falls back to the MODEL+ // endpoint when the error carries no `claudishEndpoint`, so without this+ // the recovery banner, the `[Recovery]` log and the episode key all named a+ // host that was never contacted, and told the user to check their network+ // for it. `local.ts` attaches the same property for the same reason.+ writeAuth({ [OIDC_SCOPE]: expiredEntry() });+ let connectFailure: unknown;+ try {+ await realFetch("http://127.0.0.1:1/refused-on-purpose");+ } catch (e) {+ connectFailure = e;+ }+ globalThis.fetch = mock(async () => {+ throw connectFailure;+ }) as unknown as typeof fetch;++ const error = await rejectedError(resolveGrokAccessToken());++ expect((error as { claudishEndpoint?: string }).claudishEndpoint).toBe(+ "https://auth.x.ai/oauth2/token"+ );+ // And the syscall evidence still rides in `cause`, so this remains a+ // CONNECTION failure rather than an auth one — a network fault during a+ // refresh must not walk the user down the fallback chain onto metered+ // billing.+ expect(classifyConnectionError(error)).toEqual({+ kind: "refused",+ code: "ConnectionRefused",+ });+ });+});diff --git a/packages/cli/src/providers/grok/grok-credentials.ts b/packages/cli/src/providers/grok/grok-credentials.tsindex f0d0f175..c56a29c3 100644--- a/packages/cli/src/providers/grok/grok-credentials.ts+++ b/packages/cli/src/providers/grok/grok-credentials.ts@@ -430,9 +430,18 @@ async function performRefresh(cred: GrokCredential): Promise<string> {// FIRST network touch of a `gk@` request (ComposedHandler calls getHeaders()// before the upstream fetch), so an unclassifiable failure here escapes into// the fallback chain with `status: 0`.- throw new Error(- `Could not reach ${tokenEndpoint} to refresh the Grok token: ${(error as Error).message}`,- { cause: error }+ //+ // `claudishEndpoint` names the host that ACTUALLY failed. Without it the+ // handler falls back to the model endpoint, so an `auth.x.ai` outage was+ // reported — in the banner, in the log, and in the episode key — as+ // `api.x.ai`: the wrong host, and the wrong advice for a user trying to fix+ // it. `local.ts` attaches the same property for the same reason.+ throw Object.assign(+ new Error(+ `Could not reach ${tokenEndpoint} to refresh the Grok token: ${(error as Error).message}`,+ { cause: error }+ ),+ { claudishEndpoint: tokenEndpoint });}diff --git a/packages/cli/src/providers/transport/local.ts b/packages/cli/src/providers/transport/local.tsindex 73cf4d46..24ca212d 100644--- a/packages/cli/src/providers/transport/local.ts+++ b/packages/cli/src/providers/transport/local.ts@@ -13,6 +13,7 @@import { Agent } from "undici";import { credentials } from "../../auth/credentials/authority.js";+import { markOwnTimeout } from "../../handlers/shared/connection-error.js";import { LocalModelQueue } from "../../handlers/shared/local-queue.js";import { log } from "../../logger.js";import type { LocalProvider as LocalProviderConfig } from "../../providers/provider-registry.js";@@ -243,7 +244,13 @@ export class LocalTransport implements ProviderTransport {} catch (e: any) {// KEEP the error, do not merely log it. See `lastProbeError`'s docs: this// catch is where the connect evidence used to die.- this.lastProbeError = e;+ //+ // `markOwnTimeout` because the probe's 5 s ceiling is OURS: we asked "is+ // anything there" and got no answer inside a window we chose, which is a+ // reachability fact. Untagged, a `TimeoutError` is now deliberately+ // unclassified — see `connection-error.ts`'s `NAME_KIND` header for the+ // billing reason a transport's own inference ceiling must not be one.+ this.lastProbeError = markOwnTimeout(e);this.lastProbeUrl = healthUrl;log(`[${this.displayName}] /api/tags failed: ${e?.message || e}, trying /v1/models`);}@@ -264,7 +271,8 @@ export class LocalTransport implements ProviderTransport {}log(`[${this.displayName}] /v1/models returned ${response.status}`);} catch (e: any) {- this.lastProbeError = e;+ // Tagged for the same reason as the /api/tags probe above.+ this.lastProbeError = markOwnTimeout(e);this.lastProbeUrl = modelsUrl;log(`[${this.displayName}] /v1/models failed: ${e?.message || e}`);}diff --git a/packages/cli/src/proxy-server.ts b/packages/cli/src/proxy-server.tsindex 8e7e2917..3290cb1e 100644--- a/packages/cli/src/proxy-server.ts+++ b/packages/cli/src/proxy-server.ts@@ -54,6 +54,7 @@ import { LocalTransport } from "./providers/transport/local.js";import { OpenRouterProviderTransport } from "./providers/transport/openrouter.js";import { PoeProvider } from "./providers/transport/poe.js";import type { ProviderTransport } from "./providers/transport/types.js";+import { closeAllEpisodes } from "./recovery/coordinator.js";import { warmPricingCache } from "./services/pricing-cache.js";import type { ProxyServer } from "./types.js";@@ -1218,6 +1219,14 @@ export async function createProxyServer(// `true` = close active connections too, so a streamed request in flight// can't keep the port alive after shutdown resolves.await server.stop(true);+ // "No episode may outlive the process that owns its timers" —+ // `closeAllEpisodes`'s own doc, which until now nothing but tests+ // invoked. `server.stop(true)` aborts the client signals, so waiters+ // unwind and most episodes self-close; the residual that matters is+ // `serve`, which is long-lived and can be shut down and REBUILT+ // in-process, and would otherwise carry a stale episode — and its stale+ // banner — across the rebuild.+ closeAllEpisodes();},invalidateHandlerCache: (providerSlug?: string) => {if (!providerSlug) {diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.tsindex f58c4b2a..48b5228d 100644--- a/packages/cli/src/recovery/coordinator.test.ts+++ b/packages/cli/src/recovery/coordinator.test.ts@@ -458,16 +458,38 @@ describe("noteTargetReachable — the handoff that ends without a rejoin", () =>expect(episodeCount()).toBe(0);});- test("a DIFFERENT host does not close it — the key is provider|host", async () => {+ test("ANOTHER HOST of the same provider closes it — an auth episode is still this outage", async () => {+ // The episode key is `provider|host` and the AUTH sites seed the host from+ // the error, so a `gk@` outage during a token refresh opens an episode on+ // `auth.x.ai` while `noteTargetReachable` is called with the MODEL endpoint.+ // A strict key lookup missed it, and the pane went on painting "waiting for+ // Claude Code to retry" over a working session for the full 120-second+ // grace — the fix that shipped for the fetch path, absent for the five auth+ // sites.+ //+ // Closing by provider is sound rather than convenient: a request that+ // reached the model endpoint had to authenticate first, so it has just+ // proved every host it touched is answering.useFakeClock();- const s = seed({ providerName: "reach-2" });- const h = joinEpisode(s);+ const auth = joinEpisode(+ seed({ providerName: "reach-2", endpoint: "https://auth.example.com/oauth2/token" })+ );+ auth.recordAttemptResult("ConnectionRefused", false);+ auth.handoff();+ auth.leave();+ expect(describeEpisode(auth.episodeId)?.state).toBe("handoff");++ noteTargetReachable("reach-2", "https://api.example.com/v1/chat/completions");+ expect(describeEpisode(auth.episodeId)).toBeNull();+ });++ test("a DIFFERENT PROVIDER never closes it — one provider's success says nothing", async () => {+ useFakeClock();+ const h = joinEpisode(seed({ providerName: "reach-2b" }));h.recordAttemptResult("ConnectionRefused", false);h.handoff();h.leave();- noteTargetReachable("reach-2", "http://127.0.0.1:10/v1/chat/completions");- expect(describeEpisode(h.episodeId)?.state).toBe("handoff");noteTargetReachable("other-provider", "http://127.0.0.1:9/v1/chat/completions");expect(describeEpisode(h.episodeId)?.state).toBe("handoff");});diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 84ce467d..91f921ca 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -79,8 +79,6 @@ export interface EpisodeSeed {interface Waiter {deadlineAt: number;- tier: 1 | 2;- leftByDeadline: boolean;/** Non-null exactly while this waiter is parked. */wake: ((outcome: WaitOutcome) => void) | null;fail: ((err: unknown) => void) | null;@@ -158,6 +156,7 @@ export function describeEpisode(episodeId: string): Readonly<Record<string, unknmaxTier: ep.maxTier,lastOutcome: ep.lastOutcome,nextAttemptAtPerf: ep.nextAttemptAtPerf,+ paneRequestedAtPerf: ep.paneRequestedAtPerf,};}@@ -346,7 +345,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {const key = `${seed.providerName}|${host}`;let ep = episodes.get(key);- let tier: 1 | 2 = 1;if (ep && (ep.state === "recovered" || ep.state === "abandoned")) {episodes.delete(key);@@ -366,7 +364,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {ep.clientRetries++;ep.state = "attempting";ep.maxTier = 2;- tier = 2;log(`[Recovery] ${ep.providerDisplayName} rejoined episode ${ep.episodeId} ` +`(client retry ${ep.clientRetries}, ladder ${ep.ladderIndex}, attempts ${ep.attempts})`@@ -434,8 +431,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {const episode = ep;const waiter: Waiter = {deadlineAt: seed.deadlineAt,- tier,- leftByDeadline: false,wake: null,fail: null,detach: null,@@ -520,7 +515,9 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {},handoff(): void {- waiter.leftByDeadline = true;+ // EPISODE-level only. A per-waiter copy of this was write-only, and the+ // decision it feeds — what the LAST waiter out closes the episode as —+ // is an episode-level question by construction.episode.anyLeftByDeadline = true;},@@ -658,15 +655,35 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {* closing it underneath them would strand or gave-up requests that another* request's success says nothing about. A `handoff` episode has no waiters by* construction — that is what `handoff` means.+ *+ * ── WHY IT CLOSES BY PROVIDER AND NOT BY HOST ───────────────────────────────+ *+ * The key is `provider|host`, and the AUTH sites seed the host from the error —+ * so a `gk@` outage during a token refresh can open an episode keyed on+ * `auth.x.ai` while this function is called with the MODEL endpoint. Keyed+ * lookup missed it by host, and the fix that shipped for the fetch path was+ * absent for the five auth sites `network-recovery.md` §5 enumerates: the pane+ * went on painting "waiting for Claude Code to retry" over a working session+ * for the full 120-second grace.+ *+ * Closing every `handoff` episode for the provider is sound rather than merely+ * convenient: a request that reached the model endpoint HAD to authenticate+ * first, so it has just proved every host it touched is answering. And a+ * `handoff` episode holds no waiters, so nothing is closed out from under+ * anyone either way.*/export function noteTargetReachable(providerName: string, endpoint: string): void {- const ep = episodes.get(`${providerName}|${hostOf(endpoint)}`);- if (!ep || ep.state !== "handoff") return;- log(- `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +- `closing episode ${ep.episodeId} instead of waiting out its grace`- );- closeEpisode(ep, "recovered");+ const exact = `${providerName}|${hostOf(endpoint)}`;+ const prefix = `${providerName}|`;+ for (const [key, ep] of [...episodes]) {+ if (key !== exact && !key.startsWith(prefix)) continue;+ if (ep.state !== "handoff") continue;+ log(+ `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` ++ `closing episode ${ep.episodeId} instead of waiting out its grace`+ );+ closeEpisode(ep, "recovered");+ }}/**diff --git a/packages/cli/src/recovery/magmux-ui.ts b/packages/cli/src/recovery/magmux-ui.tsindex f09bd8bd..f78c8ad3 100644--- a/packages/cli/src/recovery/magmux-ui.ts+++ b/packages/cli/src/recovery/magmux-ui.ts@@ -1,7 +1,14 @@/*** The recovery UI manager — the single owner of the socket, the pane, the- * `paneOpen` fact, the post-`bye` suppression window, the cross-process lock- * and the lease.+ * `paneOpen` fact, the cross-process lock and the lease.+ *+ * THE POST-`bye` SUPPRESSION IS NOT OWNED HERE, and that is a correction. It+ * used to be `state.suppressUntilPerf`, read only by `ensureRecoveryUi`, so+ * `[q] give up` stopped the BANNER while the next request against the same+ * dead target still held its socket for the full derived deadline with the+ * reason legible nowhere. The fact now lives in `recovery/settings.ts` where+ * `shouldSkipTier1` can read it too: one number, suppressing the hold and the+ * surface together.** OWNERSHIP IS PROCESS-SCOPED; ONLY EPISODES ARE EPISODE-SCOPED. There is ONE* pane and it multiplexes every episode, so "a pane exists" is a property of@@ -55,6 +62,7 @@ import {renderableEpisodeFrame,tryNow,} from "./coordinator.js";+import { noteRecoveryGaveUp, recoveryGiveUpActive, resetRecoveryGiveUp } from "./settings.js";import {type RecoverySocketServer,cleanupRecoverySocket,@@ -121,8 +129,6 @@ interface ManagerState {paneIndex: number | null;/** episodeId → PROCESS ms of the last heartbeat naming it. */leases: Map<string, number>;- /** PROCESS ms before which no pane may be opened (post-`bye`). */- suppressUntilPerf: number;/** Guards the async open against re-entry. */opening: boolean;lockPath: string | null;@@ -137,7 +143,6 @@ const state: ManagerState = {paneOpen: false,paneIndex: null,leases: new Map(),- suppressUntilPerf: 0,opening: false,lockPath: null,lingerTimer: null,@@ -339,7 +344,14 @@ function onCommand(cmd: RecoveryCommand): void {break;case "bye":log(`[Recovery] pane said bye (${cmd.reason}) — giving up on every live episode`);- state.suppressUntilPerf = recoveryClock().now() + UI_SUPPRESS_AFTER_BYE_MS;+ // ONE fact, two readers. `giveUpAll()` ends the episodes alive at this+ // instant; this ends the ones that do not exist yet. Without it the next+ // request against the same dead target opened a new episode, found the+ // pane suppressed, and held its socket for the full deadline with the+ // reason legible nowhere — the forbidden state, reached from the key+ // whose whole purpose is to avoid it. The hold and the surface are+ // suppressed together, for the same window, from the same number.+ noteRecoveryGaveUp(recoveryClock().now() + UI_SUPPRESS_AFTER_BYE_MS);giveUpAll();// The pane the user just closed must not reappear two seconds later, and// closing it is ALSO what revokes every lease: `teardown()` drops@@ -459,7 +471,7 @@ export function ensureRecoveryUi(episodeId: string): void {state.lingerTimer = null;}if (state.paneOpen || state.opening) return;- if (recoveryClock().now() < state.suppressUntilPerf) {+ if (recoveryGiveUpActive()) {log("[Recovery] pane suppressed — the user pressed [q] less than a minute ago");return;}@@ -534,7 +546,7 @@ export async function shutdownRecoveryUi(): Promise<void> {await closePane();registerRecoveryUi(null);installed = false;- state.suppressUntilPerf = 0;+ resetRecoveryGiveUp();explicitControlSock = null;}@@ -554,7 +566,7 @@ export function __resetUiStateForTests(): void {state.leases.clear();state.paneOpen = false;state.paneIndex = null;- state.suppressUntilPerf = 0;+ resetRecoveryGiveUp();state.opening = false;state.socketPath = null;state.server = null;diff --git a/packages/cli/src/recovery/pane-app.ts b/packages/cli/src/recovery/pane-app.tsindex ff95f971..ee1a4a9b 100644--- a/packages/cli/src/recovery/pane-app.ts+++ b/packages/cli/src/recovery/pane-app.ts@@ -256,10 +256,21 @@ export function renderPane(state: PaneViewState, width: number): string {// and a user with a narrow window gets a narrow pane. Below 64 columns the// full sentence wraps or is cut, so the hint sheds the part a user can work// out — where to click — and keeps the part they cannot: which keys exist.+ //+ // `[r]` IS HIDDEN IN `handoff`, because in that state it does nothing:+ // `tryNow` returns early when no attempt timer is armed, and a handed-off+ // episode has no waiters and therefore no timer. Advertising a key that+ // prints "retrying now…" over an episode nobody is retrying is the same+ // class of defect as a give-up that does not give up — a control that+ // appears to act and does not.+ const keys =+ ep.state === "handoff"+ ? `${C.BOLD}[q]${C.RESET} give up`+ : `${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`;lines.push(w < 64- ? `${C.GRAY} ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`- : `${C.GRAY} click here or Ctrl-G Tab, then ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`+ ? `${C.GRAY} ${C.RESET}${keys}`+ : `${C.GRAY} click here or Ctrl-G Tab, then ${C.RESET}${keys}`);if (state.notice) lines.push(`${C.GREEN} ${state.notice}${C.RESET}`);@@ -439,7 +450,10 @@ export async function runRecoveryPane(opts: RecoveryPaneOptions): Promise<numberreturn;}if (ch === "r" || ch === "R" || ch === "\r" || ch === "\n") {- if (state.frame && client?.connected) {+ // Not in `handoff`: there is no armed timer to collapse there, so the+ // key would only print a reassurance that is not true. The banner hides+ // it in that state too — one rule, both places.+ if (state.frame && state.frame.state !== "handoff" && client?.connected) {client.send({v: RECOVERY_PROTOCOL_VERSION,type: "retry_now",@@ -484,6 +498,24 @@ export async function recoveryPaneCommand(argv: string[]): Promise<void> {console.error("Usage: claudish recovery-pane --socket <path>");process.exit(2);}+ // The pane is its own PROCESS, reached from `index.ts` before `runCli()`, so+ // nothing has published a theme mode for it. `cliAnsi()` is read at paint+ // time — which is right, and which the module-load snapshot trap makes+ // necessary — but it cannot use a mode that was never detected, so the pane+ // stayed on the unknown/classic palette even where OSC 11, `COLORFGBG` or an+ // explicit `CLAUDISH_THEME=light` would have chosen the light one.+ //+ // It runs HERE rather than inside `runRecoveryPane` for the same reason the+ // TUI runs it before `createCliRenderer`: the OSC query puts stdin in raw+ // mode and reads it, so it must finish before the pane's own key handler+ // owns the stream. Bounded at 150 ms, a no-op without a TTY on both ends, and+ // never fatal — a banner with the wrong palette still beats no banner.+ try {+ const { detectAndSetThemeMode } = await import("../theme/theme-mode.js");+ await detectAndSetThemeMode();+ } catch {+ /* the classic palette is a working fallback */+ }const code = await runRecoveryPane({ socketPath });process.exit(code);}diff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.tsindex 3aa33b9a..4f13f4fd 100644--- a/packages/cli/src/recovery/settings.ts+++ b/packages/cli/src/recovery/settings.ts@@ -13,6 +13,7 @@import { ENV } from "../config.js";import { log } from "../logger.js";import { readRecoveryEnabled, readRecoveryUi } from "../profile-config.js";+import { recoveryClock } from "./clock.js";/** Flag layer. Set once by `cli.ts`'s arg loop; undefined means "not given". */interface RecoveryFlagOverrides {@@ -87,6 +88,52 @@ export function resolveRecoveryUi(): boolean {return true;}+// ─── "The user said stop" ────────────────────────────────────────────────────++/**+ * PROCESS ms until which `[q] give up` suppresses recovery. 0 = not suppressed.+ *+ * ── WHY THIS IS A PROCESS FACT AND NOT THE PANE'S OWN STATE ─────────────────+ *+ * `[q]` used to do two things, both inside the UI manager: suppress re-opening+ * the pane for a minute, and `giveUpAll()` — which iterates the episodes alive+ * AT THAT INSTANT. Nothing recorded that the user had asked recovery to stop,+ * so the next request against the same dead target opened a NEW episode, found+ * the pane suppressed, got no pane and no lease, and then held its socket for+ * the full derived deadline (~4.5 minutes at the default) before answering 400+ * — with the reason legible NOWHERE, which is the state this design calls+ * "strictly worse than the bug this feature exists to remove", reached from the+ * one affordance whose entire purpose is to end it. Before recovery existed+ * those requests failed in milliseconds.+ *+ * It is not an edge case: Claude Code issues concurrent requests during an+ * outage (main loop, title model, subagents), so a new request inside the+ * suppression window is the EXPECTED case. A control that appears to stop+ * something and does not is worse than no control.+ *+ * So the give-up suppresses the HOLD and the SURFACE together, for the same+ * window, from ONE fact — read by `shouldSkipTier1` (the hold) and by+ * `ensureRecoveryUi` (the surface). Two facts would be two things to keep in+ * step, and the asymmetry between them is exactly what created the forbidden+ * state.+ */+let giveUpUntilPerf = 0;++/** `[q] give up`: suppress recovery until `untilPerf` (PROCESS ms). */+export function noteRecoveryGaveUp(untilPerf: number): void {+ giveUpUntilPerf = Math.max(giveUpUntilPerf, untilPerf);+}++/** Is the user's give-up still in force? Reads the clock LIVE — never cached. */+export function recoveryGiveUpActive(): boolean {+ return giveUpUntilPerf > 0 && recoveryClock().now() < giveUpUntilPerf;+}++/** Pane teardown, `serve` rebuilds, and tests. */+export function resetRecoveryGiveUp(): void {+ giveUpUntilPerf = 0;+}+// ─── The deadline ────────────────────────────────────────────────────────────/**@@ -232,14 +279,32 @@ export function resetDeadlineNotice(): void {* one process-wide switch, and there is no narrower lever: the variable is* fixed at spawn and cannot key on whether a banner exists.*- * ── WHY IT IS GATED ON THE UI SWITCH ────────────────────────────────────────+ * ── THE THREE GATES, AND WHY IT IS NOT JUST THE UI SWITCH ───────────────────+ *+ * All three must hold, because the cost above is only worth paying by a launch+ * that gets the benefit:+ *+ * 1. `resolveRecoveryEnabled()` — the ladder itself. `--no-recovery` /+ * `CLAUDISH_RECOVERY=0` restores pre-recovery behaviour "byte for byte"+ * (RISK-7), and an exhaustion that answers today's immediate 400 hands+ * nothing back for the client to re-ask. A CI run that explicitly opted+ * out must not be left looping ~300 times on an unrelated 503.+ * 2. `resolveRecoveryUi()` — the user's configuration, flag > env > project >+ * global > true. Off means no pane, therefore no lease, therefore an+ * inline 400 at exhaustion. Surface and reach move together or not at all.+ * 3. `paneEligible` — whether THIS LAUNCH can obtain a surface at all, from+ * `magmuxPaneCapability()`. It is a launch-order fact, not a preference:+ * false in `-p`, in `--stdin`, with no TTY, and on a machine without+ * magmux. None of those can ever hold the lease a recovery 503 requires,+ * so for them the watchdog is the RISK-6 exposure with none of the+ * recovery benefit, applied to every unrelated 503 the session sees.*- * `resolveRecoveryUi()` is the user's configuration — flag > env > project >- * global > true. Off means no pane, therefore no lease, therefore an inline- * 400 at exhaustion, therefore nothing handed back for the client to re-ask —- * so extending its budget would buy only the side effect above. Surface and- * reach move together or not at all.+ * Gate 3 is why the caller must ask about eligibility BEFORE finalising the+ * child environment — see `magmuxPaneCapability`, which is side-effect free for+ * exactly that reason.*/-export function retryWatchdogEnv(): Record<string, string> {+export function retryWatchdogEnv(opts: { paneEligible: boolean }): Record<string, string> {+ if (!opts.paneEligible) return {};+ if (!resolveRecoveryEnabled()) return {};return resolveRecoveryUi() ? { CLAUDE_CODE_RETRY_WATCHDOG: "1" } : {};}diff --git a/packages/cli/src/recovery/types.test.ts b/packages/cli/src/recovery/types.test.tsindex 6a2f8378..0b47013e 100644--- a/packages/cli/src/recovery/types.test.ts+++ b/packages/cli/src/recovery/types.test.ts@@ -62,6 +62,32 @@ describe("recovery wire contract", () => {expect(parseFrame(JSON.stringify({ v: 1, type: "episode" }))).toBeNull(); // no episodeId});+ test("the SKEW NOTICE parses at the wrong version — it is the one frame that must", () => {+ // The mirror image of `hello` below, and for the same reason. The proxy+ // answers a mismatched pane with a `closed` / `protocol_mismatch` frame+ // carrying the PROXY'S version, so a blanket version gate dropped the very+ // message addressed to the build that cannot read the others:+ // `PaneViewState.protocolMismatch` was unreachable in a real skew and the+ // pane printed "the claudish proxy closed this connection" instead — a safe+ // failure with the wrong diagnosis, on the one day it mattered.+ const notice = JSON.stringify({+ v: 99,+ type: "closed",+ episodeId: "",+ outcome: "protocol_mismatch",+ });+ const parsed = parseFrame(notice);+ expect(parsed).not.toBeNull();+ expect(parsed?.type).toBe("closed");+ expect((parsed as { outcome?: string }).outcome).toBe("protocol_mismatch");++ // And ONLY that outcome. A skewed `closed` frame of any other kind is still+ // refused — the exemption is for the notice, not for the type.+ expect(+ parseFrame(JSON.stringify({ v: 99, type: "closed", episodeId: "e", outcome: "recovered" }))+ ).toBeNull();+ });+test("`hello` parses even at the WRONG version — the mismatch must be answerable", () => {// The server cannot tell a skewed client it is skewed if the one message it// needs to answer is the one it drops.diff --git a/packages/cli/src/recovery/types.ts b/packages/cli/src/recovery/types.tsindex 790f2999..c19ecbcd 100644--- a/packages/cli/src/recovery/types.ts+++ b/packages/cli/src/recovery/types.ts@@ -147,6 +147,23 @@ function isRecord(value: unknown): value is Record<string, unknown> {* Null for malformed JSON, for an unknown `type`, and for a `v` this build does* not speak — a reader that guesses at an unknown shape is how a version skew* becomes a rendering bug instead of a one-line notice.+ *+ * ── THE ONE EXEMPTION, AND WHY IT IS NOT AN INCONSISTENCY ───────────────────+ *+ * The skew NOTICE itself is exempt, exactly as `parseCommand` exempts `hello`+ * for the mirror-image reason. The proxy answers a mismatched pane with a+ * `closed` / `protocol_mismatch` frame carrying the PROXY'S version — so a+ * blanket version gate dropped the very message addressed to the build that+ * cannot read the others, `PaneViewState.protocolMismatch` was unreachable in+ * any real skew, and the pane printed "the claudish proxy closed this+ * connection" instead. Mixed-build grids and sessions started during an update+ * failed safely with the WRONG diagnosis: the skew notice did not appear on the+ * one day it was needed.+ *+ * Exempting it is sound where exempting a data frame would not be. It carries+ * no payload to misread — a type, an empty `episodeId` and a terminal outcome —+ * and it is the LAST thing on the connection, so nothing can follow it that a+ * mis-parse could corrupt.*/export function parseFrame(line: string): RecoveryFrame | null {let parsed: unknown;@@ -156,6 +173,14 @@ export function parseFrame(line: string): RecoveryFrame | null {return null;}if (!isRecord(parsed)) return null;+ if (parsed.type === "closed" && parsed.outcome === "protocol_mismatch") {+ return {+ v: RECOVERY_PROTOCOL_VERSION,+ type: "closed",+ episodeId: typeof parsed.episodeId === "string" ? parsed.episodeId : "",+ outcome: "protocol_mismatch",+ } as unknown as RecoveryClosedFrame;+ }if (parsed.v !== RECOVERY_PROTOCOL_VERSION) return null;if (parsed.type === "episode" && typeof parsed.episodeId === "string") {return parsed as unknown as RecoveryEpisodeFrame;