Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

fix(recovery): stop the ladder faking attempts, billing latency, and ignoring [q]

Jack Rudenko4e45c0cdb4c0
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.md
index dd5792ee..d4d79e0b 100644
--- a/ai-docs/architecture/network-recovery.md
+++ b/ai-docs/architecture/network-recovery.md
@@ -59,8 +59,10 @@ The second tier hands the retry back to the client. What the client will then sp
- **~11 attempts** on its default budget, spanning **~174 s** for a 503 chain (backoff caps at
38.4 s per gap, and `retry-after` is honoured verbatim);
-- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets when the recovery UI is
- enabled — reaching **~a day** of unattended recovery on a hung connection.
+- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets only when recovery is
+ enabled, the UI is allowed, AND this launch can actually obtain a pane — reaching **~a day** of
+ unattended recovery on a hung connection. All three gates, because the watchdog amplifies EVERY
+ 503 the session sees and is worth its cost only where a surface can exist (§7).
It is a count, not a clock. That is why `EPISODE_GRACE_MS` is 120 s: the client's worst gap is
38.4 s, so 120 s is 3.1× inside it and a returning client always finds its episode still open.
@@ -79,7 +81,8 @@ POST /v1/messages ──► ComposedHandler.handle() startTime = performance.n
classifyConnectionError() ── null ──► rethrow, unchanged
│ non-null
▼
- shouldSkipTier1()? probe header · recovery disabled · no budget left
+ shouldSkipTier1()? probe header · recovery disabled ·
+ user pressed [q] · no budget left
│ no
┌───────────────────▼──────────────────────────────────────────┐
│ TIER 1 — in-request, PRE-FLUSH, deadline derived from §1 │
@@ -165,9 +168,28 @@ is the only honest discriminator, and it reads the address, not the error.
| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the pane, so the retry may be handed back |
| exhausted, **no lease** | **400** `connection_error` | — | nothing can display the reason, so it must ride the status |
| `[q] give up` | **400** `connection_error` | — | guarded on `result.kind`, never on the lease — see below |
+| **a request arriving within 60 s of `[q]`** | **400** `connection_error` | — | the ladder is SKIPPED outright: the user said stop |
| client disconnected | 499 (unread) | — | the socket is already gone |
| unclassifiable throw on any attempt | rethrown unchanged | — | recovery must not widen what "transient" means |
+### `[q] give up` suppresses the HOLD, not only the banner
+
+`bye` originally did two things, both inside the UI manager: it suppressed re-opening the pane for
+60 s, and it called `giveUpAll()` — which iterates the episodes alive *at that instant*. Nothing
+recorded that the user had asked recovery to stop, and `shouldSkipTier1` had no gate that could
+notice. So the next request against the same dead target opened a NEW episode, found the pane
+suppressed, got no pane and no lease, and then held its socket for the full ~4.5-minute deadline
+before answering 400 — **a long hold with no surface, which is the state this design calls "strictly
+worse than the bug this feature exists to remove", reached from the one affordance whose entire
+purpose is to end it.** Before recovery existed those requests failed in milliseconds.
+
+It is not an edge case. Claude Code issues concurrent requests during an outage (main loop, title
+model, subagents), so a request arriving inside the suppression window is the EXPECTED one.
+
+The fact is therefore process-level — `recovery/settings.ts`'s `recoveryGiveUpActive()` — and is read
+by **both** `shouldSkipTier1` (the hold) and `ensureRecoveryUi` (the surface). One number, one
+window, two readers. A control that appears to stop something and does not is worse than no control.
+
### 429 would have been a live billing bug
`fallback-handler.ts` contains, verbatim:
@@ -320,6 +342,28 @@ common local failure — never reached tier 1 at all. Same class of fix in `vert
`healthChecked` latches on **success only**, so a retried `refreshAuth()` re-probes instead of
returning a false success to attempt two.
+### Three things the auth path needs that the fetch path gets for free
+
+- **The clamp has to be ENFORCED, not handed over.** `op` takes the per-attempt signal, and the fetch
+ path threads it into `fetch`. `refreshAuth()` and `getHeaders()` take no arguments, so
+ `() => this.provider.refreshAuth!()` discards it — and Grok's token exchange then performs an
+ unbounded `fetch(auth.x.ai/oauth2/token)`. A swallowed connection there outlived both the 45 s cap
+ and the client's own disconnect while `withConnectionRetry` awaited a promise nothing could settle:
+ the unbounded hold this feature exists to remove, reached from inside the machinery that removes
+ it. `untilAborted` now enforces the ceiling at the one place that owns it. The operation is
+ *abandoned*, not cancelled — threading a real signal through every transport's auth call is the
+ deeper fix and is still worth doing.
+- **The error must name the host that actually failed.** `connectionEndpointFor` falls back to the
+ MODEL endpoint when the error carries no `claudishEndpoint`, so an `auth.x.ai` outage was reported
+ — in the banner, in the log and in the episode key — as `api.x.ai`. Grok's refresh wrapper now
+ attaches it, as `local.ts` already did.
+- **`noteTargetReachable` closes by PROVIDER, not by host.** Because an auth episode can be keyed on
+ a different host than the one a later success reaches, a strict key lookup missed it and the pane
+ painted "waiting for Claude Code to retry" over a working session for the full 120 s grace. A
+ request that reached the model endpoint had to authenticate first, so it has proved every host it
+ touched is answering — and a `handoff` episode holds no waiters, so nothing is closed from under
+ anyone.
+
---
## 6. Stats — and the one step whose omission is silent
@@ -398,6 +442,38 @@ ladder, and then accepted it again, with the arithmetic in front of them, at the
`recovery_episode_id` is what makes the exposure *attributable* — without it, attribution breaks
exactly across a tier-2 handoff, which is the highest-exposure path.
+**What RISK-6 does NOT extend to, and the two ways it leaked past its own boundary.** Both were
+found in review and closed; both are the same mistake — applying the accepted arithmetic to a case
+the user was never shown.
+
+1. **A LATENCY event is not a network fault.** `classifyConnectionError` mapped the NAME
+ `TimeoutError` to `unreachable`. That name is also what `AbortSignal.timeout` on a transport's own
+ *inference* request rejects with — `vertex-oauth.ts` puts a 30 s ceiling on the model call itself.
+ A `vx@` turn whose time-to-first-byte exceeded 30 s (ordinary for a thinking model) was therefore
+ classified as "the host is unreachable", entered the ladder, took the tier-2 handoff, and was
+ re-POSTed ~300 times — **each one running one more real billed inference against a host that had
+ answered the TCP connect and was already generating.** The discriminator is now the SIGNAL'S
+ ORIGIN, not the name: our per-attempt clamp and our reachability probes carry a `markOwnTimeout`
+ own-property and classify; everything else keeps its pre-recovery route out, unclassified and
+ rethrown. The user-facing sentence was wrong too — it told the user to check VPN and DNS for a
+ host that was mid-generation.
+2. **The watchdog needs three gates, not one.** `CLAUDE_CODE_RETRY_WATCHDOG=1` was exported from the
+ UI preference alone, so `-p`, `--no-recovery`, a pipe and a machine without magmux all expanded
+ every UNRELATED 503 to ~300 client attempts while being structurally incapable of holding the
+ lease a recovery 503 requires. It now requires `resolveRecoveryEnabled()` **and**
+ `resolveRecoveryUi()` **and** `magmuxPaneCapability() !== none` — the last asked BEFORE the child
+ environment is finalised, which is why that predicate is side-effect free and is the same one
+ `planMagmuxWrap` consumes.
+
+**The transport's `getRequestInit()` is re-minted per attempt.** It was hoisted once, before the
+primary fetch, and spread into every re-issue. A transport that returns a one-shot
+`AbortSignal.timeout` therefore poisoned the whole ladder the moment it fired:
+`AbortSignal.any([fired, live])` is ALREADY ABORTED, so every later attempt rejected without opening
+a socket. For `vx@` that made tier 1 silently dead past t+30 s — the request held the full deadline
+making **zero** real connect attempts while the log and the pane counted attempts that never left the
+process. `mergeSignalIntoInit` now also drops an `own` that is already aborted, as the belt behind
+that brace.
+
**RISK-7 — a CI run holding a dead endpoint for the full deadline.** `--no-recovery` /
`CLAUDISH_RECOVERY=0` restores today's immediate 400 everywhere, byte for byte. `--no-recovery-ui` /
`CLAUDISH_RECOVERY_UI=0` keeps the retries and drops only the pane (and with it the 503 arm, since
diff --git a/packages/cli/src/claude-runner.ts b/packages/cli/src/claude-runner.ts
index e8b1966a..ee3c8e10 100644
--- a/packages/cli/src/claude-runner.ts
+++ b/packages/cli/src/claude-runner.ts
@@ -17,7 +17,7 @@ import { isatty } from "node:tty";
import { lookupModelForProvider } from "./adapters/model-catalog.js";
import { classifierPassthroughEnabled } from "./classifier-passthrough.js";
import { ENV } from "./config.js";
-import { planMagmuxWrap } from "./launcher/magmux-wrapper.js";
+import { magmuxPaneCapability, planMagmuxWrap } from "./launcher/magmux-wrapper.js";
// Aliased: runClaudeWithProxy declares its own local `log` (a quiet-aware
// console printer), and an unaliased import would be shadowed inside it.
import { log as debugLog, logStderr } from "./logger.js";
@@ -1634,12 +1634,22 @@ export async function runClaudeWithProxy(
...advisorToolEnv.vars,
};
+ // Can this launch put a recovery banner on screen at all? Asked HERE, before
+ // the child environment is finalised, because the answer gates the watchdog
+ // below — and asked through a side-effect-free function so asking is free and
+ // repeatable. `planMagmuxWrap` further down consumes the same predicate, so
+ // "may we wrap" and "may we amplify the client's retries" cannot drift.
+ const paneCapability = magmuxPaneCapability({
+ interactive: Boolean(config.interactive),
+ stdoutIsTty: Boolean(process.stdout.isTTY),
+ });
+
// The client half of "retry forever". Tier 1 holds one request for the
// derived deadline and then hands the retry back as a 503; how far recovery
// actually reaches is decided by Claude Code's willingness to re-ask, which
- // this sets. The arithmetic, the accepted cost and the side effect on
- // claudish's OTHER 503s are all stated at `retryWatchdogEnv()`.
- Object.assign(env, retryWatchdogEnv());
+ // this sets. The arithmetic, the accepted cost, the side effect on claudish's
+ // OTHER 503s and all three gates are stated at `retryWatchdogEnv()`.
+ Object.assign(env, retryWatchdogEnv({ paneEligible: paneCapability.kind !== "none" }));
// Provider display name, best-effort and FREE. Only an explicit `provider@model`
// spec names its provider without routing, and route() would touch credentials /
@@ -1958,16 +1968,22 @@ export async function runClaudeWithProxy(
// decides whether a retry happens. Off means no pane, therefore no lease,
// therefore an inline error at exhaustion — which is the 503-vs-400 decision
// made structurally rather than by a second predicate somewhere else.
- const wrap = resolveRecoveryUi()
- ? planMagmuxWrap({
- claudeBinary,
- claudeArgs,
- childEnv: env,
- cwd: process.cwd(),
- interactive: Boolean(config.interactive),
- stdoutIsTty: Boolean(process.stdout.isTTY),
- })
- : null;
+ //
+ // `paneCapability` was resolved before the environment was finalised, and is
+ // re-used rather than re-derived: the watchdog above is exported on exactly
+ // the launches that reach `planMagmuxWrap` or the ambient branch below.
+ const wrap =
+ resolveRecoveryUi() && paneCapability.kind === "wrap"
+ ? planMagmuxWrap({
+ claudeBinary,
+ claudeArgs,
+ childEnv: env,
+ cwd: process.cwd(),
+ interactive: Boolean(config.interactive),
+ stdoutIsTty: Boolean(process.stdout.isTTY),
+ magmuxBinary: paneCapability.magmux,
+ })
+ : null;
const proc = wrap
? spawn(wrap.command, wrap.args, { env: wrap.env, stdio, shell: false })
diff --git a/packages/cli/src/handlers/composed-handler-recovery.test.ts b/packages/cli/src/handlers/composed-handler-recovery.test.ts
index 856cd33e..f412275b 100644
--- a/packages/cli/src/handlers/composed-handler-recovery.test.ts
+++ b/packages/cli/src/handlers/composed-handler-recovery.test.ts
@@ -22,15 +22,24 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import type { Context } from "hono";
import { probeLink } from "../providers/probe-live.js";
import type { ProviderTransport } from "../providers/transport/types.js";
-import { resetRecoveryClock } from "../recovery/clock.js";
+import { resetRecoveryClock, setRecoveryClock } from "../recovery/clock.js";
import { closeAllEpisodes, episodeCount } from "../recovery/coordinator.js";
-import { resetRecoveryFlagOverrides, setRecoveryFlagOverrides } from "../recovery/settings.js";
+import { UI_SUPPRESS_AFTER_BYE_MS, __handleCommandForTests } from "../recovery/magmux-ui.js";
+import {
+ recoveryGiveUpActive,
+ resetRecoveryFlagOverrides,
+ resetRecoveryGiveUp,
+ setRecoveryFlagOverrides,
+} from "../recovery/settings.js";
import {
capturedRecoveryLines,
startLogCapture,
stopLogCapture,
} from "../recovery/test-helpers/capture-log.js";
+import { FakeClock } from "../recovery/test-helpers/fake-clock.js";
+import { RECOVERY_PROTOCOL_VERSION } from "../recovery/types.js";
import { ComposedHandler } from "./composed-handler.js";
+import { classifyConnectionError, markOwnTimeout } from "./shared/connection-error.js";
/** A port nothing is listening on, taken and released so it is VERIFIED free. */
function freePort(): number {
@@ -52,6 +61,9 @@ afterEach(() => {
closeAllEpisodes();
resetRecoveryClock();
resetRecoveryFlagOverrides();
+ // PROCESS state, shared with every sibling test file in this Bun process. A
+ // give-up left set would skip the ladder in tests that never pressed a key.
+ resetRecoveryGiveUp();
});
let seq = 0;
@@ -302,3 +314,246 @@ describe("shouldSkipTier1 — three gates, and no loopback carve-out", () => {
await p;
});
});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The transport's own signal, per attempt
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("every ladder attempt gets a FRESH transport init", () => {
+ /**
+ * The defect this pins, in one sentence: `getRequestInit()` was called once,
+ * before the primary fetch, and the same object was spread into every
+ * re-issue — so a transport that returns a ONE-SHOT `AbortSignal.timeout`
+ * (`vertex-oauth.ts` returns a 30-second one) poisoned the whole ladder the
+ * moment that signal fired. `AbortSignal.any([fired, live])` is ALREADY
+ * ABORTED, so the attempt rejected before it opened a socket.
+ *
+ * The visible cost was not a slow retry but a FAKE one: tier 1 held the
+ * request for the full ~270 s deadline making ZERO real connect attempts,
+ * while the `[Recovery]` log and the pane both counted attempts that never
+ * left the process.
+ *
+ * So the assertion is about what the NETWORK saw, and it needs the real
+ * runtime: a real `AbortSignal.timeout`, real `fetch`, and a real refused
+ * port. A stubbed `globalThis.fetch` cannot fail this test, because the
+ * rejection being tested happens inside fetch itself.
+ */
+ test("a retry after the transport's own signal has fired still reaches the socket", async () => {
+ const clock = new FakeClock(0);
+ setRecoveryClock(clock);
+
+ // Shorter than the gap the test then waits, so by attempt 2 the FIRST
+ // signal minted here is certainly dead.
+ const CEILING_MS = 40;
+ let initCalls = 0;
+ const transport = {
+ ...(deadTransport(deadPort) as object),
+ getRequestInit: () => {
+ initCalls++;
+ return { signal: AbortSignal.timeout(CEILING_MS) };
+ },
+ } as unknown as ProviderTransport;
+
+ const handler = new ComposedHandler(transport, "dead-model", "dead-model", 8080, {});
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ const { c } = contextFor(req);
+ const p = handler.handle(c, PAYLOAD);
+
+ // Attempt 1 fails on a refused socket and the ladder parks on its 5 s rung.
+ // The wait is REAL time and is what kills the first signal.
+ await new Promise((r) => setTimeout(r, 150));
+ expect(episodeCount()).toBe(1);
+ expect(initCalls).toBe(1);
+
+ // Fire the rung. Attempt 2 runs here.
+ await clock.advance(5_000);
+ await new Promise((r) => setTimeout(r, 150));
+
+ const lines = await capturedRecoveryLines();
+ const attempt2 = lines.find((l) => /attempt 2 failed/.test(l)) ?? "";
+
+ // THE ASSERTION. `ConnectionRefused` is the runtime saying it reached the
+ // network and the port said no. `TimeoutError` here would mean the attempt
+ // was refused by an expired signal without a socket ever being opened —
+ // and with a signal whose origin is not ours it is not even classifiable,
+ // so the ladder would break out entirely.
+ expect(attempt2).toContain("ConnectionRefused");
+ expect(attempt2).not.toContain("TimeoutError");
+ // Re-minted, not re-used: once for the primary fetch, once per re-issue.
+ expect(initCalls).toBeGreaterThanOrEqual(2);
+
+ ac.abort(new DOMException("done", "AbortError"));
+ await p.catch(() => {});
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Whose timeout was it?
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("a transport's own request timeout is NOT a connection failure", () => {
+ /**
+ * `TimeoutError` used to mean `unreachable` whatever raised it, and
+ * `AbortSignal.timeout` on a transport's INFERENCE request raises exactly
+ * that name. So a turn whose time-to-first-byte exceeded the transport's own
+ * ceiling — ordinary for a thinking model on a streaming endpoint — was
+ * classified as "the host is unreachable", entered the ladder, took the
+ * tier-2 handoff, and with the retry watchdog on was re-POSTed ~300 times.
+ * Each of those re-POSTs runs ONE MORE REAL BILLED INFERENCE against a host
+ * that answered the TCP connect and was already generating.
+ *
+ * RISK-6 was accepted for `ECONNRESET`/`EPIPE` on a genuine network fault.
+ * A latency event is neither a network fault nor transient, and was not
+ * priced. The discriminator is therefore the SIGNAL'S ORIGIN, not the name.
+ */
+ test("a hung upstream that trips the TRANSPORT's ceiling never opens an episode", async () => {
+ // A server that accepts the connection and then never answers: the exact
+ // shape a slow model presents, and the one a network classifier must not
+ // confuse with an unreachable host.
+ const hung = Bun.serve({
+ port: 0,
+ fetch: () => new Promise<Response>(() => {}),
+ });
+ try {
+ const transport = {
+ name: `hung-${Date.now()}`,
+ displayName: "Slow Model",
+ streamFormat: "openai-sse",
+ getEndpoint: () => `http://127.0.0.1:${hung.port}/v1/chat/completions`,
+ getHeaders: async () => ({}),
+ getRequestInit: () => ({ signal: AbortSignal.timeout(60) }),
+ } as unknown as ProviderTransport;
+
+ const handler = new ComposedHandler(transport, "slow-model", "slow-model", 8080, {});
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ const { c } = contextFor(req);
+
+ // Raced explicitly rather than simply awaited, because the defect's
+ // signature is a HOLD: misclassified, this request parks on the ladder
+ // for the full derived deadline (~270 s), which a bare `await` would
+ // report as a hung test run rather than as a failure.
+ const p = handler.handle(c, PAYLOAD);
+ const outcome = await Promise.race([
+ p.then(
+ () => "resolved" as const,
+ (e: unknown) => e
+ ),
+ new Promise<"still-holding">((r) => setTimeout(() => r("still-holding"), 2_000)),
+ ]);
+
+ // Unclassified means rethrown unchanged — the pre-recovery route, which
+ // is the whole point. No ladder, no hold, no 300 re-POSTs.
+ expect(outcome).not.toBe("still-holding");
+ expect(outcome).not.toBe("resolved");
+ expect((outcome as Error).name).toBe("TimeoutError");
+ expect(episodeCount()).toBe(0);
+ expect((await capturedRecoveryLines()).some((l) => l.includes("opened for"))).toBe(false);
+
+ ac.abort(new DOMException("done", "AbortError"));
+ await p.catch(() => {});
+ } finally {
+ hung.stop(true);
+ }
+ }, 15_000);
+
+ test("but OUR OWN clamp still is one — the ladder can retry what it gave up on", async () => {
+ // The other half, and the reason the discriminator cannot simply be "drop
+ // TimeoutError": the per-attempt clamp aborts with that same name, and an
+ // attempt WE cut short is a failed attempt to be retried, not an error
+ // that escapes the ladder. It carries `markOwnTimeout`, so it classifies.
+ const clampAbort = markOwnTimeout(new DOMException("attempt cap", "TimeoutError"));
+ expect(classifyConnectionError(clampAbort)).toEqual({
+ kind: "unreachable",
+ code: "TimeoutError",
+ });
+ // Provoked, not hand-written: this is what a real `AbortSignal.timeout`
+ // rejection looks like, and it must NOT classify.
+ const hung = Bun.serve({ port: 0, fetch: () => new Promise<Response>(() => {}) });
+ try {
+ let raised: unknown;
+ try {
+ await fetch(`http://127.0.0.1:${hung.port}/`, { signal: AbortSignal.timeout(40) });
+ } catch (e) {
+ raised = e;
+ }
+ expect((raised as Error).name).toBe("TimeoutError");
+ expect(classifyConnectionError(raised)).toBeNull();
+ } finally {
+ hung.stop(true);
+ }
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// `[q] give up`
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("[q] give up stops the HOLD, not only the banner", () => {
+ /**
+ * `bye` suppressed the pane for 60 s and gave up on the episodes alive at
+ * that instant — and nothing else. The next request against the same dead
+ * target opened a NEW episode, found the pane suppressed, got no pane and no
+ * lease, and then held its socket for the full ~4.5-minute deadline with the
+ * reason legible NOWHERE before answering 400.
+ *
+ * Claude Code issues concurrent requests during an outage — the main loop,
+ * the title model, subagents — so a request arriving inside the suppression
+ * window is the EXPECTED case. Before this feature existed those requests
+ * failed in milliseconds.
+ */
+ test("a request arriving after the key does not hang — it fails fast, as it used to", async () => {
+ // The real command, through the real handler the socket server calls.
+ __handleCommandForTests({
+ v: RECOVERY_PROTOCOL_VERSION,
+ type: "bye",
+ reason: "user_quit",
+ });
+
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const req = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });
+ const { c, captured } = contextFor(req);
+
+ const started = performance.now();
+ await handler.handle(c, PAYLOAD);
+ const elapsed = performance.now() - started;
+
+ expect(elapsed).toBeLessThan(1_000);
+ expect(captured.status).toBe(400);
+ expect(captured.body?.error?.type).toBe("connection_error");
+ // No episode: the hold is what the user asked to stop.
+ expect(episodeCount()).toBe(0);
+ expect((await capturedRecoveryLines()).some((l) => l.includes("skipped (gave-up)"))).toBe(true);
+ });
+
+ test("and the suppression expires with the banner's, not before or after it", async () => {
+ const clock = new FakeClock(0);
+ setRecoveryClock(clock);
+ __handleCommandForTests({
+ v: RECOVERY_PROTOCOL_VERSION,
+ type: "bye",
+ reason: "user_quit",
+ });
+ expect(recoveryGiveUpActive()).toBe(true);
+
+ // One number, read by the hold and by the surface. A minute later the
+ // ladder is available again — the key is a pause, not an off switch.
+ await clock.advance(UI_SUPPRESS_AFTER_BYE_MS - 1);
+ expect(recoveryGiveUpActive()).toBe(true);
+ await clock.advance(2);
+ expect(recoveryGiveUpActive()).toBe(false);
+ });
+});
diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.ts
index 17049fdb..53e7b623 100644
--- a/packages/cli/src/handlers/composed-handler.ts
+++ b/packages/cli/src/handlers/composed-handler.ts
@@ -45,7 +45,7 @@ import {
noteTargetReachable,
uiLeaseValid,
} from "../recovery/coordinator.js";
-import { resolveRecoveryEnabled } from "../recovery/settings.js";
+import { recoveryGiveUpActive, resolveRecoveryEnabled } from "../recovery/settings.js";
import {
type OpenAIImageBlock,
type VisionProxyAuthHeaders,
@@ -624,8 +624,9 @@ export class ComposedHandler implements ModelHandler {
* Should the retry ladder be skipped entirely, answering today's immediate
* 400? Returns the REASON, or null to proceed.
*
- * THREE GATES, AND DELIBERATELY NO FOURTH. Each is a fact available at this
- * instant — a request header, configuration, and the clock.
+ * FOUR GATES, AND DELIBERATELY NO FIFTH. Each is a fact available at this
+ * instant — a request header, configuration, the user's explicit give-up,
+ * and the clock.
*
* There is NO special case for a refused loopback address, and that absence
* is a decision rather than an omission. An earlier design skipped the ladder
@@ -649,6 +650,12 @@ export class ComposedHandler implements ModelHandler {
// The CI / scripted-`-p` master switch. Off restores today's behaviour
// everywhere, byte for byte.
if (!resolveRecoveryEnabled()) return "recovery-disabled";
+ // The user pressed `[q] give up`. It has to be read HERE and not only in
+ // the UI manager: suppressing the banner alone left the NEXT request —
+ // and during an outage Claude Code always has one — holding its socket for
+ // the full deadline with the reason legible nowhere. The key says stop, so
+ // the hold stops too, for the same window the surface does.
+ if (recoveryGiveUpActive()) return "gave-up";
// Attempt 1 already spent the budget. A guard, not a hope.
if (recoveryClock().now() + MIN_ATTEMPT_SLOT_MS > deadlineAt) return "no-budget";
return null;
@@ -1290,6 +1297,20 @@ export class ComposedHandler implements ModelHandler {
// that finally CONNECTS behave differently from the one that failed —
// and at the moment a network returns, N woken waiters would stampede
// unqueued into a provider that has just come back.
+ //
+ // `getRequestInit()` IS CALLED AGAIN, PER ATTEMPT, and that is not a
+ // tidiness preference. A transport may return a ONE-SHOT signal from it —
+ // `vertex-oauth.ts` returns `AbortSignal.timeout(30000)`, `local.ts` a
+ // ten-minute one — and `mergeSignalIntoInit` composes it with the clamp
+ // via `AbortSignal.any`. Re-using the hoisted object meant that from 30 s
+ // after the FIRST call, every ladder attempt was handed an
+ // already-aborted signal and rejected instantly without touching the
+ // network: tier 1 silently dead past t+30 s, the request held for the
+ // full ~270 s deadline making ZERO real connect attempts, while the
+ // `[Recovery]` log and the pane both reported attempts that never left
+ // the process. A transport ceiling that must survive across attempts has
+ // to be RE-MINTED, never re-used. `doParamRetry` and `doAuthRetry` below
+ // already call it freshly per attempt; this is the same rule.
const doFetchWith = (sig: AbortSignal) =>
fetch(
endpoint,
@@ -1298,7 +1319,7 @@ export class ComposedHandler implements ModelHandler {
method: "POST",
headers,
body: serialized?.body ?? JSON.stringify(requestPayload),
- ...requestInit,
+ ...(this.provider.getRequestInit?.() || {}),
},
sig
)
diff --git a/packages/cli/src/handlers/recovery-status.test.ts b/packages/cli/src/handlers/recovery-status.test.ts
index 7ccab88d..9a5f5290 100644
--- a/packages/cli/src/handlers/recovery-status.test.ts
+++ b/packages/cli/src/handlers/recovery-status.test.ts
@@ -647,31 +647,60 @@ describe("the client's re-POST rejoins the SAME episode", () => {
// The client's retry budget
// ───────────────────────────────────────────────────────────────────────────
-describe("CLAUDE_CODE_RETRY_WATCHDOG follows the recovery UI switch", () => {
- const KEY = "CLAUDISH_RECOVERY_UI";
- let saved: string | undefined;
+describe("CLAUDE_CODE_RETRY_WATCHDOG needs all three gates", () => {
+ const UI = "CLAUDISH_RECOVERY_UI";
+ const ON = "CLAUDISH_RECOVERY";
+ let savedUi: string | undefined;
+ let savedOn: string | undefined;
beforeEach(() => {
- saved = process.env[KEY];
+ savedUi = process.env[UI];
+ savedOn = process.env[ON];
});
afterEach(() => {
- if (saved === undefined) delete process.env[KEY];
- else process.env[KEY] = saved;
+ if (savedUi === undefined) delete process.env[UI];
+ else process.env[UI] = savedUi;
+ if (savedOn === undefined) delete process.env[ON];
+ else process.env[ON] = savedOn;
});
- test("set when the UI is enabled — this is what makes the reach ~a day", () => {
- delete process.env[KEY];
- expect(retryWatchdogEnv()).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });
+ test("set when the UI is enabled and a pane is reachable — the reach is ~a day", () => {
+ delete process.env[UI];
+ delete process.env[ON];
+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({ CLAUDE_CODE_RETRY_WATCHDOG: "1" });
});
test("absent when the user turned the UI off", () => {
// No pane means no lease means an inline 400 at exhaustion means nothing to
// hand back — so extending the client's budget would buy only the side
// effect on claudish's other 503s.
- process.env[KEY] = "0";
- expect(retryWatchdogEnv()).toEqual({});
- expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv()).toBe(false);
+ delete process.env[ON];
+ process.env[UI] = "0";
+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});
+ expect("CLAUDE_CODE_RETRY_WATCHDOG" in retryWatchdogEnv({ paneEligible: true })).toBe(false);
+ });
+
+ test("absent when THIS LAUNCH can never obtain a pane — `-p`, no TTY, no magmux", () => {
+ // The gate that was missing. Those launches cannot hold the lease a
+ // recovery 503 requires, so the watchdog would apply the accepted
+ // duplicate-request exposure to every UNRELATED 503 the session sees and
+ // return nothing for it: ~300 attempts on an upstream overload, for hours,
+ // in exactly the configurations that opted out.
+ delete process.env[UI];
+ delete process.env[ON];
+ expect(retryWatchdogEnv({ paneEligible: false })).toEqual({});
+ });
+
+ test("absent under --no-recovery, however the UI is configured", () => {
+ // RISK-7's promise is that `--no-recovery` restores pre-recovery behaviour
+ // byte for byte. A ladder that never runs hands nothing back, so a client
+ // looping 300 times is pure cost — and CI is where this switch lives.
+ process.env[ON] = "0";
+ delete process.env[UI];
+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});
+ process.env[UI] = "1";
+ expect(retryWatchdogEnv({ paneEligible: true })).toEqual({});
});
});
diff --git a/packages/cli/src/handlers/shared/connection-error.ts b/packages/cli/src/handlers/shared/connection-error.ts
index b04f08f9..eaf91bfc 100644
--- a/packages/cli/src/handlers/shared/connection-error.ts
+++ b/packages/cli/src/handlers/shared/connection-error.ts
@@ -49,20 +49,43 @@ const BUN_CONNECT_MESSAGE = /unable to connect\. is the computer able to access
/**
* Error NAMES that mean the same thing as a connect code, matched separately
- * because they do not arrive as one.
+ * because they do not arrive as one — AND ONLY WHEN THE SIGNAL WAS OURS.
*
* `AbortSignal.timeout(...)` rejects with a `DOMException` whose `name` is
* `TimeoutError` and whose `code` is the NUMBER 23 — so `findConnectionCode`'s
* `typeof e.code === "string"` test skipped it, its message ("The operation
* timed out.") matched no fallback, and classification returned **null**. A
- * provider that accepts the TCP connection and then never answers therefore
- * fell straight through into a bare 500, with none of the actionable wording
- * every other unreachable-host failure gets.
+ * REACHABILITY PROBE that hung therefore fell straight through into a bare
+ * 500, with none of the actionable wording every other unreachable-host
+ * failure gets.
*
- * "Refused is covered, hung is not" was the whole of the gap. It bites hardest
- * on local providers, whose health probes carry an `AbortSignal.timeout` and
- * whose inference requests carry a ten-minute one — a wedged Ollama is exactly
- * the fault a retry ladder is for.
+ * ── WHY THE NAME ALONE IS NOT THE DISCRIMINATOR ─────────────────────────────
+ *
+ * `TimeoutError` is also what a TRANSPORT'S OWN request ceiling rejects with.
+ * `vertex-oauth.ts` puts `AbortSignal.timeout(30000)` on the *inference* call
+ * and `local.ts` a ten-minute one. Classifying by name alone turned "the model
+ * took longer than its transport's ceiling" — a LATENCY event, ordinary for a
+ * thinking model on a streaming endpoint — into `unreachable`: it entered the
+ * retry ladder, took the tier-2 handoff, and with `CLAUDE_CODE_RETRY_WATCHDOG`
+ * on it was re-POSTed ~300 times, each one running ONE MORE REAL BILLED
+ * INFERENCE against a host that had answered the TCP connect and was already
+ * generating. RISK-6 (`network-recovery.md` §7) was accepted for
+ * `ECONNRESET`/`EPIPE` on a genuine network fault. It was never accepted for a
+ * latency event, which is neither a network fault nor transient — and
+ * `buildConnectionErrorMessage` then told the user to check their VPN and DNS
+ * for a host that was mid-generation.
+ *
+ * So the discriminator is the SIGNAL'S ORIGIN, carried on the error as an own
+ * property, not the name:
+ *
+ * - the ladder's per-attempt clamp (`transient-retry.ts`) — WE gave up on
+ * this attempt, and re-issuing is the entire point;
+ * - a reachability PROBE we issued with our own short timeout (`local.ts`'s
+ * 5 s `/api/tags`) — WE asked "is anything there" and got no answer.
+ *
+ * Both mean "we could not get a response out of this host inside a window WE
+ * chose". A transport's inference ceiling means something else, so it keeps
+ * its pre-recovery route out of here: unclassified, rethrown, untouched.
*
* ONLY `TimeoutError`. `AbortError` is deliberately absent and must stay
* absent: it is how a client says it has gone away and how our own per-attempt
@@ -73,6 +96,43 @@ const NAME_KIND: Record<string, ConnectionErrorKind> = {
TimeoutError: "unreachable",
};
+/**
+ * The own-property that says "this timeout is CLAUDISH'S OWN".
+ *
+ * Non-enumerable on purpose: it must not change how an error serialises into a
+ * log, a stats record or an upstream error body.
+ */
+const OWN_TIMEOUT_FLAG = "claudishOwnTimeout";
+
+/**
+ * Tag an error as raised by a timeout CLAUDISH set on a reachability question
+ * of its own — the retry ladder's per-attempt clamp, or a health probe.
+ *
+ * Returns the same object, so it composes inside an `abort(...)` call or an
+ * assignment. Defensive about frozen errors: a failure to tag degrades to
+ * "unclassified", which is the safe direction (no ladder, no re-issue).
+ */
+export function markOwnTimeout<T>(error: T): T {
+ if (error && typeof error === "object") {
+ try {
+ Object.defineProperty(error, OWN_TIMEOUT_FLAG, {
+ value: true,
+ enumerable: false,
+ configurable: true,
+ writable: true,
+ });
+ } catch {
+ /* frozen — falls through to unclassified, which is the safe direction */
+ }
+ }
+ return error;
+}
+
+/** Was this error raised by one of OUR timeouts? See `markOwnTimeout`. */
+function isOwnTimeout(error: unknown): boolean {
+ return Boolean((error as Record<string, unknown> | null | undefined)?.[OWN_TIMEOUT_FLAG]);
+}
+
/**
* Walk an error and its `cause` chain (undici's `TypeError: fetch failed` wraps
* the real syscall error in `.cause`) and return the first known connection
@@ -89,7 +149,11 @@ function findConnectionCode(error: unknown): string | null {
// on it. Walked at the same depth as `.code`, because a transport that
// rethrows with `{ cause }` buries a hung-probe timeout just as deeply as
// it buries a refusal.
- if (typeof e.name === "string" && e.name in NAME_KIND) return e.name;
+ //
+ // `isOwnTimeout(e)` is the load-bearing half: a `TimeoutError` raised by a
+ // transport's own INFERENCE ceiling is not a reachability fact and must not
+ // enter the ladder. See `NAME_KIND`'s header for what that cost.
+ if (typeof e.name === "string" && e.name in NAME_KIND && isOwnTimeout(e)) return e.name;
e = e.cause;
}
const msg = String((error as any)?.message ?? error ?? "");
diff --git a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts
index 5d29e155..9353031a 100644
--- a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts
+++ b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts
@@ -40,7 +40,7 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test";
import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";
import { closeAllEpisodes } from "../../recovery/coordinator.js";
import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";
-import { classifyConnectionError } from "./connection-error.js";
+import { classifyConnectionError, markOwnTimeout } from "./connection-error.js";
import {
PER_ATTEMPT_CONNECT_CAP_MS,
mergeSignalIntoInit,
@@ -167,9 +167,12 @@ describe("F-SLOW — a host that swallows the connection", () => {
// Why the ladder can treat its own timeout as a retryable network fault:
// `AbortSignal.timeout` rejects with a DOMException whose `name` is
// `TimeoutError` and whose `code` is the NUMBER 23, so a classifier keyed
- // only on a string `code` returned null and a hung upstream fell through
- // into a bare 500.
- const clampAbort = new DOMException("attempt cap", "TimeoutError");
+ // only on a string `code` returned null and a hung probe fell through into
+ // a bare 500.
+ //
+ // `markOwnTimeout` is what the clamp adds — see the sibling test below for
+ // why the NAME alone must not be enough.
+ const clampAbort = markOwnTimeout(new DOMException("attempt cap", "TimeoutError"));
const conn = classifyConnectionError(clampAbort);
expect(conn).not.toBeNull();
expect(conn?.kind).toBe("unreachable");
diff --git a/packages/cli/src/handlers/shared/transient-retry.test.ts b/packages/cli/src/handlers/shared/transient-retry.test.ts
index addcbbcc..a024b81c 100644
--- a/packages/cli/src/handlers/shared/transient-retry.test.ts
+++ b/packages/cli/src/handlers/shared/transient-retry.test.ts
@@ -568,3 +568,76 @@ describe("mergeSignalIntoInit composes rather than replaces", () => {
expect(mergeSignalIntoInit(bare, undefined)).toBe(bare as unknown as RequestInit);
});
});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The clamp binds operations that cannot take a signal
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("the per-attempt clamp bounds an op that IGNORES its signal", () => {
+ /**
+ * The fetch path threads the signal into `fetch`. THE AUTH PATH CANNOT:
+ * `refreshAuth()` and `getHeaders()` take no arguments, so the handler's
+ * `() => this.provider.refreshAuth!()` discards the signal, and Grok's token
+ * exchange then performs an unbounded `fetch(auth.x.ai/oauth2/token)`. A
+ * swallowed connection there outlived the 45-second cap AND the client's own
+ * disconnect, while this function sat awaiting a promise nothing could
+ * settle — the unbounded hold this whole feature exists to remove, reached
+ * from inside the machinery that removes it.
+ *
+ * A deadline one path can ignore is not a deadline, so it is enforced at the
+ * one place that owns it.
+ */
+ test("an attempt that never settles ends at the cap and the ladder goes on", async () => {
+ useFakeClock();
+ const started: number[] = [];
+ let settled = 0;
+ // Signal ignored entirely, exactly like `() => provider.refreshAuth!()`.
+ const deafOp = async () => {
+ started.push(clock.now());
+ if (started.length === 1) {
+ await new Promise(() => {}); // never settles, never observes the signal
+ }
+ settled++;
+ throw CONNECT_FAILURE;
+ };
+
+ const result = await advanceUntilSettled(
+ clock,
+ withConnectionRetry(deafOp, CONNECT_FAILURE, ctx({ deadlineAt: 200_000 })),
+ 400_000
+ );
+
+ // Attempt 1 was abandoned at the cap rather than awaited forever, so the
+ // ladder reached its later rungs and the request answered inside its
+ // budget. Without the enforcement this promise never resolves at all.
+ expect(started.length).toBeGreaterThan(1);
+ // Attempt 1 ended AT THE CAP — not at the deadline, and not never — and the
+ // next rung's own gap follows it.
+ expect((started[1] as number) - (started[0] as number)).toBe(
+ PER_ATTEMPT_CONNECT_CAP_MS + connectionRetryDelayMs(1)
+ );
+ expect(result.kind).toBe("exhausted");
+ expect(settled).toBeGreaterThan(0);
+ }, 15_000);
+
+ test("a client disconnect releases such an op too, not only a fetch", async () => {
+ useFakeClock();
+ const ac = new AbortController();
+ const deafOp = async () => {
+ ac.abort(new DOMException("client gone", "AbortError"));
+ await new Promise(() => {});
+ return "never" as unknown;
+ };
+
+ const result = await advanceUntilSettled(
+ clock,
+ withConnectionRetry(deafOp, CONNECT_FAILURE, ctx({ signal: ac.signal })),
+ 400_000
+ );
+
+ // NFR-2 for the auth path: the socket is already gone, so the waiter must
+ // unwind now rather than when an unbounded auth fetch eventually gives up.
+ expect(result.kind).toBe("client_gone");
+ expect(episodeCount()).toBe(0);
+ }, 15_000);
+});
diff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.ts
index 5c6043ba..9cc7bb36 100644
--- a/packages/cli/src/handlers/shared/transient-retry.ts
+++ b/packages/cli/src/handlers/shared/transient-retry.ts
@@ -33,6 +33,7 @@ import {
type ConnectionErrorKind,
buildConnectionErrorMessage,
classifyConnectionError,
+ markOwnTimeout,
} from "./connection-error.js";
export { EPISODE_GRACE_MS } from "../../recovery/coordinator.js";
@@ -142,6 +143,19 @@ export interface ConnectionErrorInfo {
* overwriting is a ten-minute `AbortSignal.timeout`. Compose rather than
* replace: the per-attempt clamp then bounds the attempt while the transport's
* own ceiling still applies, which is what both of them were for.
+ *
+ * ── AN ALREADY-FIRED `own` IS DROPPED, NOT COMPOSED ─────────────────────────
+ *
+ * `AbortSignal.any([fired, live])` returns a signal that is ALREADY ABORTED, so
+ * the fetch rejects before it opens a socket — measured on this machine's Bun.
+ * A transport's `getRequestInit()` signal is one-shot, so a caller that hoists
+ * the init out of its retry loop poisons every later attempt with a ceiling
+ * that expired during attempt 1: real connect attempts stop happening while the
+ * ladder, the log and the pane all go on reporting them.
+ *
+ * Callers must re-mint the init per attempt (`composed-handler.ts` does), and
+ * this is the belt behind that brace: an expired ceiling has already had its
+ * say, and carrying it forward can only convert a live attempt into a no-op.
*/
export function mergeSignalIntoInit(
init: Record<string, unknown>,
@@ -149,7 +163,8 @@ export function mergeSignalIntoInit(
): RequestInit {
if (!signal) return init as RequestInit;
const own = init.signal as AbortSignal | undefined;
- return { ...init, signal: own ? AbortSignal.any([own, signal]) : signal } as RequestInit;
+ if (!own || own.aborted) return { ...init, signal } as RequestInit;
+ return { ...init, signal: AbortSignal.any([own, signal]) } as RequestInit;
}
export interface ConnectionRetryContext {
@@ -234,6 +249,54 @@ function isClientAbort(signal: AbortSignal, err: unknown): boolean {
return (err as { name?: string })?.name === "AbortError";
}
+/**
+ * Await `p`, but stop waiting when `signal` fires.
+ *
+ * ── WHY THE CLAMP CANNOT JUST BE HANDED OVER AND TRUSTED ────────────────────
+ *
+ * `op` takes the signal, and the fetch path threads it all the way to `fetch`.
+ * THE AUTH PATH CANNOT: `refreshAuth()` and `getHeaders()` take no arguments on
+ * the transport interface, so `() => this.provider.refreshAuth!()` closes over
+ * nothing and discards the signal entirely. Grok's token exchange then performs
+ * an UNBOUNDED `fetch(auth.x.ai/oauth2/token)` — so a swallowed connection
+ * outlived both the 45-second attempt cap and the client's own disconnect,
+ * while `withConnectionRetry` sat awaiting a promise nothing could settle. A
+ * deadline that one path can ignore is not a deadline.
+ *
+ * So the clamp is enforced HERE, at the one place that owns it, rather than in
+ * N transports that would each have to remember. The operation is abandoned,
+ * not cancelled — it may still be running — but the waiter unwinds, the ladder
+ * advances, and the request answers inside its budget, which is what the budget
+ * was for. Threading a real signal into every auth call is the deeper fix and
+ * is worth doing; this is what makes the ceiling true in the meantime.
+ */
+function untilAborted<T>(p: Promise<T>, signal: AbortSignal): Promise<T> {
+ if (signal.aborted) {
+ // Nothing else will ever observe `p`, so claim its rejection here or Bun
+ // reports an unhandled one.
+ void p.catch(() => {});
+ return Promise.reject(signal.reason ?? new DOMException("aborted", "AbortError"));
+ }
+ return new Promise<T>((resolve, reject) => {
+ const onAbort = () => {
+ reject(signal.reason ?? new DOMException("aborted", "AbortError"));
+ };
+ signal.addEventListener("abort", onAbort, { once: true });
+ // Registered unconditionally, so a late settle of an abandoned operation is
+ // always handled — `resolve`/`reject` after settlement are no-ops.
+ p.then(
+ (v) => {
+ signal.removeEventListener("abort", onAbort);
+ resolve(v);
+ },
+ (e) => {
+ signal.removeEventListener("abort", onAbort);
+ reject(e);
+ }
+ );
+ });
+}
+
/**
* Re-issue `op` on a classified connection error, on the shared episode clock.
*
@@ -241,6 +304,8 @@ function isClientAbort(signal: AbortSignal, err: unknown): boolean {
* single contract cover both the fetch path and the auth path: the per-attempt
* clamp is built here, merged with the client signal here, and handed in, so
* the clamp bounds the OPERATION rather than only the `fetch` buried inside it.
+ * `untilAborted` then makes that binding rather than advisory, for the auth
+ * closures that structurally cannot accept a signal at all.
*
* Classification runs on EVERY attempt, not just the first. A retry that throws
* an auth failure, a transport bug or a programming error must not be
@@ -348,15 +413,19 @@ export async function withConnectionRetry<T>(
MIN_ATTEMPT_SLOT_MS,
Math.min(PER_ATTEMPT_CONNECT_CAP_MS, remaining)
);
+ // `markOwnTimeout` is what makes this abort re-classifiable at all.
+ // `classifyConnectionError` keys `TimeoutError` on the SIGNAL'S ORIGIN
+ // rather than on the name, precisely so a transport's own inference
+ // ceiling stays out of the ladder — so the clamp has to say it is ours.
const clampTimer = clock.setTimeout(
- () => attemptAc.abort(new DOMException("attempt cap", "TimeoutError")),
+ () => attemptAc.abort(markOwnTimeout(new DOMException("attempt cap", "TimeoutError"))),
clampMs
);
const merged = AbortSignal.any([ctx.signal, attemptAc.signal]);
try {
requestRetries++;
- const value = await op(merged);
+ const value = await untilAborted(op(merged), merged);
handle.recordAttemptResult("ok", true);
log(
`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +
diff --git a/packages/cli/src/launcher/magmux-wrapper.test.ts b/packages/cli/src/launcher/magmux-wrapper.test.ts
index a7e05d3b..65527496 100644
--- a/packages/cli/src/launcher/magmux-wrapper.test.ts
+++ b/packages/cli/src/launcher/magmux-wrapper.test.ts
@@ -10,13 +10,14 @@
*/
import { afterEach, describe, expect, test } from "bun:test";
-import { mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
-import { join } from "node:path";
+import { dirname, join } from "node:path";
import {
STRIPPED_CHILD_VARS,
buildLauncherScript,
envDeltas,
+ magmuxPaneCapability,
planMagmuxWrap,
} from "./magmux-wrapper.js";
@@ -195,7 +196,7 @@ describe("planMagmuxWrap", () => {
expect(script).not.toContain("sk-inherited-secret");
expect(script).not.toContain("PATH=");
expect(statSync(plan?.scriptPath as string).mode & 0o777).toBe(0o600);
- expect(statSync(join(root, "claudish-launch-4242")).mode & 0o777).toBe(0o700);
+ expect(statSync(dirname(plan?.scriptPath as string)).mode & 0o777).toBe(0o700);
});
test("MAGMUX_SCROLLBACK defaults to 10000 but never overrides the user's own", () => {
@@ -238,3 +239,141 @@ describe("planMagmuxWrap", () => {
expect(plan?.paneExitCode()).toBeNull();
});
});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The launcher script is a SECRET on disk
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("the script's directory is unguessable and cannot be pre-created", () => {
+ /**
+ * The file this guards is sourced by the LOGIN SHELL immediately before
+ * `exec`, and it carries claudish's env deltas — including, on the proxy
+ * path, the `ANTHROPIC_API_KEY` Claude Code authenticates to us with.
+ *
+ * It used to live at `<tmp>/claudish-launch-<pid>`, created with
+ * `mkdirSync(..., { recursive: true })`. `recursive: true` neither throws on
+ * `EEXIST` nor applies `mode` to a directory that already exists, and the
+ * name contains only the pid, which is enumerable — so on a multi-user host
+ * another uid could pre-create (or symlink) that path and have it adopted
+ * without a word, then read the session's secrets or swap the script between
+ * the write and the `exec`. CWE-377 / CWE-59.
+ *
+ * The rest of this change already knew the rule: `socket-server.ts` uses
+ * `randomBytes(12)` for its directory and `O_EXCL` for the pane lock.
+ */
+ test("the path does not contain the pid, and two plans never collide", () => {
+ const root = scratch();
+ const a = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: root });
+ const b = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: root });
+
+ // Same pid, same tmpRoot, different directory: the name is random, so it
+ // cannot be pre-created by someone who knows the pid.
+ expect(dirname(a?.scriptPath as string)).not.toBe(dirname(b?.scriptPath as string));
+ expect(a?.scriptPath).not.toContain(String(BASE.pid));
+ expect(statSync(dirname(a?.scriptPath as string)).mode & 0o777).toBe(0o700);
+ expect(statSync(a?.scriptPath as string).mode & 0o777).toBe(0o600);
+ });
+
+ test("a directory planted at the OLD predictable path is never adopted", () => {
+ // The attack, verbatim: another uid creates the path first — world-writable
+ // or a symlink into its own tree — and waits for claudish to write the
+ // session's environment into it. `recursive: true` adopted it silently and
+ // left its mode alone, and `writeFileSync` follows whatever is already
+ // there. `mkdtempSync` + `wx` makes both impossible.
+ const root = scratch();
+ const planted = join(root, `claudish-launch-${BASE.pid}`);
+ mkdirSync(planted, { recursive: true, mode: 0o777 });
+ const decoy = join(planted, "claude-launch.sh");
+ writeFileSync(decoy, "# planted");
+
+ const plan = planMagmuxWrap({
+ ...BASE,
+ childEnv: { ANTHROPIC_API_KEY: "sk-session-secret" },
+ tmpRoot: root,
+ });
+
+ expect(dirname(plan?.scriptPath as string)).not.toBe(planted);
+ // The secret did not land anywhere the planter can read.
+ expect(readFileSync(decoy, "utf-8")).toBe("# planted");
+ expect(readFileSync(plan?.scriptPath as string, "utf-8")).toContain("sk-session-secret");
+ expect(statSync(dirname(plan?.scriptPath as string)).mode & 0o777).toBe(0o700);
+ });
+
+ test("a plan that is never watched still takes its secret off disk at exit", () => {
+ // `watch()` cleans up on the CHILD's exit, which covers the ordinary path
+ // only. A SIGTERM to claudish, or a launch that never spawns, would
+ // otherwise leave the key in /tmp.
+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });
+ expect(process.listeners("exit").length).toBeGreaterThan(0);
+ plan?.cleanup();
+ expect(() => statSync(plan?.scriptPath as string)).toThrow();
+ // Idempotent, and it unhooks itself: a long-lived host must not accumulate
+ // one exit listener per launch.
+ plan?.cleanup();
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Eligibility, asked separately because the watchdog has to ask it EARLY
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("magmuxPaneCapability", () => {
+ test("wrap when interactive, on a TTY, with a binary and no ambient socket", () => {
+ expect(
+ magmuxPaneCapability({
+ interactive: true,
+ stdoutIsTty: true,
+ magmuxBinary: "/opt/homebrew/bin/magmux",
+ parentEnv: {},
+ })
+ ).toEqual({ kind: "wrap", magmux: "/opt/homebrew/bin/magmux" });
+ });
+
+ test("ambient inside someone else's magmux — a pane is reachable without wrapping", () => {
+ // `team --grid --mode interactive`, or a user who launched claudish in a
+ // pane by hand. Nothing to wrap, but there IS a multiplexer to ask.
+ expect(
+ magmuxPaneCapability({
+ interactive: true,
+ stdoutIsTty: true,
+ magmuxBinary: "/opt/homebrew/bin/magmux",
+ parentEnv: { MAGMUX_SOCK: "/tmp/magmux-someone-else.sock" },
+ })
+ ).toEqual({ kind: "ambient", sock: "/tmp/magmux-someone-else.sock" });
+ });
+
+ test("none for `-p`, for a pipe, and for a machine without magmux", () => {
+ const env = {};
+ expect(
+ magmuxPaneCapability({ interactive: false, stdoutIsTty: true, parentEnv: env }).kind
+ ).toBe("none");
+ expect(
+ magmuxPaneCapability({ interactive: true, stdoutIsTty: false, parentEnv: env }).kind
+ ).toBe("none");
+ expect(
+ magmuxPaneCapability({
+ interactive: true,
+ stdoutIsTty: true,
+ magmuxBinary: null,
+ parentEnv: env,
+ }).kind
+ ).toBe("none");
+ });
+
+ test("it is the SAME predicate planMagmuxWrap uses — they cannot drift", () => {
+ // The watchdog is decided from `magmuxPaneCapability` before the child
+ // environment is finalised; the wrap is decided from `planMagmuxWrap`
+ // afterwards. Two independent gates would mean a launch that exports the
+ // ~300-attempt retry budget and then turns out to have no surface for it.
+ for (const input of [
+ { interactive: false, stdoutIsTty: true },
+ { interactive: true, stdoutIsTty: false },
+ { interactive: true, stdoutIsTty: true, magmuxBinary: null },
+ { interactive: true, stdoutIsTty: true },
+ ] as const) {
+ const shape = { ...BASE, ...input, childEnv: {}, tmpRoot: scratch(), parentEnv: {} };
+ const wrapped = planMagmuxWrap(shape) !== null;
+ expect(wrapped).toBe(magmuxPaneCapability(shape).kind === "wrap");
+ }
+ });
+});
diff --git a/packages/cli/src/launcher/magmux-wrapper.ts b/packages/cli/src/launcher/magmux-wrapper.ts
index a1c6da55..0c4b5a5b 100644
--- a/packages/cli/src/launcher/magmux-wrapper.ts
+++ b/packages/cli/src/launcher/magmux-wrapper.ts
@@ -35,7 +35,16 @@
*/
import type { ChildProcess } from "node:child_process";
-import { chmodSync, existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
+import {
+ chmodSync,
+ closeSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ openSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
import { type Socket, connect as netConnect } from "node:net";
import { join } from "node:path";
import { findMagmuxBinaryOrNull } from "./magmux-binary.js";
@@ -140,14 +149,34 @@ export function buildLauncherScript(
}
/**
- * Build the plan, or null when this launch must not be wrapped.
+ * Can this launch obtain a recovery pane at all, and how?
*
- * Returns null — never throws — for every "not applicable" case, because a
- * missing multiplexer means the session should launch exactly as it did before
- * this feature existed.
+ * SIDE-EFFECT FREE — it writes nothing, spawns nothing, and may therefore be
+ * asked BEFORE the child environment is finalised. That ordering is the whole
+ * reason it exists as its own function: `CLAUDE_CODE_RETRY_WATCHDOG` expands
+ * Claude Code's retry budget from ~11 attempts to ~300 for EVERY 503 the
+ * session sees, so it may only be exported by a launch that can actually put
+ * the reason on a surface. Exporting it from the UI preference alone applied
+ * the accepted duplicate-request exposure (RISK-6) in `-p`, under
+ * `--no-recovery`, and on machines without magmux — the configurations that
+ * opted out — where it buys nothing at all.
+ *
+ * `wrap` — we will start magmux ourselves and own the control socket.
+ * `ambient` — already inside someone else's magmux (`team --grid`, or a user
+ * who launched claudish in a pane): nothing to wrap, but there IS a
+ * multiplexer to ask for a pane.
+ * `none` — no surface is reachable from this launch.
*/
-export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {
- if (!input.interactive || !input.stdoutIsTty) return null;
+export type MagmuxPaneCapability =
+ | { kind: "wrap"; magmux: string }
+ | { kind: "ambient"; sock: string }
+ | { kind: "none"; reason: "not-interactive" | "no-tty" | "no-magmux" };
+
+export function magmuxPaneCapability(
+ input: Pick<MagmuxWrapInput, "interactive" | "stdoutIsTty" | "parentEnv" | "magmuxBinary">
+): MagmuxPaneCapability {
+ const parentEnv = input.parentEnv ?? process.env;
+ if (!input.interactive) return { kind: "none", reason: "not-interactive" };
// ALREADY INSIDE A MULTIPLEXER — do not nest one.
//
// `team --grid --mode interactive` launches one `claudish --model X -i` per
@@ -160,32 +189,72 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {
// installed against that AMBIENT socket instead, and the cross-process
// `O_EXCL` lock in `magmux-ui.ts` is what keeps the grid to ONE banner rather
// than N.
- if ((input.parentEnv ?? process.env).MAGMUX_SOCK) return null;
+ //
+ // Checked BEFORE the TTY gate on purpose, to match `claude-runner.ts`'s
+ // ambient branch, which asks only for `interactive && MAGMUX_SOCK`.
+ const ambient = parentEnv.MAGMUX_SOCK;
+ if (ambient) return { kind: "ambient", sock: ambient };
+ if (!input.stdoutIsTty) return { kind: "none", reason: "no-tty" };
const magmux = input.magmuxBinary === undefined ? findMagmuxBinaryOrNull() : input.magmuxBinary;
- if (!magmux) return null;
+ if (!magmux) return { kind: "none", reason: "no-magmux" };
+ return { kind: "wrap", magmux };
+}
+
+/**
+ * Build the plan, or null when this launch must not be wrapped.
+ *
+ * Returns null — never throws — for every "not applicable" case, because a
+ * missing multiplexer means the session should launch exactly as it did before
+ * this feature existed. The "not applicable" set is `magmuxPaneCapability`'s,
+ * so the gate that decides the watchdog and the gate that decides the wrap
+ * cannot drift apart.
+ */
+export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {
+ const capability = magmuxPaneCapability(input);
+ if (capability.kind !== "wrap") return null;
+ const magmux = capability.magmux;
const pid = input.pid ?? process.pid;
const id = `claudish-${pid}`;
const controlSocket = `/tmp/magmux-${id}.sock`;
- // 0700 directory, 0600 file. The script carries claudish's deltas, which on
- // the proxy path include the key Claude Code authenticates to us with.
+ // ── 0700 UNGUESSABLE directory, 0600 O_EXCL file ──────────────────────────
+ //
+ // The file below is SOURCED by the login shell immediately before `exec`, and
+ // it carries claudish's env deltas — on the proxy path that includes the
+ // `ANTHROPIC_API_KEY` Claude Code authenticates to us with. So both halves of
+ // this matter, and the previous shape got both wrong:
+ //
+ // - the name was `claudish-launch-<pid>`, which is ENUMERABLE, and
+ // `mkdirSync(..., { recursive: true })` neither throws on `EEXIST` nor
+ // applies `mode` to a directory that already exists — so a directory
+ // another uid had pre-created (or symlinked elsewhere) was adopted
+ // silently, and `writeFileSync` follows an existing path. That is
+ // CWE-377/CWE-59: read the session's secrets, or swap the script between
+ // the write and the `exec`, and get code execution as the claudish user.
+ // - `writeFileSync`'s `mode` is a CREATION mode; on an existing file it is
+ // ignored, so the `chmod` was load-bearing and silent when it failed.
+ //
+ // `mkdtempSync` gives an unguessable name AND fails rather than adopting, and
+ // `wx` is `O_CREAT | O_EXCL`, so the file cannot pre-exist as a symlink to
+ // somewhere else. The same rule the rest of this change already follows:
+ // `socket-server.ts` uses `randomBytes(12)` for its directory and `O_EXCL`
+ // for the pane lock.
const tmpRoot = input.tmpRoot ?? "/tmp";
- const scriptDir = join(tmpRoot, `claudish-launch-${pid}`);
- mkdirSync(scriptDir, { mode: 0o700, recursive: true });
+ mkdirSync(tmpRoot, { recursive: true });
+ const scriptDir = mkdtempSync(join(tmpRoot, "claudish-launch-"));
+ chmodSync(scriptDir, 0o700);
// Named for Claude Code on purpose: magmux attaches its controller by
// spotting `claude`/`claudish` in the pane's command.
const scriptPath = join(scriptDir, "claude-launch.sh");
const parentEnv = input.parentEnv ?? process.env;
const deltas = envDeltas(parentEnv, input.childEnv);
- writeFileSync(scriptPath, buildLauncherScript(input.claudeBinary, input.claudeArgs, deltas), {
- mode: 0o600,
- });
+ const fd = openSync(scriptPath, "wx", 0o600);
try {
- chmodSync(scriptPath, 0o600);
- } catch {
- /* the 0700 directory is the real gate */
+ writeFileSync(fd, buildLauncherScript(input.claudeBinary, input.claudeArgs, deltas));
+ } finally {
+ closeSync(fd);
}
const env: Record<string, string> = { ...input.childEnv };
@@ -214,8 +283,12 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {
let paneExit: number | null = null;
const exitListeners: Array<(code: number | null) => void> = [];
let control: Socket | null = null;
+ let cleaned = false;
const cleanup = () => {
+ if (cleaned) return;
+ cleaned = true;
+ process.removeListener("exit", cleanup);
try {
control?.destroy();
} catch {
@@ -236,6 +309,13 @@ export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {
}
};
+ // A file holding the session's `ANTHROPIC_API_KEY` must not outlive the
+ // process on a SIGTERM or a crash-out — `watch()`'s `proc.exit` hook only
+ // covers the ordinary path, and claudish is not always the one that dies
+ // last. Same discipline as `socket-server.ts`'s socket unlink; the listener
+ // removes itself in `cleanup` so a long-lived host cannot accumulate them.
+ process.on("exit", cleanup);
+
return {
command: magmux,
args,
diff --git a/packages/cli/src/profile-config.ts b/packages/cli/src/profile-config.ts
index cb8bcedc..924b6c45 100644
--- a/packages/cli/src/profile-config.ts
+++ b/packages/cli/src/profile-config.ts
@@ -264,10 +264,22 @@ export interface ClaudishProfileConfig {
* Whether claudish may own a terminal surface on which a recovery episode's
* reason is legible. Default ON.
*
- * In this phase it decides one thing only: a REFUSED connection to a
- * loopback endpoint (a stopped Ollama) skips the ladder entirely when the
- * user has turned this off, because nothing could ever display why the
- * request is being held and they have said they do not want it to be.
+ * What it actually decides, and the blast radius is larger than "a banner":
+ *
+ * - whether the interactive session is WRAPPED in a magmux pane at launch
+ * (`claude-runner.ts` → `planMagmuxWrap`);
+ * - whether `CLAUDE_CODE_RETRY_WATCHDOG=1` is exported to Claude Code,
+ * which moves its retry budget from ~11 attempts to ~300 for every 503
+ * the session sees (it is one of three gates — see `retryWatchdogEnv`);
+ * - and through those, whether an exhausted hold can ever answer a
+ * retryable 503 rather than an inline 400, since the 503 requires a live
+ * pane lease.
+ *
+ * It never decides whether a RETRY happens. There is no loopback carve-out
+ * and its absence is deliberate (`composed-handler.ts`'s `shouldSkipTier1`);
+ * an earlier draft of this comment described one, and it had already been
+ * deleted.
+ *
* Precedence: --recovery-ui/--no-recovery-ui flag > CLAUDISH_RECOVERY_UI env
* > project `.claudish.json` > this field > true. The scoped read is
* `readRecoveryUi()`, NOT this allowlist — see its doc comment.
diff --git a/packages/cli/src/providers/grok/grok-credentials.test.ts b/packages/cli/src/providers/grok/grok-credentials.test.ts
index 25284a72..62cdf449 100644
--- a/packages/cli/src/providers/grok/grok-credentials.test.ts
+++ b/packages/cli/src/providers/grok/grok-credentials.test.ts
@@ -3,6 +3,7 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { GROK_PUBLIC_CLIENT_ID } from "../../auth/grok-oauth.js";
+import { classifyConnectionError } from "../../handlers/shared/connection-error.js";
import {
DEFAULT_GROK_PROXY_URL,
FALLBACK_GROK_CLIENT_VERSION,
@@ -599,3 +600,38 @@ describe("forceRefreshGrokAccessToken", () => {
expect(successfulFetch).toHaveBeenCalledTimes(1);
});
});
+
+describe("a refused token exchange names the AUTH host, not the inference host", () => {
+ test("the error carries claudishEndpoint = the token endpoint", async () => {
+ // `getHeaders()` is a `gk@` request's FIRST network touch and it reaches
+ // `auth.x.ai` — never `api.x.ai`. `ComposedHandler` falls back to the MODEL
+ // endpoint when the error carries no `claudishEndpoint`, so without this
+ // the recovery banner, the `[Recovery]` log and the episode key all named a
+ // host that was never contacted, and told the user to check their network
+ // for it. `local.ts` attaches the same property for the same reason.
+ writeAuth({ [OIDC_SCOPE]: expiredEntry() });
+ let connectFailure: unknown;
+ try {
+ await realFetch("http://127.0.0.1:1/refused-on-purpose");
+ } catch (e) {
+ connectFailure = e;
+ }
+ globalThis.fetch = mock(async () => {
+ throw connectFailure;
+ }) as unknown as typeof fetch;
+
+ const error = await rejectedError(resolveGrokAccessToken());
+
+ expect((error as { claudishEndpoint?: string }).claudishEndpoint).toBe(
+ "https://auth.x.ai/oauth2/token"
+ );
+ // And the syscall evidence still rides in `cause`, so this remains a
+ // CONNECTION failure rather than an auth one — a network fault during a
+ // refresh must not walk the user down the fallback chain onto metered
+ // billing.
+ expect(classifyConnectionError(error)).toEqual({
+ kind: "refused",
+ code: "ConnectionRefused",
+ });
+ });
+});
diff --git a/packages/cli/src/providers/grok/grok-credentials.ts b/packages/cli/src/providers/grok/grok-credentials.ts
index f0d0f175..c56a29c3 100644
--- a/packages/cli/src/providers/grok/grok-credentials.ts
+++ b/packages/cli/src/providers/grok/grok-credentials.ts
@@ -430,9 +430,18 @@ async function performRefresh(cred: GrokCredential): Promise<string> {
// FIRST network touch of a `gk@` request (ComposedHandler calls getHeaders()
// before the upstream fetch), so an unclassifiable failure here escapes into
// the fallback chain with `status: 0`.
- throw new Error(
- `Could not reach ${tokenEndpoint} to refresh the Grok token: ${(error as Error).message}`,
- { cause: error }
+ //
+ // `claudishEndpoint` names the host that ACTUALLY failed. Without it the
+ // handler falls back to the model endpoint, so an `auth.x.ai` outage was
+ // reported — in the banner, in the log, and in the episode key — as
+ // `api.x.ai`: the wrong host, and the wrong advice for a user trying to fix
+ // it. `local.ts` attaches the same property for the same reason.
+ throw Object.assign(
+ new Error(
+ `Could not reach ${tokenEndpoint} to refresh the Grok token: ${(error as Error).message}`,
+ { cause: error }
+ ),
+ { claudishEndpoint: tokenEndpoint }
);
}
diff --git a/packages/cli/src/providers/transport/local.ts b/packages/cli/src/providers/transport/local.ts
index 73cf4d46..24ca212d 100644
--- a/packages/cli/src/providers/transport/local.ts
+++ b/packages/cli/src/providers/transport/local.ts
@@ -13,6 +13,7 @@
import { Agent } from "undici";
import { credentials } from "../../auth/credentials/authority.js";
+import { markOwnTimeout } from "../../handlers/shared/connection-error.js";
import { LocalModelQueue } from "../../handlers/shared/local-queue.js";
import { log } from "../../logger.js";
import type { LocalProvider as LocalProviderConfig } from "../../providers/provider-registry.js";
@@ -243,7 +244,13 @@ export class LocalTransport implements ProviderTransport {
} catch (e: any) {
// KEEP the error, do not merely log it. See `lastProbeError`'s docs: this
// catch is where the connect evidence used to die.
- this.lastProbeError = e;
+ //
+ // `markOwnTimeout` because the probe's 5 s ceiling is OURS: we asked "is
+ // anything there" and got no answer inside a window we chose, which is a
+ // reachability fact. Untagged, a `TimeoutError` is now deliberately
+ // unclassified — see `connection-error.ts`'s `NAME_KIND` header for the
+ // billing reason a transport's own inference ceiling must not be one.
+ this.lastProbeError = markOwnTimeout(e);
this.lastProbeUrl = healthUrl;
log(`[${this.displayName}] /api/tags failed: ${e?.message || e}, trying /v1/models`);
}
@@ -264,7 +271,8 @@ export class LocalTransport implements ProviderTransport {
}
log(`[${this.displayName}] /v1/models returned ${response.status}`);
} catch (e: any) {
- this.lastProbeError = e;
+ // Tagged for the same reason as the /api/tags probe above.
+ this.lastProbeError = markOwnTimeout(e);
this.lastProbeUrl = modelsUrl;
log(`[${this.displayName}] /v1/models failed: ${e?.message || e}`);
}
diff --git a/packages/cli/src/proxy-server.ts b/packages/cli/src/proxy-server.ts
index 8e7e2917..3290cb1e 100644
--- a/packages/cli/src/proxy-server.ts
+++ b/packages/cli/src/proxy-server.ts
@@ -54,6 +54,7 @@ import { LocalTransport } from "./providers/transport/local.js";
import { OpenRouterProviderTransport } from "./providers/transport/openrouter.js";
import { PoeProvider } from "./providers/transport/poe.js";
import type { ProviderTransport } from "./providers/transport/types.js";
+import { closeAllEpisodes } from "./recovery/coordinator.js";
import { warmPricingCache } from "./services/pricing-cache.js";
import type { ProxyServer } from "./types.js";
@@ -1218,6 +1219,14 @@ export async function createProxyServer(
// `true` = close active connections too, so a streamed request in flight
// can't keep the port alive after shutdown resolves.
await server.stop(true);
+ // "No episode may outlive the process that owns its timers" —
+ // `closeAllEpisodes`'s own doc, which until now nothing but tests
+ // invoked. `server.stop(true)` aborts the client signals, so waiters
+ // unwind and most episodes self-close; the residual that matters is
+ // `serve`, which is long-lived and can be shut down and REBUILT
+ // in-process, and would otherwise carry a stale episode — and its stale
+ // banner — across the rebuild.
+ closeAllEpisodes();
},
invalidateHandlerCache: (providerSlug?: string) => {
if (!providerSlug) {
diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.ts
index f58c4b2a..48b5228d 100644
--- a/packages/cli/src/recovery/coordinator.test.ts
+++ b/packages/cli/src/recovery/coordinator.test.ts
@@ -458,16 +458,38 @@ describe("noteTargetReachable — the handoff that ends without a rejoin", () =>
expect(episodeCount()).toBe(0);
});
- test("a DIFFERENT host does not close it — the key is provider|host", async () => {
+ test("ANOTHER HOST of the same provider closes it — an auth episode is still this outage", async () => {
+ // The episode key is `provider|host` and the AUTH sites seed the host from
+ // the error, so a `gk@` outage during a token refresh opens an episode on
+ // `auth.x.ai` while `noteTargetReachable` is called with the MODEL endpoint.
+ // A strict key lookup missed it, and the pane went on painting "waiting for
+ // Claude Code to retry" over a working session for the full 120-second
+ // grace — the fix that shipped for the fetch path, absent for the five auth
+ // sites.
+ //
+ // Closing by provider is sound rather than convenient: a request that
+ // reached the model endpoint had to authenticate first, so it has just
+ // proved every host it touched is answering.
useFakeClock();
- const s = seed({ providerName: "reach-2" });
- const h = joinEpisode(s);
+ const auth = joinEpisode(
+ seed({ providerName: "reach-2", endpoint: "https://auth.example.com/oauth2/token" })
+ );
+ auth.recordAttemptResult("ConnectionRefused", false);
+ auth.handoff();
+ auth.leave();
+ expect(describeEpisode(auth.episodeId)?.state).toBe("handoff");
+
+ noteTargetReachable("reach-2", "https://api.example.com/v1/chat/completions");
+ expect(describeEpisode(auth.episodeId)).toBeNull();
+ });
+
+ test("a DIFFERENT PROVIDER never closes it — one provider's success says nothing", async () => {
+ useFakeClock();
+ const h = joinEpisode(seed({ providerName: "reach-2b" }));
h.recordAttemptResult("ConnectionRefused", false);
h.handoff();
h.leave();
- noteTargetReachable("reach-2", "http://127.0.0.1:10/v1/chat/completions");
- expect(describeEpisode(h.episodeId)?.state).toBe("handoff");
noteTargetReachable("other-provider", "http://127.0.0.1:9/v1/chat/completions");
expect(describeEpisode(h.episodeId)?.state).toBe("handoff");
});
diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.ts
index 84ce467d..91f921ca 100644
--- a/packages/cli/src/recovery/coordinator.ts
+++ b/packages/cli/src/recovery/coordinator.ts
@@ -79,8 +79,6 @@ export interface EpisodeSeed {
interface Waiter {
deadlineAt: number;
- tier: 1 | 2;
- leftByDeadline: boolean;
/** Non-null exactly while this waiter is parked. */
wake: ((outcome: WaitOutcome) => void) | null;
fail: ((err: unknown) => void) | null;
@@ -158,6 +156,7 @@ export function describeEpisode(episodeId: string): Readonly<Record<string, unkn
maxTier: ep.maxTier,
lastOutcome: ep.lastOutcome,
nextAttemptAtPerf: ep.nextAttemptAtPerf,
+ paneRequestedAtPerf: ep.paneRequestedAtPerf,
};
}
@@ -346,7 +345,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
const key = `${seed.providerName}|${host}`;
let ep = episodes.get(key);
- let tier: 1 | 2 = 1;
if (ep && (ep.state === "recovered" || ep.state === "abandoned")) {
episodes.delete(key);
@@ -366,7 +364,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
ep.clientRetries++;
ep.state = "attempting";
ep.maxTier = 2;
- tier = 2;
log(
`[Recovery] ${ep.providerDisplayName} rejoined episode ${ep.episodeId} ` +
`(client retry ${ep.clientRetries}, ladder ${ep.ladderIndex}, attempts ${ep.attempts})`
@@ -434,8 +431,6 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
const episode = ep;
const waiter: Waiter = {
deadlineAt: seed.deadlineAt,
- tier,
- leftByDeadline: false,
wake: null,
fail: null,
detach: null,
@@ -520,7 +515,9 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
},
handoff(): void {
- waiter.leftByDeadline = true;
+ // EPISODE-level only. A per-waiter copy of this was write-only, and the
+ // decision it feeds — what the LAST waiter out closes the episode as —
+ // is an episode-level question by construction.
episode.anyLeftByDeadline = true;
},
@@ -658,15 +655,35 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {
* closing it underneath them would strand or gave-up requests that another
* request's success says nothing about. A `handoff` episode has no waiters by
* construction — that is what `handoff` means.
+ *
+ * ── WHY IT CLOSES BY PROVIDER AND NOT BY HOST ───────────────────────────────
+ *
+ * The key is `provider|host`, and the AUTH sites seed the host from the error —
+ * so a `gk@` outage during a token refresh can open an episode keyed on
+ * `auth.x.ai` while this function is called with the MODEL endpoint. Keyed
+ * lookup missed it by host, and the fix that shipped for the fetch path was
+ * absent for the five auth sites `network-recovery.md` §5 enumerates: the pane
+ * went on painting "waiting for Claude Code to retry" over a working session
+ * for the full 120-second grace.
+ *
+ * Closing every `handoff` episode for the provider is sound rather than merely
+ * convenient: a request that reached the model endpoint HAD to authenticate
+ * first, so it has just proved every host it touched is answering. And a
+ * `handoff` episode holds no waiters, so nothing is closed out from under
+ * anyone either way.
*/
export function noteTargetReachable(providerName: string, endpoint: string): void {
- const ep = episodes.get(`${providerName}|${hostOf(endpoint)}`);
- if (!ep || ep.state !== "handoff") return;
- log(
- `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +
- `closing episode ${ep.episodeId} instead of waiting out its grace`
- );
- closeEpisode(ep, "recovered");
+ const exact = `${providerName}|${hostOf(endpoint)}`;
+ const prefix = `${providerName}|`;
+ for (const [key, ep] of [...episodes]) {
+ if (key !== exact && !key.startsWith(prefix)) continue;
+ if (ep.state !== "handoff") continue;
+ log(
+ `[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +
+ `closing episode ${ep.episodeId} instead of waiting out its grace`
+ );
+ closeEpisode(ep, "recovered");
+ }
}
/**
diff --git a/packages/cli/src/recovery/magmux-ui.ts b/packages/cli/src/recovery/magmux-ui.ts
index f09bd8bd..f78c8ad3 100644
--- a/packages/cli/src/recovery/magmux-ui.ts
+++ b/packages/cli/src/recovery/magmux-ui.ts
@@ -1,7 +1,14 @@
/**
* The recovery UI manager — the single owner of the socket, the pane, the
- * `paneOpen` fact, the post-`bye` suppression window, the cross-process lock
- * and the lease.
+ * `paneOpen` fact, the cross-process lock and the lease.
+ *
+ * THE POST-`bye` SUPPRESSION IS NOT OWNED HERE, and that is a correction. It
+ * used to be `state.suppressUntilPerf`, read only by `ensureRecoveryUi`, so
+ * `[q] give up` stopped the BANNER while the next request against the same
+ * dead target still held its socket for the full derived deadline with the
+ * reason legible nowhere. The fact now lives in `recovery/settings.ts` where
+ * `shouldSkipTier1` can read it too: one number, suppressing the hold and the
+ * surface together.
*
* OWNERSHIP IS PROCESS-SCOPED; ONLY EPISODES ARE EPISODE-SCOPED. There is ONE
* pane and it multiplexes every episode, so "a pane exists" is a property of
@@ -55,6 +62,7 @@ import {
renderableEpisodeFrame,
tryNow,
} from "./coordinator.js";
+import { noteRecoveryGaveUp, recoveryGiveUpActive, resetRecoveryGiveUp } from "./settings.js";
import {
type RecoverySocketServer,
cleanupRecoverySocket,
@@ -121,8 +129,6 @@ interface ManagerState {
paneIndex: number | null;
/** episodeId → PROCESS ms of the last heartbeat naming it. */
leases: Map<string, number>;
- /** PROCESS ms before which no pane may be opened (post-`bye`). */
- suppressUntilPerf: number;
/** Guards the async open against re-entry. */
opening: boolean;
lockPath: string | null;
@@ -137,7 +143,6 @@ const state: ManagerState = {
paneOpen: false,
paneIndex: null,
leases: new Map(),
- suppressUntilPerf: 0,
opening: false,
lockPath: null,
lingerTimer: null,
@@ -339,7 +344,14 @@ function onCommand(cmd: RecoveryCommand): void {
break;
case "bye":
log(`[Recovery] pane said bye (${cmd.reason}) — giving up on every live episode`);
- state.suppressUntilPerf = recoveryClock().now() + UI_SUPPRESS_AFTER_BYE_MS;
+ // ONE fact, two readers. `giveUpAll()` ends the episodes alive at this
+ // instant; this ends the ones that do not exist yet. Without it the next
+ // request against the same dead target opened a new episode, found the
+ // pane suppressed, and held its socket for the full deadline with the
+ // reason legible nowhere — the forbidden state, reached from the key
+ // whose whole purpose is to avoid it. The hold and the surface are
+ // suppressed together, for the same window, from the same number.
+ noteRecoveryGaveUp(recoveryClock().now() + UI_SUPPRESS_AFTER_BYE_MS);
giveUpAll();
// The pane the user just closed must not reappear two seconds later, and
// closing it is ALSO what revokes every lease: `teardown()` drops
@@ -459,7 +471,7 @@ export function ensureRecoveryUi(episodeId: string): void {
state.lingerTimer = null;
}
if (state.paneOpen || state.opening) return;
- if (recoveryClock().now() < state.suppressUntilPerf) {
+ if (recoveryGiveUpActive()) {
log("[Recovery] pane suppressed — the user pressed [q] less than a minute ago");
return;
}
@@ -534,7 +546,7 @@ export async function shutdownRecoveryUi(): Promise<void> {
await closePane();
registerRecoveryUi(null);
installed = false;
- state.suppressUntilPerf = 0;
+ resetRecoveryGiveUp();
explicitControlSock = null;
}
@@ -554,7 +566,7 @@ export function __resetUiStateForTests(): void {
state.leases.clear();
state.paneOpen = false;
state.paneIndex = null;
- state.suppressUntilPerf = 0;
+ resetRecoveryGiveUp();
state.opening = false;
state.socketPath = null;
state.server = null;
diff --git a/packages/cli/src/recovery/pane-app.ts b/packages/cli/src/recovery/pane-app.ts
index ff95f971..ee1a4a9b 100644
--- a/packages/cli/src/recovery/pane-app.ts
+++ b/packages/cli/src/recovery/pane-app.ts
@@ -256,10 +256,21 @@ export function renderPane(state: PaneViewState, width: number): string {
// and a user with a narrow window gets a narrow pane. Below 64 columns the
// full sentence wraps or is cut, so the hint sheds the part a user can work
// out — where to click — and keeps the part they cannot: which keys exist.
+ //
+ // `[r]` IS HIDDEN IN `handoff`, because in that state it does nothing:
+ // `tryNow` returns early when no attempt timer is armed, and a handed-off
+ // episode has no waiters and therefore no timer. Advertising a key that
+ // prints "retrying now…" over an episode nobody is retrying is the same
+ // class of defect as a give-up that does not give up — a control that
+ // appears to act and does not.
+ const keys =
+ ep.state === "handoff"
+ ? `${C.BOLD}[q]${C.RESET} give up`
+ : `${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`;
lines.push(
w < 64
- ? `${C.GRAY} ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`
- : `${C.GRAY} click here or Ctrl-G Tab, then ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`
+ ? `${C.GRAY} ${C.RESET}${keys}`
+ : `${C.GRAY} click here or Ctrl-G Tab, then ${C.RESET}${keys}`
);
if (state.notice) lines.push(`${C.GREEN} ${state.notice}${C.RESET}`);
@@ -439,7 +450,10 @@ export async function runRecoveryPane(opts: RecoveryPaneOptions): Promise<number
return;
}
if (ch === "r" || ch === "R" || ch === "\r" || ch === "\n") {
- if (state.frame && client?.connected) {
+ // Not in `handoff`: there is no armed timer to collapse there, so the
+ // key would only print a reassurance that is not true. The banner hides
+ // it in that state too — one rule, both places.
+ if (state.frame && state.frame.state !== "handoff" && client?.connected) {
client.send({
v: RECOVERY_PROTOCOL_VERSION,
type: "retry_now",
@@ -484,6 +498,24 @@ export async function recoveryPaneCommand(argv: string[]): Promise<void> {
console.error("Usage: claudish recovery-pane --socket <path>");
process.exit(2);
}
+ // The pane is its own PROCESS, reached from `index.ts` before `runCli()`, so
+ // nothing has published a theme mode for it. `cliAnsi()` is read at paint
+ // time — which is right, and which the module-load snapshot trap makes
+ // necessary — but it cannot use a mode that was never detected, so the pane
+ // stayed on the unknown/classic palette even where OSC 11, `COLORFGBG` or an
+ // explicit `CLAUDISH_THEME=light` would have chosen the light one.
+ //
+ // It runs HERE rather than inside `runRecoveryPane` for the same reason the
+ // TUI runs it before `createCliRenderer`: the OSC query puts stdin in raw
+ // mode and reads it, so it must finish before the pane's own key handler
+ // owns the stream. Bounded at 150 ms, a no-op without a TTY on both ends, and
+ // never fatal — a banner with the wrong palette still beats no banner.
+ try {
+ const { detectAndSetThemeMode } = await import("../theme/theme-mode.js");
+ await detectAndSetThemeMode();
+ } catch {
+ /* the classic palette is a working fallback */
+ }
const code = await runRecoveryPane({ socketPath });
process.exit(code);
}
diff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.ts
index 3aa33b9a..4f13f4fd 100644
--- a/packages/cli/src/recovery/settings.ts
+++ b/packages/cli/src/recovery/settings.ts
@@ -13,6 +13,7 @@
import { ENV } from "../config.js";
import { log } from "../logger.js";
import { readRecoveryEnabled, readRecoveryUi } from "../profile-config.js";
+import { recoveryClock } from "./clock.js";
/** Flag layer. Set once by `cli.ts`'s arg loop; undefined means "not given". */
interface RecoveryFlagOverrides {
@@ -87,6 +88,52 @@ export function resolveRecoveryUi(): boolean {
return true;
}
+// ─── "The user said stop" ────────────────────────────────────────────────────
+
+/**
+ * PROCESS ms until which `[q] give up` suppresses recovery. 0 = not suppressed.
+ *
+ * ── WHY THIS IS A PROCESS FACT AND NOT THE PANE'S OWN STATE ─────────────────
+ *
+ * `[q]` used to do two things, both inside the UI manager: suppress re-opening
+ * the pane for a minute, and `giveUpAll()` — which iterates the episodes alive
+ * AT THAT INSTANT. Nothing recorded that the user had asked recovery to stop,
+ * so the next request against the same dead target opened a NEW episode, found
+ * the pane suppressed, got no pane and no lease, and then held its socket for
+ * the full derived deadline (~4.5 minutes at the default) before answering 400
+ * — with the reason legible NOWHERE, which is the state this design calls
+ * "strictly worse than the bug this feature exists to remove", reached from the
+ * one affordance whose entire purpose is to end it. Before recovery existed
+ * those requests failed in milliseconds.
+ *
+ * It is not an edge case: Claude Code issues concurrent requests during an
+ * outage (main loop, title model, subagents), so a new request inside the
+ * suppression window is the EXPECTED case. A control that appears to stop
+ * something and does not is worse than no control.
+ *
+ * So the give-up suppresses the HOLD and the SURFACE together, for the same
+ * window, from ONE fact — read by `shouldSkipTier1` (the hold) and by
+ * `ensureRecoveryUi` (the surface). Two facts would be two things to keep in
+ * step, and the asymmetry between them is exactly what created the forbidden
+ * state.
+ */
+let giveUpUntilPerf = 0;
+
+/** `[q] give up`: suppress recovery until `untilPerf` (PROCESS ms). */
+export function noteRecoveryGaveUp(untilPerf: number): void {
+ giveUpUntilPerf = Math.max(giveUpUntilPerf, untilPerf);
+}
+
+/** Is the user's give-up still in force? Reads the clock LIVE — never cached. */
+export function recoveryGiveUpActive(): boolean {
+ return giveUpUntilPerf > 0 && recoveryClock().now() < giveUpUntilPerf;
+}
+
+/** Pane teardown, `serve` rebuilds, and tests. */
+export function resetRecoveryGiveUp(): void {
+ giveUpUntilPerf = 0;
+}
+
// ─── The deadline ────────────────────────────────────────────────────────────
/**
@@ -232,14 +279,32 @@ export function resetDeadlineNotice(): void {
* one process-wide switch, and there is no narrower lever: the variable is
* fixed at spawn and cannot key on whether a banner exists.
*
- * ── WHY IT IS GATED ON THE UI SWITCH ────────────────────────────────────────
+ * ── THE THREE GATES, AND WHY IT IS NOT JUST THE UI SWITCH ───────────────────
+ *
+ * All three must hold, because the cost above is only worth paying by a launch
+ * that gets the benefit:
+ *
+ * 1. `resolveRecoveryEnabled()` — the ladder itself. `--no-recovery` /
+ * `CLAUDISH_RECOVERY=0` restores pre-recovery behaviour "byte for byte"
+ * (RISK-7), and an exhaustion that answers today's immediate 400 hands
+ * nothing back for the client to re-ask. A CI run that explicitly opted
+ * out must not be left looping ~300 times on an unrelated 503.
+ * 2. `resolveRecoveryUi()` — the user's configuration, flag > env > project >
+ * global > true. Off means no pane, therefore no lease, therefore an
+ * inline 400 at exhaustion. Surface and reach move together or not at all.
+ * 3. `paneEligible` — whether THIS LAUNCH can obtain a surface at all, from
+ * `magmuxPaneCapability()`. It is a launch-order fact, not a preference:
+ * false in `-p`, in `--stdin`, with no TTY, and on a machine without
+ * magmux. None of those can ever hold the lease a recovery 503 requires,
+ * so for them the watchdog is the RISK-6 exposure with none of the
+ * recovery benefit, applied to every unrelated 503 the session sees.
*
- * `resolveRecoveryUi()` is the user's configuration — flag > env > project >
- * global > true. Off means no pane, therefore no lease, therefore an inline
- * 400 at exhaustion, therefore nothing handed back for the client to re-ask —
- * so extending its budget would buy only the side effect above. Surface and
- * reach move together or not at all.
+ * Gate 3 is why the caller must ask about eligibility BEFORE finalising the
+ * child environment — see `magmuxPaneCapability`, which is side-effect free for
+ * exactly that reason.
*/
-export function retryWatchdogEnv(): Record<string, string> {
+export function retryWatchdogEnv(opts: { paneEligible: boolean }): Record<string, string> {
+ if (!opts.paneEligible) return {};
+ if (!resolveRecoveryEnabled()) return {};
return resolveRecoveryUi() ? { CLAUDE_CODE_RETRY_WATCHDOG: "1" } : {};
}
diff --git a/packages/cli/src/recovery/types.test.ts b/packages/cli/src/recovery/types.test.ts
index 6a2f8378..0b47013e 100644
--- a/packages/cli/src/recovery/types.test.ts
+++ b/packages/cli/src/recovery/types.test.ts
@@ -62,6 +62,32 @@ describe("recovery wire contract", () => {
expect(parseFrame(JSON.stringify({ v: 1, type: "episode" }))).toBeNull(); // no episodeId
});
+ test("the SKEW NOTICE parses at the wrong version — it is the one frame that must", () => {
+ // The mirror image of `hello` below, and for the same reason. The proxy
+ // answers a mismatched pane with a `closed` / `protocol_mismatch` frame
+ // carrying the PROXY'S version, so a blanket version gate dropped the very
+ // message addressed to the build that cannot read the others:
+ // `PaneViewState.protocolMismatch` was unreachable in a real skew and the
+ // pane printed "the claudish proxy closed this connection" instead — a safe
+ // failure with the wrong diagnosis, on the one day it mattered.
+ const notice = JSON.stringify({
+ v: 99,
+ type: "closed",
+ episodeId: "",
+ outcome: "protocol_mismatch",
+ });
+ const parsed = parseFrame(notice);
+ expect(parsed).not.toBeNull();
+ expect(parsed?.type).toBe("closed");
+ expect((parsed as { outcome?: string }).outcome).toBe("protocol_mismatch");
+
+ // And ONLY that outcome. A skewed `closed` frame of any other kind is still
+ // refused — the exemption is for the notice, not for the type.
+ expect(
+ parseFrame(JSON.stringify({ v: 99, type: "closed", episodeId: "e", outcome: "recovered" }))
+ ).toBeNull();
+ });
+
test("`hello` parses even at the WRONG version — the mismatch must be answerable", () => {
// The server cannot tell a skewed client it is skewed if the one message it
// needs to answer is the one it drops.
diff --git a/packages/cli/src/recovery/types.ts b/packages/cli/src/recovery/types.ts
index 790f2999..c19ecbcd 100644
--- a/packages/cli/src/recovery/types.ts
+++ b/packages/cli/src/recovery/types.ts
@@ -147,6 +147,23 @@ function isRecord(value: unknown): value is Record<string, unknown> {
* Null for malformed JSON, for an unknown `type`, and for a `v` this build does
* not speak — a reader that guesses at an unknown shape is how a version skew
* becomes a rendering bug instead of a one-line notice.
+ *
+ * ── THE ONE EXEMPTION, AND WHY IT IS NOT AN INCONSISTENCY ───────────────────
+ *
+ * The skew NOTICE itself is exempt, exactly as `parseCommand` exempts `hello`
+ * for the mirror-image reason. The proxy answers a mismatched pane with a
+ * `closed` / `protocol_mismatch` frame carrying the PROXY'S version — so a
+ * blanket version gate dropped the very message addressed to the build that
+ * cannot read the others, `PaneViewState.protocolMismatch` was unreachable in
+ * any real skew, and the pane printed "the claudish proxy closed this
+ * connection" instead. Mixed-build grids and sessions started during an update
+ * failed safely with the WRONG diagnosis: the skew notice did not appear on the
+ * one day it was needed.
+ *
+ * Exempting it is sound where exempting a data frame would not be. It carries
+ * no payload to misread — a type, an empty `episodeId` and a terminal outcome —
+ * and it is the LAST thing on the connection, so nothing can follow it that a
+ * mis-parse could corrupt.
*/
export function parseFrame(line: string): RecoveryFrame | null {
let parsed: unknown;
@@ -156,6 +173,14 @@ export function parseFrame(line: string): RecoveryFrame | null {
return null;
}
if (!isRecord(parsed)) return null;
+ if (parsed.type === "closed" && parsed.outcome === "protocol_mismatch") {
+ return {
+ v: RECOVERY_PROTOCOL_VERSION,
+ type: "closed",
+ episodeId: typeof parsed.episodeId === "string" ? parsed.episodeId : "",
+ outcome: "protocol_mismatch",
+ } as unknown as RecoveryClosedFrame;
+ }
if (parsed.v !== RECOVERY_PROTOCOL_VERSION) return null;
if (parsed.type === "episode" && typeof parsed.episodeId === "string") {
return parsed as unknown as RecoveryEpisodeFrame;