Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

test(recovery): verify the Tier-1 ladder, which shipped with no tests at all

Jack Rudenkoa38f4b7b48af
diff --git a/ai-docs/reports/network-recovery-phase2-verification-20260911.md b/ai-docs/reports/network-recovery-phase2-verification-20260911.md
new file mode 100644
index 00000000..a27dcea7
--- /dev/null
+++ b/ai-docs/reports/network-recovery-phase2-verification-20260911.md
@@ -0,0 +1,147 @@
+# Network recovery, design-Phase 2 — verification measurements
+
+> Promoted out of the gitignored session directory deliberately, exactly as
+> `network-recovery-phase0-measurements.md` was. CLAUDE.md records three write-ups already lost
+> because they lived in `ai-docs/sessions/`, which does not survive a fresh clone or
+> `git worktree remove`.
+>
+> Session: `dev-feature-network-recovery-20260910-110514-0ad0a5a0` · measured 2026-09-11 on macOS
+> 25.6.0, bun 1.4.0. The per-file test inventory and the phase narrative stay in that session's
+> `implementation-log-phase2b.md`; what is here is the evidence and the findings.
+
+Phase 2 shipped the Tier-1 retry ladder with **no tests for any of its four new modules**. The suite
+was green because the pre-existing tests had been opted out of the ladder with
+`x-claudish-no-recovery` — not because the ladder had been verified. This is what verifying it
+produced.
+
+Suite: **3175 → 3271 pass**, 17 → 18 skip, **2 → 2 fail** (the two pre-existing `displayWidth`
+Unicode-oracle cases, red on clean main under bun 1.4.0 while CI pins 1.3.10). +96 tests, zero new
+failures. `typecheck` and `lint` clean.
+
+---
+
+## 1. The ladder's real gaps (C-2)
+
+Real clock, real refused loopback socket, driven through `ComposedHandler.handle()` for the full
+derived deadline. Deltas computed from the `[Recovery]` log timestamps:
+
+| attempt | fires at | gap | ladder index |
+|---|---|---|---|
+| 1 | t+0 | — | 0 |
+| 2 | t+5.002 s | **5002 ms** | 1 |
+| 3 | t+15.004 s | **10002 ms** | 2 |
+| 4 | t+45.009 s | **30005 ms** | 3 |
+| 5 | t+105.010 s | **60001 ms** | 4 (clamped) |
+| 6 | t+165.011 s | **60001 ms** | 5 (clamped) |
+| 7 | t+225.012 s | **60001 ms** | 6 (clamped) |
+| — | handoff at **t+225.018 s**, deadline 270 s | | |
+
+Total drift over 225 seconds: **18 ms**. The schedule is `5 / 10 / 30 / 60 / 60 / 60`, and the last
+array element repeating is what makes the sixth and seventh attempts happen at all — the DEADLINE
+ends the ladder, not the array. That inverts `STREAM_RETRY_DELAYS_MS`, where the array running out
+*is* the budget.
+
+**`validation-criteria.md`'s C-2 text is stale and the implementation is right.** It says "six
+attempts in total, with the handoff at t≈165s", derived for the superseded `TIER1_DEADLINE_MS =
+180_000`. ADD-1 replaced that constant with a derivation that yields 270 000 ms at the default, and
+270 s fits one more 60 s rung: **seven attempts, handoff at t≈225 s.** The gap sequence the criterion
+headlines is unchanged.
+
+---
+
+## 2. F-SLOW fidelity, re-measured (ADD-2 and ADD-3 confirmed)
+
+`192.0.2.1` is TEST-NET-1 (RFC 5737), unrouted **by convention** — a captive portal or an aggressive
+corporate resolver can answer it, in which case every F-SLOW assertion passes for the wrong reason.
+Checked before use:
+
+| target | elapsed | outcome |
+|---|---|---|
+| `http://127.0.0.1:1/` (control) | **0.368 ms** | `TypeError` / `ConnectionRefused` |
+| `https://192.0.2.1/v1` (F-SLOW) | **75005.18 ms** | `TypeError` / **`ConnectionRefused`** |
+
+Both report the SAME CODE. **Slow and fast connect failures are indistinguishable by code on macOS;
+only elapsed time separates them** — ADD-2, reproduced independently. Any logic that branches on the
+code to tell "slow unreachable" from "fast refused" is wrong on this platform.
+
+Phase 0 measured the OS give-up at 75.004 s; this machine, a day later, 75.005 s. That stability is
+what makes ADD-3 binding: a `PER_ATTEMPT_CONNECT_CAP_MS` of 75 000 races the operating system by
+**four milliseconds**, and which one wins is decided per run.
+
+## 3. The per-attempt clamp against that fault
+
+Real clock, real hung connect, the ladder's own clamp:
+
+```
+[F-SLOW live] attempt 45.001s (cap 45s, macOS gives up at 75.004s) · total 50.005s · exhausted
+```
+
+The attempt ended at **45.001 s** — a 30-second margin under the OS's 75.005 s — and the whole
+episode finished at 50.005 s inside a 60 s budget, `exhausted` rather than overrun. The clamp's own
+abort re-classifies as a failed attempt of the same kind (a `DOMException` named `TimeoutError`,
+whose `code` is the NUMBER 23, which is why the classifier had to match on `name`) rather than
+escaping the ladder as an unrelated error.
+
+---
+
+## 4. Mutation proof: 23 mutations, 23 killed, 0 survivors
+
+Each mutation reverts ONE property, runs the tests that claim to pin it, and restores **by file
+copy — never git**, because the stash stack and the index are shared with the main checkout and
+every sibling worktree.
+
+Killed: the repeating last delay · the `API_TIMEOUT_MS` derivation · the deadline floor · the
+shortened-hold notice · one-timer-per-episode · waiter-deadline isolation · the Observer unsubscribe ·
+rejoin keeping its ladder position · ladder-advance-per-round · the deadline bounding the attempt
+*after* the sleep · the per-attempt clamp · client-disconnect detection · re-classification on every
+attempt · signal composition · the shared inbound deadline anchor · the `x-claudish-no-recovery` gate ·
+`probe-live` sending that header · the `CLAUDISH_RECOVERY` master switch · the success-only health
+latch · the `{ cause }` carrying local connect evidence · the `loadConfig` allowlist entries · `{}`
+meaning "no opinion" in the scoped reader · the injectable clock.
+
+---
+
+## 5. Two traps for whoever writes the Phase-3 tests
+
+### 5.1 The debug-log buffer is flushed against `logFilePath` AT FLUSH TIME
+
+`logger.ts`:
+
+```ts
+function flushLogBuffer(): void {
+ if (!logFilePath || logBuffer.length === 0) return; // early return…
+ appendFile(logFilePath, logBuffer.join(""), …); // …KEEPS the buffer
+}
+```
+
+The early return does not clear `logBuffer`. Lines pushed while log file A was installed, but not yet
+flushed when `logFilePath` became null, are appended to log file B the moment one is created — so a
+test file's trailing log lines surface inside the NEXT file's capture.
+
+Symptom, and it is the expensive shape: **an assertion that passes alone and fails in the full
+suite.** Harmless in production (one log file per process), so the fix belongs in the test helper —
+stamp a marker through the same FIFO buffer at capture start and read only what follows it. See
+`packages/cli/src/recovery/test-helpers/capture-log.ts`.
+
+### 5.2 Two different `[Recovery]` lines contain the substring `recovered after`
+
+`withConnectionRetry` writes `… recovered after N attempts …`; `closeEpisode` writes
+`… closed: recovered after …ms …`. A loose `includes("recovered after")` counts one recovery twice.
+Any assertion about recovery counts — and C-17's assertion 3 is exactly one — must use the two
+distinct patterns.
+
+---
+
+## 6. One recorded deviation from a criterion's literal text
+
+C-17 assertion 4 requires the `[Recovery]` lines to name `…/api/tags`. They name `…/v1/models`:
+
+```
+[Recovery] episode f04bc428-… opened for Ollama at http://127.0.0.1:53801/v1/models (refused/ConnectionRefused)
+```
+
+`checkHealth()` probes `/api/tags` first and falls back to `/v1/models`, overwriting `lastProbeUrl`
+each time, so what is reported is the LAST probe attempted. That is the more general behaviour — LM
+Studio and vLLM have no `/api/tags` at all, so pinning the first would name a URL those providers
+never touch — and it satisfies what FR-5 actually requires: the right HOST, and a probe path rather
+than the inference path. The test is written to the requirement rather than to the example.
diff --git a/packages/cli/src/handlers/composed-handler-recovery.test.ts b/packages/cli/src/handlers/composed-handler-recovery.test.ts
new file mode 100644
index 00000000..856cd33e
--- /dev/null
+++ b/packages/cli/src/handlers/composed-handler-recovery.test.ts
@@ -0,0 +1,304 @@
+/**
+ * The ladder as it is actually reached: through `ComposedHandler.handle()`,
+ * over a real socket, against a real refused port.
+ *
+ * Two validation criteria live here and both are demonstrated rather than
+ * asserted in the abstract:
+ *
+ * C-9 — a client disconnect stops the loop and releases the request within
+ * a second. Reachable because `Request.signal` was measured firing in
+ * 0.49–0.86 ms in the HELD shape (handler awaiting, no headers sent).
+ * C-11 — `--probe` and the config TUI's Test All still fail fast. Without the
+ * `x-claudish-no-recovery` guard this feature breaks both tools for
+ * exactly the fault they exist to diagnose, and misreports it as
+ * `timeout` rather than `network error`.
+ *
+ * C-11 runs through a REAL `Bun.serve` and the REAL `probeLink`, not a
+ * simulation of either: the header has to survive an actual HTTP hop for the
+ * criterion to mean anything.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import type { Context } from "hono";
+import { probeLink } from "../providers/probe-live.js";
+import type { ProviderTransport } from "../providers/transport/types.js";
+import { resetRecoveryClock } from "../recovery/clock.js";
+import { closeAllEpisodes, episodeCount } from "../recovery/coordinator.js";
+import { resetRecoveryFlagOverrides, setRecoveryFlagOverrides } from "../recovery/settings.js";
+import {
+ capturedRecoveryLines,
+ startLogCapture,
+ stopLogCapture,
+} from "../recovery/test-helpers/capture-log.js";
+import { ComposedHandler } from "./composed-handler.js";
+
+/** A port nothing is listening on, taken and released so it is VERIFIED free. */
+function freePort(): number {
+ const s = Bun.serve({ port: 0, fetch: () => new Response("x") });
+ const port = s.port as number;
+ s.stop(true);
+ return port;
+}
+
+let deadPort: number;
+
+beforeEach(() => {
+ deadPort = freePort();
+ startLogCapture();
+});
+
+afterEach(() => {
+ stopLogCapture();
+ closeAllEpisodes();
+ resetRecoveryClock();
+ resetRecoveryFlagOverrides();
+});
+
+let seq = 0;
+function deadTransport(port: number): ProviderTransport {
+ return {
+ name: `dead-${++seq}`,
+ displayName: "Dead Endpoint",
+ streamFormat: "openai-sse",
+ getEndpoint: () => `http://127.0.0.1:${port}/v1/chat/completions`,
+ getHeaders: async () => ({}),
+ } as unknown as ProviderTransport;
+}
+
+const PAYLOAD = {
+ model: "dead-model",
+ max_tokens: 16,
+ messages: [{ role: "user", content: "hi" }],
+};
+
+/** A Hono-shaped Context over a REAL `Request`, so `c.req.raw.signal` is real. */
+function contextFor(req: Request): { c: Context; captured: { status?: number; body?: any } } {
+ const captured: { status?: number; body?: any } = {};
+ const c = {
+ req: {
+ raw: req,
+ header: (name?: string) => (name === undefined ? {} : (req.headers.get(name) ?? undefined)),
+ },
+ header: () => {},
+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),
+ json: (body: unknown, status?: number) => {
+ captured.body = body;
+ captured.status = status;
+ return new Response(JSON.stringify(body), {
+ status: status ?? 200,
+ headers: { "content-type": "application/json" },
+ });
+ },
+ } as unknown as Context;
+ return { c, captured };
+}
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-9
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-9 — a client disconnect stops the retry loop within a second", () => {
+ test("the held request is released, the episode closed, and no attempt follows", async () => {
+ const handler = new ComposedHandler(deadTransport(deadPort), "dead-model", "dead-model", 8080, {
+ // no x-claudish-no-recovery header: this request DOES enter the ladder
+ });
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ const { c } = contextFor(req);
+
+ const started = performance.now();
+ const p = handler.handle(c, PAYLOAD);
+
+ // Let the first attempt fail and the ladder park on its 5 s first rung,
+ // then go away — the shape of Claude Code being killed mid-outage.
+ await new Promise((r) => setTimeout(r, 120));
+ expect(episodeCount()).toBe(1);
+
+ const abortedAt = performance.now();
+ ac.abort(new DOMException("client gone", "AbortError"));
+ const response = await p;
+ const releasedAt = performance.now();
+
+ // NFR-2: released promptly, not after the 5 s rung and emphatically not
+ // after the 120 s handoff grace, which is for a DELIBERATE tier-2 handoff.
+ expect(releasedAt - abortedAt).toBeLessThan(1_000);
+ expect(releasedAt - started).toBeLessThan(5_000);
+ expect(response.status).toBe(499);
+
+ // The loop exited and the episode released its timer at once.
+ expect(episodeCount()).toBe(0);
+
+ const lines = await capturedRecoveryLines();
+ expect(lines.some((l) => l.includes("client gone during wait"))).toBe(true);
+ expect(lines.some((l) => l.includes("client disconnected after"))).toBe(true);
+ // Exactly one attempt was ever made — the one that opened the episode.
+ // A second would mean the loop kept running after the client left.
+ const attemptLines = lines.filter((l) => /attempt \d+ failed/.test(l));
+ expect(attemptLines.length).toBe(1);
+ });
+
+ test("a client that is ALREADY gone is answered without entering the ladder at all", async () => {
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ ac.abort(new DOMException("client gone", "AbortError"));
+ const { c } = contextFor(req);
+
+ const started = performance.now();
+ const response = await handler.handle(c, PAYLOAD);
+ expect(performance.now() - started).toBeLessThan(1_000);
+ expect(response.status).toBe(499);
+ expect(episodeCount()).toBe(0);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-11
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-11 — probes still fail fast against a dead endpoint", () => {
+ test("the header gate answers connection_error 400 in well under a second", async () => {
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ headers: { "x-claudish-no-recovery": "1" },
+ });
+ const { c, captured } = contextFor(req);
+
+ const started = performance.now();
+ await handler.handle(c, PAYLOAD);
+ const elapsed = performance.now() - started;
+
+ expect(elapsed).toBeLessThan(1_000);
+ expect(captured.status).toBe(400);
+ expect(captured.body?.error?.type).toBe("connection_error");
+ // Nothing was held: no episode was ever opened.
+ expect(episodeCount()).toBe(0);
+ const lines = await capturedRecoveryLines();
+ expect(lines.some((l) => l.includes("skipped (probe)"))).toBe(true);
+ });
+
+ test("REAL probeLink over a REAL socket reports `network-error`, not `timeout`", async () => {
+ // The whole hop: probeLink → HTTP → ComposedHandler → the header gate →
+ // an immediate connection_error 400 → probe-live's classifier. A
+ // simulation of any one of those links would leave the criterion unproven.
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const proxy = Bun.serve({
+ port: 0,
+ async fetch(req) {
+ const payload = await req.json();
+ const { c } = contextFor(req);
+ return await handler.handle(c, payload as Record<string, unknown>);
+ },
+ });
+
+ try {
+ const started = performance.now();
+ const result = await probeLink(
+ `http://127.0.0.1:${proxy.port}`,
+ { provider: "dead", modelSpec: "dead@dead-model", hasCredentials: true },
+ // Generously long, so "under a second" is a real discrimination: an
+ // unguarded ladder would burn this whole budget and then report the
+ // probe's own abort as `timeout`.
+ 8_000
+ );
+ const elapsed = performance.now() - started;
+
+ expect(result.state).toBe("network-error");
+ expect(result.state).not.toBe("timeout");
+ expect(elapsed).toBeLessThan(1_000);
+ expect(result.errorMessage ?? "").toMatch(/reach|connect|network/i);
+ expect(episodeCount()).toBe(0);
+ } finally {
+ proxy.stop(true);
+ }
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The other two skip gates, and the absence of a fourth
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("shouldSkipTier1 — three gates, and no loopback carve-out", () => {
+ test("the master switch restores today's immediate 400 (RISK-7, the CI escape)", async () => {
+ setRecoveryFlagOverrides({ recovery: false });
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const req = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });
+ const { c, captured } = contextFor(req);
+
+ const started = performance.now();
+ await handler.handle(c, PAYLOAD);
+
+ expect(performance.now() - started).toBeLessThan(1_000);
+ expect(captured.status).toBe(400);
+ expect(captured.body?.error?.type).toBe("connection_error");
+ expect(episodeCount()).toBe(0);
+ expect(
+ (await capturedRecoveryLines()).some((l) => l.includes("skipped (recovery-disabled)"))
+ ).toBe(true);
+ });
+
+ test("A REFUSED LOOPBACK ADDRESS ENTERS THE LADDER — there is no carve-out", async () => {
+ // DEC-1, and this is the assertion that pins it. Two successive designs
+ // carved out `refused` + loopback, and each revision of the predicate
+ // either fired universally (skipping recovery everywhere) or never. The
+ // rule is now general: every classified connection failure enters the
+ // ladder regardless of address or UI state.
+ setRecoveryFlagOverrides({ recoveryUi: false }); // the old carve-out's key
+ const handler = new ComposedHandler(
+ deadTransport(deadPort),
+ "dead-model",
+ "dead-model",
+ 8080,
+ {}
+ );
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ const { c } = contextFor(req);
+
+ const p = handler.handle(c, PAYLOAD);
+ await new Promise((r) => setTimeout(r, 120));
+
+ // 127.0.0.1, refused, recovery UI OFF — and still held.
+ expect(episodeCount()).toBe(1);
+ const lines = await capturedRecoveryLines();
+ expect(lines.some((l) => l.includes("episode") && l.includes("opened for"))).toBe(true);
+ expect(lines.some((l) => l.includes("skipped ("))).toBe(false);
+
+ ac.abort(new DOMException("done", "AbortError"));
+ await p;
+ });
+});
diff --git a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts
new file mode 100644
index 00000000..5d29e155
--- /dev/null
+++ b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts
@@ -0,0 +1,240 @@
+/**
+ * F-SLOW — the fault class the rest of the suite structurally cannot exercise.
+ *
+ * EVERY other fault in this feature's validation set is a loopback refusal, and
+ * a refusal comes back in about one millisecond. The originating incident was
+ * the opposite: a host that SWALLOWS the connection, where a single connect
+ * burns tens of seconds. `192.0.2.1` is TEST-NET-1 (RFC 5737) — reserved for
+ * documentation and unrouted — so a TCP connect to it hangs until the OS gives
+ * up. Measured on macOS: **75.004 s**, against 0.4 ms for the refused-loopback
+ * control.
+ *
+ * TWO THINGS THAT MATTER AND ARE NOT OBVIOUS:
+ *
+ * 1. **Slow and fast are indistinguishable BY CODE on macOS.** `192.0.2.1:443`
+ * reports `ConnectionRefused`, the same code as an instantly-refused
+ * loopback port. Only ELAPSED TIME separates them, so any logic that
+ * branches on the code to tell them apart is wrong here.
+ * 2. **`PER_ATTEMPT_CONNECT_CAP_MS` must clear the OS's own give-up.** At the
+ * design's original 75 000 the cap raced the operating system by four
+ * milliseconds, and which one won was decided per run. It is 45 000, which
+ * clears it by thirty seconds.
+ *
+ * WHAT IS IN THE DEFAULT SUITE AND WHAT IS NOT — a deliberate split:
+ *
+ * - The test below runs by default. It makes a REAL connect to 192.0.2.1 that
+ * really hangs, and aborts it through the ladder's own clamp on the injected
+ * clock. Real hang, real abort, ~0.1 s of wall clock. What it proves is the
+ * mechanism: the clamp reaches a connect that the OS would otherwise hold
+ * for 75 s, and the resulting abort is re-classified as a failed attempt
+ * rather than escaping as an unrelated error.
+ * - The FULL-FIDELITY run — the real 45 s clamp against a real 75 s connect,
+ * on the real clock — is gated behind `CLAUDISH_SLOW_NET_E2E=1` and costs
+ * ~50 s. Putting it in the default path would add half the suite's total
+ * runtime to every run, for one assertion, and the project's own note about
+ * credential-gated live tests is the precedent for gating rather than
+ * deleting. Its captured output is in the phase log.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";
+import { closeAllEpisodes } from "../../recovery/coordinator.js";
+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";
+import { classifyConnectionError } from "./connection-error.js";
+import {
+ PER_ATTEMPT_CONNECT_CAP_MS,
+ mergeSignalIntoInit,
+ withConnectionRetry,
+} from "./transient-retry.js";
+
+/** TEST-NET-1. Unrouted by RFC, so the connect hangs rather than being refused. */
+const SLOW_URL = "https://192.0.2.1/v1/chat/completions";
+/** macOS's own measured give-up against an unrouted address. */
+const OS_GIVE_UP_MS = 75_004;
+
+let clock: FakeClock;
+
+beforeEach(() => {
+ clock = new FakeClock(0);
+ setRecoveryClock(clock);
+});
+
+afterEach(() => {
+ closeAllEpisodes();
+ resetRecoveryClock();
+});
+
+const CONNECT_FAILURE_SEED = Object.assign(
+ new TypeError("Unable to connect. Is the computer able to access the url?"),
+ { code: "ConnectionRefused", path: SLOW_URL }
+);
+
+describe("F-SLOW — a host that swallows the connection", () => {
+ test("the fault is REAL on this machine: the connect does not settle quickly", async () => {
+ // The criterion's own precondition: 192.0.2.1 is unrouted BY CONVENTION,
+ // and a captive portal or an aggressive corporate resolver can answer it.
+ // If it does, every assertion below would pass for the wrong reason, so the
+ // fidelity check runs first and says so out loud.
+ const ac = new AbortController();
+ let settled = false;
+ const p = fetch(SLOW_URL, { signal: ac.signal }).then(
+ () => {
+ settled = true;
+ },
+ () => {
+ settled = true;
+ }
+ );
+ const t0 = performance.now();
+ await new Promise((r) => setTimeout(r, 1_500));
+ const stillHanging = !settled;
+ ac.abort();
+ await p.catch(() => {});
+
+ if (!stillHanging) {
+ throw new Error(
+ `192.0.2.1 settled in ${(performance.now() - t0).toFixed(0)} ms — something on this ` +
+ "network answers TEST-NET-1, so F-SLOW is not reproducible here. Substitute a " +
+ "locally blackholed route (which needs privileges) and note the substitution."
+ );
+ }
+ expect(stillHanging).toBe(true);
+ }, 10_000);
+
+ test("the per-attempt clamp aborts a REAL hung connect far inside the OS's give-up", async () => {
+ let opStarted = 0;
+ let opSettledAt: number | null = null;
+ let abortSeen = false;
+ const realElapsed: number[] = [];
+
+ const op = async (signal: AbortSignal) => {
+ opStarted++;
+ const t0 = performance.now();
+ signal.addEventListener("abort", () => {
+ abortSeen = true;
+ });
+ try {
+ // A REAL fetch, to a REAL unrouted address, carrying the ladder's own
+ // merged signal exactly as `doFetchWith` threads it into the init. The
+ // spread order is the point: a signal written before the transport's
+ // own init is silently overwritten by it.
+ const init = mergeSignalIntoInit({ method: "POST" }, signal);
+ return await fetch(SLOW_URL, init);
+ } finally {
+ realElapsed.push(performance.now() - t0);
+ opSettledAt = clock.now();
+ }
+ };
+
+ const result = await advanceUntilSettled(
+ clock,
+ withConnectionRetry(op, CONNECT_FAILURE_SEED, {
+ providerName: "slow-net",
+ providerDisplayName: "Slow Net",
+ resolveEndpoint: () => SLOW_URL,
+ // One rung of budget: 5 s gap, then the attempt, then no room for
+ // another (5 + 45 + 60 + 3 > 60 000).
+ deadlineAt: 60_000,
+ signal: new AbortController().signal,
+ }),
+ 200_000
+ );
+
+ expect(opStarted).toBe(1);
+ // The clamp fired — the connect did NOT return on its own.
+ expect(abortSeen).toBe(true);
+ // On the ladder's clock the attempt ended at the cap, not at the OS's
+ // 75.004 s and not at the deadline.
+ expect((opSettledAt as unknown as number) - 5_000).toBe(PER_ATTEMPT_CONNECT_CAP_MS);
+ // And in REAL time it ended immediately, because the abort reached a live
+ // socket rather than waiting the operating system out. This is the number
+ // that says the clamp is real and not merely scheduled.
+ expect(realElapsed[0] as number).toBeLessThan(5_000);
+ expect(PER_ATTEMPT_CONNECT_CAP_MS).toBeLessThan(OS_GIVE_UP_MS - 20_000);
+
+ // The clamp's own abort is a FAILED ATTEMPT of the same kind, not an
+ // unrelated error that escapes the ladder.
+ expect(result.kind).toBe("exhausted");
+ if (result.kind === "exhausted") {
+ expect(result.conn.kind).toBeDefined();
+ }
+ // The deadline is honoured from the inbound request start: everything
+ // above happened inside the budget.
+ expect(clock.now()).toBeLessThanOrEqual(60_000);
+ }, 30_000);
+
+ test("a clamp abort re-classifies as a connection error rather than escaping", () => {
+ // Why the ladder can treat its own timeout as a retryable network fault:
+ // `AbortSignal.timeout` rejects with a DOMException whose `name` is
+ // `TimeoutError` and whose `code` is the NUMBER 23, so a classifier keyed
+ // only on a string `code` returned null and a hung upstream fell through
+ // into a bare 500.
+ const clampAbort = new DOMException("attempt cap", "TimeoutError");
+ const conn = classifyConnectionError(clampAbort);
+ expect(conn).not.toBeNull();
+ expect(conn?.kind).toBe("unreachable");
+ });
+
+ test("slow and fast are indistinguishable BY CODE — only elapsed time separates them", () => {
+ // Recorded as an executable note. macOS reports ECONNREFUSED /
+ // ConnectionRefused for an unrouted address, the same code an instantly
+ // refused loopback port gives. Branching on the code to tell "slow
+ // unreachable" from "fast refused" is therefore wrong on this platform.
+ const slow = Object.assign(new TypeError("Unable to connect."), {
+ code: "ConnectionRefused",
+ });
+ const fast = Object.assign(new TypeError("Unable to connect."), {
+ code: "ConnectionRefused",
+ });
+ expect(classifyConnectionError(slow)?.kind).toBe(classifyConnectionError(fast)?.kind);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Full fidelity: the real 45 s clamp on the real clock. ~50 s. Gated.
+// ───────────────────────────────────────────────────────────────────────────
+
+const LIVE = process.env.CLAUDISH_SLOW_NET_E2E === "1";
+
+describe.skipIf(!LIVE)("F-SLOW, full fidelity (CLAUDISH_SLOW_NET_E2E=1)", () => {
+ test("the real clamp ends the attempt at ~45 s, well before the OS's 75.004 s", async () => {
+ resetRecoveryClock(); // REAL clock: this is the whole point of the gate.
+ let elapsed = 0;
+ const op = async (signal: AbortSignal) => {
+ const t0 = performance.now();
+ try {
+ return await fetch(SLOW_URL, mergeSignalIntoInit({ method: "POST" }, signal));
+ } finally {
+ elapsed = performance.now() - t0;
+ }
+ };
+
+ const started = performance.now();
+ const result = await withConnectionRetry(op, CONNECT_FAILURE_SEED, {
+ providerName: "slow-net-live",
+ providerDisplayName: "Slow Net Live",
+ resolveEndpoint: () => SLOW_URL,
+ deadlineAt: performance.now() + 60_000,
+ signal: new AbortController().signal,
+ });
+ const total = performance.now() - started;
+
+ // The numbers ARE the evidence for this criterion, so print them: a gated
+ // test that only asserts leaves nothing behind to paste into a phase log.
+ console.log(
+ `[F-SLOW live] attempt ${(elapsed / 1000).toFixed(3)}s (cap ${
+ PER_ATTEMPT_CONNECT_CAP_MS / 1000
+ }s, macOS gives up at ${OS_GIVE_UP_MS / 1000}s) · total ${(total / 1000).toFixed(3)}s · ${result.kind}`
+ );
+
+ // The attempt ended at the cap, with a wide margin on both sides: not
+ // early (it really did hang) and not at the OS's own give-up.
+ expect(elapsed).toBeGreaterThan(PER_ATTEMPT_CONNECT_CAP_MS - 2_000);
+ expect(elapsed).toBeLessThan(PER_ATTEMPT_CONNECT_CAP_MS + 2_000);
+ expect(elapsed).toBeLessThan(OS_GIVE_UP_MS - 20_000);
+ // And the deadline held from the inbound request start.
+ expect(total).toBeLessThan(60_000);
+ expect(result.kind).toBe("exhausted");
+ await drain();
+ }, 180_000);
+});
diff --git a/packages/cli/src/handlers/shared/transient-retry.test.ts b/packages/cli/src/handlers/shared/transient-retry.test.ts
new file mode 100644
index 00000000..addcbbcc
--- /dev/null
+++ b/packages/cli/src/handlers/shared/transient-retry.test.ts
@@ -0,0 +1,570 @@
+/**
+ * The tier-1 retry ladder.
+ *
+ * This file carries **C-2** (the backoff schedule) and the unit half of **C-9**
+ * (client disconnect). Both run on the injected clock, so the 5/10/30/60/60
+ * sequence and the 270-second deadline are asserted in milliseconds of real
+ * time rather than in minutes.
+ *
+ * The ladder's single most mis-readable property: **the last delay REPEATS.**
+ * `CONNECTION_RETRY_DELAYS_MS` has four elements and the schedule is
+ * 5/10/30/60/60/60/… forever — the DEADLINE ends it, not the array. That
+ * inverts `STREAM_RETRY_DELAYS_MS`, where the array running out IS the budget,
+ * and an implementation that copied the sibling's shape would stop after four
+ * gaps with most of the budget unspent.
+ *
+ * Fixtures are PROVOKED from a real `fetch` refusal, never hand-written: Bun's
+ * fetch uses neither Node's errno names nor `.cause`, so a hand-written
+ * `{ code: "ECONNREFUSED" }` would test the classifier's table instead of the
+ * runtime's behaviour.
+ */
+
+import { afterEach, beforeAll, describe, expect, test } from "bun:test";
+import type { Context } from "hono";
+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";
+import { closeAllEpisodes, describeEpisode, episodeCount } from "../../recovery/coordinator.js";
+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";
+import {
+ CONNECTION_RETRY_DELAYS_MS,
+ MIN_ATTEMPT_SLOT_MS,
+ PER_ATTEMPT_CONNECT_CAP_MS,
+ connectionRetryDelayMs,
+ inboundStartedAtPerf,
+ mergeSignalIntoInit,
+ refreshDeadlineAt,
+ tier1DeadlineAt,
+ withConnectionRetry,
+} from "./transient-retry.js";
+
+// ───────────────────────────────────────────────────────────────────────────
+// Fixture — provoked, not hand-written
+// ───────────────────────────────────────────────────────────────────────────
+
+const realFetch = globalThis.fetch;
+let CONNECT_FAILURE: unknown;
+
+beforeAll(async () => {
+ try {
+ await realFetch("http://127.0.0.1:1/refused-on-purpose");
+ throw new Error("127.0.0.1:1 accepted a connection — this fixture needs a closed port");
+ } catch (e) {
+ CONNECT_FAILURE = e;
+ }
+ // Guard the guard. If Bun ever stops setting a recognisable code, every
+ // assertion below would go green for the wrong reason.
+ expect((CONNECT_FAILURE as { code?: string }).code).toBe("ConnectionRefused");
+});
+
+let clock: FakeClock;
+function useFakeClock(start = 0): FakeClock {
+ clock = new FakeClock(start);
+ setRecoveryClock(clock);
+ return clock;
+}
+
+afterEach(() => {
+ closeAllEpisodes();
+ resetRecoveryClock();
+});
+
+let seq = 0;
+function ctx(overrides: Record<string, unknown> = {}) {
+ return {
+ providerName: `ladder-${++seq}`,
+ providerDisplayName: "Ladder Probe",
+ resolveEndpoint: () => "http://127.0.0.1:1/v1/chat/completions",
+ deadlineAt: 270_000,
+ signal: new AbortController().signal,
+ ...overrides,
+ } as Parameters<typeof withConnectionRetry>[2];
+}
+
+/** An `op` that always fails the way a dead socket does, recording WHEN. */
+function failingOp(at: number[]) {
+ return async (_signal: AbortSignal) => {
+ at.push(clock.now());
+ throw CONNECT_FAILURE;
+ };
+}
+
+// ───────────────────────────────────────────────────────────────────────────
+// The schedule itself
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("CONNECTION_RETRY_DELAYS_MS", () => {
+ test("is exactly [5000, 10000, 30000, 60000]", () => {
+ expect([...CONNECTION_RETRY_DELAYS_MS]).toEqual([5_000, 10_000, 30_000, 60_000]);
+ });
+
+ test("is not the stream-retry schedule — the two budgets are unrelated", async () => {
+ const { STREAM_RETRY_DELAYS_MS } = (await import("../composed-handler.js")) as unknown as {
+ STREAM_RETRY_DELAYS_MS?: readonly number[];
+ };
+ // Not exported today; the point of the assertion is that if it ever is,
+ // the two schedules must still differ. Merging them couples the sniffer's
+ // 12 s budget to the connect ladder.
+ if (STREAM_RETRY_DELAYS_MS) {
+ expect([...STREAM_RETRY_DELAYS_MS]).not.toEqual([...CONNECTION_RETRY_DELAYS_MS]);
+ }
+ expect(CONNECTION_RETRY_DELAYS_MS.length).toBe(4);
+ });
+});
+
+describe("connectionRetryDelayMs — the LAST element repeats", () => {
+ test("indexes the array for 0..2", () => {
+ expect(connectionRetryDelayMs(0)).toBe(5_000);
+ expect(connectionRetryDelayMs(1)).toBe(10_000);
+ expect(connectionRetryDelayMs(2)).toBe(30_000);
+ });
+
+ test("returns 60000 for EVERY n ≥ 3 — the budget ends the ladder, not the array", () => {
+ for (const n of [3, 4, 5, 6, 10, 99, 1_000, Number.MAX_SAFE_INTEGER]) {
+ expect(connectionRetryDelayMs(n)).toBe(60_000);
+ }
+ // An implementation that ran off the end would return undefined here, and
+ // `setTimeout(fn, undefined)` fires immediately — a tight loop against a
+ // dead host rather than a one-minute gap.
+ expect(connectionRetryDelayMs(4)).not.toBeUndefined();
+ });
+
+ test("a negative index clamps to the first rung rather than to undefined", () => {
+ expect(connectionRetryDelayMs(-1)).toBe(5_000);
+ expect(connectionRetryDelayMs(-100)).toBe(5_000);
+ });
+});
+
+describe("the per-attempt cap clears the OS's own give-up", () => {
+ test("is 45 s — NOT 75 000, which races macOS's 75.004 s by 4 ms", () => {
+ expect(PER_ATTEMPT_CONNECT_CAP_MS).toBe(45_000);
+ // Measured: macOS gives up on an unrouted address at 75.004 s. A cap at
+ // 75 000 is decided differently on different runs, which makes every
+ // assertion keyed on "the attempt ended at the clamp" flaky by
+ // construction. The margin below is the whole reason for the number.
+ expect(75_004 - PER_ATTEMPT_CONNECT_CAP_MS).toBeGreaterThan(20_000);
+ });
+
+ test("refreshDeadlineAt reserves exactly one unclamped attempt", () => {
+ expect(refreshDeadlineAt(270_000)).toBe(270_000 - PER_ATTEMPT_CONNECT_CAP_MS);
+ expect(refreshDeadlineAt(270_000)).toBe(225_000);
+ });
+
+ test("MIN_ATTEMPT_SLOT_MS is 3 s", () => {
+ expect(MIN_ATTEMPT_SLOT_MS).toBe(3_000);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-2 — the observed gaps
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-2 — the backoff schedule is the specified one", () => {
+ test("gaps are 5/10/30/60/60/60 and the ladder exhausts at the computed attempt", async () => {
+ const c = useFakeClock(0);
+ const at: number[] = [];
+ const deadlineAt = 270_000; // the default, derived from API_TIMEOUT_MS
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt })),
+ 400_000
+ );
+
+ // Attempt 1 is the caller's own failure, outside this function, so `at`
+ // holds the RE-ISSUES. Their absolute times:
+ expect(at).toEqual([5_000, 15_000, 45_000, 105_000, 165_000, 225_000]);
+
+ const gaps = at.map((t, i) => t - (i === 0 ? 0 : (at[i - 1] as number)));
+ expect(gaps).toEqual([5_000, 10_000, 30_000, 60_000, 60_000, 60_000]);
+
+ // Seven attempts in the episode: the opening failure plus six re-issues.
+ expect(result.kind).toBe("exhausted");
+ expect(result.attempts).toBe(7);
+
+ // Exhaustion is a CEILING by construction: the last failure lands before
+ // the deadline, and the next rung would not fit.
+ const last = at[at.length - 1] as number;
+ expect(last).toBeLessThan(deadlineAt);
+ expect(last + connectionRetryDelayMs(6) + MIN_ATTEMPT_SLOT_MS).toBeGreaterThan(deadlineAt);
+ expect(c.now()).toBeLessThanOrEqual(deadlineAt);
+ });
+
+ test("the sequence is deadline-driven, not array-driven: a 30 s budget stops at 2 gaps", async () => {
+ // The shape a user with API_TIMEOUT_MS=60000 gets. The gaps are the SAME
+ // first rungs; only how many of them fit changes.
+ const c = useFakeClock(0);
+ const at: number[] = [];
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 30_000 })),
+ 120_000
+ );
+ expect(at).toEqual([5_000, 15_000]);
+ expect(result.kind).toBe("exhausted");
+ expect(result.attempts).toBe(3);
+ });
+
+ test("a 15 s floor budget still buys one real retry", async () => {
+ const c = useFakeClock(0);
+ const at: number[] = [];
+ await advanceUntilSettled(
+ c,
+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 15_000 })),
+ 60_000
+ );
+ // 5 + 3 = 8 ≤ 15, so one rung fits; 5 + 10 + 3 = 18 > 15, so a second
+ // does not. Anything under 8 s would make the ladder a pure delay in
+ // front of the same 400.
+ expect(at).toEqual([5_000]);
+ });
+
+ test("the deadline check bounds the ATTEMPT THAT FOLLOWS the sleep, not just the sleep", async () => {
+ // 165 000 + 60 000 = 225 000 < 240 000, so a check that bounded only the
+ // SLEEP would start a seventh attempt with 15 s of budget left. It must
+ // not: MIN_ATTEMPT_SLOT_MS is part of the comparison.
+ const c = useFakeClock(0);
+ const at: number[] = [];
+ await advanceUntilSettled(
+ c,
+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 226_000 })),
+ 400_000
+ );
+ expect(at).toEqual([5_000, 15_000, 45_000, 105_000, 165_000]);
+ expect(at).not.toContain(225_000);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Recovery, and the outcomes that are not recovery
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("a re-issue that succeeds returns the value and stops the ladder", () => {
+ test("`ok` on the third attempt — no fourth attempt is ever made", async () => {
+ const c = useFakeClock(0);
+ const at: number[] = [];
+ const op = async (_s: AbortSignal) => {
+ at.push(c.now());
+ if (at.length < 3) throw CONNECT_FAILURE;
+ return { body: "a real upstream Response would be here" };
+ };
+
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(op, CONNECT_FAILURE, ctx()),
+ 400_000
+ );
+
+ expect(result.kind).toBe("ok");
+ if (result.kind !== "ok") throw new Error("unreachable");
+ expect(result.value).toEqual({ body: "a real upstream Response would be here" });
+ // Two failed re-issues, then the success — and then nothing.
+ expect(at).toEqual([5_000, 15_000, 45_000]);
+ expect(result.attempts).toBe(4);
+ expect(result.recoveryMs).toBe(45_000);
+ expect(result.episodeId).not.toBe("");
+ // Ladder stopped: no timer left armed, no episode left open.
+ expect(c.pending()).toBe(0);
+ expect(episodeCount()).toBe(0);
+ });
+
+ test("an immediate success on the FIRST re-issue costs exactly one gap", async () => {
+ const c = useFakeClock(0);
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(async () => "ok", CONNECT_FAILURE, ctx()),
+ 60_000
+ );
+ expect(result.kind).toBe("ok");
+ expect(c.now()).toBe(5_000);
+ expect(result.attempts).toBe(2);
+ });
+});
+
+describe("outcomes that are not recovery", () => {
+ test("an unclassifiable later throw comes back as `threw`, not as the network error", async () => {
+ // HIGH-10. A retry that throws an auth failure, a transport bug or a
+ // programming error must not be relabelled as the previous connection
+ // error — that would quietly widen recovery past the classified set.
+ const c = useFakeClock(0);
+ const boom = new TypeError("undefined is not a function");
+ let n = 0;
+ const op = async () => {
+ n++;
+ throw boom;
+ };
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(op, CONNECT_FAILURE, ctx()),
+ 60_000
+ );
+ expect(result.kind).toBe("threw");
+ if (result.kind !== "threw") throw new Error("unreachable");
+ expect(result.error).toBe(boom);
+ expect(n).toBe(1);
+ expect(c.pending()).toBe(0);
+ });
+
+ test("an unclassifiable FIRST failure is refused rather than retried", async () => {
+ const c = useFakeClock(0);
+ let called = false;
+ const result = await withConnectionRetry(
+ async () => {
+ called = true;
+ return "never";
+ },
+ new TypeError("not a network failure"),
+ ctx()
+ );
+ expect(result.kind).toBe("threw");
+ expect(called).toBe(false);
+ expect(episodeCount()).toBe(0);
+ expect(c.pending()).toBe(0);
+ });
+
+ test("`gave_up` when the episode is abandoned under a parked waiter", async () => {
+ useFakeClock(0);
+ const at: number[] = [];
+ const p = withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx());
+ await drain();
+ closeAllEpisodes("gave_up");
+ const result = await p;
+ expect(result.kind).toBe("gave_up");
+ expect(at).toEqual([]);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// C-9 — the client going away
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-9 — a client disconnect stops the loop and cancels the upstream fetch", () => {
+ test("aborting mid-gap resolves `client_gone`, clears the timer and releases the episode", async () => {
+ const c = useFakeClock(0);
+ const ac = new AbortController();
+ const at: number[] = [];
+ const p = withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ signal: ac.signal }));
+ await drain();
+ expect(c.pending()).toBe(1);
+
+ ac.abort(new DOMException("client gone", "AbortError"));
+ const result = await p;
+
+ expect(result.kind).toBe("client_gone");
+ // No attempt was made after the abort.
+ expect(at).toEqual([]);
+ // The timer is gone and the episode is released AT ONCE — not after the
+ // 120 s handoff grace, which applies only to a deliberate tier-2 handoff.
+ expect(c.pending()).toBe(0);
+ expect(episodeCount()).toBe(0);
+ expect(describeEpisode(result.episodeId)).toBeNull();
+ });
+
+ test("the signal handed to `op` aborts with the client — the upstream fetch is cancelled", async () => {
+ const c = useFakeClock(0);
+ const ac = new AbortController();
+ let inFlight: AbortSignal | null = null;
+ let abortedDuringOp = false;
+
+ const op = (signal: AbortSignal) =>
+ new Promise<string>((_resolve, reject) => {
+ inFlight = signal;
+ signal.addEventListener("abort", () => {
+ abortedDuringOp = true;
+ reject(signal.reason);
+ });
+ });
+
+ const p = withConnectionRetry(op, CONNECT_FAILURE, ctx({ signal: ac.signal }));
+ await drain();
+ await c.advance(5_000); // first rung fires, op starts and hangs
+ expect(inFlight).not.toBeNull();
+ expect((inFlight as unknown as AbortSignal).aborted).toBe(false);
+
+ ac.abort(new DOMException("client gone", "AbortError"));
+ const result = await p;
+
+ // This is what "the upstream fetch is cancelled" means concretely: the
+ // signal `op` was handed — the one merged into the real fetch init — is
+ // the one that fired.
+ expect(abortedDuringOp).toBe(true);
+ expect(result.kind).toBe("client_gone");
+ expect(c.pending()).toBe(0);
+ expect(episodeCount()).toBe(0);
+ });
+
+ test("an already-aborted client is answered without a single attempt", async () => {
+ const c = useFakeClock(0);
+ const ac = new AbortController();
+ ac.abort(new DOMException("gone", "AbortError"));
+ const at: number[] = [];
+ const result = await withConnectionRetry(
+ failingOp(at),
+ CONNECT_FAILURE,
+ ctx({ signal: ac.signal })
+ );
+ expect(result.kind).toBe("client_gone");
+ expect(at).toEqual([]);
+ expect(c.pending()).toBe(0);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The per-attempt clamp
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("the per-attempt clamp bounds the OPERATION, not just the fetch inside it", () => {
+ test("a hung attempt is aborted at the cap and counted as a failed attempt", async () => {
+ const c = useFakeClock(0);
+ const starts: number[] = [];
+ const ends: number[] = [];
+ const op = (signal: AbortSignal) =>
+ new Promise<string>((_res, rej) => {
+ starts.push(c.now());
+ signal.addEventListener("abort", () => {
+ ends.push(c.now());
+ rej(signal.reason);
+ });
+ });
+
+ const result = await advanceUntilSettled(
+ c,
+ withConnectionRetry(op, CONNECT_FAILURE, ctx({ deadlineAt: 270_000 })),
+ 400_000
+ );
+
+ expect(starts[0]).toBe(5_000);
+ // Aborted at the cap, not at the OS's 75 s and not at the deadline.
+ expect((ends[0] as number) - (starts[0] as number)).toBe(PER_ATTEMPT_CONNECT_CAP_MS);
+ // The clamp's own abort is re-classified as a failed attempt of the same
+ // kind rather than escaping as an unrelated error.
+ expect(result.kind).toBe("exhausted");
+ // Every attempt costs its gap plus the full cap, so the deadline is still
+ // honoured from the inbound request start.
+ expect(c.now()).toBeLessThanOrEqual(270_000);
+ });
+
+ test("the clamp shrinks to the REMAINING budget when that is smaller than the cap", async () => {
+ const c = useFakeClock(0);
+ const starts: number[] = [];
+ const ends: number[] = [];
+ const op = (signal: AbortSignal) =>
+ new Promise<string>((_res, rej) => {
+ starts.push(c.now());
+ signal.addEventListener("abort", () => {
+ ends.push(c.now());
+ rej(signal.reason);
+ });
+ });
+ // 5 s gap, then only 15 s of budget left — less than the 45 s cap.
+ await advanceUntilSettled(
+ c,
+ withConnectionRetry(op, CONNECT_FAILURE, ctx({ deadlineAt: 20_000 })),
+ 120_000
+ );
+ expect(starts[0]).toBe(5_000);
+ expect((ends[0] as number) - (starts[0] as number)).toBe(15_000);
+ });
+
+ test("the clamp is disarmed the instant the operation settles", async () => {
+ const c = useFakeClock(0);
+ await advanceUntilSettled(
+ c,
+ withConnectionRetry(async () => "ok", CONNECT_FAILURE, ctx()),
+ 60_000
+ );
+ // If the clamp outlived the attempt it would still be armed here, and on
+ // the real clock it would abort a signal already merged into a live
+ // streaming body.
+ expect(c.pending()).toBe(0);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The deadline's origin
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("the deadline is anchored to the INBOUND request, not to the candidate", () => {
+ function fakeContext(raw?: Request): Context {
+ return { req: { raw, header: () => undefined } } as unknown as Context;
+ }
+
+ test("every candidate in one chain inherits the FIRST candidate's start", () => {
+ const c = useFakeClock(1_000);
+ const raw = new Request("http://127.0.0.1:1/v1/messages", { method: "POST" });
+ const first = inboundStartedAtPerf(fakeContext(raw));
+ expect(first).toBe(1_000);
+
+ // The fallback handler calls handle() once PER CANDIDATE, and each mints
+ // its own `startTime`. Without the shared anchor a three-candidate chain
+ // could hold one socket for three full deadlines.
+ c.advance(50_000);
+ const second = inboundStartedAtPerf(fakeContext(raw));
+ expect(second).toBe(1_000);
+ });
+
+ test("a different inbound request gets its own anchor", async () => {
+ const c = useFakeClock(0);
+ const a = new Request("http://127.0.0.1:1/a", { method: "POST" });
+ const b = new Request("http://127.0.0.1:1/b", { method: "POST" });
+ expect(inboundStartedAtPerf(fakeContext(a))).toBe(0);
+ await c.advance(7_000);
+ expect(inboundStartedAtPerf(fakeContext(b))).toBe(7_000);
+ expect(inboundStartedAtPerf(fakeContext(a))).toBe(0);
+ });
+
+ test("a synthesised Context with no raw Request still yields a usable anchor", () => {
+ useFakeClock(4_242);
+ expect(inboundStartedAtPerf(fakeContext(undefined))).toBe(4_242);
+ });
+
+ test("tier1DeadlineAt = anchor + the DERIVED budget, and it moves with API_TIMEOUT_MS", () => {
+ useFakeClock(1_000);
+ const raw = new Request("http://127.0.0.1:1/deadline", { method: "POST" });
+ const saved = process.env.API_TIMEOUT_MS;
+ try {
+ delete process.env.API_TIMEOUT_MS;
+ expect(tier1DeadlineAt(fakeContext(raw))).toBe(1_000 + 270_000);
+
+ const raw2 = new Request("http://127.0.0.1:1/deadline2", { method: "POST" });
+ process.env.API_TIMEOUT_MS = "60000";
+ // Same anchor arithmetic, different budget — the number is derived, not
+ // pinned. A user with a 60 s client must not be handed a 270 s hold.
+ expect(tier1DeadlineAt(fakeContext(raw2))).toBe(1_000 + 30_000);
+ } finally {
+ if (saved === undefined) delete process.env.API_TIMEOUT_MS;
+ else process.env.API_TIMEOUT_MS = saved;
+ }
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Signal composition
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("mergeSignalIntoInit composes rather than replaces", () => {
+ test("the transport's own signal survives beside the clamp", async () => {
+ const own = new AbortController();
+ const ours = new AbortController();
+ const init = mergeSignalIntoInit({ method: "POST", signal: own.signal }, ours.signal);
+ const merged = init.signal as AbortSignal;
+ expect(merged).not.toBe(own.signal);
+ expect(merged).not.toBe(ours.signal);
+
+ // Either one aborts the merged signal — which is what makes the local
+ // transport's ten-minute ceiling and our 45 s clamp both apply.
+ expect(merged.aborted).toBe(false);
+ own.abort(new DOMException("transport ceiling", "TimeoutError"));
+ await drain(1);
+ expect(merged.aborted).toBe(true);
+
+ const init2 = mergeSignalIntoInit({ signal: new AbortController().signal }, ours.signal);
+ ours.abort(new DOMException("attempt cap", "TimeoutError"));
+ await drain(1);
+ expect((init2.signal as AbortSignal).aborted).toBe(true);
+ });
+
+ test("no own signal ⇒ ours is used directly; no signal of ours ⇒ the init is untouched", () => {
+ const ours = new AbortController().signal;
+ expect(mergeSignalIntoInit({ method: "POST" }, ours).signal).toBe(ours);
+ const bare = { method: "POST" };
+ expect(mergeSignalIntoInit(bare, undefined)).toBe(bare as unknown as RequestInit);
+ });
+});
diff --git a/packages/cli/src/profile-config-recovery.test.ts b/packages/cli/src/profile-config-recovery.test.ts
new file mode 100644
index 00000000..2b2bc44d
--- /dev/null
+++ b/packages/cli/src/profile-config-recovery.test.ts
@@ -0,0 +1,111 @@
+/**
+ * The CLAUDE.md silent-drop invariant, discharged for the two recovery fields.
+ *
+ * > A new `ClaudishProfileConfig` field MUST be added to `loadConfig`'s
+ * > allowlist in `profile-config.ts`; otherwise it survives on disk until the
+ * > first global save and is then dropped.
+ *
+ * `loadConfig` is a HAND-WRITTEN allowlist, not a spread, so a field absent
+ * from it round-trips to nothing. The failure is silent in the worst way: the
+ * user's setting works until some unrelated code path calls
+ * `saveConfig(loadConfig())` — the config TUI saving any other setting will do
+ * — and is then erased with no error. For `recovery.enabled: false` that means
+ * a deliberate OFF-SWITCH silently turning back ON.
+ *
+ * The trap has caught this codebase twice before (`onepasswordEnvironments`,
+ * then `keychain`), which is why the round-trip is pinned the moment the field
+ * is defined rather than the moment it is first consumed.
+ *
+ * Hermetic via `--config`'s own override seam, which is what `bun run
+ * test:safe` relies on to stay off the real machine file.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { setConfigFileOverride } from "./config-override.js";
+import { loadConfig, saveConfig } from "./profile-config.js";
+
+let dir: string;
+let file: string;
+
+beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), "claudish-recovery-roundtrip-"));
+ file = join(dir, "config.json");
+ setConfigFileOverride(file);
+});
+
+afterEach(() => {
+ setConfigFileOverride(null);
+ rmSync(dir, { recursive: true, force: true });
+});
+
+function onDisk(): Record<string, unknown> {
+ return JSON.parse(readFileSync(file, "utf-8")) as Record<string, unknown>;
+}
+
+describe("saveConfig(loadConfig()) round-trip", () => {
+ test("preserves recoveryUi", () => {
+ writeFileSync(
+ file,
+ JSON.stringify({ openRouterApiKey: "sk-test", recoveryUi: { enabled: false } })
+ );
+
+ // Exactly what the config TUI does when it saves any unrelated setting.
+ saveConfig(loadConfig());
+
+ expect(onDisk().recoveryUi).toEqual({ enabled: false });
+ });
+
+ test("preserves recovery (the master switch) — an off-switch must not turn itself on", () => {
+ writeFileSync(
+ file,
+ JSON.stringify({ openRouterApiKey: "sk-test", recovery: { enabled: false } })
+ );
+ saveConfig(loadConfig());
+ expect(onDisk().recovery).toEqual({ enabled: false });
+ });
+
+ test("survives repeated save/load cycles, which is how the drop actually shows up", () => {
+ writeFileSync(
+ file,
+ JSON.stringify({ recovery: { enabled: false }, recoveryUi: { enabled: true } })
+ );
+ for (let i = 0; i < 3; i++) saveConfig(loadConfig());
+ expect(onDisk().recovery).toEqual({ enabled: false });
+ expect(onDisk().recoveryUi).toEqual({ enabled: true });
+ });
+
+ test("an empty object survives too — `{}` is a meaningful value, not an absence", () => {
+ // `{}` means "no opinion, fall through to the next scope". Dropping it is
+ // a smaller bug than dropping `{enabled:false}` but it is the same bug,
+ // and an allowlist entry guarded on truthiness rather than on
+ // `!== undefined` would drop exactly this shape.
+ writeFileSync(file, JSON.stringify({ recoveryUi: {} }));
+ saveConfig(loadConfig());
+ expect(onDisk().recoveryUi).toEqual({});
+ });
+
+ test("absent stays absent — the allowlist must not invent a default onto disk", () => {
+ writeFileSync(file, JSON.stringify({ openRouterApiKey: "sk-test" }));
+ saveConfig(loadConfig());
+ const disk = onDisk();
+ expect("recovery" in disk).toBe(false);
+ expect("recoveryUi" in disk).toBe(false);
+ });
+
+ test("the sibling fields it sits beside still round-trip, so the allowlist was not disturbed", () => {
+ writeFileSync(
+ file,
+ JSON.stringify({
+ openRouterApiKey: "sk-test",
+ proOnUltracode: true,
+ recoveryUi: { enabled: false },
+ })
+ );
+ saveConfig(loadConfig());
+ expect(onDisk().proOnUltracode).toBe(true);
+ expect(onDisk().recoveryUi).toEqual({ enabled: false });
+ });
+});
diff --git a/packages/cli/src/providers/transport/local-recovery.test.ts b/packages/cli/src/providers/transport/local-recovery.test.ts
new file mode 100644
index 00000000..b10971ff
--- /dev/null
+++ b/packages/cli/src/providers/transport/local-recovery.test.ts
@@ -0,0 +1,321 @@
+/**
+ * C-17 — a stopped LOCAL provider reaches the ladder at all.
+ *
+ * WHY THIS FILE EXISTS SEPARATELY FROM EVERY OTHER RECOVERY TEST. The rest of
+ * the suite induces its faults with a CUSTOM ENDPOINT, and custom endpoints are
+ * built on the OpenAI / Anthropic / LiteLLM transports — none of which has a
+ * `refreshAuth`, and all of which now classify a connect failure correctly. So
+ * the whole of C-1…C-6 can go green over a completely broken local path: a
+ * stopped Ollama fails inside `checkHealth()`, which used to swallow the
+ * syscall error, and `refreshAuth()` rethrew a bare `Error` carrying neither a
+ * `code` nor a `cause`. `classifyConnectionError` returned `null`, and the
+ * single most common local failure never reached recovery at all.
+ *
+ * Four assertions, and only the first was in the original criterion:
+ *
+ * 1. the ladder runs at all;
+ * 2. exactly ONE episode across the whole recovery;
+ * 3. NO `recovered` is reported while the server stays down;
+ * 4. the `[Recovery]` lines name the PROBE endpoint, not the inference one.
+ *
+ * Assertions 2 and 3 exist because assertion 1 passes even when the fix is a
+ * no-op. `refreshAuth` opens with `if (this.healthChecked) return;` and
+ * `checkHealth()` used to latch that flag ON FAILURE, so a retried
+ * `refreshAuth()` issued no probe and did not throw — it returned SUCCESS for a
+ * server that was still dead. The episode would close `recovered` for an outage
+ * that never ended, and the banner and ladder assertion 1 sees would belong to
+ * a SECOND episode opened milliseconds later by the fetch path against the same
+ * dead port.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import type { Context } from "hono";
+import { ComposedHandler } from "../../handlers/composed-handler.js";
+import { classifyConnectionError } from "../../handlers/shared/connection-error.js";
+import { withConnectionRetry } from "../../handlers/shared/transient-retry.js";
+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";
+import { closeAllEpisodes, episodeCount } from "../../recovery/coordinator.js";
+import {
+ capturedRecoveryLines,
+ startLogCapture,
+ stopLogCapture,
+} from "../../recovery/test-helpers/capture-log.js";
+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";
+import type { LocalProvider as LocalProviderConfig } from "../provider-registry.js";
+import { LocalTransport } from "./local.js";
+
+/** A port nothing is listening on, taken and released so it is VERIFIED free. */
+function freePort(): number {
+ const s = Bun.serve({ port: 0, fetch: () => new Response("x") });
+ const port = s.port as number;
+ s.stop(true);
+ return port;
+}
+
+const realFetch = globalThis.fetch;
+let clock: FakeClock;
+let deadPort: number;
+
+beforeEach(() => {
+ deadPort = freePort();
+ clock = new FakeClock(0);
+ setRecoveryClock(clock);
+ startLogCapture();
+});
+
+afterEach(() => {
+ globalThis.fetch = realFetch;
+ stopLogCapture();
+ closeAllEpisodes();
+ resetRecoveryClock();
+});
+
+function ollamaConfig(port: number): LocalProviderConfig {
+ return {
+ name: "ollama",
+ baseUrl: `http://127.0.0.1:${port}`,
+ apiPath: "/v1/chat/completions",
+ envVar: "OLLAMA_API_KEY",
+ prefixes: ["ollama"],
+ };
+}
+
+/** Count the health probes the transport actually issues, per URL. */
+function countProbes(): { urls: string[] } {
+ const urls: string[] = [];
+ const wrapped = (async (input: RequestInfo | URL, init?: RequestInit) => {
+ const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;
+ if (url.includes("/api/tags") || url.includes("/v1/models") || url.includes("/api/show")) {
+ urls.push(url);
+ }
+ return realFetch(input as RequestInfo, init);
+ }) as unknown as typeof fetch;
+ globalThis.fetch = wrapped;
+ return { urls };
+}
+
+const PAYLOAD = {
+ model: "llama3.2",
+ max_tokens: 16,
+ messages: [{ role: "user", content: "hi" }],
+};
+
+function contextFor(req: Request): { c: Context; captured: { status?: number; body?: any } } {
+ const captured: { status?: number; body?: any } = {};
+ const c = {
+ req: {
+ raw: req,
+ header: (name?: string) => (name === undefined ? {} : (req.headers.get(name) ?? undefined)),
+ },
+ header: () => {},
+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),
+ json: (body: unknown, status?: number) => {
+ captured.body = body;
+ captured.status = status;
+ return new Response(JSON.stringify(body), { status: status ?? 200 });
+ },
+ } as unknown as Context;
+ return { c, captured };
+}
+
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("C-17 — a stopped local provider enters the ladder through refreshAuth", () => {
+ test("a bare `new Error` from refreshAuth would NOT classify — the cause is what carries it", async () => {
+ // The control that makes the rest of this file meaningful. Without
+ // `{ cause }` the classifier returns null and everything downstream is a
+ // no-op for ollama / lmstudio / vllm.
+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");
+ let thrown: unknown;
+ try {
+ await transport.refreshAuth();
+ } catch (e) {
+ thrown = e;
+ }
+ expect(thrown).toBeDefined();
+ expect(classifyConnectionError(thrown)).not.toBeNull();
+ expect(classifyConnectionError(thrown)?.kind).toBe("refused");
+ // And the endpoint it carries is a PROBE url, not the inference one.
+ //
+ // DEVIATION FROM THE CRITERION'S LITERAL TEXT, recorded here rather than
+ // hidden: C-17 assertion 4 says the lines must name `…/api/tags`. They name
+ // `…/v1/models`, because `checkHealth()` tries `/api/tags` FIRST and then
+ // falls back to `/v1/models`, overwriting `lastProbeUrl` each time — so the
+ // endpoint reported is the LAST probe attempted rather than the first. That
+ // is the more general behaviour (LM Studio and vLLM have no `/api/tags` at
+ // all, so pinning the first would name a URL those providers never use) and
+ // it satisfies what FR-5 actually requires: the right HOST, and a probe
+ // path rather than the inference path. The assertion is written to the
+ // requirement, not to the example.
+ const endpoint = (thrown as { claudishEndpoint?: string }).claudishEndpoint ?? "";
+ expect(endpoint).toMatch(/\/(api\/tags|v1\/models)$/);
+ expect(endpoint).not.toContain("/v1/chat/completions");
+ expect(endpoint).toContain(`127.0.0.1:${deadPort}`);
+ // A bare rethrow — what the transport used to do — classifies as null.
+ expect(classifyConnectionError(new Error((thrown as Error).message))).toBeNull();
+ });
+
+ test("the ladder RE-PROBES on every attempt and never reports a false `recovered`", async () => {
+ const { urls } = countProbes();
+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");
+
+ let first: unknown;
+ try {
+ await transport.refreshAuth();
+ } catch (e) {
+ first = e;
+ }
+ const probesAfterFirst = urls.length;
+ expect(probesAfterFirst).toBeGreaterThan(0);
+
+ const conn = classifyConnectionError(first);
+ expect(conn).not.toBeNull();
+
+ const result = await advanceUntilSettled(
+ clock,
+ withConnectionRetry(() => transport.refreshAuth(), first, {
+ providerName: "ollama",
+ providerDisplayName: "Ollama",
+ resolveEndpoint: (err) =>
+ (err as { claudishEndpoint?: string })?.claudishEndpoint ??
+ `http://127.0.0.1:${deadPort}`,
+ deadlineAt: 30_000, // two rungs is enough to see the latch
+ signal: new AbortController().signal,
+ }),
+ 120_000
+ );
+
+ // 1. The ladder ran.
+ expect(result.kind).toBe("exhausted");
+ expect(result.attempts).toBe(3);
+
+ // 2. Each retried refreshAuth RE-PROBED. With the success-only latch
+ // reverted, this number does not move at all past the first call.
+ expect(urls.length).toBeGreaterThan(probesAfterFirst);
+ expect(urls.filter((u) => u.includes("/api/tags")).length).toBeGreaterThanOrEqual(3);
+
+ // 3. No false recovery: the server never came back, so nothing may say it
+ // did — this is the assertion that fails when the latch is reverted.
+ const lines = await capturedRecoveryLines();
+ expect(lines.some((l) => l.includes("Ollama recovered after"))).toBe(false);
+ expect(lines.some((l) => l.includes("closed: recovered"))).toBe(false);
+
+ // 4. Exactly ONE episode across the whole recovery.
+ const opened = lines.filter((l) => l.includes("opened for"));
+ expect(opened.length).toBe(1);
+ const ids = new Set(
+ lines.map((l) => /episode ([0-9a-f-]{36})/.exec(l)?.[1]).filter(Boolean) as string[]
+ );
+ expect(ids.size).toBe(1);
+
+ // 5. The lines name a PROBE endpoint, not the inference endpoint. See the
+ // recorded deviation in the first test for why it is `/v1/models`.
+ expect(opened[0]).toMatch(/\/(api\/tags|v1\/models) /);
+ expect(opened[0]).not.toContain("/v1/chat/completions");
+ });
+
+ test("a server that comes back MID-LADDER ends the episode with a real success", async () => {
+ // Assertion 5 of C-17. It requires the health probe to actually run again,
+ // which the reverted latch makes impossible: attempt 2 would resolve
+ // instantly, and `fetchContextWindow()` — guarded by the same flag — would
+ // never run at all.
+ const { urls } = countProbes();
+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");
+
+ let first: unknown;
+ try {
+ await transport.refreshAuth();
+ } catch (e) {
+ first = e;
+ }
+
+ let server: ReturnType<typeof Bun.serve> | null = null;
+ let attempt = 0;
+ const op = async () => {
+ attempt++;
+ // The user starts Ollama between the second and third attempt.
+ if (attempt === 3 && !server) {
+ server = Bun.serve({
+ port: deadPort,
+ fetch(req) {
+ const url = new URL(req.url);
+ if (url.pathname === "/api/tags") return Response.json({ models: [] });
+ if (url.pathname === "/api/show")
+ return Response.json({ model_info: { "llama.context_length": 131072 } });
+ return new Response("not found", { status: 404 });
+ },
+ });
+ }
+ return transport.refreshAuth();
+ };
+
+ try {
+ const result = await advanceUntilSettled(
+ clock,
+ withConnectionRetry(op, first, {
+ providerName: "ollama",
+ providerDisplayName: "Ollama",
+ resolveEndpoint: (err) =>
+ (err as { claudishEndpoint?: string })?.claudishEndpoint ??
+ `http://127.0.0.1:${deadPort}`,
+ deadlineAt: 270_000,
+ signal: new AbortController().signal,
+ }),
+ 400_000
+ );
+
+ expect(result.kind).toBe("ok");
+ expect(attempt).toBe(3);
+ // The recovered server ran fetchContextWindow — which the latch used to
+ // make unreachable forever, leaving the provider on a stale window.
+ expect(urls.some((u) => u.includes("/api/show"))).toBe(true);
+ expect(transport.getContextWindow()).toBe(131072);
+
+ const lines = await capturedRecoveryLines();
+ // ONE episode, and it recovered exactly once. The two patterns are
+ // deliberately distinct: `closed: recovered after …` ALSO contains the
+ // substring "recovered after", so a loose match counts one recovery twice.
+ expect(lines.filter((l) => l.includes("opened for")).length).toBe(1);
+ expect(lines.filter((l) => l.includes("Ollama recovered after")).length).toBe(1);
+ expect(lines.filter((l) => l.includes("closed: recovered")).length).toBe(1);
+ expect(episodeCount()).toBe(0);
+ } finally {
+ (server as ReturnType<typeof Bun.serve> | null)?.stop(true);
+ }
+ });
+
+ test("through the REAL ComposedHandler: a stopped local provider is held, not 401'd", async () => {
+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");
+ const handler = new ComposedHandler(
+ transport as unknown as ConstructorParameters<typeof ComposedHandler>[0],
+ "llama3.2",
+ "llama3.2",
+ 8080,
+ {}
+ );
+ const ac = new AbortController();
+ const req = new Request("http://127.0.0.1:8080/v1/messages", {
+ method: "POST",
+ signal: ac.signal,
+ });
+ const { c, captured } = contextFor(req);
+
+ const p = handler.handle(c, PAYLOAD);
+ await drain(40);
+
+ // It is HELD — an episode is open — rather than answered 401 and walked
+ // down the fallback chain onto metered billing.
+ expect(episodeCount()).toBe(1);
+ const opened = (await capturedRecoveryLines()).filter((l) => l.includes("opened for"));
+ expect(opened.length).toBe(1);
+ expect(opened[0]).toContain("Ollama");
+ expect(opened[0]).toMatch(/\/(api\/tags|v1\/models) /);
+ expect(opened[0]).not.toContain("/v1/chat/completions");
+
+ ac.abort(new DOMException("client gone", "AbortError"));
+ const response = await p;
+ expect(response.status).toBe(499);
+ expect(captured.status).not.toBe(401);
+ expect(episodeCount()).toBe(0);
+ });
+});
diff --git a/packages/cli/src/recovery/clock.test.ts b/packages/cli/src/recovery/clock.test.ts
new file mode 100644
index 00000000..79b3a979
--- /dev/null
+++ b/packages/cli/src/recovery/clock.test.ts
@@ -0,0 +1,102 @@
+/**
+ * The injectable clock.
+ *
+ * One property is being pinned, and it is the one every other recovery test
+ * depends on: the ladder reads `recoveryClock()` at the moment it schedules,
+ * so injecting a clock genuinely CONTROLS the schedule rather than merely
+ * shadowing a value nobody consults. If that stops being true, the ladder tests
+ * do not fail — they hang for the real 270-second deadline and are killed by
+ * the runner, which reads as flakiness rather than as a defect.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { recoveryClock, resetRecoveryClock, setRecoveryClock } from "./clock.js";
+import { closeAllEpisodes, joinEpisode } from "./coordinator.js";
+import { FakeClock, drain } from "./test-helpers/fake-clock.js";
+
+afterEach(() => {
+ closeAllEpisodes();
+ resetRecoveryClock();
+});
+
+describe("recovery clock", () => {
+ test("defaults to the real clock, in PROCESS ms", () => {
+ const before = performance.now();
+ const t = recoveryClock().now();
+ const after = performance.now();
+ expect(t).toBeGreaterThanOrEqual(before);
+ expect(t).toBeLessThanOrEqual(after);
+ // PROCESS ms, not epoch ms. Differencing the two units is the mistake the
+ // separation exists to make unrepresentable, and epoch ms is ~1.8e12.
+ expect(t).toBeLessThan(1e10);
+ });
+
+ test("setRecoveryClock installs, resetRecoveryClock restores", () => {
+ const fake = new FakeClock(1234);
+ setRecoveryClock(fake);
+ expect(recoveryClock()).toBe(fake);
+ expect(recoveryClock().now()).toBe(1234);
+ resetRecoveryClock();
+ expect(recoveryClock()).not.toBe(fake);
+ // Back to a live reading rather than the fake's frozen 1234.
+ const a = recoveryClock().now();
+ expect(a).not.toBe(1234);
+ expect(Math.abs(a - performance.now())).toBeLessThan(50);
+ });
+
+ test("the injected clock really drives the ladder timer", async () => {
+ const fake = new FakeClock(0);
+ setRecoveryClock(fake);
+ const handle = joinEpisode({
+ providerName: "clock-probe",
+ providerDisplayName: "Clock Probe",
+ endpoint: "http://127.0.0.1:9/v1",
+ kind: "refused",
+ code: "ConnectionRefused",
+ reason: "unreachable",
+ deadlineAt: 270_000,
+ });
+
+ let woke = false;
+ const parked = handle.waitForNextAttempt(new AbortController().signal, 5_000).then(() => {
+ woke = true;
+ });
+ await drain();
+
+ // Armed on the FAKE clock, not the real one.
+ expect(fake.pending()).toBe(1);
+ expect(fake.pendingAt()).toEqual([5_000]);
+
+ // Real time passing changes nothing: the gap is measured on the injected
+ // clock alone.
+ await drain(10);
+ expect(woke).toBe(false);
+
+ await fake.advance(4_999);
+ expect(woke).toBe(false);
+ await fake.advance(1);
+ await parked;
+ expect(woke).toBe(true);
+ expect(fake.pending()).toBe(0);
+
+ handle.leave();
+ });
+
+ test("unref is offered for the grace timer and is optional on the interface", () => {
+ const fake = new FakeClock(0);
+ setRecoveryClock(fake);
+ const h = fake.setTimeout(() => {}, 10);
+ recoveryClock().unref?.(h);
+ expect(fake.unrefed.has(h as number)).toBe(true);
+
+ // A clock without `unref` must not crash a caller. The real one has it;
+ // the contract says optional, so the call site is `?.`-guarded.
+ const minimal = {
+ now: () => 0,
+ setTimeout: () => 1,
+ clearTimeout: () => {},
+ };
+ setRecoveryClock(minimal);
+ expect(() => recoveryClock().unref?.(1)).not.toThrow();
+ });
+});
diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.ts
new file mode 100644
index 00000000..3ccd21f7
--- /dev/null
+++ b/packages/cli/src/recovery/coordinator.test.ts
@@ -0,0 +1,535 @@
+/**
+ * The episode coordinator.
+ *
+ * Four properties are load-bearing and each one was a review finding before it
+ * was a test:
+ *
+ * 1. N waiters on one target share ONE timer. Without this an outage costs N
+ * ladders, N sets of connect attempts against a dead network, and N
+ * unsynchronised countdowns.
+ * 2. A waiter past its OWN deadline answers alone and LEAVES THE OTHERS
+ * RUNNING. The previous revision let one waiter's deadline move the shared
+ * episode into `handoff` — a state with no timer — so every other parked
+ * waiter silently stopped being retried and answered having attempted
+ * nothing. The visible symptom was `waiters: 4` with one of them actually
+ * being retried.
+ * 3. The waiter set is EMPTY after an aborted wait, and the abort listener is
+ * detached. This is the Observer leak the design named as its own most
+ * likely failure mode.
+ * 4. A request arriving inside `EPISODE_GRACE_MS` REJOINS: same episode id,
+ * same ladder position. Restarting the ladder at 5 s would hammer a dead
+ * host harder the longer the outage lasted.
+ *
+ * Every test runs on the injected clock. None of them sleeps.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { resetRecoveryClock, setRecoveryClock } from "./clock.js";
+import {
+ EPISODE_GRACE_MS,
+ closeAllEpisodes,
+ describeEpisode,
+ episodeCount,
+ giveUp,
+ joinEpisode,
+ tryNow,
+ uiLeaseValid,
+} from "./coordinator.js";
+import type { WaitOutcome } from "./coordinator.js";
+import { FakeClock, countingSignal, drain } from "./test-helpers/fake-clock.js";
+
+let clock: FakeClock;
+
+function useFakeClock(start = 0): FakeClock {
+ clock = new FakeClock(start);
+ setRecoveryClock(clock);
+ return clock;
+}
+
+afterEach(() => {
+ closeAllEpisodes();
+ resetRecoveryClock();
+});
+
+let seq = 0;
+/** A fresh provider name per test, because episodes are keyed by provider|host. */
+function seed(overrides: Partial<Parameters<typeof joinEpisode>[0]> = {}) {
+ const name = overrides.providerName ?? `coord-${++seq}`;
+ return {
+ providerName: name,
+ providerDisplayName: "Coord Probe",
+ endpoint: "http://127.0.0.1:9/v1/chat/completions",
+ kind: "refused" as const,
+ code: "ConnectionRefused",
+ reason: "Cannot reach Coord Probe",
+ deadlineAt: 270_000,
+ ...overrides,
+ };
+}
+
+const never = () => new AbortController().signal;
+
+// ───────────────────────────────────────────────────────────────────────────
+// 1. One episode, one timer
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("one episode per target, one timer for every waiter", () => {
+ test("two waiters on the same key share ONE episode and ONE timer", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode(s);
+ const b = joinEpisode(s);
+
+ expect(a.episodeId).toBe(b.episodeId);
+ expect(episodeCount()).toBe(1);
+ expect(describeEpisode(a.episodeId)?.waiters).toBe(2);
+
+ let wokeA = false;
+ let wokeB = false;
+ const pa = a.waitForNextAttempt(never(), 5_000).then(() => {
+ wokeA = true;
+ });
+ await drain();
+ // B joins the round already in flight and INHERITS the running gap rather
+ // than arming a second timer or restarting the first.
+ const pb = b.waitForNextAttempt(never(), 5_000).then(() => {
+ wokeB = true;
+ });
+ await drain();
+
+ expect(clock.pending()).toBe(1);
+ expect(clock.pendingAt()).toEqual([5_000]);
+
+ await clock.advance(5_000);
+ await Promise.all([pa, pb]);
+ expect(wokeA).toBe(true);
+ expect(wokeB).toBe(true);
+ expect(clock.pending()).toBe(0);
+
+ a.leave();
+ b.leave();
+ });
+
+ test("a waiter joining MID-GAP inherits the remaining time, it does not restart it", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode(s);
+ const pa = a.waitForNextAttempt(never(), 60_000);
+ await drain();
+ expect(clock.pendingAt()).toEqual([60_000]);
+
+ await clock.advance(50_000); // t = 50 000, 10 s of the gap left
+ const b = joinEpisode(s);
+ const pb = b.waitForNextAttempt(never(), 60_000);
+ await drain();
+
+ // Still one timer, still due at 60 000 — not at 110 000.
+ expect(clock.pending()).toBe(1);
+ expect(clock.pendingAt()).toEqual([60_000]);
+
+ await clock.advance(10_000);
+ await Promise.all([pa, pb]);
+ expect(clock.now()).toBe(60_000);
+ a.leave();
+ b.leave();
+ });
+
+ test("different targets get different episodes", () => {
+ useFakeClock();
+ const a = joinEpisode(seed({ providerName: "alpha" }));
+ const b = joinEpisode(seed({ providerName: "beta" }));
+ expect(a.episodeId).not.toBe(b.episodeId);
+ expect(episodeCount()).toBe(2);
+ a.leave();
+ b.leave();
+ });
+
+ test("the ladder advances ONCE per round however many waiters attempted it", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode(s);
+ const b = joinEpisode(s);
+ const c = joinEpisode(s);
+
+ // The opening failure: three waiters, one round, index must stay at 0 so
+ // the first gap is the ladder's FIRST rung.
+ a.recordAttemptResult("ConnectionRefused", false);
+ b.recordAttemptResult("ConnectionRefused", false);
+ c.recordAttemptResult("ConnectionRefused", false);
+ expect(a.ladderIndex()).toBe(0);
+ expect(a.attempts()).toBe(3);
+
+ const parked = Promise.all([
+ a.waitForNextAttempt(never(), 5_000),
+ b.waitForNextAttempt(never(), 5_000),
+ c.waitForNextAttempt(never(), 5_000),
+ ]);
+ await drain();
+ await clock.advance(5_000);
+ await parked;
+
+ a.recordAttemptResult("ConnectionRefused", false);
+ b.recordAttemptResult("ConnectionRefused", false);
+ c.recordAttemptResult("ConnectionRefused", false);
+ // One round, one advance — not three.
+ expect(a.ladderIndex()).toBe(1);
+ expect(a.attempts()).toBe(6);
+
+ a.leave();
+ b.leave();
+ c.leave();
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// 2. A waiter's deadline is its own
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("a waiter past its own deadline leaves WITHOUT stranding the others", () => {
+ test("the shared ladder keeps running for the waiter that still has budget", async () => {
+ useFakeClock();
+ const s = seed();
+ const early = joinEpisode({ ...s, deadlineAt: 20_000 });
+ const late = joinEpisode({ ...s, deadlineAt: 270_000 });
+ expect(early.episodeId).toBe(late.episodeId);
+ const id = early.episodeId;
+
+ // Round 1: both park, both wake.
+ const r1 = Promise.all([
+ early.waitForNextAttempt(never(), 5_000),
+ late.waitForNextAttempt(never(), 5_000),
+ ]);
+ await drain();
+ await clock.advance(5_000);
+ await r1;
+ early.recordAttemptResult("ConnectionRefused", false);
+ late.recordAttemptResult("ConnectionRefused", false);
+
+ // `early` now runs out of budget: it declares a handoff and leaves.
+ early.handoff();
+ early.leave();
+
+ // THE EPISODE IS STILL ALIVE and still has the other waiter.
+ const after = describeEpisode(id);
+ expect(after).not.toBeNull();
+ expect(after?.waiters).toBe(1);
+ // `state` describes the retry LOOP, not the departed waiter.
+ expect(after?.state).not.toBe("handoff");
+
+ // And `late` still gets its scheduled attempts.
+ let lateWoke = false;
+ const r2 = late.waitForNextAttempt(never(), 10_000).then((o) => {
+ lateWoke = o.kind === "attempt";
+ });
+ await drain();
+ expect(clock.pending()).toBe(1);
+ await clock.advance(10_000);
+ await r2;
+ expect(lateWoke).toBe(true);
+
+ late.leave();
+ });
+
+ test("the episode enters handoff only when the LAST waiter has left by deadline", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode({ ...s, deadlineAt: 20_000 });
+ const b = joinEpisode({ ...s, deadlineAt: 40_000 });
+ const id = a.episodeId;
+
+ a.handoff();
+ a.leave();
+ expect(describeEpisode(id)?.state).not.toBe("handoff");
+
+ b.handoff();
+ b.leave();
+ expect(describeEpisode(id)?.state).toBe("handoff");
+ // The grace timer is armed AND unref'd — it outlives the request that
+ // created it and must never be the reason the process refuses to exit.
+ expect(clock.pendingAt()).toEqual([EPISODE_GRACE_MS]);
+ expect(clock.unrefed.size).toBe(1);
+ });
+
+ test("a handoff waiter leaving does NOT clear the shared attempt timer", async () => {
+ useFakeClock();
+ const s = seed();
+ const early = joinEpisode({ ...s, deadlineAt: 20_000 });
+ const late = joinEpisode({ ...s, deadlineAt: 270_000 });
+
+ const parked = late.waitForNextAttempt(never(), 30_000);
+ await drain();
+ expect(clock.pending()).toBe(1);
+
+ early.handoff();
+ early.leave();
+ // Still armed. Clearing it here is exactly how the previous revision
+ // stranded the remaining waiters.
+ expect(clock.pending()).toBe(1);
+
+ await clock.advance(30_000);
+ expect((await parked).kind).toBe("attempt");
+ late.leave();
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// 3. The Observer leak
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("no Observer leak: the waiter set empties and listeners detach", () => {
+ test("an aborted wait rejects, empties the set and releases the episode at once", async () => {
+ useFakeClock();
+ const ac = new AbortController();
+ const { signal, counts } = countingSignal(ac);
+ const h = joinEpisode(seed());
+ const id = h.episodeId;
+
+ const parked = h.waitForNextAttempt(signal, 60_000);
+ await drain();
+ expect(describeEpisode(id)?.waiters).toBe(1);
+ expect(counts.added).toBe(1);
+
+ ac.abort(new DOMException("client gone", "AbortError"));
+ await expect(parked).rejects.toThrow();
+
+ // The caller's `finally`.
+ h.leave();
+
+ // The set is empty, the timer is gone, and the episode is RELEASED —
+ // no 120 s grace, because the grace is for a deliberate tier-2 handoff
+ // and not a general retention policy.
+ expect(describeEpisode(id)).toBeNull();
+ expect(episodeCount()).toBe(0);
+ expect(clock.pending()).toBe(0);
+ // The listener came off. Every indirect proxy for this (episode count,
+ // timer count) is also true of an implementation that leaks it.
+ expect(counts.removed).toBeGreaterThanOrEqual(counts.added);
+ });
+
+ test("a NORMALLY woken waiter also detaches its abort listener", async () => {
+ useFakeClock();
+ const ac = new AbortController();
+ const { signal, counts } = countingSignal(ac);
+ const h = joinEpisode(seed());
+
+ const parked = h.waitForNextAttempt(signal, 5_000);
+ await drain();
+ expect(counts.added).toBe(1);
+ await clock.advance(5_000);
+ expect((await parked).kind).toBe("attempt");
+ // Detached on the WAKE path, not only on the abort path — otherwise every
+ // rung of a long ladder adds another listener to the client's signal.
+ expect(counts.removed).toBe(1);
+ h.leave();
+ });
+
+ test("leave() is idempotent — it is called from a `finally` that can run twice", () => {
+ useFakeClock();
+ const h = joinEpisode(seed());
+ const id = h.episodeId;
+ h.leave();
+ h.leave();
+ h.leave();
+ expect(describeEpisode(id)).toBeNull();
+ expect(episodeCount()).toBe(0);
+ });
+
+ test("an already-aborted signal rejects without ever arming a timer", async () => {
+ useFakeClock();
+ const ac = new AbortController();
+ ac.abort(new DOMException("gone", "AbortError"));
+ const h = joinEpisode(seed());
+ await expect(h.waitForNextAttempt(ac.signal, 5_000)).rejects.toThrow();
+ expect(clock.pending()).toBe(0);
+ h.leave();
+ });
+
+ test("closeAllEpisodes wakes every parked waiter rather than stranding it", async () => {
+ useFakeClock();
+ const h = joinEpisode(seed());
+ const parked = h.waitForNextAttempt(never(), 60_000);
+ await drain();
+ closeAllEpisodes("shutdown");
+ // The timer has just been cleared; without the explicit wake this promise
+ // could never settle and the process would hang on shutdown.
+ expect((await parked).kind).toBe("gave_up");
+ expect(episodeCount()).toBe(0);
+ h.leave();
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// 4. Rejoining inside the grace window
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("an episode rejoined within EPISODE_GRACE_MS continues", () => {
+ test("same episode id, ladder position carried forward, clientRetries++", async () => {
+ useFakeClock();
+ const s = seed();
+ const first = joinEpisode(s);
+ const id = first.episodeId;
+
+ // Walk the ladder up three rungs.
+ for (let i = 0; i < 3; i++) {
+ first.recordAttemptResult("ConnectionRefused", false);
+ const p = first.waitForNextAttempt(never(), 5_000);
+ await drain();
+ await clock.advance(5_000);
+ await p;
+ }
+ first.recordAttemptResult("ConnectionRefused", false);
+ const ladderAtHandoff = first.ladderIndex();
+ const attemptsAtHandoff = first.attempts();
+ expect(ladderAtHandoff).toBeGreaterThan(0);
+
+ first.handoff();
+ first.leave();
+ expect(describeEpisode(id)?.state).toBe("handoff");
+
+ // The client comes back inside the grace window.
+ await clock.advance(EPISODE_GRACE_MS - 1_000);
+ const second = joinEpisode(s);
+
+ expect(second.episodeId).toBe(id);
+ // The ladder does NOT reset. The target has been down for the whole of it,
+ // and starting again at 5 s would hammer a dead host harder the longer the
+ // outage lasted.
+ expect(second.ladderIndex()).toBe(ladderAtHandoff);
+ expect(second.attempts()).toBe(attemptsAtHandoff);
+ expect(describeEpisode(id)?.clientRetries).toBe(1);
+ expect(describeEpisode(id)?.maxTier).toBe(2);
+ expect(describeEpisode(id)?.state).toBe("attempting");
+ // The grace timer was cancelled on rejoin, not left to fire later.
+ expect(clock.pending()).toBe(0);
+ second.leave();
+ });
+
+ test("past the grace window the episode is gone and a NEW one is minted", async () => {
+ useFakeClock();
+ const s = seed();
+ const first = joinEpisode(s);
+ const id = first.episodeId;
+ first.recordAttemptResult("ConnectionRefused", false);
+ first.handoff();
+ first.leave();
+ expect(describeEpisode(id)?.state).toBe("handoff");
+
+ await clock.advance(EPISODE_GRACE_MS + 1);
+ expect(describeEpisode(id)).toBeNull();
+
+ const second = joinEpisode(s);
+ expect(second.episodeId).not.toBe(id);
+ expect(second.ladderIndex()).toBe(0);
+ expect(second.attempts()).toBe(0);
+ second.leave();
+ });
+
+ test("EPISODE_GRACE_MS is the measured 120 s, not a guess", () => {
+ // Claude Code's own post-503 backoff was measured to cap at ~38.4 s, so
+ // this is 3.1x the worst observed rejoin gap.
+ expect(EPISODE_GRACE_MS).toBe(120_000);
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// Success, and the seams that are deliberately inert in this phase
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("a successful attempt", () => {
+ test("resets the ladder and wakes the other waiters instead of stranding them", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode(s);
+ const b = joinEpisode(s);
+
+ a.recordAttemptResult("ConnectionRefused", false);
+ const p1 = Promise.all([
+ a.waitForNextAttempt(never(), 5_000),
+ b.waitForNextAttempt(never(), 5_000),
+ ]);
+ await drain();
+ await clock.advance(5_000);
+ await p1;
+ a.recordAttemptResult("ConnectionRefused", false);
+ expect(a.ladderIndex()).toBe(1);
+
+ // b parks again; a succeeds while b is parked.
+ const bWoke: WaitOutcome["kind"][] = [];
+ const pb = b.waitForNextAttempt(never(), 10_000).then((o) => {
+ bWoke.push(o.kind);
+ });
+ await drain();
+ expect(clock.pending()).toBe(1);
+
+ a.recordAttemptResult("ok", true);
+ await drain();
+
+ // The host answered a second ago: it has not earned a sixty-second gap.
+ expect(a.ladderIndex()).toBe(0);
+ // And b was woken rather than left on a timer that has just been cleared.
+ await pb;
+ expect(bWoke).toEqual(["attempt"]);
+ expect(clock.pending()).toBe(0);
+
+ a.leave();
+ b.leave();
+ });
+
+ test("the episode closes `recovered` when the last waiter leaves after a success", () => {
+ useFakeClock();
+ const h = joinEpisode(seed());
+ const id = h.episodeId;
+ h.recordAttemptResult("ConnectionRefused", false);
+ h.recordAttemptResult("ok", true);
+ h.leave();
+ expect(describeEpisode(id)).toBeNull();
+ expect(episodeCount()).toBe(0);
+ });
+});
+
+describe("the wake paths that later phases drive", () => {
+ test("tryNow collapses the wait and advances the round", async () => {
+ useFakeClock();
+ const h = joinEpisode(seed());
+ const parked = h.waitForNextAttempt(never(), 60_000);
+ await drain();
+ expect(clock.pending()).toBe(1);
+
+ tryNow(h.episodeId);
+ expect((await parked).kind).toBe("attempt");
+ // Woken at t=0, not at t=60 000.
+ expect(clock.now()).toBe(0);
+ expect(clock.pending()).toBe(0);
+ h.leave();
+ });
+
+ test("giveUp wakes every waiter with gave_up and latches", async () => {
+ useFakeClock();
+ const s = seed();
+ const a = joinEpisode(s);
+ const b = joinEpisode(s);
+ const pa = a.waitForNextAttempt(never(), 60_000);
+ const pb = b.waitForNextAttempt(never(), 60_000);
+ await drain();
+
+ giveUp(a.episodeId);
+ expect((await pa).kind).toBe("gave_up");
+ expect((await pb).kind).toBe("gave_up");
+ // Latched: a waiter parking after [q] must not be parked again.
+ expect((await a.waitForNextAttempt(never(), 60_000)).kind).toBe("gave_up");
+ a.leave();
+ b.leave();
+ });
+
+ test("uiLeaseValid is inert in this phase and answers false", () => {
+ useFakeClock();
+ const h = joinEpisode(seed());
+ // There is no socket, no pane and no heartbeat yet, so the honest answer is
+ // false and the exhaustion arm keeps answering today's 400. A `true` here
+ // would ship the design's own stated worst case: a retryable status with no
+ // surface on which the reason is legible.
+ expect(uiLeaseValid(h.episodeId)).toBe(false);
+ expect(h.uiLeaseValid()).toBe(false);
+ h.leave();
+ });
+});
diff --git a/packages/cli/src/recovery/settings.test.ts b/packages/cli/src/recovery/settings.test.ts
new file mode 100644
index 00000000..f6ba188e
--- /dev/null
+++ b/packages/cli/src/recovery/settings.test.ts
@@ -0,0 +1,270 @@
+/**
+ * The tier-1 deadline and the two switches.
+ *
+ * THE DEADLINE IS THE SHARPEST REQUIREMENT IN THIS PHASE, and it is a
+ * DERIVATION rather than a constant. Phase 0 measured Claude Code's client
+ * abort against a custom base URL at 359.607 s and then PROVED that the number
+ * is `API_TIMEOUT_MS`'s 360 000 ms default rather than a fixed watchdog, by
+ * setting the variable to 20 000 and observing six aborts at exactly 20.000 s.
+ *
+ * So a user who has set `API_TIMEOUT_MS=60000` has a client that gives up at
+ * 60 s. Hold their request for a deadline derived from the DEFAULT and every
+ * episode ends in `client_gone`: recovery never engages, on that machine,
+ * silently, forever, with nothing printed anywhere. That is the failure this
+ * file exists to make impossible, and it is why the first test asserts the
+ * value MOVES with the environment rather than asserting any single number.
+ */
+
+import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { readRecoveryEnabled, readRecoveryUi } from "../profile-config.js";
+import {
+ CLIENT_CEILING_CLAMP_MS,
+ DEADLINE_MARGIN_MS,
+ DEFAULT_API_TIMEOUT_MS,
+ DEFAULT_TIER1_DEADLINE_MS,
+ TIER1_DEADLINE_FLOOR_MS,
+ logDeadlineIfShortened,
+ resetDeadlineNotice,
+ resetRecoveryFlagOverrides,
+ resolveRecoveryEnabled,
+ resolveRecoveryUi,
+ resolveTier1DeadlineMs,
+ setRecoveryFlagOverrides,
+} from "./settings.js";
+import {
+ capturedRecoveryLines,
+ startLogCapture,
+ stopLogCapture,
+} from "./test-helpers/capture-log.js";
+
+const ENV_KEYS = ["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"] as const;
+const saved: Record<string, string | undefined> = {};
+
+beforeEach(() => {
+ for (const k of ENV_KEYS) {
+ saved[k] = process.env[k];
+ delete process.env[k];
+ }
+ resetRecoveryFlagOverrides();
+ resetDeadlineNotice();
+});
+
+afterEach(() => {
+ for (const k of ENV_KEYS) {
+ if (saved[k] === undefined) delete process.env[k];
+ else process.env[k] = saved[k] as string;
+ }
+ resetRecoveryFlagOverrides();
+ resetDeadlineNotice();
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The deadline
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("resolveTier1DeadlineMs — derived from API_TIMEOUT_MS, never hardcoded", () => {
+ test("the measured default is what the derivation is anchored to", () => {
+ // Recorded so a later edit that 'simplifies' the anchor has to argue with
+ // the measurement rather than with a bare number.
+ expect(DEFAULT_API_TIMEOUT_MS).toBe(360_000);
+ expect(CLIENT_CEILING_CLAMP_MS).toBe(300_000);
+ expect(DEADLINE_MARGIN_MS).toBe(30_000);
+ expect(TIER1_DEADLINE_FLOOR_MS).toBe(15_000);
+ });
+
+ test("unset ⇒ 270 000 ms (min(default,300k) − 30k)", () => {
+ expect(resolveTier1DeadlineMs({} as NodeJS.ProcessEnv)).toBe(270_000);
+ expect(DEFAULT_TIER1_DEADLINE_MS).toBe(270_000);
+ });
+
+ test("API_TIMEOUT_MS=60000 ⇒ 30 000 ms, far BELOW the default hold", () => {
+ // The whole point. A hardcoded 270 s here would outlive this user's client
+ // by 210 s and end every episode in client_gone.
+ const d = resolveTier1DeadlineMs({ API_TIMEOUT_MS: "60000" } as NodeJS.ProcessEnv);
+ expect(d).toBe(30_000);
+ expect(d).toBeLessThan(60_000);
+ expect(d).toBeLessThan(DEFAULT_TIER1_DEADLINE_MS);
+ });
+
+ test("the value MOVES with the environment — it is not a constant", () => {
+ // Mutation-relevant: an implementation that returned a constant passes the
+ // default case and fails here, which is the only case that matters.
+ const seen = new Set(
+ ["45000", "60000", "120000", "240000", "360000"].map((v) =>
+ resolveTier1DeadlineMs({ API_TIMEOUT_MS: v } as NodeJS.ProcessEnv)
+ )
+ );
+ expect(seen.size).toBeGreaterThan(1);
+ expect([...seen].sort((a, b) => a - b)).toEqual([15_000, 30_000, 90_000, 210_000, 270_000]);
+ });
+
+ test("an absurdly small API_TIMEOUT_MS clamps UP to the floor, not to zero or negative", () => {
+ // 20 000 − 30 000 is negative. A raw subtraction would give a deadline in
+ // the past, so shouldSkipTier1's no-budget gate would fire on every single
+ // request and the feature would be off with no trace.
+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "20000" } as NodeJS.ProcessEnv)).toBe(15_000);
+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "1" } as NodeJS.ProcessEnv)).toBe(15_000);
+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "30000" } as NodeJS.ProcessEnv)).toBe(15_000);
+ });
+
+ test("a generous API_TIMEOUT_MS is clamped by the client ceiling", () => {
+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "900000" } as NodeJS.ProcessEnv)).toBe(270_000);
+ });
+
+ test("garbage, zero and negative all fall back to the default ceiling", () => {
+ for (const raw of ["", "abc", "0", "-5000", "NaN"]) {
+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: raw } as NodeJS.ProcessEnv)).toBe(270_000);
+ }
+ });
+
+ test("reads process.env by default", () => {
+ process.env.API_TIMEOUT_MS = "60000";
+ expect(resolveTier1DeadlineMs()).toBe(30_000);
+ });
+});
+
+describe("logDeadlineIfShortened", () => {
+ // Through the REAL logger — see capture-log.ts for why not `mock.module()`.
+ beforeAll(() => startLogCapture());
+ afterAll(() => stopLogCapture());
+
+ test("says nothing at the default — a healthy default must not be noisy", async () => {
+ logDeadlineIfShortened(resolveTier1DeadlineMs({} as NodeJS.ProcessEnv));
+ expect(await capturedRecoveryLines()).toEqual([]);
+ });
+
+ test("fires once when the environment shortens the hold, naming both numbers", async () => {
+ process.env.API_TIMEOUT_MS = "60000";
+ logDeadlineIfShortened(resolveTier1DeadlineMs());
+ logDeadlineIfShortened(resolveTier1DeadlineMs());
+ logDeadlineIfShortened(resolveTier1DeadlineMs());
+ const lines = await capturedRecoveryLines();
+ // Once per process: a shortened budget is not an error and must not be
+ // repeated on every failing request.
+ expect(lines.length).toBe(1);
+ expect(lines[0]).toContain("tier-1 hold shortened to 30s");
+ // Both halves matter: what you got, and what you would have got. A line
+ // carrying only the first is not actionable.
+ expect(lines[0]).toContain("API_TIMEOUT_MS=60000");
+ expect(lines[0]).toContain("default would be 270s");
+ });
+});
+
+// ───────────────────────────────────────────────────────────────────────────
+// The switches
+// ───────────────────────────────────────────────────────────────────────────
+
+describe("resolveRecoveryEnabled / resolveRecoveryUi — flag > env > scoped > true", () => {
+ test("nothing set ⇒ true for both (the default is ON)", () => {
+ expect(resolveRecoveryEnabled()).toBe(true);
+ expect(resolveRecoveryUi()).toBe(true);
+ });
+
+ test("env alone decides when no flag is given", () => {
+ process.env.CLAUDISH_RECOVERY = "0";
+ process.env.CLAUDISH_RECOVERY_UI = "false";
+ expect(resolveRecoveryEnabled()).toBe(false);
+ expect(resolveRecoveryUi()).toBe(false);
+ process.env.CLAUDISH_RECOVERY = "1";
+ process.env.CLAUDISH_RECOVERY_UI = "true";
+ expect(resolveRecoveryEnabled()).toBe(true);
+ expect(resolveRecoveryUi()).toBe(true);
+ });
+
+ test("the flag BEATS the environment, in both directions", () => {
+ process.env.CLAUDISH_RECOVERY = "0";
+ process.env.CLAUDISH_RECOVERY_UI = "0";
+ setRecoveryFlagOverrides({ recovery: true, recoveryUi: true });
+ expect(resolveRecoveryEnabled()).toBe(true);
+ expect(resolveRecoveryUi()).toBe(true);
+
+ process.env.CLAUDISH_RECOVERY = "1";
+ process.env.CLAUDISH_RECOVERY_UI = "1";
+ setRecoveryFlagOverrides({ recovery: false, recoveryUi: false });
+ expect(resolveRecoveryEnabled()).toBe(false);
+ expect(resolveRecoveryUi()).toBe(false);
+ });
+
+ test("the two switches are independent — the UI switch never gates the ladder", () => {
+ // DEC-1: the loopback carve-out is deleted. `recoveryUi` decides one thing
+ // only, and it is not whether a retry happens.
+ setRecoveryFlagOverrides({ recoveryUi: false });
+ expect(resolveRecoveryUi()).toBe(false);
+ expect(resolveRecoveryEnabled()).toBe(true);
+ });
+
+ test("an unparseable env value is NO opinion, not false", () => {
+ // "yes"/"on"/"" must not silently disable recovery: a typo'd off-switch
+ // that half-works is worse than one that does not work at all.
+ for (const raw of ["yes", "on", "", " ", "2", "off"]) {
+ process.env.CLAUDISH_RECOVERY = raw;
+ expect(resolveRecoveryEnabled()).toBe(true);
+ }
+ });
+
+ test("whitespace and case are tolerated on the values that DO parse", () => {
+ process.env.CLAUDISH_RECOVERY = " FALSE ";
+ expect(resolveRecoveryEnabled()).toBe(false);
+ process.env.CLAUDISH_RECOVERY = " True ";
+ expect(resolveRecoveryEnabled()).toBe(true);
+ });
+
+ test("resetRecoveryFlagOverrides clears the flag layer", () => {
+ setRecoveryFlagOverrides({ recovery: false });
+ expect(resolveRecoveryEnabled()).toBe(false);
+ resetRecoveryFlagOverrides();
+ expect(resolveRecoveryEnabled()).toBe(true);
+ });
+});
+
+describe("the scoped layers: project .claudish.json > global config.json", () => {
+ let dir: string;
+ beforeEach(() => {
+ dir = mkdtempSync(join(tmpdir(), "claudish-recovery-scope-"));
+ });
+ afterEach(() => rmSync(dir, { recursive: true, force: true }));
+
+ const paths = () => ({
+ global: () => join(dir, "config.json"),
+ project: () => join(dir, ".claudish.json"),
+ });
+
+ test("project beats global", () => {
+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: true } }));
+ writeFileSync(join(dir, ".claudish.json"), JSON.stringify({ recoveryUi: { enabled: false } }));
+ expect(readRecoveryUi(paths())).toBe(false);
+ });
+
+ test("global answers when the project file says nothing", () => {
+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recovery: { enabled: false } }));
+ expect(readRecoveryEnabled(paths())).toBe(false);
+ });
+
+ test("`{}` is NO OPINION and falls through to the next scope", () => {
+ // The reason both fields are objects rather than bare booleans. A bare
+ // boolean cannot express 'present but undecided', so a project file that
+ // merely mentions the key would override the global one with `false`.
+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: false } }));
+ writeFileSync(join(dir, ".claudish.json"), JSON.stringify({ recoveryUi: {} }));
+ expect(readRecoveryUi(paths())).toBe(false);
+ });
+
+ test("neither scope stating a boolean ⇒ undefined, so the caller's default applies", () => {
+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: "yes" } }));
+ expect(readRecoveryUi(paths())).toBeUndefined();
+ expect(readRecoveryEnabled(paths())).toBeUndefined();
+ });
+
+ test("a garbled file is skipped rather than fatal", () => {
+ writeFileSync(join(dir, ".claudish.json"), "{ not json");
+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recovery: { enabled: false } }));
+ expect(readRecoveryEnabled(paths())).toBe(false);
+ });
+
+ test("a missing file is skipped rather than fatal", () => {
+ expect(readRecoveryUi(paths())).toBeUndefined();
+ });
+});
diff --git a/packages/cli/src/recovery/test-helpers/capture-log.ts b/packages/cli/src/recovery/test-helpers/capture-log.ts
new file mode 100644
index 00000000..db54f50f
--- /dev/null
+++ b/packages/cli/src/recovery/test-helpers/capture-log.ts
@@ -0,0 +1,81 @@
+/**
+ * Read what `logger.log()` actually wrote, through the real logger.
+ *
+ * `log()` is a no-op unless a log path is configured, so a `[Recovery]` line is
+ * invisible to a plain assertion. The alternative — `mock.module("../logger")` —
+ * is forbidden here: Bun's module registry bleeds a module mock into sibling
+ * test FILES, which has broken this suite's e2e files before. So the capture
+ * goes through the logger's own debug-log path instead of replacing it.
+ *
+ * `noLogs: true` is deliberate: it suppresses the ALWAYS-ON log, which lives in
+ * `~/.claudish/logs`. A test must not write to the user's real home. The debug
+ * log lands in `<cwd>/logs`, which is gitignored, and is deleted on stop.
+ *
+ * ─── WHY THERE IS A MARKER, WHICH LOOKS LIKE OVER-ENGINEERING AND IS NOT ───
+ *
+ * `logger.log()` is ASYNCHRONOUSLY BUFFERED. It pushes onto a module-level
+ * `logBuffer` and a 100 ms interval writes that buffer to whatever
+ * `logFilePath` holds **at flush time** — not at push time:
+ *
+ * function flushLogBuffer() {
+ * if (!logFilePath || logBuffer.length === 0) return; // ← early return
+ * appendFile(logFilePath, logBuffer.join(""), …); // KEEPS the buffer
+ * }
+ *
+ * So lines pushed while capture file A was installed, but not yet flushed when
+ * `stopLogCapture()` set `logFilePath` to null, are NOT discarded: they sit in
+ * the buffer and are appended to capture file B the moment the next test
+ * installs one. A whole test file's trailing log lines can therefore surface
+ * inside the NEXT file's capture — which is exactly what happened: the C-17
+ * "exactly one episode id" assertion passed alone and saw two ids when
+ * `composed-handler-recovery.test.ts` ran before it in the same process.
+ *
+ * That is a property of the production logger, not a bug this helper
+ * introduced, and it is not worth changing the logger for. The fix here is to
+ * stamp a unique marker through the SAME buffer at capture start and read only
+ * what follows it: the buffer is FIFO, so anything stale was pushed earlier and
+ * lands ahead of the marker.
+ */
+
+import { randomUUID } from "node:crypto";
+import { existsSync, readFileSync, rmSync } from "node:fs";
+import { getLogFilePath, initLogger, log } from "../../logger.js";
+
+let capturePath: string | null = null;
+let marker = "";
+
+export function startLogCapture(): void {
+ initLogger(true, "debug", true);
+ capturePath = getLogFilePath();
+ marker = `[CaptureMarker] ${randomUUID()}`;
+ // Through `log()`, so it enters the same FIFO buffer the stale lines are in.
+ log(marker);
+}
+
+/** Everything written since `startLogCapture`. Waits out the 100 ms flush. */
+export async function capturedLines(): Promise<string[]> {
+ await new Promise((r) => setTimeout(r, 250));
+ if (!capturePath || !existsSync(capturePath)) return [];
+ const all = readFileSync(capturePath, "utf-8").split("\n").filter(Boolean);
+ const at = all.findLastIndex((l) => l.includes(marker));
+ return at >= 0 ? all.slice(at + 1) : all;
+}
+
+/** Lines carrying `[Recovery]`, with the ISO timestamp prefix left intact. */
+export async function capturedRecoveryLines(): Promise<string[]> {
+ return (await capturedLines()).filter((l) => l.includes("[Recovery]"));
+}
+
+export function stopLogCapture(): void {
+ const path = capturePath;
+ capturePath = null;
+ marker = "";
+ initLogger(false, "info", true);
+ if (path) {
+ try {
+ rmSync(path, { force: true });
+ } catch {
+ // A pending async append can hold it briefly; the directory is gitignored.
+ }
+ }
+}
diff --git a/packages/cli/src/recovery/test-helpers/fake-clock.ts b/packages/cli/src/recovery/test-helpers/fake-clock.ts
new file mode 100644
index 00000000..ab356758
--- /dev/null
+++ b/packages/cli/src/recovery/test-helpers/fake-clock.ts
@@ -0,0 +1,194 @@
+/**
+ * A deterministic `RecoveryClock` for the retry-ladder tests.
+ *
+ * WHY THIS EXISTS RATHER THAN `mock.module()`. The ladder's own gaps are
+ * 5/10/30/60/60 seconds and its deadline is derived from `API_TIMEOUT_MS`
+ * (270 s at the default), so asserting the schedule against the real clock
+ * would cost ~four minutes per assertion and would be flaky at every margin.
+ * The project's standing rule forbids mocking shared infrastructure — Bun's
+ * module registry bleeds a `mock.module()` into sibling test FILES — so
+ * `recovery/clock.ts` exposes an explicit injection seam and this is the thing
+ * injected into it.
+ *
+ * It is not a general fake-timers shim: it implements exactly the four methods
+ * `RecoveryClock` declares, and it drives ONLY the recovery subsystem. Real
+ * `setTimeout` still works underneath, which is what `drain()` uses to let the
+ * awaiting production code make progress between two fired timers.
+ */
+
+import type { RecoveryClock } from "../clock.js";
+
+interface FakeTimer {
+ at: number;
+ fn: () => void;
+}
+
+export class FakeClock implements RecoveryClock {
+ private t: number;
+ private seq = 0;
+ private readonly timers = new Map<number, FakeTimer>();
+ /** Handles this clock was asked to `unref`. The grace timer must be in here. */
+ readonly unrefed = new Set<number>();
+
+ constructor(start = 0) {
+ this.t = start;
+ }
+
+ now(): number {
+ return this.t;
+ }
+
+ setTimeout(fn: () => void, ms: number): unknown {
+ const id = ++this.seq;
+ this.timers.set(id, { at: this.t + Math.max(0, ms), fn });
+ return id;
+ }
+
+ clearTimeout(handle: unknown): void {
+ this.timers.delete(handle as number);
+ }
+
+ unref(handle: unknown): void {
+ this.unrefed.add(handle as number);
+ }
+
+ /** Armed timer count. A leaked timer is a leaked episode, so tests read this. */
+ pending(): number {
+ return this.timers.size;
+ }
+
+ /** The fire times of every armed timer, ascending. */
+ pendingAt(): number[] {
+ return [...this.timers.values()].map((t) => t.at).sort((a, b) => a - b);
+ }
+
+ /**
+ * Move time forward, firing every timer that comes due, in order, and
+ * yielding to the real event loop after each one so the code awaiting it can
+ * run before the next fires.
+ */
+ async advance(ms: number): Promise<void> {
+ const target = this.t + ms;
+ for (;;) {
+ let nextId: number | null = null;
+ let nextAt = Number.POSITIVE_INFINITY;
+ for (const [id, timer] of this.timers) {
+ if (timer.at <= target && timer.at < nextAt) {
+ nextAt = timer.at;
+ nextId = id;
+ }
+ }
+ if (nextId === null) break;
+ this.t = Math.max(this.t, nextAt);
+ const timer = this.timers.get(nextId) as FakeTimer;
+ this.timers.delete(nextId);
+ timer.fn();
+ await drain();
+ }
+ this.t = target;
+ await drain();
+ }
+}
+
+/** Let the real microtask and macrotask queues catch up. */
+export async function drain(ticks = 6): Promise<void> {
+ for (let i = 0; i < ticks; i++) await new Promise((r) => setTimeout(r, 0));
+}
+
+/**
+ * Step `clock` forward until `p` settles, then return its value.
+ *
+ * The ladder decides its own next gap, so a test cannot know in advance how far
+ * to advance; stepping in small increments and re-checking is both simpler and
+ * closer to what the real clock does. `maxMs` is FAKE milliseconds — the test
+ * itself never sleeps for them.
+ */
+export async function advanceUntilSettled<T>(
+ clock: FakeClock,
+ p: Promise<T>,
+ maxMs: number,
+ stepMs = 1_000
+): Promise<T> {
+ let settled = false;
+ let value: T | undefined;
+ let error: unknown;
+ let threw = false;
+ p.then(
+ (v) => {
+ value = v;
+ settled = true;
+ },
+ (e) => {
+ error = e;
+ threw = true;
+ settled = true;
+ }
+ );
+ await drain();
+ let spent = 0;
+ let idle = 0;
+ while (!settled && spent < maxMs) {
+ // Jump straight to the next armed timer when there is one. Stepping in
+ // fixed increments works too but costs a real event-loop drain per step,
+ // which turned a 270-second fake ladder into six real seconds of ticking.
+ const next = clock.pendingAt()[0];
+ if (next === undefined) {
+ // Nothing armed and nothing settled: the production code is between
+ // rungs doing REAL async work — a refused connect resolves in about a
+ // millisecond, which is more than a few microtasks. Give the real event
+ // loop a moment before concluding the fake clock has to move.
+ if (idle < 40) {
+ idle++;
+ await new Promise((r) => setTimeout(r, 1));
+ continue;
+ }
+ idle = 0;
+ await clock.advance(stepMs);
+ spent += stepMs;
+ continue;
+ }
+ idle = 0;
+ const step = Math.max(1, next - clock.now());
+ await clock.advance(step);
+ spent += step;
+ }
+ await drain();
+ if (!settled) throw new Error(`promise did not settle within ${maxMs} fake ms`);
+ if (threw) throw error;
+ return value as T;
+}
+
+/**
+ * A real `AbortSignal` wrapped so a test can COUNT listener registrations.
+ *
+ * The coordinator's named failure mode is the subscribe-without-unsubscribe
+ * leak: a waiter that parks, leaves, and never detaches keeps its closure — and
+ * therefore the whole episode — alive for as long as the client's signal
+ * exists. `added === removed` is the only direct evidence of the `finally`
+ * doing its job; every indirect proxy for it (episode count, timer count) is
+ * also true of an implementation that leaks the listener.
+ */
+export function countingSignal(controller: AbortController): {
+ signal: AbortSignal;
+ counts: { added: number; removed: number };
+} {
+ const counts = { added: 0, removed: 0 };
+ const real = controller.signal;
+ const proxy = {
+ get aborted() {
+ return real.aborted;
+ },
+ get reason() {
+ return real.reason;
+ },
+ addEventListener(type: string, fn: EventListener, opts?: AddEventListenerOptions) {
+ counts.added++;
+ real.addEventListener(type, fn, opts);
+ },
+ removeEventListener(type: string, fn: EventListener) {
+ counts.removed++;
+ real.removeEventListener(type, fn);
+ },
+ };
+ return { signal: proxy as unknown as AbortSignal, counts };
+}