Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
test(recovery): verify the Tier-1 ladder, which shipped with no tests at all
Jack Rudenkoa38f4b7b48af
diff --git a/ai-docs/reports/network-recovery-phase2-verification-20260911.md b/ai-docs/reports/network-recovery-phase2-verification-20260911.mdnew file mode 100644index 00000000..a27dcea7--- /dev/null+++ b/ai-docs/reports/network-recovery-phase2-verification-20260911.md@@ -0,0 +1,147 @@+# Network recovery, design-Phase 2 — verification measurements++> Promoted out of the gitignored session directory deliberately, exactly as+> `network-recovery-phase0-measurements.md` was. CLAUDE.md records three write-ups already lost+> because they lived in `ai-docs/sessions/`, which does not survive a fresh clone or+> `git worktree remove`.+>+> Session: `dev-feature-network-recovery-20260910-110514-0ad0a5a0` · measured 2026-09-11 on macOS+> 25.6.0, bun 1.4.0. The per-file test inventory and the phase narrative stay in that session's+> `implementation-log-phase2b.md`; what is here is the evidence and the findings.++Phase 2 shipped the Tier-1 retry ladder with **no tests for any of its four new modules**. The suite+was green because the pre-existing tests had been opted out of the ladder with+`x-claudish-no-recovery` — not because the ladder had been verified. This is what verifying it+produced.++Suite: **3175 → 3271 pass**, 17 → 18 skip, **2 → 2 fail** (the two pre-existing `displayWidth`+Unicode-oracle cases, red on clean main under bun 1.4.0 while CI pins 1.3.10). +96 tests, zero new+failures. `typecheck` and `lint` clean.++---++## 1. The ladder's real gaps (C-2)++Real clock, real refused loopback socket, driven through `ComposedHandler.handle()` for the full+derived deadline. Deltas computed from the `[Recovery]` log timestamps:++| attempt | fires at | gap | ladder index |+|---|---|---|---|+| 1 | t+0 | — | 0 |+| 2 | t+5.002 s | **5002 ms** | 1 |+| 3 | t+15.004 s | **10002 ms** | 2 |+| 4 | t+45.009 s | **30005 ms** | 3 |+| 5 | t+105.010 s | **60001 ms** | 4 (clamped) |+| 6 | t+165.011 s | **60001 ms** | 5 (clamped) |+| 7 | t+225.012 s | **60001 ms** | 6 (clamped) |+| — | handoff at **t+225.018 s**, deadline 270 s | | |++Total drift over 225 seconds: **18 ms**. The schedule is `5 / 10 / 30 / 60 / 60 / 60`, and the last+array element repeating is what makes the sixth and seventh attempts happen at all — the DEADLINE+ends the ladder, not the array. That inverts `STREAM_RETRY_DELAYS_MS`, where the array running out+*is* the budget.++**`validation-criteria.md`'s C-2 text is stale and the implementation is right.** It says "six+attempts in total, with the handoff at t≈165s", derived for the superseded `TIER1_DEADLINE_MS =+180_000`. ADD-1 replaced that constant with a derivation that yields 270 000 ms at the default, and+270 s fits one more 60 s rung: **seven attempts, handoff at t≈225 s.** The gap sequence the criterion+headlines is unchanged.++---++## 2. F-SLOW fidelity, re-measured (ADD-2 and ADD-3 confirmed)++`192.0.2.1` is TEST-NET-1 (RFC 5737), unrouted **by convention** — a captive portal or an aggressive+corporate resolver can answer it, in which case every F-SLOW assertion passes for the wrong reason.+Checked before use:++| target | elapsed | outcome |+|---|---|---|+| `http://127.0.0.1:1/` (control) | **0.368 ms** | `TypeError` / `ConnectionRefused` |+| `https://192.0.2.1/v1` (F-SLOW) | **75005.18 ms** | `TypeError` / **`ConnectionRefused`** |++Both report the SAME CODE. **Slow and fast connect failures are indistinguishable by code on macOS;+only elapsed time separates them** — ADD-2, reproduced independently. Any logic that branches on the+code to tell "slow unreachable" from "fast refused" is wrong on this platform.++Phase 0 measured the OS give-up at 75.004 s; this machine, a day later, 75.005 s. That stability is+what makes ADD-3 binding: a `PER_ATTEMPT_CONNECT_CAP_MS` of 75 000 races the operating system by+**four milliseconds**, and which one wins is decided per run.++## 3. The per-attempt clamp against that fault++Real clock, real hung connect, the ladder's own clamp:++```+[F-SLOW live] attempt 45.001s (cap 45s, macOS gives up at 75.004s) · total 50.005s · exhausted+```++The attempt ended at **45.001 s** — a 30-second margin under the OS's 75.005 s — and the whole+episode finished at 50.005 s inside a 60 s budget, `exhausted` rather than overrun. The clamp's own+abort re-classifies as a failed attempt of the same kind (a `DOMException` named `TimeoutError`,+whose `code` is the NUMBER 23, which is why the classifier had to match on `name`) rather than+escaping the ladder as an unrelated error.++---++## 4. Mutation proof: 23 mutations, 23 killed, 0 survivors++Each mutation reverts ONE property, runs the tests that claim to pin it, and restores **by file+copy — never git**, because the stash stack and the index are shared with the main checkout and+every sibling worktree.++Killed: the repeating last delay · the `API_TIMEOUT_MS` derivation · the deadline floor · the+shortened-hold notice · one-timer-per-episode · waiter-deadline isolation · the Observer unsubscribe ·+rejoin keeping its ladder position · ladder-advance-per-round · the deadline bounding the attempt+*after* the sleep · the per-attempt clamp · client-disconnect detection · re-classification on every+attempt · signal composition · the shared inbound deadline anchor · the `x-claudish-no-recovery` gate ·+`probe-live` sending that header · the `CLAUDISH_RECOVERY` master switch · the success-only health+latch · the `{ cause }` carrying local connect evidence · the `loadConfig` allowlist entries · `{}`+meaning "no opinion" in the scoped reader · the injectable clock.++---++## 5. Two traps for whoever writes the Phase-3 tests++### 5.1 The debug-log buffer is flushed against `logFilePath` AT FLUSH TIME++`logger.ts`:++```ts+function flushLogBuffer(): void {+ if (!logFilePath || logBuffer.length === 0) return; // early return…+ appendFile(logFilePath, logBuffer.join(""), …); // …KEEPS the buffer+}+```++The early return does not clear `logBuffer`. Lines pushed while log file A was installed, but not yet+flushed when `logFilePath` became null, are appended to log file B the moment one is created — so a+test file's trailing log lines surface inside the NEXT file's capture.++Symptom, and it is the expensive shape: **an assertion that passes alone and fails in the full+suite.** Harmless in production (one log file per process), so the fix belongs in the test helper —+stamp a marker through the same FIFO buffer at capture start and read only what follows it. See+`packages/cli/src/recovery/test-helpers/capture-log.ts`.++### 5.2 Two different `[Recovery]` lines contain the substring `recovered after`++`withConnectionRetry` writes `… recovered after N attempts …`; `closeEpisode` writes+`… closed: recovered after …ms …`. A loose `includes("recovered after")` counts one recovery twice.+Any assertion about recovery counts — and C-17's assertion 3 is exactly one — must use the two+distinct patterns.++---++## 6. One recorded deviation from a criterion's literal text++C-17 assertion 4 requires the `[Recovery]` lines to name `…/api/tags`. They name `…/v1/models`:++```+[Recovery] episode f04bc428-… opened for Ollama at http://127.0.0.1:53801/v1/models (refused/ConnectionRefused)+```++`checkHealth()` probes `/api/tags` first and falls back to `/v1/models`, overwriting `lastProbeUrl`+each time, so what is reported is the LAST probe attempted. That is the more general behaviour — LM+Studio and vLLM have no `/api/tags` at all, so pinning the first would name a URL those providers+never touch — and it satisfies what FR-5 actually requires: the right HOST, and a probe path rather+than the inference path. The test is written to the requirement rather than to the example.diff --git a/packages/cli/src/handlers/composed-handler-recovery.test.ts b/packages/cli/src/handlers/composed-handler-recovery.test.tsnew file mode 100644index 00000000..856cd33e--- /dev/null+++ b/packages/cli/src/handlers/composed-handler-recovery.test.ts@@ -0,0 +1,304 @@+/**+ * The ladder as it is actually reached: through `ComposedHandler.handle()`,+ * over a real socket, against a real refused port.+ *+ * Two validation criteria live here and both are demonstrated rather than+ * asserted in the abstract:+ *+ * C-9 — a client disconnect stops the loop and releases the request within+ * a second. Reachable because `Request.signal` was measured firing in+ * 0.49–0.86 ms in the HELD shape (handler awaiting, no headers sent).+ * C-11 — `--probe` and the config TUI's Test All still fail fast. Without the+ * `x-claudish-no-recovery` guard this feature breaks both tools for+ * exactly the fault they exist to diagnose, and misreports it as+ * `timeout` rather than `network error`.+ *+ * C-11 runs through a REAL `Bun.serve` and the REAL `probeLink`, not a+ * simulation of either: the header has to survive an actual HTTP hop for the+ * criterion to mean anything.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import type { Context } from "hono";+import { probeLink } from "../providers/probe-live.js";+import type { ProviderTransport } from "../providers/transport/types.js";+import { resetRecoveryClock } from "../recovery/clock.js";+import { closeAllEpisodes, episodeCount } from "../recovery/coordinator.js";+import { resetRecoveryFlagOverrides, setRecoveryFlagOverrides } from "../recovery/settings.js";+import {+ capturedRecoveryLines,+ startLogCapture,+ stopLogCapture,+} from "../recovery/test-helpers/capture-log.js";+import { ComposedHandler } from "./composed-handler.js";++/** A port nothing is listening on, taken and released so it is VERIFIED free. */+function freePort(): number {+ const s = Bun.serve({ port: 0, fetch: () => new Response("x") });+ const port = s.port as number;+ s.stop(true);+ return port;+}++let deadPort: number;++beforeEach(() => {+ deadPort = freePort();+ startLogCapture();+});++afterEach(() => {+ stopLogCapture();+ closeAllEpisodes();+ resetRecoveryClock();+ resetRecoveryFlagOverrides();+});++let seq = 0;+function deadTransport(port: number): ProviderTransport {+ return {+ name: `dead-${++seq}`,+ displayName: "Dead Endpoint",+ streamFormat: "openai-sse",+ getEndpoint: () => `http://127.0.0.1:${port}/v1/chat/completions`,+ getHeaders: async () => ({}),+ } as unknown as ProviderTransport;+}++const PAYLOAD = {+ model: "dead-model",+ max_tokens: 16,+ messages: [{ role: "user", content: "hi" }],+};++/** A Hono-shaped Context over a REAL `Request`, so `c.req.raw.signal` is real. */+function contextFor(req: Request): { c: Context; captured: { status?: number; body?: any } } {+ const captured: { status?: number; body?: any } = {};+ const c = {+ req: {+ raw: req,+ header: (name?: string) => (name === undefined ? {} : (req.headers.get(name) ?? undefined)),+ },+ header: () => {},+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),+ json: (body: unknown, status?: number) => {+ captured.body = body;+ captured.status = status;+ return new Response(JSON.stringify(body), {+ status: status ?? 200,+ headers: { "content-type": "application/json" },+ });+ },+ } as unknown as Context;+ return { c, captured };+}++// ───────────────────────────────────────────────────────────────────────────+// C-9+// ───────────────────────────────────────────────────────────────────────────++describe("C-9 — a client disconnect stops the retry loop within a second", () => {+ test("the held request is released, the episode closed, and no attempt follows", async () => {+ const handler = new ComposedHandler(deadTransport(deadPort), "dead-model", "dead-model", 8080, {+ // no x-claudish-no-recovery header: this request DOES enter the ladder+ });+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ const { c } = contextFor(req);++ const started = performance.now();+ const p = handler.handle(c, PAYLOAD);++ // Let the first attempt fail and the ladder park on its 5 s first rung,+ // then go away — the shape of Claude Code being killed mid-outage.+ await new Promise((r) => setTimeout(r, 120));+ expect(episodeCount()).toBe(1);++ const abortedAt = performance.now();+ ac.abort(new DOMException("client gone", "AbortError"));+ const response = await p;+ const releasedAt = performance.now();++ // NFR-2: released promptly, not after the 5 s rung and emphatically not+ // after the 120 s handoff grace, which is for a DELIBERATE tier-2 handoff.+ expect(releasedAt - abortedAt).toBeLessThan(1_000);+ expect(releasedAt - started).toBeLessThan(5_000);+ expect(response.status).toBe(499);++ // The loop exited and the episode released its timer at once.+ expect(episodeCount()).toBe(0);++ const lines = await capturedRecoveryLines();+ expect(lines.some((l) => l.includes("client gone during wait"))).toBe(true);+ expect(lines.some((l) => l.includes("client disconnected after"))).toBe(true);+ // Exactly one attempt was ever made — the one that opened the episode.+ // A second would mean the loop kept running after the client left.+ const attemptLines = lines.filter((l) => /attempt \d+ failed/.test(l));+ expect(attemptLines.length).toBe(1);+ });++ test("a client that is ALREADY gone is answered without entering the ladder at all", async () => {+ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ ac.abort(new DOMException("client gone", "AbortError"));+ const { c } = contextFor(req);++ const started = performance.now();+ const response = await handler.handle(c, PAYLOAD);+ expect(performance.now() - started).toBeLessThan(1_000);+ expect(response.status).toBe(499);+ expect(episodeCount()).toBe(0);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// C-11+// ───────────────────────────────────────────────────────────────────────────++describe("C-11 — probes still fail fast against a dead endpoint", () => {+ test("the header gate answers connection_error 400 in well under a second", async () => {+ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ headers: { "x-claudish-no-recovery": "1" },+ });+ const { c, captured } = contextFor(req);++ const started = performance.now();+ await handler.handle(c, PAYLOAD);+ const elapsed = performance.now() - started;++ expect(elapsed).toBeLessThan(1_000);+ expect(captured.status).toBe(400);+ expect(captured.body?.error?.type).toBe("connection_error");+ // Nothing was held: no episode was ever opened.+ expect(episodeCount()).toBe(0);+ const lines = await capturedRecoveryLines();+ expect(lines.some((l) => l.includes("skipped (probe)"))).toBe(true);+ });++ test("REAL probeLink over a REAL socket reports `network-error`, not `timeout`", async () => {+ // The whole hop: probeLink → HTTP → ComposedHandler → the header gate →+ // an immediate connection_error 400 → probe-live's classifier. A+ // simulation of any one of those links would leave the criterion unproven.+ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const proxy = Bun.serve({+ port: 0,+ async fetch(req) {+ const payload = await req.json();+ const { c } = contextFor(req);+ return await handler.handle(c, payload as Record<string, unknown>);+ },+ });++ try {+ const started = performance.now();+ const result = await probeLink(+ `http://127.0.0.1:${proxy.port}`,+ { provider: "dead", modelSpec: "dead@dead-model", hasCredentials: true },+ // Generously long, so "under a second" is a real discrimination: an+ // unguarded ladder would burn this whole budget and then report the+ // probe's own abort as `timeout`.+ 8_000+ );+ const elapsed = performance.now() - started;++ expect(result.state).toBe("network-error");+ expect(result.state).not.toBe("timeout");+ expect(elapsed).toBeLessThan(1_000);+ expect(result.errorMessage ?? "").toMatch(/reach|connect|network/i);+ expect(episodeCount()).toBe(0);+ } finally {+ proxy.stop(true);+ }+ });+});++// ───────────────────────────────────────────────────────────────────────────+// The other two skip gates, and the absence of a fourth+// ───────────────────────────────────────────────────────────────────────────++describe("shouldSkipTier1 — three gates, and no loopback carve-out", () => {+ test("the master switch restores today's immediate 400 (RISK-7, the CI escape)", async () => {+ setRecoveryFlagOverrides({ recovery: false });+ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const req = new Request("http://127.0.0.1:8080/v1/messages", { method: "POST" });+ const { c, captured } = contextFor(req);++ const started = performance.now();+ await handler.handle(c, PAYLOAD);++ expect(performance.now() - started).toBeLessThan(1_000);+ expect(captured.status).toBe(400);+ expect(captured.body?.error?.type).toBe("connection_error");+ expect(episodeCount()).toBe(0);+ expect(+ (await capturedRecoveryLines()).some((l) => l.includes("skipped (recovery-disabled)"))+ ).toBe(true);+ });++ test("A REFUSED LOOPBACK ADDRESS ENTERS THE LADDER — there is no carve-out", async () => {+ // DEC-1, and this is the assertion that pins it. Two successive designs+ // carved out `refused` + loopback, and each revision of the predicate+ // either fired universally (skipping recovery everywhere) or never. The+ // rule is now general: every classified connection failure enters the+ // ladder regardless of address or UI state.+ setRecoveryFlagOverrides({ recoveryUi: false }); // the old carve-out's key+ const handler = new ComposedHandler(+ deadTransport(deadPort),+ "dead-model",+ "dead-model",+ 8080,+ {}+ );+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ const { c } = contextFor(req);++ const p = handler.handle(c, PAYLOAD);+ await new Promise((r) => setTimeout(r, 120));++ // 127.0.0.1, refused, recovery UI OFF — and still held.+ expect(episodeCount()).toBe(1);+ const lines = await capturedRecoveryLines();+ expect(lines.some((l) => l.includes("episode") && l.includes("opened for"))).toBe(true);+ expect(lines.some((l) => l.includes("skipped ("))).toBe(false);++ ac.abort(new DOMException("done", "AbortError"));+ await p;+ });+});diff --git a/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.tsnew file mode 100644index 00000000..5d29e155--- /dev/null+++ b/packages/cli/src/handlers/shared/transient-retry-slow-net.test.ts@@ -0,0 +1,240 @@+/**+ * F-SLOW — the fault class the rest of the suite structurally cannot exercise.+ *+ * EVERY other fault in this feature's validation set is a loopback refusal, and+ * a refusal comes back in about one millisecond. The originating incident was+ * the opposite: a host that SWALLOWS the connection, where a single connect+ * burns tens of seconds. `192.0.2.1` is TEST-NET-1 (RFC 5737) — reserved for+ * documentation and unrouted — so a TCP connect to it hangs until the OS gives+ * up. Measured on macOS: **75.004 s**, against 0.4 ms for the refused-loopback+ * control.+ *+ * TWO THINGS THAT MATTER AND ARE NOT OBVIOUS:+ *+ * 1. **Slow and fast are indistinguishable BY CODE on macOS.** `192.0.2.1:443`+ * reports `ConnectionRefused`, the same code as an instantly-refused+ * loopback port. Only ELAPSED TIME separates them, so any logic that+ * branches on the code to tell them apart is wrong here.+ * 2. **`PER_ATTEMPT_CONNECT_CAP_MS` must clear the OS's own give-up.** At the+ * design's original 75 000 the cap raced the operating system by four+ * milliseconds, and which one won was decided per run. It is 45 000, which+ * clears it by thirty seconds.+ *+ * WHAT IS IN THE DEFAULT SUITE AND WHAT IS NOT — a deliberate split:+ *+ * - The test below runs by default. It makes a REAL connect to 192.0.2.1 that+ * really hangs, and aborts it through the ladder's own clamp on the injected+ * clock. Real hang, real abort, ~0.1 s of wall clock. What it proves is the+ * mechanism: the clamp reaches a connect that the OS would otherwise hold+ * for 75 s, and the resulting abort is re-classified as a failed attempt+ * rather than escaping as an unrelated error.+ * - The FULL-FIDELITY run — the real 45 s clamp against a real 75 s connect,+ * on the real clock — is gated behind `CLAUDISH_SLOW_NET_E2E=1` and costs+ * ~50 s. Putting it in the default path would add half the suite's total+ * runtime to every run, for one assertion, and the project's own note about+ * credential-gated live tests is the precedent for gating rather than+ * deleting. Its captured output is in the phase log.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";+import { closeAllEpisodes } from "../../recovery/coordinator.js";+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";+import { classifyConnectionError } from "./connection-error.js";+import {+ PER_ATTEMPT_CONNECT_CAP_MS,+ mergeSignalIntoInit,+ withConnectionRetry,+} from "./transient-retry.js";++/** TEST-NET-1. Unrouted by RFC, so the connect hangs rather than being refused. */+const SLOW_URL = "https://192.0.2.1/v1/chat/completions";+/** macOS's own measured give-up against an unrouted address. */+const OS_GIVE_UP_MS = 75_004;++let clock: FakeClock;++beforeEach(() => {+ clock = new FakeClock(0);+ setRecoveryClock(clock);+});++afterEach(() => {+ closeAllEpisodes();+ resetRecoveryClock();+});++const CONNECT_FAILURE_SEED = Object.assign(+ new TypeError("Unable to connect. Is the computer able to access the url?"),+ { code: "ConnectionRefused", path: SLOW_URL }+);++describe("F-SLOW — a host that swallows the connection", () => {+ test("the fault is REAL on this machine: the connect does not settle quickly", async () => {+ // The criterion's own precondition: 192.0.2.1 is unrouted BY CONVENTION,+ // and a captive portal or an aggressive corporate resolver can answer it.+ // If it does, every assertion below would pass for the wrong reason, so the+ // fidelity check runs first and says so out loud.+ const ac = new AbortController();+ let settled = false;+ const p = fetch(SLOW_URL, { signal: ac.signal }).then(+ () => {+ settled = true;+ },+ () => {+ settled = true;+ }+ );+ const t0 = performance.now();+ await new Promise((r) => setTimeout(r, 1_500));+ const stillHanging = !settled;+ ac.abort();+ await p.catch(() => {});++ if (!stillHanging) {+ throw new Error(+ `192.0.2.1 settled in ${(performance.now() - t0).toFixed(0)} ms — something on this ` ++ "network answers TEST-NET-1, so F-SLOW is not reproducible here. Substitute a " ++ "locally blackholed route (which needs privileges) and note the substitution."+ );+ }+ expect(stillHanging).toBe(true);+ }, 10_000);++ test("the per-attempt clamp aborts a REAL hung connect far inside the OS's give-up", async () => {+ let opStarted = 0;+ let opSettledAt: number | null = null;+ let abortSeen = false;+ const realElapsed: number[] = [];++ const op = async (signal: AbortSignal) => {+ opStarted++;+ const t0 = performance.now();+ signal.addEventListener("abort", () => {+ abortSeen = true;+ });+ try {+ // A REAL fetch, to a REAL unrouted address, carrying the ladder's own+ // merged signal exactly as `doFetchWith` threads it into the init. The+ // spread order is the point: a signal written before the transport's+ // own init is silently overwritten by it.+ const init = mergeSignalIntoInit({ method: "POST" }, signal);+ return await fetch(SLOW_URL, init);+ } finally {+ realElapsed.push(performance.now() - t0);+ opSettledAt = clock.now();+ }+ };++ const result = await advanceUntilSettled(+ clock,+ withConnectionRetry(op, CONNECT_FAILURE_SEED, {+ providerName: "slow-net",+ providerDisplayName: "Slow Net",+ resolveEndpoint: () => SLOW_URL,+ // One rung of budget: 5 s gap, then the attempt, then no room for+ // another (5 + 45 + 60 + 3 > 60 000).+ deadlineAt: 60_000,+ signal: new AbortController().signal,+ }),+ 200_000+ );++ expect(opStarted).toBe(1);+ // The clamp fired — the connect did NOT return on its own.+ expect(abortSeen).toBe(true);+ // On the ladder's clock the attempt ended at the cap, not at the OS's+ // 75.004 s and not at the deadline.+ expect((opSettledAt as unknown as number) - 5_000).toBe(PER_ATTEMPT_CONNECT_CAP_MS);+ // And in REAL time it ended immediately, because the abort reached a live+ // socket rather than waiting the operating system out. This is the number+ // that says the clamp is real and not merely scheduled.+ expect(realElapsed[0] as number).toBeLessThan(5_000);+ expect(PER_ATTEMPT_CONNECT_CAP_MS).toBeLessThan(OS_GIVE_UP_MS - 20_000);++ // The clamp's own abort is a FAILED ATTEMPT of the same kind, not an+ // unrelated error that escapes the ladder.+ expect(result.kind).toBe("exhausted");+ if (result.kind === "exhausted") {+ expect(result.conn.kind).toBeDefined();+ }+ // The deadline is honoured from the inbound request start: everything+ // above happened inside the budget.+ expect(clock.now()).toBeLessThanOrEqual(60_000);+ }, 30_000);++ test("a clamp abort re-classifies as a connection error rather than escaping", () => {+ // Why the ladder can treat its own timeout as a retryable network fault:+ // `AbortSignal.timeout` rejects with a DOMException whose `name` is+ // `TimeoutError` and whose `code` is the NUMBER 23, so a classifier keyed+ // only on a string `code` returned null and a hung upstream fell through+ // into a bare 500.+ const clampAbort = new DOMException("attempt cap", "TimeoutError");+ const conn = classifyConnectionError(clampAbort);+ expect(conn).not.toBeNull();+ expect(conn?.kind).toBe("unreachable");+ });++ test("slow and fast are indistinguishable BY CODE — only elapsed time separates them", () => {+ // Recorded as an executable note. macOS reports ECONNREFUSED /+ // ConnectionRefused for an unrouted address, the same code an instantly+ // refused loopback port gives. Branching on the code to tell "slow+ // unreachable" from "fast refused" is therefore wrong on this platform.+ const slow = Object.assign(new TypeError("Unable to connect."), {+ code: "ConnectionRefused",+ });+ const fast = Object.assign(new TypeError("Unable to connect."), {+ code: "ConnectionRefused",+ });+ expect(classifyConnectionError(slow)?.kind).toBe(classifyConnectionError(fast)?.kind);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Full fidelity: the real 45 s clamp on the real clock. ~50 s. Gated.+// ───────────────────────────────────────────────────────────────────────────++const LIVE = process.env.CLAUDISH_SLOW_NET_E2E === "1";++describe.skipIf(!LIVE)("F-SLOW, full fidelity (CLAUDISH_SLOW_NET_E2E=1)", () => {+ test("the real clamp ends the attempt at ~45 s, well before the OS's 75.004 s", async () => {+ resetRecoveryClock(); // REAL clock: this is the whole point of the gate.+ let elapsed = 0;+ const op = async (signal: AbortSignal) => {+ const t0 = performance.now();+ try {+ return await fetch(SLOW_URL, mergeSignalIntoInit({ method: "POST" }, signal));+ } finally {+ elapsed = performance.now() - t0;+ }+ };++ const started = performance.now();+ const result = await withConnectionRetry(op, CONNECT_FAILURE_SEED, {+ providerName: "slow-net-live",+ providerDisplayName: "Slow Net Live",+ resolveEndpoint: () => SLOW_URL,+ deadlineAt: performance.now() + 60_000,+ signal: new AbortController().signal,+ });+ const total = performance.now() - started;++ // The numbers ARE the evidence for this criterion, so print them: a gated+ // test that only asserts leaves nothing behind to paste into a phase log.+ console.log(+ `[F-SLOW live] attempt ${(elapsed / 1000).toFixed(3)}s (cap ${+ PER_ATTEMPT_CONNECT_CAP_MS / 1000+ }s, macOS gives up at ${OS_GIVE_UP_MS / 1000}s) · total ${(total / 1000).toFixed(3)}s · ${result.kind}`+ );++ // The attempt ended at the cap, with a wide margin on both sides: not+ // early (it really did hang) and not at the OS's own give-up.+ expect(elapsed).toBeGreaterThan(PER_ATTEMPT_CONNECT_CAP_MS - 2_000);+ expect(elapsed).toBeLessThan(PER_ATTEMPT_CONNECT_CAP_MS + 2_000);+ expect(elapsed).toBeLessThan(OS_GIVE_UP_MS - 20_000);+ // And the deadline held from the inbound request start.+ expect(total).toBeLessThan(60_000);+ expect(result.kind).toBe("exhausted");+ await drain();+ }, 180_000);+});diff --git a/packages/cli/src/handlers/shared/transient-retry.test.ts b/packages/cli/src/handlers/shared/transient-retry.test.tsnew file mode 100644index 00000000..addcbbcc--- /dev/null+++ b/packages/cli/src/handlers/shared/transient-retry.test.ts@@ -0,0 +1,570 @@+/**+ * The tier-1 retry ladder.+ *+ * This file carries **C-2** (the backoff schedule) and the unit half of **C-9**+ * (client disconnect). Both run on the injected clock, so the 5/10/30/60/60+ * sequence and the 270-second deadline are asserted in milliseconds of real+ * time rather than in minutes.+ *+ * The ladder's single most mis-readable property: **the last delay REPEATS.**+ * `CONNECTION_RETRY_DELAYS_MS` has four elements and the schedule is+ * 5/10/30/60/60/60/… forever — the DEADLINE ends it, not the array. That+ * inverts `STREAM_RETRY_DELAYS_MS`, where the array running out IS the budget,+ * and an implementation that copied the sibling's shape would stop after four+ * gaps with most of the budget unspent.+ *+ * Fixtures are PROVOKED from a real `fetch` refusal, never hand-written: Bun's+ * fetch uses neither Node's errno names nor `.cause`, so a hand-written+ * `{ code: "ECONNREFUSED" }` would test the classifier's table instead of the+ * runtime's behaviour.+ */++import { afterEach, beforeAll, describe, expect, test } from "bun:test";+import type { Context } from "hono";+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";+import { closeAllEpisodes, describeEpisode, episodeCount } from "../../recovery/coordinator.js";+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";+import {+ CONNECTION_RETRY_DELAYS_MS,+ MIN_ATTEMPT_SLOT_MS,+ PER_ATTEMPT_CONNECT_CAP_MS,+ connectionRetryDelayMs,+ inboundStartedAtPerf,+ mergeSignalIntoInit,+ refreshDeadlineAt,+ tier1DeadlineAt,+ withConnectionRetry,+} from "./transient-retry.js";++// ───────────────────────────────────────────────────────────────────────────+// Fixture — provoked, not hand-written+// ───────────────────────────────────────────────────────────────────────────++const realFetch = globalThis.fetch;+let CONNECT_FAILURE: unknown;++beforeAll(async () => {+ try {+ await realFetch("http://127.0.0.1:1/refused-on-purpose");+ throw new Error("127.0.0.1:1 accepted a connection — this fixture needs a closed port");+ } catch (e) {+ CONNECT_FAILURE = e;+ }+ // Guard the guard. If Bun ever stops setting a recognisable code, every+ // assertion below would go green for the wrong reason.+ expect((CONNECT_FAILURE as { code?: string }).code).toBe("ConnectionRefused");+});++let clock: FakeClock;+function useFakeClock(start = 0): FakeClock {+ clock = new FakeClock(start);+ setRecoveryClock(clock);+ return clock;+}++afterEach(() => {+ closeAllEpisodes();+ resetRecoveryClock();+});++let seq = 0;+function ctx(overrides: Record<string, unknown> = {}) {+ return {+ providerName: `ladder-${++seq}`,+ providerDisplayName: "Ladder Probe",+ resolveEndpoint: () => "http://127.0.0.1:1/v1/chat/completions",+ deadlineAt: 270_000,+ signal: new AbortController().signal,+ ...overrides,+ } as Parameters<typeof withConnectionRetry>[2];+}++/** An `op` that always fails the way a dead socket does, recording WHEN. */+function failingOp(at: number[]) {+ return async (_signal: AbortSignal) => {+ at.push(clock.now());+ throw CONNECT_FAILURE;+ };+}++// ───────────────────────────────────────────────────────────────────────────+// The schedule itself+// ───────────────────────────────────────────────────────────────────────────++describe("CONNECTION_RETRY_DELAYS_MS", () => {+ test("is exactly [5000, 10000, 30000, 60000]", () => {+ expect([...CONNECTION_RETRY_DELAYS_MS]).toEqual([5_000, 10_000, 30_000, 60_000]);+ });++ test("is not the stream-retry schedule — the two budgets are unrelated", async () => {+ const { STREAM_RETRY_DELAYS_MS } = (await import("../composed-handler.js")) as unknown as {+ STREAM_RETRY_DELAYS_MS?: readonly number[];+ };+ // Not exported today; the point of the assertion is that if it ever is,+ // the two schedules must still differ. Merging them couples the sniffer's+ // 12 s budget to the connect ladder.+ if (STREAM_RETRY_DELAYS_MS) {+ expect([...STREAM_RETRY_DELAYS_MS]).not.toEqual([...CONNECTION_RETRY_DELAYS_MS]);+ }+ expect(CONNECTION_RETRY_DELAYS_MS.length).toBe(4);+ });+});++describe("connectionRetryDelayMs — the LAST element repeats", () => {+ test("indexes the array for 0..2", () => {+ expect(connectionRetryDelayMs(0)).toBe(5_000);+ expect(connectionRetryDelayMs(1)).toBe(10_000);+ expect(connectionRetryDelayMs(2)).toBe(30_000);+ });++ test("returns 60000 for EVERY n ≥ 3 — the budget ends the ladder, not the array", () => {+ for (const n of [3, 4, 5, 6, 10, 99, 1_000, Number.MAX_SAFE_INTEGER]) {+ expect(connectionRetryDelayMs(n)).toBe(60_000);+ }+ // An implementation that ran off the end would return undefined here, and+ // `setTimeout(fn, undefined)` fires immediately — a tight loop against a+ // dead host rather than a one-minute gap.+ expect(connectionRetryDelayMs(4)).not.toBeUndefined();+ });++ test("a negative index clamps to the first rung rather than to undefined", () => {+ expect(connectionRetryDelayMs(-1)).toBe(5_000);+ expect(connectionRetryDelayMs(-100)).toBe(5_000);+ });+});++describe("the per-attempt cap clears the OS's own give-up", () => {+ test("is 45 s — NOT 75 000, which races macOS's 75.004 s by 4 ms", () => {+ expect(PER_ATTEMPT_CONNECT_CAP_MS).toBe(45_000);+ // Measured: macOS gives up on an unrouted address at 75.004 s. A cap at+ // 75 000 is decided differently on different runs, which makes every+ // assertion keyed on "the attempt ended at the clamp" flaky by+ // construction. The margin below is the whole reason for the number.+ expect(75_004 - PER_ATTEMPT_CONNECT_CAP_MS).toBeGreaterThan(20_000);+ });++ test("refreshDeadlineAt reserves exactly one unclamped attempt", () => {+ expect(refreshDeadlineAt(270_000)).toBe(270_000 - PER_ATTEMPT_CONNECT_CAP_MS);+ expect(refreshDeadlineAt(270_000)).toBe(225_000);+ });++ test("MIN_ATTEMPT_SLOT_MS is 3 s", () => {+ expect(MIN_ATTEMPT_SLOT_MS).toBe(3_000);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// C-2 — the observed gaps+// ───────────────────────────────────────────────────────────────────────────++describe("C-2 — the backoff schedule is the specified one", () => {+ test("gaps are 5/10/30/60/60/60 and the ladder exhausts at the computed attempt", async () => {+ const c = useFakeClock(0);+ const at: number[] = [];+ const deadlineAt = 270_000; // the default, derived from API_TIMEOUT_MS+ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt })),+ 400_000+ );++ // Attempt 1 is the caller's own failure, outside this function, so `at`+ // holds the RE-ISSUES. Their absolute times:+ expect(at).toEqual([5_000, 15_000, 45_000, 105_000, 165_000, 225_000]);++ const gaps = at.map((t, i) => t - (i === 0 ? 0 : (at[i - 1] as number)));+ expect(gaps).toEqual([5_000, 10_000, 30_000, 60_000, 60_000, 60_000]);++ // Seven attempts in the episode: the opening failure plus six re-issues.+ expect(result.kind).toBe("exhausted");+ expect(result.attempts).toBe(7);++ // Exhaustion is a CEILING by construction: the last failure lands before+ // the deadline, and the next rung would not fit.+ const last = at[at.length - 1] as number;+ expect(last).toBeLessThan(deadlineAt);+ expect(last + connectionRetryDelayMs(6) + MIN_ATTEMPT_SLOT_MS).toBeGreaterThan(deadlineAt);+ expect(c.now()).toBeLessThanOrEqual(deadlineAt);+ });++ test("the sequence is deadline-driven, not array-driven: a 30 s budget stops at 2 gaps", async () => {+ // The shape a user with API_TIMEOUT_MS=60000 gets. The gaps are the SAME+ // first rungs; only how many of them fit changes.+ const c = useFakeClock(0);+ const at: number[] = [];+ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 30_000 })),+ 120_000+ );+ expect(at).toEqual([5_000, 15_000]);+ expect(result.kind).toBe("exhausted");+ expect(result.attempts).toBe(3);+ });++ test("a 15 s floor budget still buys one real retry", async () => {+ const c = useFakeClock(0);+ const at: number[] = [];+ await advanceUntilSettled(+ c,+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 15_000 })),+ 60_000+ );+ // 5 + 3 = 8 ≤ 15, so one rung fits; 5 + 10 + 3 = 18 > 15, so a second+ // does not. Anything under 8 s would make the ladder a pure delay in+ // front of the same 400.+ expect(at).toEqual([5_000]);+ });++ test("the deadline check bounds the ATTEMPT THAT FOLLOWS the sleep, not just the sleep", async () => {+ // 165 000 + 60 000 = 225 000 < 240 000, so a check that bounded only the+ // SLEEP would start a seventh attempt with 15 s of budget left. It must+ // not: MIN_ATTEMPT_SLOT_MS is part of the comparison.+ const c = useFakeClock(0);+ const at: number[] = [];+ await advanceUntilSettled(+ c,+ withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ deadlineAt: 226_000 })),+ 400_000+ );+ expect(at).toEqual([5_000, 15_000, 45_000, 105_000, 165_000]);+ expect(at).not.toContain(225_000);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Recovery, and the outcomes that are not recovery+// ───────────────────────────────────────────────────────────────────────────++describe("a re-issue that succeeds returns the value and stops the ladder", () => {+ test("`ok` on the third attempt — no fourth attempt is ever made", async () => {+ const c = useFakeClock(0);+ const at: number[] = [];+ const op = async (_s: AbortSignal) => {+ at.push(c.now());+ if (at.length < 3) throw CONNECT_FAILURE;+ return { body: "a real upstream Response would be here" };+ };++ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(op, CONNECT_FAILURE, ctx()),+ 400_000+ );++ expect(result.kind).toBe("ok");+ if (result.kind !== "ok") throw new Error("unreachable");+ expect(result.value).toEqual({ body: "a real upstream Response would be here" });+ // Two failed re-issues, then the success — and then nothing.+ expect(at).toEqual([5_000, 15_000, 45_000]);+ expect(result.attempts).toBe(4);+ expect(result.recoveryMs).toBe(45_000);+ expect(result.episodeId).not.toBe("");+ // Ladder stopped: no timer left armed, no episode left open.+ expect(c.pending()).toBe(0);+ expect(episodeCount()).toBe(0);+ });++ test("an immediate success on the FIRST re-issue costs exactly one gap", async () => {+ const c = useFakeClock(0);+ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(async () => "ok", CONNECT_FAILURE, ctx()),+ 60_000+ );+ expect(result.kind).toBe("ok");+ expect(c.now()).toBe(5_000);+ expect(result.attempts).toBe(2);+ });+});++describe("outcomes that are not recovery", () => {+ test("an unclassifiable later throw comes back as `threw`, not as the network error", async () => {+ // HIGH-10. A retry that throws an auth failure, a transport bug or a+ // programming error must not be relabelled as the previous connection+ // error — that would quietly widen recovery past the classified set.+ const c = useFakeClock(0);+ const boom = new TypeError("undefined is not a function");+ let n = 0;+ const op = async () => {+ n++;+ throw boom;+ };+ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(op, CONNECT_FAILURE, ctx()),+ 60_000+ );+ expect(result.kind).toBe("threw");+ if (result.kind !== "threw") throw new Error("unreachable");+ expect(result.error).toBe(boom);+ expect(n).toBe(1);+ expect(c.pending()).toBe(0);+ });++ test("an unclassifiable FIRST failure is refused rather than retried", async () => {+ const c = useFakeClock(0);+ let called = false;+ const result = await withConnectionRetry(+ async () => {+ called = true;+ return "never";+ },+ new TypeError("not a network failure"),+ ctx()+ );+ expect(result.kind).toBe("threw");+ expect(called).toBe(false);+ expect(episodeCount()).toBe(0);+ expect(c.pending()).toBe(0);+ });++ test("`gave_up` when the episode is abandoned under a parked waiter", async () => {+ useFakeClock(0);+ const at: number[] = [];+ const p = withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx());+ await drain();+ closeAllEpisodes("gave_up");+ const result = await p;+ expect(result.kind).toBe("gave_up");+ expect(at).toEqual([]);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// C-9 — the client going away+// ───────────────────────────────────────────────────────────────────────────++describe("C-9 — a client disconnect stops the loop and cancels the upstream fetch", () => {+ test("aborting mid-gap resolves `client_gone`, clears the timer and releases the episode", async () => {+ const c = useFakeClock(0);+ const ac = new AbortController();+ const at: number[] = [];+ const p = withConnectionRetry(failingOp(at), CONNECT_FAILURE, ctx({ signal: ac.signal }));+ await drain();+ expect(c.pending()).toBe(1);++ ac.abort(new DOMException("client gone", "AbortError"));+ const result = await p;++ expect(result.kind).toBe("client_gone");+ // No attempt was made after the abort.+ expect(at).toEqual([]);+ // The timer is gone and the episode is released AT ONCE — not after the+ // 120 s handoff grace, which applies only to a deliberate tier-2 handoff.+ expect(c.pending()).toBe(0);+ expect(episodeCount()).toBe(0);+ expect(describeEpisode(result.episodeId)).toBeNull();+ });++ test("the signal handed to `op` aborts with the client — the upstream fetch is cancelled", async () => {+ const c = useFakeClock(0);+ const ac = new AbortController();+ let inFlight: AbortSignal | null = null;+ let abortedDuringOp = false;++ const op = (signal: AbortSignal) =>+ new Promise<string>((_resolve, reject) => {+ inFlight = signal;+ signal.addEventListener("abort", () => {+ abortedDuringOp = true;+ reject(signal.reason);+ });+ });++ const p = withConnectionRetry(op, CONNECT_FAILURE, ctx({ signal: ac.signal }));+ await drain();+ await c.advance(5_000); // first rung fires, op starts and hangs+ expect(inFlight).not.toBeNull();+ expect((inFlight as unknown as AbortSignal).aborted).toBe(false);++ ac.abort(new DOMException("client gone", "AbortError"));+ const result = await p;++ // This is what "the upstream fetch is cancelled" means concretely: the+ // signal `op` was handed — the one merged into the real fetch init — is+ // the one that fired.+ expect(abortedDuringOp).toBe(true);+ expect(result.kind).toBe("client_gone");+ expect(c.pending()).toBe(0);+ expect(episodeCount()).toBe(0);+ });++ test("an already-aborted client is answered without a single attempt", async () => {+ const c = useFakeClock(0);+ const ac = new AbortController();+ ac.abort(new DOMException("gone", "AbortError"));+ const at: number[] = [];+ const result = await withConnectionRetry(+ failingOp(at),+ CONNECT_FAILURE,+ ctx({ signal: ac.signal })+ );+ expect(result.kind).toBe("client_gone");+ expect(at).toEqual([]);+ expect(c.pending()).toBe(0);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// The per-attempt clamp+// ───────────────────────────────────────────────────────────────────────────++describe("the per-attempt clamp bounds the OPERATION, not just the fetch inside it", () => {+ test("a hung attempt is aborted at the cap and counted as a failed attempt", async () => {+ const c = useFakeClock(0);+ const starts: number[] = [];+ const ends: number[] = [];+ const op = (signal: AbortSignal) =>+ new Promise<string>((_res, rej) => {+ starts.push(c.now());+ signal.addEventListener("abort", () => {+ ends.push(c.now());+ rej(signal.reason);+ });+ });++ const result = await advanceUntilSettled(+ c,+ withConnectionRetry(op, CONNECT_FAILURE, ctx({ deadlineAt: 270_000 })),+ 400_000+ );++ expect(starts[0]).toBe(5_000);+ // Aborted at the cap, not at the OS's 75 s and not at the deadline.+ expect((ends[0] as number) - (starts[0] as number)).toBe(PER_ATTEMPT_CONNECT_CAP_MS);+ // The clamp's own abort is re-classified as a failed attempt of the same+ // kind rather than escaping as an unrelated error.+ expect(result.kind).toBe("exhausted");+ // Every attempt costs its gap plus the full cap, so the deadline is still+ // honoured from the inbound request start.+ expect(c.now()).toBeLessThanOrEqual(270_000);+ });++ test("the clamp shrinks to the REMAINING budget when that is smaller than the cap", async () => {+ const c = useFakeClock(0);+ const starts: number[] = [];+ const ends: number[] = [];+ const op = (signal: AbortSignal) =>+ new Promise<string>((_res, rej) => {+ starts.push(c.now());+ signal.addEventListener("abort", () => {+ ends.push(c.now());+ rej(signal.reason);+ });+ });+ // 5 s gap, then only 15 s of budget left — less than the 45 s cap.+ await advanceUntilSettled(+ c,+ withConnectionRetry(op, CONNECT_FAILURE, ctx({ deadlineAt: 20_000 })),+ 120_000+ );+ expect(starts[0]).toBe(5_000);+ expect((ends[0] as number) - (starts[0] as number)).toBe(15_000);+ });++ test("the clamp is disarmed the instant the operation settles", async () => {+ const c = useFakeClock(0);+ await advanceUntilSettled(+ c,+ withConnectionRetry(async () => "ok", CONNECT_FAILURE, ctx()),+ 60_000+ );+ // If the clamp outlived the attempt it would still be armed here, and on+ // the real clock it would abort a signal already merged into a live+ // streaming body.+ expect(c.pending()).toBe(0);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// The deadline's origin+// ───────────────────────────────────────────────────────────────────────────++describe("the deadline is anchored to the INBOUND request, not to the candidate", () => {+ function fakeContext(raw?: Request): Context {+ return { req: { raw, header: () => undefined } } as unknown as Context;+ }++ test("every candidate in one chain inherits the FIRST candidate's start", () => {+ const c = useFakeClock(1_000);+ const raw = new Request("http://127.0.0.1:1/v1/messages", { method: "POST" });+ const first = inboundStartedAtPerf(fakeContext(raw));+ expect(first).toBe(1_000);++ // The fallback handler calls handle() once PER CANDIDATE, and each mints+ // its own `startTime`. Without the shared anchor a three-candidate chain+ // could hold one socket for three full deadlines.+ c.advance(50_000);+ const second = inboundStartedAtPerf(fakeContext(raw));+ expect(second).toBe(1_000);+ });++ test("a different inbound request gets its own anchor", async () => {+ const c = useFakeClock(0);+ const a = new Request("http://127.0.0.1:1/a", { method: "POST" });+ const b = new Request("http://127.0.0.1:1/b", { method: "POST" });+ expect(inboundStartedAtPerf(fakeContext(a))).toBe(0);+ await c.advance(7_000);+ expect(inboundStartedAtPerf(fakeContext(b))).toBe(7_000);+ expect(inboundStartedAtPerf(fakeContext(a))).toBe(0);+ });++ test("a synthesised Context with no raw Request still yields a usable anchor", () => {+ useFakeClock(4_242);+ expect(inboundStartedAtPerf(fakeContext(undefined))).toBe(4_242);+ });++ test("tier1DeadlineAt = anchor + the DERIVED budget, and it moves with API_TIMEOUT_MS", () => {+ useFakeClock(1_000);+ const raw = new Request("http://127.0.0.1:1/deadline", { method: "POST" });+ const saved = process.env.API_TIMEOUT_MS;+ try {+ delete process.env.API_TIMEOUT_MS;+ expect(tier1DeadlineAt(fakeContext(raw))).toBe(1_000 + 270_000);++ const raw2 = new Request("http://127.0.0.1:1/deadline2", { method: "POST" });+ process.env.API_TIMEOUT_MS = "60000";+ // Same anchor arithmetic, different budget — the number is derived, not+ // pinned. A user with a 60 s client must not be handed a 270 s hold.+ expect(tier1DeadlineAt(fakeContext(raw2))).toBe(1_000 + 30_000);+ } finally {+ if (saved === undefined) delete process.env.API_TIMEOUT_MS;+ else process.env.API_TIMEOUT_MS = saved;+ }+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Signal composition+// ───────────────────────────────────────────────────────────────────────────++describe("mergeSignalIntoInit composes rather than replaces", () => {+ test("the transport's own signal survives beside the clamp", async () => {+ const own = new AbortController();+ const ours = new AbortController();+ const init = mergeSignalIntoInit({ method: "POST", signal: own.signal }, ours.signal);+ const merged = init.signal as AbortSignal;+ expect(merged).not.toBe(own.signal);+ expect(merged).not.toBe(ours.signal);++ // Either one aborts the merged signal — which is what makes the local+ // transport's ten-minute ceiling and our 45 s clamp both apply.+ expect(merged.aborted).toBe(false);+ own.abort(new DOMException("transport ceiling", "TimeoutError"));+ await drain(1);+ expect(merged.aborted).toBe(true);++ const init2 = mergeSignalIntoInit({ signal: new AbortController().signal }, ours.signal);+ ours.abort(new DOMException("attempt cap", "TimeoutError"));+ await drain(1);+ expect((init2.signal as AbortSignal).aborted).toBe(true);+ });++ test("no own signal ⇒ ours is used directly; no signal of ours ⇒ the init is untouched", () => {+ const ours = new AbortController().signal;+ expect(mergeSignalIntoInit({ method: "POST" }, ours).signal).toBe(ours);+ const bare = { method: "POST" };+ expect(mergeSignalIntoInit(bare, undefined)).toBe(bare as unknown as RequestInit);+ });+});diff --git a/packages/cli/src/profile-config-recovery.test.ts b/packages/cli/src/profile-config-recovery.test.tsnew file mode 100644index 00000000..2b2bc44d--- /dev/null+++ b/packages/cli/src/profile-config-recovery.test.ts@@ -0,0 +1,111 @@+/**+ * The CLAUDE.md silent-drop invariant, discharged for the two recovery fields.+ *+ * > A new `ClaudishProfileConfig` field MUST be added to `loadConfig`'s+ * > allowlist in `profile-config.ts`; otherwise it survives on disk until the+ * > first global save and is then dropped.+ *+ * `loadConfig` is a HAND-WRITTEN allowlist, not a spread, so a field absent+ * from it round-trips to nothing. The failure is silent in the worst way: the+ * user's setting works until some unrelated code path calls+ * `saveConfig(loadConfig())` — the config TUI saving any other setting will do+ * — and is then erased with no error. For `recovery.enabled: false` that means+ * a deliberate OFF-SWITCH silently turning back ON.+ *+ * The trap has caught this codebase twice before (`onepasswordEnvironments`,+ * then `keychain`), which is why the round-trip is pinned the moment the field+ * is defined rather than the moment it is first consumed.+ *+ * Hermetic via `--config`'s own override seam, which is what `bun run+ * test:safe` relies on to stay off the real machine file.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { setConfigFileOverride } from "./config-override.js";+import { loadConfig, saveConfig } from "./profile-config.js";++let dir: string;+let file: string;++beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "claudish-recovery-roundtrip-"));+ file = join(dir, "config.json");+ setConfigFileOverride(file);+});++afterEach(() => {+ setConfigFileOverride(null);+ rmSync(dir, { recursive: true, force: true });+});++function onDisk(): Record<string, unknown> {+ return JSON.parse(readFileSync(file, "utf-8")) as Record<string, unknown>;+}++describe("saveConfig(loadConfig()) round-trip", () => {+ test("preserves recoveryUi", () => {+ writeFileSync(+ file,+ JSON.stringify({ openRouterApiKey: "sk-test", recoveryUi: { enabled: false } })+ );++ // Exactly what the config TUI does when it saves any unrelated setting.+ saveConfig(loadConfig());++ expect(onDisk().recoveryUi).toEqual({ enabled: false });+ });++ test("preserves recovery (the master switch) — an off-switch must not turn itself on", () => {+ writeFileSync(+ file,+ JSON.stringify({ openRouterApiKey: "sk-test", recovery: { enabled: false } })+ );+ saveConfig(loadConfig());+ expect(onDisk().recovery).toEqual({ enabled: false });+ });++ test("survives repeated save/load cycles, which is how the drop actually shows up", () => {+ writeFileSync(+ file,+ JSON.stringify({ recovery: { enabled: false }, recoveryUi: { enabled: true } })+ );+ for (let i = 0; i < 3; i++) saveConfig(loadConfig());+ expect(onDisk().recovery).toEqual({ enabled: false });+ expect(onDisk().recoveryUi).toEqual({ enabled: true });+ });++ test("an empty object survives too — `{}` is a meaningful value, not an absence", () => {+ // `{}` means "no opinion, fall through to the next scope". Dropping it is+ // a smaller bug than dropping `{enabled:false}` but it is the same bug,+ // and an allowlist entry guarded on truthiness rather than on+ // `!== undefined` would drop exactly this shape.+ writeFileSync(file, JSON.stringify({ recoveryUi: {} }));+ saveConfig(loadConfig());+ expect(onDisk().recoveryUi).toEqual({});+ });++ test("absent stays absent — the allowlist must not invent a default onto disk", () => {+ writeFileSync(file, JSON.stringify({ openRouterApiKey: "sk-test" }));+ saveConfig(loadConfig());+ const disk = onDisk();+ expect("recovery" in disk).toBe(false);+ expect("recoveryUi" in disk).toBe(false);+ });++ test("the sibling fields it sits beside still round-trip, so the allowlist was not disturbed", () => {+ writeFileSync(+ file,+ JSON.stringify({+ openRouterApiKey: "sk-test",+ proOnUltracode: true,+ recoveryUi: { enabled: false },+ })+ );+ saveConfig(loadConfig());+ expect(onDisk().proOnUltracode).toBe(true);+ expect(onDisk().recoveryUi).toEqual({ enabled: false });+ });+});diff --git a/packages/cli/src/providers/transport/local-recovery.test.ts b/packages/cli/src/providers/transport/local-recovery.test.tsnew file mode 100644index 00000000..b10971ff--- /dev/null+++ b/packages/cli/src/providers/transport/local-recovery.test.ts@@ -0,0 +1,321 @@+/**+ * C-17 — a stopped LOCAL provider reaches the ladder at all.+ *+ * WHY THIS FILE EXISTS SEPARATELY FROM EVERY OTHER RECOVERY TEST. The rest of+ * the suite induces its faults with a CUSTOM ENDPOINT, and custom endpoints are+ * built on the OpenAI / Anthropic / LiteLLM transports — none of which has a+ * `refreshAuth`, and all of which now classify a connect failure correctly. So+ * the whole of C-1…C-6 can go green over a completely broken local path: a+ * stopped Ollama fails inside `checkHealth()`, which used to swallow the+ * syscall error, and `refreshAuth()` rethrew a bare `Error` carrying neither a+ * `code` nor a `cause`. `classifyConnectionError` returned `null`, and the+ * single most common local failure never reached recovery at all.+ *+ * Four assertions, and only the first was in the original criterion:+ *+ * 1. the ladder runs at all;+ * 2. exactly ONE episode across the whole recovery;+ * 3. NO `recovered` is reported while the server stays down;+ * 4. the `[Recovery]` lines name the PROBE endpoint, not the inference one.+ *+ * Assertions 2 and 3 exist because assertion 1 passes even when the fix is a+ * no-op. `refreshAuth` opens with `if (this.healthChecked) return;` and+ * `checkHealth()` used to latch that flag ON FAILURE, so a retried+ * `refreshAuth()` issued no probe and did not throw — it returned SUCCESS for a+ * server that was still dead. The episode would close `recovered` for an outage+ * that never ended, and the banner and ladder assertion 1 sees would belong to+ * a SECOND episode opened milliseconds later by the fetch path against the same+ * dead port.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import type { Context } from "hono";+import { ComposedHandler } from "../../handlers/composed-handler.js";+import { classifyConnectionError } from "../../handlers/shared/connection-error.js";+import { withConnectionRetry } from "../../handlers/shared/transient-retry.js";+import { resetRecoveryClock, setRecoveryClock } from "../../recovery/clock.js";+import { closeAllEpisodes, episodeCount } from "../../recovery/coordinator.js";+import {+ capturedRecoveryLines,+ startLogCapture,+ stopLogCapture,+} from "../../recovery/test-helpers/capture-log.js";+import { FakeClock, advanceUntilSettled, drain } from "../../recovery/test-helpers/fake-clock.js";+import type { LocalProvider as LocalProviderConfig } from "../provider-registry.js";+import { LocalTransport } from "./local.js";++/** A port nothing is listening on, taken and released so it is VERIFIED free. */+function freePort(): number {+ const s = Bun.serve({ port: 0, fetch: () => new Response("x") });+ const port = s.port as number;+ s.stop(true);+ return port;+}++const realFetch = globalThis.fetch;+let clock: FakeClock;+let deadPort: number;++beforeEach(() => {+ deadPort = freePort();+ clock = new FakeClock(0);+ setRecoveryClock(clock);+ startLogCapture();+});++afterEach(() => {+ globalThis.fetch = realFetch;+ stopLogCapture();+ closeAllEpisodes();+ resetRecoveryClock();+});++function ollamaConfig(port: number): LocalProviderConfig {+ return {+ name: "ollama",+ baseUrl: `http://127.0.0.1:${port}`,+ apiPath: "/v1/chat/completions",+ envVar: "OLLAMA_API_KEY",+ prefixes: ["ollama"],+ };+}++/** Count the health probes the transport actually issues, per URL. */+function countProbes(): { urls: string[] } {+ const urls: string[] = [];+ const wrapped = (async (input: RequestInfo | URL, init?: RequestInit) => {+ const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;+ if (url.includes("/api/tags") || url.includes("/v1/models") || url.includes("/api/show")) {+ urls.push(url);+ }+ return realFetch(input as RequestInfo, init);+ }) as unknown as typeof fetch;+ globalThis.fetch = wrapped;+ return { urls };+}++const PAYLOAD = {+ model: "llama3.2",+ max_tokens: 16,+ messages: [{ role: "user", content: "hi" }],+};++function contextFor(req: Request): { c: Context; captured: { status?: number; body?: any } } {+ const captured: { status?: number; body?: any } = {};+ const c = {+ req: {+ raw: req,+ header: (name?: string) => (name === undefined ? {} : (req.headers.get(name) ?? undefined)),+ },+ header: () => {},+ body: (body: BodyInit | null, init?: ResponseInit) => new Response(body, init),+ json: (body: unknown, status?: number) => {+ captured.body = body;+ captured.status = status;+ return new Response(JSON.stringify(body), { status: status ?? 200 });+ },+ } as unknown as Context;+ return { c, captured };+}++// ───────────────────────────────────────────────────────────────────────────++describe("C-17 — a stopped local provider enters the ladder through refreshAuth", () => {+ test("a bare `new Error` from refreshAuth would NOT classify — the cause is what carries it", async () => {+ // The control that makes the rest of this file meaningful. Without+ // `{ cause }` the classifier returns null and everything downstream is a+ // no-op for ollama / lmstudio / vllm.+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");+ let thrown: unknown;+ try {+ await transport.refreshAuth();+ } catch (e) {+ thrown = e;+ }+ expect(thrown).toBeDefined();+ expect(classifyConnectionError(thrown)).not.toBeNull();+ expect(classifyConnectionError(thrown)?.kind).toBe("refused");+ // And the endpoint it carries is a PROBE url, not the inference one.+ //+ // DEVIATION FROM THE CRITERION'S LITERAL TEXT, recorded here rather than+ // hidden: C-17 assertion 4 says the lines must name `…/api/tags`. They name+ // `…/v1/models`, because `checkHealth()` tries `/api/tags` FIRST and then+ // falls back to `/v1/models`, overwriting `lastProbeUrl` each time — so the+ // endpoint reported is the LAST probe attempted rather than the first. That+ // is the more general behaviour (LM Studio and vLLM have no `/api/tags` at+ // all, so pinning the first would name a URL those providers never use) and+ // it satisfies what FR-5 actually requires: the right HOST, and a probe+ // path rather than the inference path. The assertion is written to the+ // requirement, not to the example.+ const endpoint = (thrown as { claudishEndpoint?: string }).claudishEndpoint ?? "";+ expect(endpoint).toMatch(/\/(api\/tags|v1\/models)$/);+ expect(endpoint).not.toContain("/v1/chat/completions");+ expect(endpoint).toContain(`127.0.0.1:${deadPort}`);+ // A bare rethrow — what the transport used to do — classifies as null.+ expect(classifyConnectionError(new Error((thrown as Error).message))).toBeNull();+ });++ test("the ladder RE-PROBES on every attempt and never reports a false `recovered`", async () => {+ const { urls } = countProbes();+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");++ let first: unknown;+ try {+ await transport.refreshAuth();+ } catch (e) {+ first = e;+ }+ const probesAfterFirst = urls.length;+ expect(probesAfterFirst).toBeGreaterThan(0);++ const conn = classifyConnectionError(first);+ expect(conn).not.toBeNull();++ const result = await advanceUntilSettled(+ clock,+ withConnectionRetry(() => transport.refreshAuth(), first, {+ providerName: "ollama",+ providerDisplayName: "Ollama",+ resolveEndpoint: (err) =>+ (err as { claudishEndpoint?: string })?.claudishEndpoint ??+ `http://127.0.0.1:${deadPort}`,+ deadlineAt: 30_000, // two rungs is enough to see the latch+ signal: new AbortController().signal,+ }),+ 120_000+ );++ // 1. The ladder ran.+ expect(result.kind).toBe("exhausted");+ expect(result.attempts).toBe(3);++ // 2. Each retried refreshAuth RE-PROBED. With the success-only latch+ // reverted, this number does not move at all past the first call.+ expect(urls.length).toBeGreaterThan(probesAfterFirst);+ expect(urls.filter((u) => u.includes("/api/tags")).length).toBeGreaterThanOrEqual(3);++ // 3. No false recovery: the server never came back, so nothing may say it+ // did — this is the assertion that fails when the latch is reverted.+ const lines = await capturedRecoveryLines();+ expect(lines.some((l) => l.includes("Ollama recovered after"))).toBe(false);+ expect(lines.some((l) => l.includes("closed: recovered"))).toBe(false);++ // 4. Exactly ONE episode across the whole recovery.+ const opened = lines.filter((l) => l.includes("opened for"));+ expect(opened.length).toBe(1);+ const ids = new Set(+ lines.map((l) => /episode ([0-9a-f-]{36})/.exec(l)?.[1]).filter(Boolean) as string[]+ );+ expect(ids.size).toBe(1);++ // 5. The lines name a PROBE endpoint, not the inference endpoint. See the+ // recorded deviation in the first test for why it is `/v1/models`.+ expect(opened[0]).toMatch(/\/(api\/tags|v1\/models) /);+ expect(opened[0]).not.toContain("/v1/chat/completions");+ });++ test("a server that comes back MID-LADDER ends the episode with a real success", async () => {+ // Assertion 5 of C-17. It requires the health probe to actually run again,+ // which the reverted latch makes impossible: attempt 2 would resolve+ // instantly, and `fetchContextWindow()` — guarded by the same flag — would+ // never run at all.+ const { urls } = countProbes();+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");++ let first: unknown;+ try {+ await transport.refreshAuth();+ } catch (e) {+ first = e;+ }++ let server: ReturnType<typeof Bun.serve> | null = null;+ let attempt = 0;+ const op = async () => {+ attempt++;+ // The user starts Ollama between the second and third attempt.+ if (attempt === 3 && !server) {+ server = Bun.serve({+ port: deadPort,+ fetch(req) {+ const url = new URL(req.url);+ if (url.pathname === "/api/tags") return Response.json({ models: [] });+ if (url.pathname === "/api/show")+ return Response.json({ model_info: { "llama.context_length": 131072 } });+ return new Response("not found", { status: 404 });+ },+ });+ }+ return transport.refreshAuth();+ };++ try {+ const result = await advanceUntilSettled(+ clock,+ withConnectionRetry(op, first, {+ providerName: "ollama",+ providerDisplayName: "Ollama",+ resolveEndpoint: (err) =>+ (err as { claudishEndpoint?: string })?.claudishEndpoint ??+ `http://127.0.0.1:${deadPort}`,+ deadlineAt: 270_000,+ signal: new AbortController().signal,+ }),+ 400_000+ );++ expect(result.kind).toBe("ok");+ expect(attempt).toBe(3);+ // The recovered server ran fetchContextWindow — which the latch used to+ // make unreachable forever, leaving the provider on a stale window.+ expect(urls.some((u) => u.includes("/api/show"))).toBe(true);+ expect(transport.getContextWindow()).toBe(131072);++ const lines = await capturedRecoveryLines();+ // ONE episode, and it recovered exactly once. The two patterns are+ // deliberately distinct: `closed: recovered after …` ALSO contains the+ // substring "recovered after", so a loose match counts one recovery twice.+ expect(lines.filter((l) => l.includes("opened for")).length).toBe(1);+ expect(lines.filter((l) => l.includes("Ollama recovered after")).length).toBe(1);+ expect(lines.filter((l) => l.includes("closed: recovered")).length).toBe(1);+ expect(episodeCount()).toBe(0);+ } finally {+ (server as ReturnType<typeof Bun.serve> | null)?.stop(true);+ }+ });++ test("through the REAL ComposedHandler: a stopped local provider is held, not 401'd", async () => {+ const transport = new LocalTransport(ollamaConfig(deadPort), "llama3.2");+ const handler = new ComposedHandler(+ transport as unknown as ConstructorParameters<typeof ComposedHandler>[0],+ "llama3.2",+ "llama3.2",+ 8080,+ {}+ );+ const ac = new AbortController();+ const req = new Request("http://127.0.0.1:8080/v1/messages", {+ method: "POST",+ signal: ac.signal,+ });+ const { c, captured } = contextFor(req);++ const p = handler.handle(c, PAYLOAD);+ await drain(40);++ // It is HELD — an episode is open — rather than answered 401 and walked+ // down the fallback chain onto metered billing.+ expect(episodeCount()).toBe(1);+ const opened = (await capturedRecoveryLines()).filter((l) => l.includes("opened for"));+ expect(opened.length).toBe(1);+ expect(opened[0]).toContain("Ollama");+ expect(opened[0]).toMatch(/\/(api\/tags|v1\/models) /);+ expect(opened[0]).not.toContain("/v1/chat/completions");++ ac.abort(new DOMException("client gone", "AbortError"));+ const response = await p;+ expect(response.status).toBe(499);+ expect(captured.status).not.toBe(401);+ expect(episodeCount()).toBe(0);+ });+});diff --git a/packages/cli/src/recovery/clock.test.ts b/packages/cli/src/recovery/clock.test.tsnew file mode 100644index 00000000..79b3a979--- /dev/null+++ b/packages/cli/src/recovery/clock.test.ts@@ -0,0 +1,102 @@+/**+ * The injectable clock.+ *+ * One property is being pinned, and it is the one every other recovery test+ * depends on: the ladder reads `recoveryClock()` at the moment it schedules,+ * so injecting a clock genuinely CONTROLS the schedule rather than merely+ * shadowing a value nobody consults. If that stops being true, the ladder tests+ * do not fail — they hang for the real 270-second deadline and are killed by+ * the runner, which reads as flakiness rather than as a defect.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { recoveryClock, resetRecoveryClock, setRecoveryClock } from "./clock.js";+import { closeAllEpisodes, joinEpisode } from "./coordinator.js";+import { FakeClock, drain } from "./test-helpers/fake-clock.js";++afterEach(() => {+ closeAllEpisodes();+ resetRecoveryClock();+});++describe("recovery clock", () => {+ test("defaults to the real clock, in PROCESS ms", () => {+ const before = performance.now();+ const t = recoveryClock().now();+ const after = performance.now();+ expect(t).toBeGreaterThanOrEqual(before);+ expect(t).toBeLessThanOrEqual(after);+ // PROCESS ms, not epoch ms. Differencing the two units is the mistake the+ // separation exists to make unrepresentable, and epoch ms is ~1.8e12.+ expect(t).toBeLessThan(1e10);+ });++ test("setRecoveryClock installs, resetRecoveryClock restores", () => {+ const fake = new FakeClock(1234);+ setRecoveryClock(fake);+ expect(recoveryClock()).toBe(fake);+ expect(recoveryClock().now()).toBe(1234);+ resetRecoveryClock();+ expect(recoveryClock()).not.toBe(fake);+ // Back to a live reading rather than the fake's frozen 1234.+ const a = recoveryClock().now();+ expect(a).not.toBe(1234);+ expect(Math.abs(a - performance.now())).toBeLessThan(50);+ });++ test("the injected clock really drives the ladder timer", async () => {+ const fake = new FakeClock(0);+ setRecoveryClock(fake);+ const handle = joinEpisode({+ providerName: "clock-probe",+ providerDisplayName: "Clock Probe",+ endpoint: "http://127.0.0.1:9/v1",+ kind: "refused",+ code: "ConnectionRefused",+ reason: "unreachable",+ deadlineAt: 270_000,+ });++ let woke = false;+ const parked = handle.waitForNextAttempt(new AbortController().signal, 5_000).then(() => {+ woke = true;+ });+ await drain();++ // Armed on the FAKE clock, not the real one.+ expect(fake.pending()).toBe(1);+ expect(fake.pendingAt()).toEqual([5_000]);++ // Real time passing changes nothing: the gap is measured on the injected+ // clock alone.+ await drain(10);+ expect(woke).toBe(false);++ await fake.advance(4_999);+ expect(woke).toBe(false);+ await fake.advance(1);+ await parked;+ expect(woke).toBe(true);+ expect(fake.pending()).toBe(0);++ handle.leave();+ });++ test("unref is offered for the grace timer and is optional on the interface", () => {+ const fake = new FakeClock(0);+ setRecoveryClock(fake);+ const h = fake.setTimeout(() => {}, 10);+ recoveryClock().unref?.(h);+ expect(fake.unrefed.has(h as number)).toBe(true);++ // A clock without `unref` must not crash a caller. The real one has it;+ // the contract says optional, so the call site is `?.`-guarded.+ const minimal = {+ now: () => 0,+ setTimeout: () => 1,+ clearTimeout: () => {},+ };+ setRecoveryClock(minimal);+ expect(() => recoveryClock().unref?.(1)).not.toThrow();+ });+});diff --git a/packages/cli/src/recovery/coordinator.test.ts b/packages/cli/src/recovery/coordinator.test.tsnew file mode 100644index 00000000..3ccd21f7--- /dev/null+++ b/packages/cli/src/recovery/coordinator.test.ts@@ -0,0 +1,535 @@+/**+ * The episode coordinator.+ *+ * Four properties are load-bearing and each one was a review finding before it+ * was a test:+ *+ * 1. N waiters on one target share ONE timer. Without this an outage costs N+ * ladders, N sets of connect attempts against a dead network, and N+ * unsynchronised countdowns.+ * 2. A waiter past its OWN deadline answers alone and LEAVES THE OTHERS+ * RUNNING. The previous revision let one waiter's deadline move the shared+ * episode into `handoff` — a state with no timer — so every other parked+ * waiter silently stopped being retried and answered having attempted+ * nothing. The visible symptom was `waiters: 4` with one of them actually+ * being retried.+ * 3. The waiter set is EMPTY after an aborted wait, and the abort listener is+ * detached. This is the Observer leak the design named as its own most+ * likely failure mode.+ * 4. A request arriving inside `EPISODE_GRACE_MS` REJOINS: same episode id,+ * same ladder position. Restarting the ladder at 5 s would hammer a dead+ * host harder the longer the outage lasted.+ *+ * Every test runs on the injected clock. None of them sleeps.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { resetRecoveryClock, setRecoveryClock } from "./clock.js";+import {+ EPISODE_GRACE_MS,+ closeAllEpisodes,+ describeEpisode,+ episodeCount,+ giveUp,+ joinEpisode,+ tryNow,+ uiLeaseValid,+} from "./coordinator.js";+import type { WaitOutcome } from "./coordinator.js";+import { FakeClock, countingSignal, drain } from "./test-helpers/fake-clock.js";++let clock: FakeClock;++function useFakeClock(start = 0): FakeClock {+ clock = new FakeClock(start);+ setRecoveryClock(clock);+ return clock;+}++afterEach(() => {+ closeAllEpisodes();+ resetRecoveryClock();+});++let seq = 0;+/** A fresh provider name per test, because episodes are keyed by provider|host. */+function seed(overrides: Partial<Parameters<typeof joinEpisode>[0]> = {}) {+ const name = overrides.providerName ?? `coord-${++seq}`;+ return {+ providerName: name,+ providerDisplayName: "Coord Probe",+ endpoint: "http://127.0.0.1:9/v1/chat/completions",+ kind: "refused" as const,+ code: "ConnectionRefused",+ reason: "Cannot reach Coord Probe",+ deadlineAt: 270_000,+ ...overrides,+ };+}++const never = () => new AbortController().signal;++// ───────────────────────────────────────────────────────────────────────────+// 1. One episode, one timer+// ───────────────────────────────────────────────────────────────────────────++describe("one episode per target, one timer for every waiter", () => {+ test("two waiters on the same key share ONE episode and ONE timer", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode(s);+ const b = joinEpisode(s);++ expect(a.episodeId).toBe(b.episodeId);+ expect(episodeCount()).toBe(1);+ expect(describeEpisode(a.episodeId)?.waiters).toBe(2);++ let wokeA = false;+ let wokeB = false;+ const pa = a.waitForNextAttempt(never(), 5_000).then(() => {+ wokeA = true;+ });+ await drain();+ // B joins the round already in flight and INHERITS the running gap rather+ // than arming a second timer or restarting the first.+ const pb = b.waitForNextAttempt(never(), 5_000).then(() => {+ wokeB = true;+ });+ await drain();++ expect(clock.pending()).toBe(1);+ expect(clock.pendingAt()).toEqual([5_000]);++ await clock.advance(5_000);+ await Promise.all([pa, pb]);+ expect(wokeA).toBe(true);+ expect(wokeB).toBe(true);+ expect(clock.pending()).toBe(0);++ a.leave();+ b.leave();+ });++ test("a waiter joining MID-GAP inherits the remaining time, it does not restart it", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode(s);+ const pa = a.waitForNextAttempt(never(), 60_000);+ await drain();+ expect(clock.pendingAt()).toEqual([60_000]);++ await clock.advance(50_000); // t = 50 000, 10 s of the gap left+ const b = joinEpisode(s);+ const pb = b.waitForNextAttempt(never(), 60_000);+ await drain();++ // Still one timer, still due at 60 000 — not at 110 000.+ expect(clock.pending()).toBe(1);+ expect(clock.pendingAt()).toEqual([60_000]);++ await clock.advance(10_000);+ await Promise.all([pa, pb]);+ expect(clock.now()).toBe(60_000);+ a.leave();+ b.leave();+ });++ test("different targets get different episodes", () => {+ useFakeClock();+ const a = joinEpisode(seed({ providerName: "alpha" }));+ const b = joinEpisode(seed({ providerName: "beta" }));+ expect(a.episodeId).not.toBe(b.episodeId);+ expect(episodeCount()).toBe(2);+ a.leave();+ b.leave();+ });++ test("the ladder advances ONCE per round however many waiters attempted it", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode(s);+ const b = joinEpisode(s);+ const c = joinEpisode(s);++ // The opening failure: three waiters, one round, index must stay at 0 so+ // the first gap is the ladder's FIRST rung.+ a.recordAttemptResult("ConnectionRefused", false);+ b.recordAttemptResult("ConnectionRefused", false);+ c.recordAttemptResult("ConnectionRefused", false);+ expect(a.ladderIndex()).toBe(0);+ expect(a.attempts()).toBe(3);++ const parked = Promise.all([+ a.waitForNextAttempt(never(), 5_000),+ b.waitForNextAttempt(never(), 5_000),+ c.waitForNextAttempt(never(), 5_000),+ ]);+ await drain();+ await clock.advance(5_000);+ await parked;++ a.recordAttemptResult("ConnectionRefused", false);+ b.recordAttemptResult("ConnectionRefused", false);+ c.recordAttemptResult("ConnectionRefused", false);+ // One round, one advance — not three.+ expect(a.ladderIndex()).toBe(1);+ expect(a.attempts()).toBe(6);++ a.leave();+ b.leave();+ c.leave();+ });+});++// ───────────────────────────────────────────────────────────────────────────+// 2. A waiter's deadline is its own+// ───────────────────────────────────────────────────────────────────────────++describe("a waiter past its own deadline leaves WITHOUT stranding the others", () => {+ test("the shared ladder keeps running for the waiter that still has budget", async () => {+ useFakeClock();+ const s = seed();+ const early = joinEpisode({ ...s, deadlineAt: 20_000 });+ const late = joinEpisode({ ...s, deadlineAt: 270_000 });+ expect(early.episodeId).toBe(late.episodeId);+ const id = early.episodeId;++ // Round 1: both park, both wake.+ const r1 = Promise.all([+ early.waitForNextAttempt(never(), 5_000),+ late.waitForNextAttempt(never(), 5_000),+ ]);+ await drain();+ await clock.advance(5_000);+ await r1;+ early.recordAttemptResult("ConnectionRefused", false);+ late.recordAttemptResult("ConnectionRefused", false);++ // `early` now runs out of budget: it declares a handoff and leaves.+ early.handoff();+ early.leave();++ // THE EPISODE IS STILL ALIVE and still has the other waiter.+ const after = describeEpisode(id);+ expect(after).not.toBeNull();+ expect(after?.waiters).toBe(1);+ // `state` describes the retry LOOP, not the departed waiter.+ expect(after?.state).not.toBe("handoff");++ // And `late` still gets its scheduled attempts.+ let lateWoke = false;+ const r2 = late.waitForNextAttempt(never(), 10_000).then((o) => {+ lateWoke = o.kind === "attempt";+ });+ await drain();+ expect(clock.pending()).toBe(1);+ await clock.advance(10_000);+ await r2;+ expect(lateWoke).toBe(true);++ late.leave();+ });++ test("the episode enters handoff only when the LAST waiter has left by deadline", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode({ ...s, deadlineAt: 20_000 });+ const b = joinEpisode({ ...s, deadlineAt: 40_000 });+ const id = a.episodeId;++ a.handoff();+ a.leave();+ expect(describeEpisode(id)?.state).not.toBe("handoff");++ b.handoff();+ b.leave();+ expect(describeEpisode(id)?.state).toBe("handoff");+ // The grace timer is armed AND unref'd — it outlives the request that+ // created it and must never be the reason the process refuses to exit.+ expect(clock.pendingAt()).toEqual([EPISODE_GRACE_MS]);+ expect(clock.unrefed.size).toBe(1);+ });++ test("a handoff waiter leaving does NOT clear the shared attempt timer", async () => {+ useFakeClock();+ const s = seed();+ const early = joinEpisode({ ...s, deadlineAt: 20_000 });+ const late = joinEpisode({ ...s, deadlineAt: 270_000 });++ const parked = late.waitForNextAttempt(never(), 30_000);+ await drain();+ expect(clock.pending()).toBe(1);++ early.handoff();+ early.leave();+ // Still armed. Clearing it here is exactly how the previous revision+ // stranded the remaining waiters.+ expect(clock.pending()).toBe(1);++ await clock.advance(30_000);+ expect((await parked).kind).toBe("attempt");+ late.leave();+ });+});++// ───────────────────────────────────────────────────────────────────────────+// 3. The Observer leak+// ───────────────────────────────────────────────────────────────────────────++describe("no Observer leak: the waiter set empties and listeners detach", () => {+ test("an aborted wait rejects, empties the set and releases the episode at once", async () => {+ useFakeClock();+ const ac = new AbortController();+ const { signal, counts } = countingSignal(ac);+ const h = joinEpisode(seed());+ const id = h.episodeId;++ const parked = h.waitForNextAttempt(signal, 60_000);+ await drain();+ expect(describeEpisode(id)?.waiters).toBe(1);+ expect(counts.added).toBe(1);++ ac.abort(new DOMException("client gone", "AbortError"));+ await expect(parked).rejects.toThrow();++ // The caller's `finally`.+ h.leave();++ // The set is empty, the timer is gone, and the episode is RELEASED —+ // no 120 s grace, because the grace is for a deliberate tier-2 handoff+ // and not a general retention policy.+ expect(describeEpisode(id)).toBeNull();+ expect(episodeCount()).toBe(0);+ expect(clock.pending()).toBe(0);+ // The listener came off. Every indirect proxy for this (episode count,+ // timer count) is also true of an implementation that leaks it.+ expect(counts.removed).toBeGreaterThanOrEqual(counts.added);+ });++ test("a NORMALLY woken waiter also detaches its abort listener", async () => {+ useFakeClock();+ const ac = new AbortController();+ const { signal, counts } = countingSignal(ac);+ const h = joinEpisode(seed());++ const parked = h.waitForNextAttempt(signal, 5_000);+ await drain();+ expect(counts.added).toBe(1);+ await clock.advance(5_000);+ expect((await parked).kind).toBe("attempt");+ // Detached on the WAKE path, not only on the abort path — otherwise every+ // rung of a long ladder adds another listener to the client's signal.+ expect(counts.removed).toBe(1);+ h.leave();+ });++ test("leave() is idempotent — it is called from a `finally` that can run twice", () => {+ useFakeClock();+ const h = joinEpisode(seed());+ const id = h.episodeId;+ h.leave();+ h.leave();+ h.leave();+ expect(describeEpisode(id)).toBeNull();+ expect(episodeCount()).toBe(0);+ });++ test("an already-aborted signal rejects without ever arming a timer", async () => {+ useFakeClock();+ const ac = new AbortController();+ ac.abort(new DOMException("gone", "AbortError"));+ const h = joinEpisode(seed());+ await expect(h.waitForNextAttempt(ac.signal, 5_000)).rejects.toThrow();+ expect(clock.pending()).toBe(0);+ h.leave();+ });++ test("closeAllEpisodes wakes every parked waiter rather than stranding it", async () => {+ useFakeClock();+ const h = joinEpisode(seed());+ const parked = h.waitForNextAttempt(never(), 60_000);+ await drain();+ closeAllEpisodes("shutdown");+ // The timer has just been cleared; without the explicit wake this promise+ // could never settle and the process would hang on shutdown.+ expect((await parked).kind).toBe("gave_up");+ expect(episodeCount()).toBe(0);+ h.leave();+ });+});++// ───────────────────────────────────────────────────────────────────────────+// 4. Rejoining inside the grace window+// ───────────────────────────────────────────────────────────────────────────++describe("an episode rejoined within EPISODE_GRACE_MS continues", () => {+ test("same episode id, ladder position carried forward, clientRetries++", async () => {+ useFakeClock();+ const s = seed();+ const first = joinEpisode(s);+ const id = first.episodeId;++ // Walk the ladder up three rungs.+ for (let i = 0; i < 3; i++) {+ first.recordAttemptResult("ConnectionRefused", false);+ const p = first.waitForNextAttempt(never(), 5_000);+ await drain();+ await clock.advance(5_000);+ await p;+ }+ first.recordAttemptResult("ConnectionRefused", false);+ const ladderAtHandoff = first.ladderIndex();+ const attemptsAtHandoff = first.attempts();+ expect(ladderAtHandoff).toBeGreaterThan(0);++ first.handoff();+ first.leave();+ expect(describeEpisode(id)?.state).toBe("handoff");++ // The client comes back inside the grace window.+ await clock.advance(EPISODE_GRACE_MS - 1_000);+ const second = joinEpisode(s);++ expect(second.episodeId).toBe(id);+ // The ladder does NOT reset. The target has been down for the whole of it,+ // and starting again at 5 s would hammer a dead host harder the longer the+ // outage lasted.+ expect(second.ladderIndex()).toBe(ladderAtHandoff);+ expect(second.attempts()).toBe(attemptsAtHandoff);+ expect(describeEpisode(id)?.clientRetries).toBe(1);+ expect(describeEpisode(id)?.maxTier).toBe(2);+ expect(describeEpisode(id)?.state).toBe("attempting");+ // The grace timer was cancelled on rejoin, not left to fire later.+ expect(clock.pending()).toBe(0);+ second.leave();+ });++ test("past the grace window the episode is gone and a NEW one is minted", async () => {+ useFakeClock();+ const s = seed();+ const first = joinEpisode(s);+ const id = first.episodeId;+ first.recordAttemptResult("ConnectionRefused", false);+ first.handoff();+ first.leave();+ expect(describeEpisode(id)?.state).toBe("handoff");++ await clock.advance(EPISODE_GRACE_MS + 1);+ expect(describeEpisode(id)).toBeNull();++ const second = joinEpisode(s);+ expect(second.episodeId).not.toBe(id);+ expect(second.ladderIndex()).toBe(0);+ expect(second.attempts()).toBe(0);+ second.leave();+ });++ test("EPISODE_GRACE_MS is the measured 120 s, not a guess", () => {+ // Claude Code's own post-503 backoff was measured to cap at ~38.4 s, so+ // this is 3.1x the worst observed rejoin gap.+ expect(EPISODE_GRACE_MS).toBe(120_000);+ });+});++// ───────────────────────────────────────────────────────────────────────────+// Success, and the seams that are deliberately inert in this phase+// ───────────────────────────────────────────────────────────────────────────++describe("a successful attempt", () => {+ test("resets the ladder and wakes the other waiters instead of stranding them", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode(s);+ const b = joinEpisode(s);++ a.recordAttemptResult("ConnectionRefused", false);+ const p1 = Promise.all([+ a.waitForNextAttempt(never(), 5_000),+ b.waitForNextAttempt(never(), 5_000),+ ]);+ await drain();+ await clock.advance(5_000);+ await p1;+ a.recordAttemptResult("ConnectionRefused", false);+ expect(a.ladderIndex()).toBe(1);++ // b parks again; a succeeds while b is parked.+ const bWoke: WaitOutcome["kind"][] = [];+ const pb = b.waitForNextAttempt(never(), 10_000).then((o) => {+ bWoke.push(o.kind);+ });+ await drain();+ expect(clock.pending()).toBe(1);++ a.recordAttemptResult("ok", true);+ await drain();++ // The host answered a second ago: it has not earned a sixty-second gap.+ expect(a.ladderIndex()).toBe(0);+ // And b was woken rather than left on a timer that has just been cleared.+ await pb;+ expect(bWoke).toEqual(["attempt"]);+ expect(clock.pending()).toBe(0);++ a.leave();+ b.leave();+ });++ test("the episode closes `recovered` when the last waiter leaves after a success", () => {+ useFakeClock();+ const h = joinEpisode(seed());+ const id = h.episodeId;+ h.recordAttemptResult("ConnectionRefused", false);+ h.recordAttemptResult("ok", true);+ h.leave();+ expect(describeEpisode(id)).toBeNull();+ expect(episodeCount()).toBe(0);+ });+});++describe("the wake paths that later phases drive", () => {+ test("tryNow collapses the wait and advances the round", async () => {+ useFakeClock();+ const h = joinEpisode(seed());+ const parked = h.waitForNextAttempt(never(), 60_000);+ await drain();+ expect(clock.pending()).toBe(1);++ tryNow(h.episodeId);+ expect((await parked).kind).toBe("attempt");+ // Woken at t=0, not at t=60 000.+ expect(clock.now()).toBe(0);+ expect(clock.pending()).toBe(0);+ h.leave();+ });++ test("giveUp wakes every waiter with gave_up and latches", async () => {+ useFakeClock();+ const s = seed();+ const a = joinEpisode(s);+ const b = joinEpisode(s);+ const pa = a.waitForNextAttempt(never(), 60_000);+ const pb = b.waitForNextAttempt(never(), 60_000);+ await drain();++ giveUp(a.episodeId);+ expect((await pa).kind).toBe("gave_up");+ expect((await pb).kind).toBe("gave_up");+ // Latched: a waiter parking after [q] must not be parked again.+ expect((await a.waitForNextAttempt(never(), 60_000)).kind).toBe("gave_up");+ a.leave();+ b.leave();+ });++ test("uiLeaseValid is inert in this phase and answers false", () => {+ useFakeClock();+ const h = joinEpisode(seed());+ // There is no socket, no pane and no heartbeat yet, so the honest answer is+ // false and the exhaustion arm keeps answering today's 400. A `true` here+ // would ship the design's own stated worst case: a retryable status with no+ // surface on which the reason is legible.+ expect(uiLeaseValid(h.episodeId)).toBe(false);+ expect(h.uiLeaseValid()).toBe(false);+ h.leave();+ });+});diff --git a/packages/cli/src/recovery/settings.test.ts b/packages/cli/src/recovery/settings.test.tsnew file mode 100644index 00000000..f6ba188e--- /dev/null+++ b/packages/cli/src/recovery/settings.test.ts@@ -0,0 +1,270 @@+/**+ * The tier-1 deadline and the two switches.+ *+ * THE DEADLINE IS THE SHARPEST REQUIREMENT IN THIS PHASE, and it is a+ * DERIVATION rather than a constant. Phase 0 measured Claude Code's client+ * abort against a custom base URL at 359.607 s and then PROVED that the number+ * is `API_TIMEOUT_MS`'s 360 000 ms default rather than a fixed watchdog, by+ * setting the variable to 20 000 and observing six aborts at exactly 20.000 s.+ *+ * So a user who has set `API_TIMEOUT_MS=60000` has a client that gives up at+ * 60 s. Hold their request for a deadline derived from the DEFAULT and every+ * episode ends in `client_gone`: recovery never engages, on that machine,+ * silently, forever, with nothing printed anywhere. That is the failure this+ * file exists to make impossible, and it is why the first test asserts the+ * value MOVES with the environment rather than asserting any single number.+ */++import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { readRecoveryEnabled, readRecoveryUi } from "../profile-config.js";+import {+ CLIENT_CEILING_CLAMP_MS,+ DEADLINE_MARGIN_MS,+ DEFAULT_API_TIMEOUT_MS,+ DEFAULT_TIER1_DEADLINE_MS,+ TIER1_DEADLINE_FLOOR_MS,+ logDeadlineIfShortened,+ resetDeadlineNotice,+ resetRecoveryFlagOverrides,+ resolveRecoveryEnabled,+ resolveRecoveryUi,+ resolveTier1DeadlineMs,+ setRecoveryFlagOverrides,+} from "./settings.js";+import {+ capturedRecoveryLines,+ startLogCapture,+ stopLogCapture,+} from "./test-helpers/capture-log.js";++const ENV_KEYS = ["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"] as const;+const saved: Record<string, string | undefined> = {};++beforeEach(() => {+ for (const k of ENV_KEYS) {+ saved[k] = process.env[k];+ delete process.env[k];+ }+ resetRecoveryFlagOverrides();+ resetDeadlineNotice();+});++afterEach(() => {+ for (const k of ENV_KEYS) {+ if (saved[k] === undefined) delete process.env[k];+ else process.env[k] = saved[k] as string;+ }+ resetRecoveryFlagOverrides();+ resetDeadlineNotice();+});++// ───────────────────────────────────────────────────────────────────────────+// The deadline+// ───────────────────────────────────────────────────────────────────────────++describe("resolveTier1DeadlineMs — derived from API_TIMEOUT_MS, never hardcoded", () => {+ test("the measured default is what the derivation is anchored to", () => {+ // Recorded so a later edit that 'simplifies' the anchor has to argue with+ // the measurement rather than with a bare number.+ expect(DEFAULT_API_TIMEOUT_MS).toBe(360_000);+ expect(CLIENT_CEILING_CLAMP_MS).toBe(300_000);+ expect(DEADLINE_MARGIN_MS).toBe(30_000);+ expect(TIER1_DEADLINE_FLOOR_MS).toBe(15_000);+ });++ test("unset ⇒ 270 000 ms (min(default,300k) − 30k)", () => {+ expect(resolveTier1DeadlineMs({} as NodeJS.ProcessEnv)).toBe(270_000);+ expect(DEFAULT_TIER1_DEADLINE_MS).toBe(270_000);+ });++ test("API_TIMEOUT_MS=60000 ⇒ 30 000 ms, far BELOW the default hold", () => {+ // The whole point. A hardcoded 270 s here would outlive this user's client+ // by 210 s and end every episode in client_gone.+ const d = resolveTier1DeadlineMs({ API_TIMEOUT_MS: "60000" } as NodeJS.ProcessEnv);+ expect(d).toBe(30_000);+ expect(d).toBeLessThan(60_000);+ expect(d).toBeLessThan(DEFAULT_TIER1_DEADLINE_MS);+ });++ test("the value MOVES with the environment — it is not a constant", () => {+ // Mutation-relevant: an implementation that returned a constant passes the+ // default case and fails here, which is the only case that matters.+ const seen = new Set(+ ["45000", "60000", "120000", "240000", "360000"].map((v) =>+ resolveTier1DeadlineMs({ API_TIMEOUT_MS: v } as NodeJS.ProcessEnv)+ )+ );+ expect(seen.size).toBeGreaterThan(1);+ expect([...seen].sort((a, b) => a - b)).toEqual([15_000, 30_000, 90_000, 210_000, 270_000]);+ });++ test("an absurdly small API_TIMEOUT_MS clamps UP to the floor, not to zero or negative", () => {+ // 20 000 − 30 000 is negative. A raw subtraction would give a deadline in+ // the past, so shouldSkipTier1's no-budget gate would fire on every single+ // request and the feature would be off with no trace.+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "20000" } as NodeJS.ProcessEnv)).toBe(15_000);+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "1" } as NodeJS.ProcessEnv)).toBe(15_000);+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "30000" } as NodeJS.ProcessEnv)).toBe(15_000);+ });++ test("a generous API_TIMEOUT_MS is clamped by the client ceiling", () => {+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: "900000" } as NodeJS.ProcessEnv)).toBe(270_000);+ });++ test("garbage, zero and negative all fall back to the default ceiling", () => {+ for (const raw of ["", "abc", "0", "-5000", "NaN"]) {+ expect(resolveTier1DeadlineMs({ API_TIMEOUT_MS: raw } as NodeJS.ProcessEnv)).toBe(270_000);+ }+ });++ test("reads process.env by default", () => {+ process.env.API_TIMEOUT_MS = "60000";+ expect(resolveTier1DeadlineMs()).toBe(30_000);+ });+});++describe("logDeadlineIfShortened", () => {+ // Through the REAL logger — see capture-log.ts for why not `mock.module()`.+ beforeAll(() => startLogCapture());+ afterAll(() => stopLogCapture());++ test("says nothing at the default — a healthy default must not be noisy", async () => {+ logDeadlineIfShortened(resolveTier1DeadlineMs({} as NodeJS.ProcessEnv));+ expect(await capturedRecoveryLines()).toEqual([]);+ });++ test("fires once when the environment shortens the hold, naming both numbers", async () => {+ process.env.API_TIMEOUT_MS = "60000";+ logDeadlineIfShortened(resolveTier1DeadlineMs());+ logDeadlineIfShortened(resolveTier1DeadlineMs());+ logDeadlineIfShortened(resolveTier1DeadlineMs());+ const lines = await capturedRecoveryLines();+ // Once per process: a shortened budget is not an error and must not be+ // repeated on every failing request.+ expect(lines.length).toBe(1);+ expect(lines[0]).toContain("tier-1 hold shortened to 30s");+ // Both halves matter: what you got, and what you would have got. A line+ // carrying only the first is not actionable.+ expect(lines[0]).toContain("API_TIMEOUT_MS=60000");+ expect(lines[0]).toContain("default would be 270s");+ });+});++// ───────────────────────────────────────────────────────────────────────────+// The switches+// ───────────────────────────────────────────────────────────────────────────++describe("resolveRecoveryEnabled / resolveRecoveryUi — flag > env > scoped > true", () => {+ test("nothing set ⇒ true for both (the default is ON)", () => {+ expect(resolveRecoveryEnabled()).toBe(true);+ expect(resolveRecoveryUi()).toBe(true);+ });++ test("env alone decides when no flag is given", () => {+ process.env.CLAUDISH_RECOVERY = "0";+ process.env.CLAUDISH_RECOVERY_UI = "false";+ expect(resolveRecoveryEnabled()).toBe(false);+ expect(resolveRecoveryUi()).toBe(false);+ process.env.CLAUDISH_RECOVERY = "1";+ process.env.CLAUDISH_RECOVERY_UI = "true";+ expect(resolveRecoveryEnabled()).toBe(true);+ expect(resolveRecoveryUi()).toBe(true);+ });++ test("the flag BEATS the environment, in both directions", () => {+ process.env.CLAUDISH_RECOVERY = "0";+ process.env.CLAUDISH_RECOVERY_UI = "0";+ setRecoveryFlagOverrides({ recovery: true, recoveryUi: true });+ expect(resolveRecoveryEnabled()).toBe(true);+ expect(resolveRecoveryUi()).toBe(true);++ process.env.CLAUDISH_RECOVERY = "1";+ process.env.CLAUDISH_RECOVERY_UI = "1";+ setRecoveryFlagOverrides({ recovery: false, recoveryUi: false });+ expect(resolveRecoveryEnabled()).toBe(false);+ expect(resolveRecoveryUi()).toBe(false);+ });++ test("the two switches are independent — the UI switch never gates the ladder", () => {+ // DEC-1: the loopback carve-out is deleted. `recoveryUi` decides one thing+ // only, and it is not whether a retry happens.+ setRecoveryFlagOverrides({ recoveryUi: false });+ expect(resolveRecoveryUi()).toBe(false);+ expect(resolveRecoveryEnabled()).toBe(true);+ });++ test("an unparseable env value is NO opinion, not false", () => {+ // "yes"/"on"/"" must not silently disable recovery: a typo'd off-switch+ // that half-works is worse than one that does not work at all.+ for (const raw of ["yes", "on", "", " ", "2", "off"]) {+ process.env.CLAUDISH_RECOVERY = raw;+ expect(resolveRecoveryEnabled()).toBe(true);+ }+ });++ test("whitespace and case are tolerated on the values that DO parse", () => {+ process.env.CLAUDISH_RECOVERY = " FALSE ";+ expect(resolveRecoveryEnabled()).toBe(false);+ process.env.CLAUDISH_RECOVERY = " True ";+ expect(resolveRecoveryEnabled()).toBe(true);+ });++ test("resetRecoveryFlagOverrides clears the flag layer", () => {+ setRecoveryFlagOverrides({ recovery: false });+ expect(resolveRecoveryEnabled()).toBe(false);+ resetRecoveryFlagOverrides();+ expect(resolveRecoveryEnabled()).toBe(true);+ });+});++describe("the scoped layers: project .claudish.json > global config.json", () => {+ let dir: string;+ beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "claudish-recovery-scope-"));+ });+ afterEach(() => rmSync(dir, { recursive: true, force: true }));++ const paths = () => ({+ global: () => join(dir, "config.json"),+ project: () => join(dir, ".claudish.json"),+ });++ test("project beats global", () => {+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: true } }));+ writeFileSync(join(dir, ".claudish.json"), JSON.stringify({ recoveryUi: { enabled: false } }));+ expect(readRecoveryUi(paths())).toBe(false);+ });++ test("global answers when the project file says nothing", () => {+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recovery: { enabled: false } }));+ expect(readRecoveryEnabled(paths())).toBe(false);+ });++ test("`{}` is NO OPINION and falls through to the next scope", () => {+ // The reason both fields are objects rather than bare booleans. A bare+ // boolean cannot express 'present but undecided', so a project file that+ // merely mentions the key would override the global one with `false`.+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: false } }));+ writeFileSync(join(dir, ".claudish.json"), JSON.stringify({ recoveryUi: {} }));+ expect(readRecoveryUi(paths())).toBe(false);+ });++ test("neither scope stating a boolean ⇒ undefined, so the caller's default applies", () => {+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recoveryUi: { enabled: "yes" } }));+ expect(readRecoveryUi(paths())).toBeUndefined();+ expect(readRecoveryEnabled(paths())).toBeUndefined();+ });++ test("a garbled file is skipped rather than fatal", () => {+ writeFileSync(join(dir, ".claudish.json"), "{ not json");+ writeFileSync(join(dir, "config.json"), JSON.stringify({ recovery: { enabled: false } }));+ expect(readRecoveryEnabled(paths())).toBe(false);+ });++ test("a missing file is skipped rather than fatal", () => {+ expect(readRecoveryUi(paths())).toBeUndefined();+ });+});diff --git a/packages/cli/src/recovery/test-helpers/capture-log.ts b/packages/cli/src/recovery/test-helpers/capture-log.tsnew file mode 100644index 00000000..db54f50f--- /dev/null+++ b/packages/cli/src/recovery/test-helpers/capture-log.ts@@ -0,0 +1,81 @@+/**+ * Read what `logger.log()` actually wrote, through the real logger.+ *+ * `log()` is a no-op unless a log path is configured, so a `[Recovery]` line is+ * invisible to a plain assertion. The alternative — `mock.module("../logger")` —+ * is forbidden here: Bun's module registry bleeds a module mock into sibling+ * test FILES, which has broken this suite's e2e files before. So the capture+ * goes through the logger's own debug-log path instead of replacing it.+ *+ * `noLogs: true` is deliberate: it suppresses the ALWAYS-ON log, which lives in+ * `~/.claudish/logs`. A test must not write to the user's real home. The debug+ * log lands in `<cwd>/logs`, which is gitignored, and is deleted on stop.+ *+ * ─── WHY THERE IS A MARKER, WHICH LOOKS LIKE OVER-ENGINEERING AND IS NOT ───+ *+ * `logger.log()` is ASYNCHRONOUSLY BUFFERED. It pushes onto a module-level+ * `logBuffer` and a 100 ms interval writes that buffer to whatever+ * `logFilePath` holds **at flush time** — not at push time:+ *+ * function flushLogBuffer() {+ * if (!logFilePath || logBuffer.length === 0) return; // ← early return+ * appendFile(logFilePath, logBuffer.join(""), …); // KEEPS the buffer+ * }+ *+ * So lines pushed while capture file A was installed, but not yet flushed when+ * `stopLogCapture()` set `logFilePath` to null, are NOT discarded: they sit in+ * the buffer and are appended to capture file B the moment the next test+ * installs one. A whole test file's trailing log lines can therefore surface+ * inside the NEXT file's capture — which is exactly what happened: the C-17+ * "exactly one episode id" assertion passed alone and saw two ids when+ * `composed-handler-recovery.test.ts` ran before it in the same process.+ *+ * That is a property of the production logger, not a bug this helper+ * introduced, and it is not worth changing the logger for. The fix here is to+ * stamp a unique marker through the SAME buffer at capture start and read only+ * what follows it: the buffer is FIFO, so anything stale was pushed earlier and+ * lands ahead of the marker.+ */++import { randomUUID } from "node:crypto";+import { existsSync, readFileSync, rmSync } from "node:fs";+import { getLogFilePath, initLogger, log } from "../../logger.js";++let capturePath: string | null = null;+let marker = "";++export function startLogCapture(): void {+ initLogger(true, "debug", true);+ capturePath = getLogFilePath();+ marker = `[CaptureMarker] ${randomUUID()}`;+ // Through `log()`, so it enters the same FIFO buffer the stale lines are in.+ log(marker);+}++/** Everything written since `startLogCapture`. Waits out the 100 ms flush. */+export async function capturedLines(): Promise<string[]> {+ await new Promise((r) => setTimeout(r, 250));+ if (!capturePath || !existsSync(capturePath)) return [];+ const all = readFileSync(capturePath, "utf-8").split("\n").filter(Boolean);+ const at = all.findLastIndex((l) => l.includes(marker));+ return at >= 0 ? all.slice(at + 1) : all;+}++/** Lines carrying `[Recovery]`, with the ISO timestamp prefix left intact. */+export async function capturedRecoveryLines(): Promise<string[]> {+ return (await capturedLines()).filter((l) => l.includes("[Recovery]"));+}++export function stopLogCapture(): void {+ const path = capturePath;+ capturePath = null;+ marker = "";+ initLogger(false, "info", true);+ if (path) {+ try {+ rmSync(path, { force: true });+ } catch {+ // A pending async append can hold it briefly; the directory is gitignored.+ }+ }+}diff --git a/packages/cli/src/recovery/test-helpers/fake-clock.ts b/packages/cli/src/recovery/test-helpers/fake-clock.tsnew file mode 100644index 00000000..ab356758--- /dev/null+++ b/packages/cli/src/recovery/test-helpers/fake-clock.ts@@ -0,0 +1,194 @@+/**+ * A deterministic `RecoveryClock` for the retry-ladder tests.+ *+ * WHY THIS EXISTS RATHER THAN `mock.module()`. The ladder's own gaps are+ * 5/10/30/60/60 seconds and its deadline is derived from `API_TIMEOUT_MS`+ * (270 s at the default), so asserting the schedule against the real clock+ * would cost ~four minutes per assertion and would be flaky at every margin.+ * The project's standing rule forbids mocking shared infrastructure — Bun's+ * module registry bleeds a `mock.module()` into sibling test FILES — so+ * `recovery/clock.ts` exposes an explicit injection seam and this is the thing+ * injected into it.+ *+ * It is not a general fake-timers shim: it implements exactly the four methods+ * `RecoveryClock` declares, and it drives ONLY the recovery subsystem. Real+ * `setTimeout` still works underneath, which is what `drain()` uses to let the+ * awaiting production code make progress between two fired timers.+ */++import type { RecoveryClock } from "../clock.js";++interface FakeTimer {+ at: number;+ fn: () => void;+}++export class FakeClock implements RecoveryClock {+ private t: number;+ private seq = 0;+ private readonly timers = new Map<number, FakeTimer>();+ /** Handles this clock was asked to `unref`. The grace timer must be in here. */+ readonly unrefed = new Set<number>();++ constructor(start = 0) {+ this.t = start;+ }++ now(): number {+ return this.t;+ }++ setTimeout(fn: () => void, ms: number): unknown {+ const id = ++this.seq;+ this.timers.set(id, { at: this.t + Math.max(0, ms), fn });+ return id;+ }++ clearTimeout(handle: unknown): void {+ this.timers.delete(handle as number);+ }++ unref(handle: unknown): void {+ this.unrefed.add(handle as number);+ }++ /** Armed timer count. A leaked timer is a leaked episode, so tests read this. */+ pending(): number {+ return this.timers.size;+ }++ /** The fire times of every armed timer, ascending. */+ pendingAt(): number[] {+ return [...this.timers.values()].map((t) => t.at).sort((a, b) => a - b);+ }++ /**+ * Move time forward, firing every timer that comes due, in order, and+ * yielding to the real event loop after each one so the code awaiting it can+ * run before the next fires.+ */+ async advance(ms: number): Promise<void> {+ const target = this.t + ms;+ for (;;) {+ let nextId: number | null = null;+ let nextAt = Number.POSITIVE_INFINITY;+ for (const [id, timer] of this.timers) {+ if (timer.at <= target && timer.at < nextAt) {+ nextAt = timer.at;+ nextId = id;+ }+ }+ if (nextId === null) break;+ this.t = Math.max(this.t, nextAt);+ const timer = this.timers.get(nextId) as FakeTimer;+ this.timers.delete(nextId);+ timer.fn();+ await drain();+ }+ this.t = target;+ await drain();+ }+}++/** Let the real microtask and macrotask queues catch up. */+export async function drain(ticks = 6): Promise<void> {+ for (let i = 0; i < ticks; i++) await new Promise((r) => setTimeout(r, 0));+}++/**+ * Step `clock` forward until `p` settles, then return its value.+ *+ * The ladder decides its own next gap, so a test cannot know in advance how far+ * to advance; stepping in small increments and re-checking is both simpler and+ * closer to what the real clock does. `maxMs` is FAKE milliseconds — the test+ * itself never sleeps for them.+ */+export async function advanceUntilSettled<T>(+ clock: FakeClock,+ p: Promise<T>,+ maxMs: number,+ stepMs = 1_000+): Promise<T> {+ let settled = false;+ let value: T | undefined;+ let error: unknown;+ let threw = false;+ p.then(+ (v) => {+ value = v;+ settled = true;+ },+ (e) => {+ error = e;+ threw = true;+ settled = true;+ }+ );+ await drain();+ let spent = 0;+ let idle = 0;+ while (!settled && spent < maxMs) {+ // Jump straight to the next armed timer when there is one. Stepping in+ // fixed increments works too but costs a real event-loop drain per step,+ // which turned a 270-second fake ladder into six real seconds of ticking.+ const next = clock.pendingAt()[0];+ if (next === undefined) {+ // Nothing armed and nothing settled: the production code is between+ // rungs doing REAL async work — a refused connect resolves in about a+ // millisecond, which is more than a few microtasks. Give the real event+ // loop a moment before concluding the fake clock has to move.+ if (idle < 40) {+ idle++;+ await new Promise((r) => setTimeout(r, 1));+ continue;+ }+ idle = 0;+ await clock.advance(stepMs);+ spent += stepMs;+ continue;+ }+ idle = 0;+ const step = Math.max(1, next - clock.now());+ await clock.advance(step);+ spent += step;+ }+ await drain();+ if (!settled) throw new Error(`promise did not settle within ${maxMs} fake ms`);+ if (threw) throw error;+ return value as T;+}++/**+ * A real `AbortSignal` wrapped so a test can COUNT listener registrations.+ *+ * The coordinator's named failure mode is the subscribe-without-unsubscribe+ * leak: a waiter that parks, leaves, and never detaches keeps its closure — and+ * therefore the whole episode — alive for as long as the client's signal+ * exists. `added === removed` is the only direct evidence of the `finally`+ * doing its job; every indirect proxy for it (episode count, timer count) is+ * also true of an implementation that leaks the listener.+ */+export function countingSignal(controller: AbortController): {+ signal: AbortSignal;+ counts: { added: number; removed: number };+} {+ const counts = { added: 0, removed: 0 };+ const real = controller.signal;+ const proxy = {+ get aborted() {+ return real.aborted;+ },+ get reason() {+ return real.reason;+ },+ addEventListener(type: string, fn: EventListener, opts?: AddEventListenerOptions) {+ counts.added++;+ real.addEventListener(type, fn, opts);+ },+ removeEventListener(type: string, fn: EventListener) {+ counts.removed++;+ real.removeEventListener(type, fn);+ },+ };+ return { signal: proxy as unknown as AbortSignal, counts };+}